Percona server 5.6.12 started with thread_pool hangs on shutdown

Bug #1201681 reported by Alexey Stroganov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
High
Sergei Glushchenko
5.1
Invalid
Undecided
Unassigned
5.5
Invalid
Undecided
Unassigned
5.6
Fix Released
High
Sergei Glushchenko

Bug Description

to reproduce this issue one need to start server with thread_pool, make connection, start shutdown.

console #1:

[root@host] cd bin64_5612.rc60.4

[root@host] ./scripts/mysql_install_db --basedir=. --datadir=data/
[root@host] ./bin/mysqld --no-defaults --basedir=. --datadir=data --thread_handling=pool-of-threads --socket=/tmp/mysql.sock --user=root
...
Version: '5.6.12-rc60.4' socket: '/tmp/mysql.sock' port: 3306 Percona Server with XtraDB (GPL), Release rc60.4, Revision 393

console #2:
[root@host] mysql -uroot test

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.12-rc60.4 Percona Server with XtraDB (GPL), Release rc60.4, Revision 393
....

console #3:
mysqladmin shutdown -uroot -S /tmp/mysql.sock
<HANG>

console #1:

2013-07-15 21:41:45 1195 [Note] ./bin/mysqld: Normal shutdown

2013-07-15 21:41:45 1195 [Note] Giving 1 client threads a chance to die gracefully
2013-07-15 21:41:45 1195 [Note] Event Scheduler: Purging the queue. 0 events
2013-07-15 21:41:45 1195 [Note] Shutting down slave threads
2013-07-15 21:41:47 1195 [Note] Forcefully disconnecting 1 remaining clients
2013-07-15 21:41:47 1195 [Warning] ./bin/mysqld: Forcing close of thread 2 user: 'root'

<HANG>

backtrace

Thread 26 (Thread 0x7f994a209700 (LWP 1197)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=0, message1=0x7f994a208d98, message2=0x7f994a208d90, type=0x7f994a208d88, space_id=0x7f994a208d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=0, message1=0x7f994a208d98, message2=0x7f994a208d90, type=0x7f994a208d88, space_id=0x7f994a208d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=0) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f9922388700 (LWP 1198)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=1, message1=0x7f9922387d98, message2=0x7f9922387d90, type=0x7f9922387d88, space_id=0x7f9922387d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=1, message1=0x7f9922387d98, message2=0x7f9922387d90, type=0x7f9922387d88, space_id=0x7f9922387d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=1) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f9921987700 (LWP 1199)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=2, message1=0x7f9921986d98, message2=0x7f9921986d90, type=0x7f9921986d88, space_id=0x7f9921986d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=2, message1=0x7f9921986d98, message2=0x7f9921986d90, type=0x7f9921986d88, space_id=0x7f9921986d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=2) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f9920f86700 (LWP 1200)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=3, message1=0x7f9920f85d98, message2=0x7f9920f85d90, type=0x7f9920f85d88, space_id=0x7f9920f85d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=3, message1=0x7f9920f85d98, message2=0x7f9920f85d90, type=0x7f9920f85d88, space_id=0x7f9920f85d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=3) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f9920585700 (LWP 1201)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=4, message1=0x7f9920584d98, message2=0x7f9920584d90, type=0x7f9920584d88, space_id=0x7f9920584d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=4, message1=0x7f9920584d98, message2=0x7f9920584d90, type=0x7f9920584d88, space_id=0x7f9920584d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=4) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f991fb84700 (LWP 1202)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=5, message1=0x7f991fb83d98, message2=0x7f991fb83d90, type=0x7f991fb83d88, space_id=0x7f991fb83d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=5, message1=0x7f991fb83d98, message2=0x7f991fb83d90, type=0x7f991fb83d88, space_id=0x7f991fb83d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=5) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f991f183700 (LWP 1203)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=6, message1=0x7f991f182d98, message2=0x7f991f182d90, type=0x7f991f182d88, space_id=0x7f991f182d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=6, message1=0x7f991f182d98, message2=0x7f991f182d90, type=0x7f991f182d88, space_id=0x7f991f182d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=6) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f991e782700 (LWP 1204)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=7, message1=0x7f991e781d98, message2=0x7f991e781d90, type=0x7f991e781d88, space_id=0x7f991e781d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=7, message1=0x7f991e781d98, message2=0x7f991e781d90, type=0x7f991e781d88, space_id=0x7f991e781d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=7) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
---Type <return> to continue, or q <return> to quit---
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f991dd81700 (LWP 1205)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=8, message1=0x7f991dd80d98, message2=0x7f991dd80d90, type=0x7f991dd80d88, space_id=0x7f991dd80d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=8, message1=0x7f991dd80d98, message2=0x7f991dd80d90, type=0x7f991dd80d88, space_id=0x7f991dd80d80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=8) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f991d380700 (LWP 1206)):
#0 0x00007f994cf43614 in ?? () from /lib64/libaio.so.1
#1 0x000000000097a462 in os_aio_linux_collect (global_seg=9, message1=0x7f991d37fd98, message2=0x7f991d37fd90, type=0x7f991d37fd88, space_id=0x7f991d37fd80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5089
#2 os_aio_linux_handle (global_seg=9, message1=0x7f991d37fd98, message2=0x7f991d37fd90, type=0x7f991d37fd88, space_id=0x7f991d37fd80)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0file.cc:5234
#3 0x0000000000a832aa in fil_aio_wait (segment=9) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fil/fil0fil.cc:5575
#4 0x00000000009e26b8 in io_handler_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0start.cc:479
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f991b8d2700 (LWP 1208)):
#0 0x00007f994d14f7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000097c92d in os_cond_wait_timed (event=0x1afe8c0, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:177
#2 os_event_wait_time_low (event=0x1afe8c0, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:723
#3 0x000000000095e663 in lock_wait_timeout_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/lock/lock0wait.cc:503
#4 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f991aed1700 (LWP 1209)):
#0 0x00007f994d14f7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000097c92d in os_cond_wait_timed (event=0x173d2f0, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:177
#2 os_event_wait_time_low (event=0x173d2f0, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:723
#3 0x00000000009df3b5 in srv_error_monitor_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0srv.cc:2084
#4 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f991a4d0700 (LWP 1210)):
#0 0x00007f994d14f7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000097c92d in os_cond_wait_timed (event=0x173d380, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:177
#2 os_event_wait_time_low (event=0x173d380, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:723
#3 0x00000000009e0ead in srv_monitor_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0srv.cc:1822
#4 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f991c2d3700 (LWP 1211)):
#0 0x00007f994b9044f3 in select () from /lib64/libc.so.6
#1 0x000000000097cccf in os_thread_sleep (tm=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0thread.cc:255
#2 0x00000000009de5b3 in srv_master_sleep (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0srv.cc:2803
#3 srv_master_thread (arg=<value optimized out>) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0srv.cc:2847
#4 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f9919acf700 (LWP 1212)):
#0 0x00007f994d14f43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000097cad2 in os_cond_wait (event=0x173d260, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:214
#2 os_event_wait_low (event=0x173d260, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:610
#3 0x00000000009e189d in srv_purge_coordinator_suspend (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0srv.cc:3140
#4 srv_purge_coordinator_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/srv/srv0srv.cc:3257
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f99190ce700 (LWP 1213)):
#0 0x00007f994b9044f3 in select () from /lib64/libc.so.6
#1 0x000000000097cccf in os_thread_sleep (tm=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0thread.cc:255
#2 0x0000000000a475cb in page_cleaner_sleep_if_needed (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/buf/buf0flu.cc:2377
#3 buf_flush_page_cleaner_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/buf/buf0flu.cc:2418
#4 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 10 (Thread 0x7f99186cd700 (LWP 1214)):
#0 0x00007f994d14f43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000097cad2 in os_cond_wait (event=0x173d410, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:214
#2 os_event_wait_low (event=0x173d410, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:610
#3 0x0000000000a42421 in buf_dump_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/buf/buf0dump.cc:594
#4 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f9917ccc700 (LWP 1215)):
#0 0x00007f994d14f7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000097c92d in os_cond_wait_timed (event=0x1b05640, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:177
#2 os_event_wait_time_low (event=0x1b05640, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:723
#3 0x0000000000a75255 in dict_stats_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/dict/dict0stats_bg.cc:348
#4 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f99172cb700 (LWP 1216)):
#0 0x00007f994d14f7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000097c92d in os_cond_wait_timed (event=0x1b35ff0, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:177
#2 os_event_wait_time_low (event=0x1b35ff0, time_in_usec=<value optimized out>, reset_sig_count=1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/os/os0sync.cc:723
#3 0x0000000000a0ac04 in ib_wqueue_timedwait (wq=0x1b35e68, wait_in_usecs=5000000)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/ut/ut0wqueue.cc:154
#4 0x0000000000aa5721 in fts_optimize_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/innobase/fts/fts0opt.cc:2932
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f99168ca700 (LWP 1233)):
#0 0x00007f994d14f7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000079c96c in inline_mysql_cond_timedwait (param=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.12-rc60.4/include/mysql/psi/mysql_thread.h:1199
#2 timer_thread (param=<value optimized out>) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:505
#3 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/perfschema/pfs.cc:1855
#4 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f994859b700 (LWP 1234)):
#0 0x00007f994d1532a5 in sigwait () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000583ec0 in signal_hand (arg=<value optimized out>) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/mysqld.cc:3159
#2 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/perfschema/pfs.cc:1855
#3 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f994855a700 (LWP 1237)):
#0 0x00007f994b90bf03 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000079aee1 in io_poll_wait (pollfd=12, native_events=0x7f9948556c30, maxevents=1024, timeout_ms=-1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:281
#2 0x000000000079bc1a in get_event (current_thread=0x7f9948559cd0, thread_group=0x1316200, abstime=0x7f9948559d60)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:643
#3 0x000000000079c064 in worker_main (param=0x1316200) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:1515
#4 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/perfschema/pfs.cc:1855
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f9948519700 (LWP 1240)):
#0 0x00007f994b90bf03 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000079aee1 in io_poll_wait (pollfd=13, native_events=0x7f9948515c30, maxevents=1024, timeout_ms=-1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:281
#2 0x000000000079bc1a in get_event (current_thread=0x7f9948518cd0, thread_group=0x1316400, abstime=0x7f9948518d60)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:643
#3 0x000000000079c064 in worker_main (param=0x1316400) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:1515
#4 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/perfschema/pfs.cc:1855
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f99484d8700 (LWP 1243)):
#0 0x00007f994b90bf03 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000079aee1 in io_poll_wait (pollfd=14, native_events=0x7f99484d4c30, maxevents=1024, timeout_ms=-1)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:281
#2 0x000000000079bc1a in get_event (current_thread=0x7f99484d7cd0, thread_group=0x1316600, abstime=0x7f99484d7d60)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:643
#3 0x000000000079c064 in worker_main (param=0x1316600) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/threadpool_unix.cc:1515
#4 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/perfschema/pfs.cc:1855
#5 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f9948497700 (LWP 1244)):
#0 0x00007f994d14f43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000058ab54 in kill_server (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.12-rc60.4/include/mysql/psi/mysql_thread.h:1162
#2 kill_server_thread (arg=<value optimized out>) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/mysqld.cc:1642
---Type <return> to continue, or q <return> to quit---
#3 0x0000000000b3fa0b in pfs_spawn_thread (arg=<value optimized out>)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/storage/perfschema/pfs.cc:1855
#4 0x00007f994d14b851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f994b90b90d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f994d56a7e0 (LWP 1195)):
#0 0x00007f994d14f43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005895a8 in inline_mysql_cond_wait (argc=7, argv=0x1633de8)
    at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.12-rc60.4/include/mysql/psi/mysql_thread.h:1162
#2 mysqld_main (argc=7, argv=0x1633de8) at /mnt/workspace/percona-server-5.6-binaries/label_exp/centos5-64/Percona-Server-5.6.11-rc60.3/sql/mysqld.cc:5690
#3 0x00007f994b841cdd in __libc_start_main () from /lib64/libc.so.6
#4 0x000000000057c799 in _start ()

Tags: tp

Related branches

Revision history for this message
Alexey Kopytov (akopytov) wrote :

Sergei, can you take a look?

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :
Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :
tags: added: 56qual
Revision history for this message
Roel Van de Paar (roel11) wrote :

Sergei, it looks like that patch is not pushed yet?

tags: removed: 56qual
tags: added: 56qual
Revision history for this message
Roel Van de Paar (roel11) wrote :

bugfix targeted for GA (review by Laurynas together with bug 1201440)

Revision history for this message
Roel Van de Paar (roel11) wrote :

--56qual (tp beta)

tags: added: tp
removed: 56qual
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-693

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.