Crash on Concurrent INSERT INTO .. ON DUPLICATE KEY UPDATE

Bug #1149735 reported by Jervin R
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
In Progress
Undecided
Seppo Jaakola

Bug Description

After a node rejoins the cluster on SST, a potential crash can occur when the next writeset being executed involves a INSERT INTO ... ON DUPLICATE KEY UPDATE. In this case, we have db01, db02 and db03. db01 left the cluster for unknown issues, after rejoining the cluster thru xtrabackup sst, it crashed with the stack trace below. Then db02 followed, after bring db02 back online, db03 crashed as well.

130228 19:37:16 [ERROR] WSREP: FSM: no such a transition EXECUTING -> COMMITTED
00:37:16 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1874
max_threads=5000
thread_count=1469
connection_count=1469
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10951434 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0xa859e260
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f6f68fc0e48 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7c83f5]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x6a1e04]
/lib64/libpthread.so.0[0x370440f500]
/lib64/libc.so.6(gsignal+0x35)[0x37040328a5]
/lib64/libc.so.6(abort+0x175)[0x3704034085]
/usr/lib64/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5StateENS1_10TransitionENS_10EmptyGuardENS_11EmptyActionEE8shift_toES2_+0x2d9)[0x7f8cff6f6529]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM11post_commitEPNS_9TrxHandleE+0xd2)[0x7f8cff715c22]
/usr/lib64/libgalera_smm.so(galera_post_commit+0x5e)[0x7f8cff72e1be]
/usr/sbin/mysqld(_Z25wsrep_cleanup_transactionP3THD+0x97)[0x6599e7]
/usr/sbin/mysqld[0x6a7b06]
/usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x4b9)[0x6a9069]
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x27)[0x644ff7]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x50c)[0x593c6c]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x333)[0x598de3]
/usr/sbin/mysqld[0x5997a0]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1a82)[0x59c072]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x59c499]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x15f)[0x63710f]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x6372d1]
/lib64/libpthread.so.0[0x3704407851]
/lib64/libc.so.6(clone+0x6d)[0x37040e811d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f6ee507d620): is an invalid pointer
Connection ID (thread ID): 1710299
Status: NOT_KILLED

Full backtrace already shared with Seppo privately, let me know if anything else is needed.

Tags: i29891
Revision history for this message
Seppo Jaakola (seppo-jaakola) wrote :

may be a duplicate of: lp:1123233

Changed in percona-xtradb-cluster:
assignee: nobody → Seppo Jaakola (seppo-jaakola)
Revision history for this message
Seppo Jaakola (seppo-jaakola) wrote :

during testing for reproducing this issue, another issue surfaced: lp:1155183, these issues can be related

Changed in percona-xtradb-cluster:
status: New → In Progress
Revision history for this message
Gabriel Féron (gabriel-feron-n) wrote :

I'm facing the same issue, and the symptoms seems to be the same. A concurrent INSERT INTO... ON DUPLICATE UPDATE.

I'm getting messages like these in the log (when wrep_debug is ON):

130801 10:51:23 [Note] WSREP: empty rbr buffer, query: INSERT INTO `catalogsearch_result` (`query_id`, `product_id`, `relevance`) VALUES (3616, 18815, 1.709000) ON DUPLICATE KEY UPDATE `relevance` = 1.709000

Until it crashes with a similar stacktrace as the one reported earlier.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.