InnoDB FTS introduced new mutex sync level in 5.6.15, broke UNIV_SYNC_DEBUG

Bug #1258154 reported by Laurynas Biveinis
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL Server
Unknown
Unknown
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
High
Laurynas Biveinis
5.1
Invalid
Undecided
Unassigned
5.5
Invalid
Undecided
Unassigned
5.6
Fix Released
High
Laurynas Biveinis

Bug Description

[5 Dec 13:06] Laurynas Biveinis

Description:
After bug 69617 was introduced (where UNIV_SYNC_DEBUG appears to be unconditionally removed from UNIV_DEBUG by mistake), it was only a matter of time before UNIV_SYNC_DEBUG started bitrotting. This happened with 5.6.15 revision [1] that introduced a new sync level, SYNC_FTS_TOKENIZE, but did not add its handling to sync_thread_add_level().

Thus, if bug 69617 is fixed and debug server is built, a server crashes every time the FTS parallel tokenization thread is created. In MTR this is seen as crashes on

    main.alter_table innodb_fts.fulltext
    innodb_fts.innodb_fts_large_records innodb_fts.innodb_fts_misc
    innodb_fts.innodb_fts_misc_1 innodb_fts.innodb_fts_multiple_index
    innodb_fts.innodb_fts_proximity innodb_fts.innodb_fts_transaction
    innodb.innodb-alter innodb_fts.innobase_drop_fts_index_table
    innodb_fts.innodb-fts-ddl innodb_fts.innodb-fts-fic
    innodb_fts.innodb-fts-stopword innodb_fts.innodb_fts_index_table
    innodb_fts.innodb_fts_misc_debug
    innodb_fts.innodb_fts_result_cache_limit
    innodb_fts.innodb_fts_stopword_charset innodb.innodb-wl5980-alter

Sample crash stacktrace:

Thread 1 (Thread 0x7f9520ff9700 (LWP 11184)):
#0 0x00007f953683ef0c in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:62
#1 0x0000000000aa8cc9 in my_write_core (sig=6) at /home/laurynas/percona/src/5.6/mysys/stacktrace.c:422
#2 0x000000000071a613 in handle_fatal_signal (sig=6) at /home/laurynas/percona/src/5.6/sql/signal_handler.cc:248
#3 <signal handler called>
#4 0x00007f9535c82f77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5 0x00007f9535c865e8 in __GI_abort () at abort.c:90
#6 0x0000000000ce9df5 in sync_thread_add_level (latch=0x7f9510033298, level=167, relock=0) at /home/laurynas/percona/src/5.6/storage/innobase/sync/sync0sync.cc:1311
#7 0x0000000000ce86b9 in mutex_set_debug_info (mutex=0x7f9510033298, file_name=0x10449b0 "/home/laurynas/percona/src/5.6/storage/innobase/row/row0ftsort.cc", line=565) at /home/laurynas/percona/src/5.6/storage/innobase/sync/sync0sync.cc:614
#8 0x0000000000c56597 in mutex_enter_func (mutex=0x7f9510033298, file_name=0x10449b0 "/home/laurynas/percona/src/5.6/storage/innobase/row/row0ftsort.cc", line=565) at /home/laurynas/percona/src/5.6/storage/innobase/include/sync0sync.ic:215
#9 0x0000000000c56622 in pfs_mutex_enter_func (mutex=0x7f9510033298, file_name=0x10449b0 "/home/laurynas/percona/src/5.6/storage/innobase/row/row0ftsort.cc", line=565) at /home/laurynas/percona/src/5.6/storage/innobase/include/sync0sync.ic:245
#10 0x0000000000c59065 in row_merge_fts_get_next_doc_item (psort_info=0x7f9510033188, doc_item=0x7f9520ff8cd0) at /home/laurynas/percona/src/5.6/storage/innobase/row/row0ftsort.cc:565
#11 0x0000000000c5937a in fts_parallel_tokenization (arg=0x7f9510033188) at /home/laurynas/percona/src/5.6/storage/innobase/row/row0ftsort.cc:635
#12 0x00007f9536839f6e in start_thread (arg=0x7f9520ff9700) at pthread_create.c:311
#13 0x00007f9535d469cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

[1]

5.6$ bzr log -r 5499
------------------------------------------------------------
revno: 5499
committer: Shaohua Wang <email address hidden>
branch nick: mysql-5.6-bug17384979
timestamp: Tue 2013-10-08 11:22:51 +0800
message:
  BUG 17384979 - YIELD ON SCANNER THREAD FOR PARALLEL FTS INDEX CREATE TO MINIMIZE MEMORY USAGE

  Solution:
  1. Let scanner thread sleep for a while if fts_doc_list memory exceeds the limit;
  2. Free doc_item once it's tokenized(a mutex is created to protect fts_doc_list).

  rb#3335 approved by Jimmy.Yang

How to repeat:
Remove the #undef UNIV_SYNC_DEBUG line from univ.i
Build debug server
Run any of the above testcases

Suggested fix:
Fix bug 69617, add SYNC_FTS_TOKENIZE case to sync_thread_add_level().

Related branches

tags: added: ci innodb low-hanging-fruit upstream
Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

Upstream fix in 5.6.16 / 5.7.4.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-746

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.