Segfault during remove_table_from_cache on slave node

Remaining nodes had the following in the log:

node1:
091114 16:45:24 [ERROR] Slave SQL: Error 'Table 'test.comm01' doesn't exist' on opening tables, Error_code: 1146
091114 16:45:24 [Warning] RBR event 2 apply warning: 1146
091114 16:45:24 [ERROR] mm_galera.c:apply_write_set():657: apply failed for: 7, in level: 4
091114 16:45:24 [Warning] mm_galera.c:process_query_write_set_applying():802: ws apply failed, rcode: 7, seqno: 8909824, last_seen: 8909773
091114 16:45:24 [Warning] mm_galera.c:process_query_write_set_applying():830: ws applying is not possible, 8909824 - 1454417
091114 16:45:24 [ERROR] mm_galera.c:process_query_write_set():953: could not apply trx: 8909824 1454417
091114 16:45:24 [Note] wsrep recv thread exiting (code:0)
091114 16:45:24 [Note] wsrep starting shutdown
091114 16:45:24 [Note] /mnt/galera/mysql/libexec/mysqld: Normal shutdown

node2:
091114 16:43:13 [ERROR] Slave SQL: Error 'Table 'test.comm01' doesn't exist' on opening tables, Error_code: 1146
091114 16:43:13 [Warning] RBR event 2 apply warning: 1146
091114 16:43:13 [ERROR] mm_galera.c:apply_write_set():657: apply failed for: 7, in level: 4
091114 16:43:13 [Warning] mm_galera.c:process_query_write_set_applying():802: ws apply failed, rcode: 7, seqno: 8909824, last_seen: 8909773
091114 16:43:13 [Warning] mm_galera.c:process_query_write_set_applying():830: ws applying is not possible, 8909824 - 1454416
091114 16:43:13 [ERROR] mm_galera.c:process_query_write_set():953: could not apply trx: 8909824 1454416
091114 16:43:13 [Note] wsrep recv thread exiting (code:0)
091114 16:43:13 [Note] wsrep starting shutdown
091114 16:43:13 [Note] /mnt/galera/mysql/libexec/mysqld: Normal shutdown

node3:
091114 16:43:13 [ERROR] Slave SQL: Error 'Table 'test.comm01' doesn't exist' on opening tables, Error_code: 1146
091114 16:43:13 [Warning] RBR event 2 apply warning: 1146
091114 16:43:13 [ERROR] mm_galera.c:apply_write_set():657: apply failed for: 7, in level: 4
091114 16:43:13 [Warning] mm_galera.c:process_query_write_set_applying():802: ws apply failed, rcode: 7, seqno: 8909824, last_seen: 8909773
091114 16:43:13 [Warning] mm_galera.c:process_query_write_set_applying():830: ws applying is not possible, 8909824 - 1454415
091114 16:43:13 [ERROR] mm_galera.c:process_query_write_set():953: could not apply trx: 8909824 1454415
091114 16:43:13 [Note] wsrep recv thread exiting (code:0)
091114 16:43:13 [Note] wsrep starting shutdown
Cascading view changes, dropping state message
091114 16:43:13 [Note] gcs_group.c:gcs_group_handle_comp_msg():266: New COMPONENT: primary = yes, my_idx = 1, memb_num = 2
091114 16:43:13 [Note] gcs_core.c:core_handle_comp_msg():620: STATE EXCHANGE: Waiting for state UUID.
091114 16:43:13 [Note] gcs_core.c:core_handle_uuid_msg():697: STATE EXCHANGE: sent state msg: b5f194ce-d166-11de-0800-37da9a464a48
091114 16:43:13 [Note] gcs_group.c:gcs_group_handle_state_msg():422: STATE EXCHANGE: got state msg: b5f194ce-d166-11de-0800-37da9a464a48 from 0 (ip-10-226-130-241)
091114 16:43:13 [Note] gcs_group.c:gcs_group_handle_state_msg():422: STATE EXCHANGE: got state msg: b5f194ce-d166-11de-0800-37da9a464a48 from 1 (ip-10-226-103-194)
091114 16:43:13 [Note] gcs_group.c:gcs_gro...

Test case is to run sqlgen load against a single node MySQL/Galera
When sqlgen is running, open mysql connection to the node, and issue some DDL
queries, like:

mysql> alter table comm00 add column (i int);
mysql> alter table comm00 drop column i;

Sooner or later, the node will segfault, here is the stack trace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xaf4a0b90 (LWP 6227)]
0x08391c47 in ha_wsrep_abort_transaction (bf_thd=0x0, victim_thd=0xa137c18)
    at handler.cc:4714
4714 for (info= bf_thd->transaction.stmt.ha_list; info; info= info->next())
(gdb) bt
#0 0x08391c47 in ha_wsrep_abort_transaction (bf_thd=0x0, victim_thd=0xa137c18)
    at handler.cc:4714
#1 0x0825a1ff in wsrep_abort_thd (bf_thd_ptr=0x0, victim_thd_ptr=0xa137c18)
    at sql_parse.cc:8412
#2 0x082ac7d3 in remove_table_from_cache (thd=0xa137c18, db=0x9f7a9b8 "test",
    table_name=0x9f7a9bd "comm00", flags=2) at sql_base.cc:8492
#3 0x083b0f81 in wait_while_table_is_used (thd=0xa137c18, table=0x9faabc8,
    function=HA_EXTRA_PREPARE_FOR_RENAME) at sql_table.cc:4207
#4 0x083bd56d in mysql_alter_table (thd=0xa137c18, new_db=0xa1a8998 "test",
    new_name=0xa1a8798 "comm00", create_info=0xaf49f240, table_list=0xa1a87c0,
    alter_info=0xaf49f530, order_num=0, order=0x0, ignore=false)
    at sql_table.cc:7304
#5 0x08264529 in mysql_execute_command (thd=0xa137c18) at sql_parse.cc:3196
#6 0x0826e1d4 in mysql_parse (thd=0xa137c18,
    inBuf=0xa1a8700 "alter table comm00 add column (i int)", length=37,
    found_semicolon=0xaf4a0030) at sql_parse.cc:6395
#7 0x0826fc20 in dispatch_command (command=COM_QUERY, thd=0xa137c18,
    packet=0xa16c661 "alter table comm00 add column (i int)", packet_length=37)
    at sql_parse.cc:1381
#8 0x08271d8e in do_command (thd=0xa137c18) at sql_parse.cc:964
#9 0x082593a0 in handle_one_connection (arg=0xa137c18) at sql_connect.cc:1165
#10 0xb7e7850f in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#11 0xb7d69a0e in clone () from /lib/tls/i686/cmov/libc.so.6

The implementation for converting locking session to transaction interfered with TO isolation of DDL queries.
This was fixed in change set 2920. The fix required to carry flag of locking session in thr lock info.

Here is a simple script to cause conflicting DDL for sqlgen load generator

for i in `seq 1000`; do
    echo "round: $i";
    mysql -uroot -prootpass -hrotweiler test -e"alter table comm00 add column (i int);";
    mysql -uroot -prootpass -hrotweiler test -e"alter table comm00 drop column i ;" ;
done

The orginal problem does not happen anymore. However, when testing with high conflict rate DML load and simultaneous DDL load, it can happen, that delegate node crashes for not being able to apply "replaying" write set.

Will close this bug and open new DDL/DML concurrency bug, with deadline for 0.7.1 release.