Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC

Boostrapping a Node in NON-PRIMARY Crashes

Bug #1286450 reported by Jervin R
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Galera
Status tracked in 3.x
2.x
Fix Released
High
Alex Yurchenko
Galera 25.2.9
3.x
Fix Released
High
Alex Yurchenko
Galera 25.3.4
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Status tracked in 5.6
5.5
Fix Released
Undecided
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC 5.5.37-25.10
5.6
Fix Released
Undecided
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC 5.6.15-25.5

Bug Description

Coming from 2 nodes, PXC 5.6.15, kill -9 the second node, wait for the first node to be in Initialized state then set pc.boostrap=1. You get the crash.

2014-03-01 01:05:22 9793 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
terminate called after throwing an instance of 'gu::NotSet'
06:05:32 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=4
max_threads=153
thread_count=4
connection_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 661734 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x3250d70
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fceb4558d38 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x902445]
/usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x680114]
/lib64/libpthread.so.0[0x31c700f500]
/lib64/libc.so.6(gsignal+0x35)[0x31c68328e5]
/lib64/libc.so.6(abort+0x175)[0x31c68340c5]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x31ca0bea5d]
/usr/lib64/libstdc++.so.6[0x31ca0bcbe6]
/usr/lib64/libstdc++.so.6[0x31ca0bcc13]
/usr/lib64/libstdc++.so.6[0x31ca0bcd0e]
/usr/lib64/libgalera_smm.so(_ZNK2gu6Config3getERKSs+0x138)[0x7fcec1de4f48]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM9param_setERKSsS2_+0x1d)[0x7fcec1edd1bd]
/usr/lib64/libgalera_smm.so(_Z16wsrep_set_paramsRN6galera10ReplicatorEPKc+0x188)[0x7fcec1edbd78]
/usr/lib64/libgalera_smm.so(galera_parameters_set+0x2c)[0x7fcec1f156dc]
/usr/sbin/mysqld(_Z29wsrep_provider_options_updateP7sys_varP3THD13enum_var_type+0x26)[0x5bad06]
/usr/sbin/mysqld(_ZN7sys_var6updateEP3THDP7set_var+0x8a)[0x67f14a]
/usr/sbin/mysqld(_ZN7set_var6updateEP3THD+0x17)[0x67f6b7]
/usr/sbin/mysqld(_Z17sql_set_variablesP3THDP4ListI12set_var_baseE+0x89)[0x67e919]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x319d)[0x7046ad]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x608)[0x7077c8]
/usr/sbin/mysqld[0x7078f1]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1ad4)[0x709b74]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1e3)[0x70af43]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x17f)[0x6d47ef]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6d49c7]
/usr/sbin/mysqld(pfs_spawn_thread+0x12a)[0xb3bdca]
/lib64/libpthread.so.0[0x31c7007851]
/lib64/libc.so.6(clone+0x6d)[0x31c68e894d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fce6c005200): is an invalid pointer
Connection ID (thread ID): 4
Status: NOT_KILLED

Related branches

lp:~codership/galera/23.2.x (Merged)
lp:galera/2.x
lp:galera
David Bennett: Pending requested
Diff: 0 lines
Revision history for this message
Jervin R (revin) wrote :

[root@pxc01 pxc-5.6.15]# rpm -qa|grep -i percona
Percona-XtraDB-Cluster-client-56-5.6.15-25.4.731.rhel6.x86_64
percona-xtrabackup-debuginfo-2.1.7-721.rhel6.x86_64
Percona-XtraDB-Cluster-galera-3-debuginfo-3.3-1.207.rhel6.x86_64
percona-xtrabackup-2.1.7-721.rhel6.x86_64
Percona-XtraDB-Cluster-devel-56-5.6.15-25.4.731.rhel6.x86_64
Percona-XtraDB-Cluster-galera-3-3.3-1.207.rhel6.x86_64
Percona-XtraDB-Cluster-server-56-5.6.15-25.4.731.rhel6.x86_64
Percona-XtraDB-Cluster-56-debuginfo-5.6.15-25.4.731.rhel6.x86_64
Percona-XtraDB-Cluster-56-5.6.15-25.4.731.rhel6.x86_64
Percona-XtraDB-Cluster-shared-56-5.6.15-25.4.731.rhel6.x86_64
Percona-Server-shared-compat-5.5.35-rel33.0.611.rhel6.x86_64

[root@pxc01 pxc-5.6.15]# cat /etc/my.cnf
[client]
port = 3306
socket = /var/lib/mysql/mysqld.sock
user = root
password = root

[mysqld]
datadir = /var/lib/mysql
port = 3306
socket = /var/lib/mysql/mysqld.sock
pid-file = /var/lib/mysql/mysqld.pid
skip-external-locking
key_buffer_size = 32M
max_allowed_packet = 128M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 0
query_cache_type = 0
query_cache_min_res_unit= 1536
skip-slave-start
expire_logs_days = 15

slow-query-log = 1
long_query_time = 2
#log-bin=mysql-bin
server-id = 10
log-slave-updates = 1
binlog_format = ROW

innodb_log_buffer_size = 8M
innodb_buffer_pool_size = 512M
innodb_log_file_size = 256M
innodb-file-per-table
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2

log-error = /var/lib/mysql/mysqld.log
slow_query_log_file = /var/lib/mysql/mysqld-slow.log

skip-name-resolve

wsrep_provider = /usr/lib64/libgalera_smm.so
wsrep_slave_threads = 2
wsrep_cluster_name = pxc
wsrep_sst_method = xtrabackup
wsrep_node_name = pxc01
wsrep_node_address = 192.168.56.42
wsrep_cluster_address = gcomm://

[mysqldump]
quick
max_allowed_packet = 128M

[mysql]
no-auto-rehash

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :
Download full text (5.5 KiB)

To simplify, just setting pc.bootstrap=1 during runtime crashes.

#0 0x00007ffff5f5e389 in raise () from /usr/lib/libc.so.6
#1 0x00007ffff5f5f788 in abort () from /usr/lib/libc.so.6
#2 0x00007ffff6848625 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#3 0x00007ffff6846786 in ?? () from /usr/lib/libstdc++.so.6
#4 0x00007ffff68467b3 in std::terminate() () from /usr/lib/libstdc++.so.6
#5 0x00007ffff68469f2 in __cxa_throw () from /usr/lib/libstdc++.so.6
#6 0x00007fffd7fb1427 in gu::Config::get (this=<optimized out>, key=...) at galerautils/src/gu_config.hpp:126
#7 0x00007fffd80b7f67 in galera::ReplicatorSMM::param_set (this=0x145d290, key=..., value=...) at galera/src/replicator_smm_params.cpp:143
#8 0x00007fffd80b55d7 in wsrep_set_params (repl=..., params=<optimized out>) at galera/src/wsrep_params.cpp:40
#9 0x00007fffd80e8657 in galera_parameters_set (gh=<optimized out>, params=<optimized out>) at galera/src/wsrep_provider.cpp:117
#10 0x0000000000633416 in wsrep_provider_options_update (self=<optimized out>, thd=<optimized out>, type=<optimized out>)
    at /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/wsrep_var.cc:287
#11 0x00000000006cdcab in sys_var::update (this=0x12f8140 <Sys_wsrep_provider_options>, thd=0x2304c40, var=<optimized out>)
    at /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/set_var.cc:194
#12 0x00000000006cdfa7 in set_var::update (this=<optimized out>, thd=<optimized out>) at /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/set_var.cc:679
#13 0x00000000006ce87c in sql_set_variables (thd=thd@entry=0x2304c40, var_list=var_list@entry=0x2307f18) at /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/set_var.cc:579
#14 0x0000000000740053 in mysql_execute_command (thd=thd@entry=0x2304c40) at /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/sql_parse.cc:4381
#15 0x0000000000742670 in mysql_parse (thd=thd@entry=0x2304c40, rawbuf=rawbuf@entry=0x7fff68005130 "set global wsrep_provider_options=\"pc.bootstrap=1\"", length=length@entry=50,
    parser_state=parser_state@entry=0x7fffbc568800) at /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/sql_parse.cc:7188
#16 0x0000000000742c4c in wsrep_mysql_parse (thd=thd@entry=0x2304c40, rawbuf=0x7fff68005130 "set global wsrep_provider_options=\"pc.bootstrap=1\"", length=length@entry=50,
    parser_state=parser_state@entry=0x7fffbc568800) at /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/sql_parse.cc:6940
#17 0x0000000000744542 in dispatch_command (command=<optimized out>, command@entry=COM_QUERY, thd=thd@entry=0x2304c40, packet=packet@entry=0x2338e11 "set global wsrep_provider_options=\"pc.bootstrap=1\"",
    packet_length=packet_length@entry=50) at /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/sql_parse.cc:1630
#18 0x0000000000744b49 in do_command (thd=0x2304c40) at /media/Tintin/Work/code/percona-xtradb-cluster/pxc56/Percona-Server/sql/sql_parse.cc:1133
#19 0x0000000000717d12 in do_handle_one_connection (thd_arg=thd_arg@entry=0x2304c40) at /media/Tintin/Work/code/perc...

Read more...

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Another regression after configuration system refactoring: exception to catch changed.
Fix pushed in:
http://bazaar.launchpad.net/~codership/galera/2.x/revision/170
http://bazaar.launchpad.net/~codership/galera/3.x/revision/174

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1634

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.