PXC crash

Bug #1110561 reported by Jay Janssen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
New
Undecided
Unassigned

Bug Description

Crash

Percona-XtraDB-Cluster-server-5.5.28-23.7.369.rhel6.x86_64
CentOS 6.3
Customer has FKs

Noticed high sys CPU util before crashes. Crash log is attached.

Revision history for this message
Jay Janssen (jay-janssen) wrote :
Revision history for this message
Jay Janssen (jay-janssen) wrote :

'perf top' output from high sys CPU nodes (I believe these are in a pre-crash state):

Events: 266K cycles
 27.18% [kernel] [k] _spin_lock
  6.44% mysqld [.] 0x54b935
  2.23% mysqld [.] ha_partition::change_partitions(st_ha_create_information*, char const*, unsigned l
  1.92% libc-2.12.so [.] memcpy
  1.63% mysqld [.] yaSSL::buildFinished(yaSSL::SSL&, yaSSL::Finished&, unsigned char const*)
  1.34% libc-2.12.so [.] _int_malloc

Revision history for this message
Jay Janssen (jay-janssen) wrote :

Another one (much higher sys):

Events: 632K cycles
 58.11% [kernel] [k] _spin_lock
  4.58% mysqld [.] my_regcomp
  3.22% mysqld [.] 0x54b935
  1.15% libc-2.12.so [.] _int_malloc
  0.93% libc-2.12.so [.] memcpy
  0.91% mysqld [.] yaSSL::buildFinished(yaSSL::SSL&, yaSSL::Finished&, unsigned char const*)
  0.78% mysqld [.] ha_partition::change_partitions(st_ha_create_information*, char const*, unsigned l
  0.76% mysqld [.] create_fromuni
  0.52% [kernel] [k] intel_idle
  0.50% mysqld [.] MYSQLparse(void*)
  0.50% mysqld [.] row_purge_remove_sec_if_poss_leaf
  0.49% [kernel] [k] schedule
  0.44% libc-2.12.so [.] _int_free
  0.41% mysqld [.] row_purge_node_create
  0.41% [kernel] [k] copy_user_generic_string
  0.38% mysqld [.] ha_partition::truncate_partition(Alter_info*, bool*)
  0.35% libc-2.12.so [.] malloc

Revision history for this message
Jay Janssen (jay-janssen) wrote :

top - 11:50:03 up 21 days, 20:12, 6 users, load average: 1.20, 1.10, 1.05
Tasks: 693 total, 1 running, 692 sleeping, 0 stopped, 0 zombie
Cpu(s): 24.4%us, 46.0%sy, 0.0%ni, 29.1%id, 0.0%wa, 0.0%hi, 0.5%si, 0.0%st
Mem: 132112432k total, 124533664k used, 7578768k free, 186180k buffers
Swap: 16777208k total, 4028k used, 16773180k free, 11420344k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16004 mysql 20 0 123g 107g 2.0g S 2259.2 85.4 54854:27 mysqld
 3265 root 20 0 124m 5100 1860 S 1.0 0.0 0:01.77 perl
 5577 root 20 0 15536 1736 944 R 1.0 0.0 0:01.01 top

Revision history for this message
Jay Janssen (jay-janssen) wrote :

Note: tried disabling transparent hugepages, no apparent effect:

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always [never]

[root@db02 mysql]# cat /sys/block/sd*/queue/scheduler
[noop] anticipatory deadline cfq
[noop] anticipatory deadline cfq

sdb (second one) is the datadir. It is SSD drives.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Similar bug reported lp:1123233

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Since there are 2 other bugs reported similar to this, following details are required:

a) If the tables have FK or not. -- This is present in description.

b) A core dump and/or backtrace

c) If not b), atleast the crash log from the error log.

b and c are required to check the nature of query (is it INSERT ... DUPLICATE?).

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

The other two bugs being lp:1149735 and lp:1123233

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.