Percona Server moved to https://jira.percona.com/projects/PS

malloc(): memory corruption with replication

Bug #915814 reported by PavelVD
46
This bug affects 8 people
Affects Status Importance Assigned to Milestone
MySQL Server
Unknown
Unknown
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
High
George Ormond Lorch III
Percona Server moved to https://jira.percona.com/projects/PS 5.5.24-26.0
5.1
Fix Released
Medium
George Ormond Lorch III
Percona Server moved to https://jira.percona.com/projects/PS 5.1.65-14.0
5.5
Fix Released
High
George Ormond Lorch III
Percona Server moved to https://jira.percona.com/projects/PS 5.5.24-26.0

Bug Description

Since update to 5.5.18
We periodically encounter this error message from the server

*** glibc detected *** /usr/sbin/mysqld: malloc(): memory corruption: 0x00007f8cb104f580 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71ad6)[0x7f930b69bad6]
/lib/libc.so.6(+0x74b6d)[0x7f930b69eb6d]
/lib/libc.so.6(__libc_malloc+0x70)[0x7f930b6a0930]
/usr/sbin/mysqld(my_malloc+0x32)[0x805572]
/usr/sbin/mysqld(alloc_root+0x104)[0x7fdfd4]
/usr/sbin/mysqld(_Z10MYSQLparsePv+0x1836b)[0x66e3db]
/usr/sbin/mysqld(_Z9parse_sqlP3THDP12Parser_stateP19Object_creation_ctx+0x9d)[0x58eb9d]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x262)[0x599012]
/usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xd5c)[0x73ea0c]
/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x125)[0x532c65]
/usr/sbin/mysqld[0x537681]
/usr/sbin/mysqld(handle_slave_sql+0xa7b)[0x539bdb]
/lib/libpthread.so.0(+0x68ba)[0x7f930c4578ba]
/lib/libc.so.6(clone+0x6d)[0x7f930b6f902d]
======= Memory map: ========
00400000-00d47000 r-xp 00000000 08:06 32 /usr/sbin/mysqld
00f47000-01063000 rw-p 00947000 08:06 32 /usr/sbin/mysqld
01063000-01090000 rw-p 00000000 00:00 0
01d1c000-2e085000 rw-p 00000000 00:00 0 [heap]
......
7f930a3dc000-7f930a3e7000 r-xp 00000000 08:02 49 /lib/libnss_files-2.11.2.so
7f930a3e7000-7f930a5e6000 ---p 0000b000 08:02 49 /lib/libnss_files-2.11.2.so
7f930a5e6000-7f930a5e7000 r--p 0000a000 08:02 49 /lib/libnss_files-2.11.2.so
7f930a5e7000-7f930a5e8000 rw-p 0000b000 08:02 49 /lib/libnss_files-2.11.2.so
7f930a5e8000-7f930a5f2000 r-xp 00000000 08:02 46 /lib/libnss_nis-2.11.2.so
7f930a5f2000-7f930a7f1000 ---p 0000a000 08:02 46 /lib/libnss_nis-2.11.2.so
7f930a7f1000-7f930a7f2000 r--p 00009000 08:02 46 /lib/libnss_nis-2.11.2.so
7f930a7f2000-7f930a7f3000 rw-p 0000a000 08:02 46 /lib/libnss_nis-2.11.2.so
7f930a7f3000-7f930a808000 r-xp 00000000 08:02 64 /lib/libnsl-2.11.2.so
7f930a808000-7f930aa07000 ---p 00015000 08:02 64 /lib/libnsl-2.11.2.so
7f930aa07000-7f930aa08000 r--p 00014000 08:02 64 /lib/libnsl-2.11.2.so
7f930aa08000-7f930aa09000 rw-p 00015000 08:02 64 /lib/libnsl-2.11.2.so
7f930aa09000-7f930aa0b000 rw-p 00000000 00:00 0
7f930aa0b000-7f930aa12000 r-xp 00000000 08:02 60 /lib/libnss_compat-2.11.2.so
7f930aa12000-7f930ac11000 ---p 00007000 08:02 60 /lib/libnss_compat-2.11.2.so
7f930ac11000-7f930ac12000 r--p 00006000 08:02 60 /lib/libnss_compat-2.11.2.so
7f930ac12000-7f930ac13000 rw-p 00007000 08:02 60 /lib/libnss_compat-2.11.2.so
7f930ac13000-7f930ac29000 r-xp 00000000 08:02 38 /lib/libgcc_s.so.1
7f930ac29000-7f930ae28000 ---p 00016000 08:02 38 /lib/libgcc_s.so.1
7f930ae28000-7f930ae29000 rw-p 00015000 08:02 38 /lib/libgcc_s.so.1
7f930ae29000-7f930ae2a000 ---p 00000000 00:00 0
7f930ae2a000-7f930b62a000 rw-p 00000000 00:00 0
7f930b62a000-7f930b782000 r-xp 00000000 08:02 53 /lib/libc-2.11.2.so
7f930b782000-7f930b981000 ---p 00158000 08:02 53 /lib/libc-2.11.2.so
7f930b981000-7f930b985000 r--p 00157000 08:02 53 /lib/libc-2.11.2.so
7f930b985000-7f930b986000 rw-p 0015b000 08:02 53 /lib/libc-2.11.2.so
7f930b986000-7f930b98b000 rw-p 00000000 00:00 0
7f930b98b000-7f930b98c000 r-xp 00000000 08:02 1063 /lib/libaio.so.1.0.1
7f930b98c000-7f930bb8b000 ---p 00001000 08:02 1063 /lib/libaio.so.1.0.1
7f930bb8b000-7f930bb8c000 rw-p 00000000 08:02 1063 /lib/libaio.so.1.0.1
7f930bb8c000-7f930bb93000 r-xp 00000000 08:02 55 /lib/librt-2.11.2.so
7f930bb93000-7f930bd92000 ---p 00007000 08:02 55 /lib/librt-2.11.2.so
7f930bd92000-7f930bd93000 r--p 00006000 08:02 55 /lib/librt-2.11.2.so
7f930bd93000-7f930bd94000 rw-p 00007000 08:02 55 /lib/librt-2.11.2.so
7f930bd94000-7f930be14000 r-xp 00000000 08:02 62 /lib/libm-2.11.2.so
7f930be14000-7f930c014000 ---p 00080000 08:02 62 /lib/libm-2.11.2.so
7f930c014000-7f930c015000 r--p 00080000 08:02 62 /lib/libm-2.11.2.so
7f930c015000-7f930c016000 rw-p 00081000 08:02 62 /lib/libm-2.11.2.so
7f930c016000-7f930c018000 r-xp 00000000 08:02 51 /lib/libdl-2.11.2.so
7f930c018000-7f930c218000 ---p 00002000 08:02 51 /lib/libdl-2.11.2.so
7f930c218000-7f930c219000 r--p 00002000 08:02 51 /lib/libdl-2.11.2.so
7f930c219000-7f930c21a000 rw-p 00003000 08:02 51 /lib/libdl-2.11.2.so

7f930c21a000-7f930c222000 r-xp 00000000 08:02 66 /lib/libcrypt-2.11.2.so
7f930c222000-7f930c421000 ---p 00008000 08:02 66 /lib/libcrypt-2.11.2.so
7f930c421000-7f930c422000 r--p 00007000 08:02 66 /lib/libcrypt-2.11.2.so
7f930c422000-7f930c423000 rw-p 00008000 08:02 66 /lib/libcrypt-2.11.2.so
7f930c423000-7f930c451000 rw-p 00000000 00:00 0
7f930c451000-7f930c468000 r-xp 00000000 08:02 54 /lib/libpthread-2.11.2.so
7f930c468000-7f930c667000 ---p 00017000 08:02 54 /lib/libpthread-2.11.2.so
7f930c667000-7f930c668000 r--p 00016000 08:02 54 /lib/libpthread-2.11.2.so
7f930c668000-7f930c669000 rw-p 00017000 08:02 54 /lib/libpthread-2.11.2.so
7f930c669000-7f930c66d000 rw-p 00000000 00:00 0
7f930c66d000-7f930c68b000 r-xp 00000000 08:02 68 /lib/ld-2.11.2.so
7f930c6a4000-7f930c6a5000 ---p 00000000 00:00 0
7f930c6a5000-7f930c6d5000 rw-p 00000000 00:00 0
7f930c6d5000-7f930c6d6000 ---p 00000000 00:00 0
7f930c6d6000-7f930c706000 rw-p 00000000 00:00 0
7f930c706000-7f930c707000 ---p 00000000 00:00 0
7f930c707000-7f930c737000 rw-p 00000000 00:00 0
7f930c737000-7f930c738000 ---p 00000000 00:00 0
7f930c738000-7f930c768000 rw-p 00000000 00:00 0
7f930c768000-7f930c769000 ---p 00000000 00:00 0
7f930c769000-7f930c799000 rw-p 00000000 00:00 0
7f930c799000-7f930c79a000 ---p 00000000 00:00 0
7f930c79a000-7f930c7ca000 rw-p 00000000 00:00 0
7f930c7ca000-7f930c7cb000 ---p 00000000 00:00 0
7f930c7cb000-7f930c7fb000 rw-p 00000000 00:00 0
7f930c7fb000-7f930c7fc000 ---p 00000000 00:00 0
7f930c7fc000-7f930c82c000 rw-p 00000000 00:00 0
7f930c82c000-7f930c82d000 ---p 00000000 00:00 0
7f930c82d000-7f930c884000 rw-p 00000000 00:00 0
7f930c887000-7f930c88a000 rw-p 00000000 00:00 0
7f930c88a000-7f930c88b000 r--p 0001d000 08:02 68 /lib/ld-2.11.2.so
7f930c88b000-7f930c88c000 rw-p 0001e000 08:02 68 /lib/ld-2.11.2.so
7f930c88c000-7f930c88d000 rw-p 00000000 00:00 0
7fff6f0a4000-7fff6f0b9000 rw-p 00000000 00:00 0 [stack]
7fff6f1aa000-7fff6f1ab000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
120113 5:43:01 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1073741824
read_buffer_size=2097152
max_used_connections=141
max_threads=400
thread_count=7
connection_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6787866 K
bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x28814b20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f930c7f9ea8 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x8094a9]
/usr/sbin/mysqld(handle_segfault+0x3c2)[0x51b402]
/lib/libpthread.so.0(+0xef60)[0x7f930c45ff60]
/lib/libc.so.6(gsignal+0x35)[0x7f930b65c165]
/lib/libc.so.6(abort+0x180)[0x7f930b65ef70]
/lib/libc.so.6(+0x6827b)[0x7f930b69227b]
/lib/libc.so.6(+0x71ad6)[0x7f930b69bad6]
/lib/libc.so.6(+0x74b6d)[0x7f930b69eb6d]
/lib/libc.so.6(__libc_malloc+0x70)[0x7f930b6a0930]
/usr/sbin/mysqld(my_malloc+0x32)[0x805572]
/usr/sbin/mysqld(alloc_root+0x104)[0x7fdfd4]
/usr/sbin/mysqld(_Z10MYSQLparsePv+0x1836b)[0x66e3db]
/usr/sbin/mysqld(_Z9parse_sqlP3THDP12Parser_stateP19Object_creation_ctx+0x9d)[0x58eb9d]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x262)[0x599012]
/usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xd5c)[0x73ea0c]
/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x125)[0x532c65]
/usr/sbin/mysqld[0x537681]
/usr/sbin/mysqld(handle_slave_sql+0xa7b)[0x539bdb]
/lib/libpthread.so.0(+0x68ba)[0x7f930c4578ba]
/lib/libc.so.6(clone+0x6d)[0x7f930b6f902d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f8cb104f128): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
mysqld: /home/jenkins/workspace/percona-server-5.5-debs/label_exp/debian5-64/target/Percona-Server-5.5.18-rel23.0/mysys/my_new.cc:52: int __cxa_pure_virtual(): Assertion `! "Aborted: pure virtual method called."'
 failed.

query_cache_strip_comments is off
+------------------------------+----------+
| Variable_name | Value |
+------------------------------+----------+
| have_query_cache | YES |
| query_cache_limit | 2097152 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 50331648 |
| query_cache_strip_comments | OFF |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
+------------------------------+----------+

Related branches

lp:~akopytov/percona-server/bug915814-5.5
Laurynas Biveinis (community): Approve
Diff: 1805 lines (+448/-1188)
1 file modified
patches/query_cache_enhance.patch (+448/-1188)
lp:~gl-az/percona-server/5.5-915814
Alexey Kopytov (community): Approve
Diff: 93 lines (+42/-3)
1 file modified
Percona-Server/sql/log_event.cc (+42/-3)
lp:~gl-az/percona-server/5.1-915814
Alexey Kopytov (community): Approve
Diff: 85 lines (+42/-3)
1 file modified
Percona-Server/sql/log_event.cc (+42/-3)
Revision history for this message
Alexey Kopytov (akopytov) wrote :

Looks exactly like the 5.5 counterpart of bug #908531.

Revision history for this message
Alexey Kopytov (akopytov) wrote :

This is reported against 5.5.18 which has the fix for bug #705688. So it cannot be a duplicate of bug #705688. Restoring the original status.

Revision history for this message
Oleg Tsarev (tsarev) wrote :

Can you please provide following information:

1) Exact query which acquired the crash (if you have specific)
2) Binary log which acquired the crash with some events before
3) Schema of your tables (if this possible, of course)

So, if this problem related to query cache enhance then disable query cache will help you.

Changed in percona-server:
assignee: nobody → Oleg Tsarev (tsarev)
importance: Undecided → High
status: New → Incomplete
Revision history for this message
PavelVD (pdobryakov) wrote :

At the moment I do not have the opportunity to provide excerpts from bin-logs, as I rolled all the problematic server to version 5.5.17 after the occurrence of this problem.
Indicate the particular query was not possible, unfortunately, because We use mixed and row based replication, and bin logs was difficult to say what caused the fall.
We have encountered this problem on about 20 different servers with very different databases and schemas.
The first thing I did when faced with this problem - I disabled the query cache, but this had no impact on the situation.

Oleg Tsarev (tsarev)
Changed in percona-server:
assignee: Oleg Tsarev (tsarev) → Patrick Crews (patrick-crews)
Revision history for this message
Oleg Tsarev (tsarev) wrote :

Statement Based Replication doesn't send to slave "just SELECT" queries.
Query Cache works with "just SELECT" queries (other are ignored).

Related to query-cache-strip-comments code doesn't allocate memory if
query "not a SELECT".

From this point of view I'm sure - this problems not related to
query-cache-strip-comments,

Revision history for this message
PavelVD (pdobryakov) wrote :
Download full text (10.5 KiB)

query on master:

USE `sas-ru`;

CREATE TABLE analytics_sections_categories(
  id INT(11) NOT NULL AUTO_INCREMENT,
  section_id SMALLINT(6) NOT NULL,
  category_id SMALLINT(6) NOT NULL,
  PRIMARY KEY (section_id, category_id),
  INDEX FK_analytics_category (category_id),
  UNIQUE INDEX id (id),
  CONSTRAINT FK_analytics_category FOREIGN KEY (category_id)
  REFERENCES analytics_category (id) ON DELETE RESTRICT ON UPDATE RESTRICT,
  CONSTRAINT FK_analytics_section FOREIGN KEY (section_id)
  REFERENCES analytics_section (id) ON DELETE RESTRICT ON UPDATE RESTRICT
)
ENGINE = INNODB
CHARACTER SET utf8
COLLATE utf8_general_ci;

USE `site-ru-v4`;
CREATE TABLE analytics_sections_categories(
  id INT(11) NOT NULL AUTO_INCREMENT,
  section_id SMALLINT(6) NOT NULL,
  category_id SMALLINT(6) NOT NULL,
  PRIMARY KEY (section_id, category_id),
  INDEX FK_analytics_category (category_id),
  UNIQUE INDEX id (id),
  CONSTRAINT FK_analytics_category FOREIGN KEY (category_id)
  REFERENCES analytics_category (id) ON DELETE RESTRICT ON UPDATE RESTRICT,
  CONSTRAINT FK_analytics_section FOREIGN KEY (section_id)
  REFERENCES analytics_section (id) ON DELETE RESTRICT ON UPDATE RESTRICT
)
ENGINE = INNODB
CHARACTER SET utf8
COLLATE utf8_general_ci;

I`ve emulate those situation and get error. Here is records in binlog
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#120203 5:25:06 server id 125009144 end_log_pos 107 Start: binlog v 4, server v 5.5.18-55-log created 120203 5:25:06
# Warning: this binlog is either in use or was not closed properly.
BINLOG '
klMrTw/4fHMHZwAAAGsAAAABAAQANS41LjE4LTU1LWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAVAAEGggAAAAICAgCAA==
'/*!*/;
# at 555729637
#120203 13:23:00 server id 125009144 end_log_pos 555729859 Query thread_id=3657294 exec_time=0 error_code=0
SET TIMESTAMP=1328268180/*!*/;
SET @@session.pseudo_thread_id=3657294/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=0/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=33/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
UPDATE `phpmyadmin`.`pma_tracking` SET `tracking_active` = '0' WHERE `db_name` = 'sas-ru' AND `table_name` = 'analytics_sections_categories' AND `version` = '0'
/*!*/;
# at 555729859
#120203 13:23:00 server id 125009144 end_log_pos 555729924 Query thread_id=3657294 exec_time=0 error_code=0
SET TIMESTAMP=1328268180/*!*/;
COMMIT
/*!*/;
# at 555729924
#120203 13:23:01 server id 125009144 end_log_pos 555730600 Query thread_id=3657295 exec_time=0 error_code=0
use site-ru-v4/*!*/;
SET TIMESTAMP=1328268181/*!*/;
CREATE TABLE analytics_sections_categories(
  id INT(11) NOT NULL AUTO_INCREMENT,
  section_id SMALLINT(6) NOT NULL,
  category_id SMALLINT(6) NOT NULL,
  PRIMARY KEY (section_id, ca...

Revision history for this message
PavelVD (pdobryakov) wrote :

What other information necessary to identify the problem?

Revision history for this message
PavelVD (pdobryakov) wrote :

there is also a bug in version 5.5.20

Revision history for this message
Dima (l-dima) wrote :

I have recently upgraded to Percona 5.5.20 and I am hitting this bug every 2-3 days.
My instances which hit that are slave-only instances for backup purposes. MySQL locks up, cannot be shutdown and has to be killed.

If somebody interested in pursuing this bug let me know what info you need me to collect and how to do it.

Thanks,
D.

Revision history for this message
PavelVD (pdobryakov) wrote :

There, do you have plans to address this bug?
Because of this error, we can not update the version 5.5.18 or older at all our production servers that have extremely nice.

Revision history for this message
PavelVD (pdobryakov) wrote :

Why does the error status is "incomplete" to 2012-02-01?
What additional information is needed to fix this error?

Revision history for this message
Vadim Tkachenko (vadim-tk) wrote :

PavelVD,

We will look into this, and let you know if additional information is needed.

Changed in percona-server:
status: Incomplete → New
Revision history for this message
Alexey Kopytov (akopytov) wrote :

Pavel,

I could not reproduce the crash in my environment using the queries you provided in comment #6, because "CREATE TABLE analytics_sections_categories..." fails to resolve the analytics_section table in an FK constraint.

You mentioned that you've emulated the error. Does this mean you have an isolated test case reproducing the crash?

Changed in percona-server:
status: New → Incomplete
Revision history for this message
PavelVD (pdobryakov) wrote :

Unfortunately, I do not have possibility to create a stand-alone test because in other cases where there was the collapse of servers in the bin log entries were different.

Revision history for this message
PavelVD (pdobryakov) wrote :

Now, I checked the logs after a further 3 falls - no strange entries in the relay-log and in the bin-logs were not.

Revision history for this message
PavelVD (pdobryakov) wrote :

Bin-logs on master at that moment

Revision history for this message
PavelVD (pdobryakov) wrote :

error logs on server at that moment

Patrick Crews (patrick-crews)
Changed in percona-server:
assignee: Patrick Crews (patrick-crews) → nobody
Alexey Kopytov (akopytov)
Changed in percona-server:
milestone: none → 5.5.21-25.0
assignee: nobody → Alexey Kopytov (akopytov)
status: Incomplete → In Progress
Alexey Kopytov (akopytov)
Changed in percona-server:
status: In Progress → Fix Committed
Revision history for this message
Steffen Boehme (boemm) wrote :

It seems for me, that percona offers stable releases once a month, so the next release may be provided in the middle of April ... am I right?
In this case we had to rollback our servers to the last version not affected by this bug.

Stewart Smith (stewart)
Changed in percona-server:
milestone: 5.5.21-25.0 → 5.5.21-25.1
Revision history for this message
Alexey Kopytov (akopytov) wrote :

Steffen,

We are going to provide 5.5.21-25.1 with a fix for this bug next week.

Revision history for this message
Steffen Boehme (boemm) wrote :

That great, thanks for your support!

Revision history for this message
Alexey Kopytov (akopytov) wrote :

We didn't have a reproducible test case for this bug. So we just reverted non-essential refactoring introduced in 5.5.18 as the most likely culprit.

It turns out it's an upstream issue after all: http://bugs.mysql.com/bug.php?id=64624, as I originally thought. At least the symptoms and affected versions are the same.

Revision history for this message
Michael S. Moody (michael-sykosoft) wrote :

This has been fixed upstream in 5.5.26 (and is blocking, which has kept us on Percona 5.5.16). Reference: http://bugs.mysql.com/bug.php?id=64624

Any chance we'll see this soon in Percona?

Revision history for this message
Stewart Smith (stewart) wrote : Re: [Bug 915814] Re: malloc(): memory corruption with 5.5.18

On Tue, 12 Jun 2012 23:17:56 -0000, "Michael S. Moody" <email address hidden> wrote:
> This has been fixed upstream in 5.5.26 (and is blocking, which has kept
> us on Percona 5.5.16). Reference: http://bugs.mysql.com/bug.php?id=64624
>
> Any chance we'll see this soon in Percona?

We commit to having a Percona Server release for each new MySQL point
release within 1 month of the MySQL release.

--
Stewart Smith

Revision history for this message
Stewart Smith (stewart) wrote :

On Tue, 12 Jun 2012 23:17:56 -0000, "Michael S. Moody" <email address hidden> wrote:
> This has been fixed upstream in 5.5.26 (and is blocking, which has kept
> us on Percona 5.5.16). Reference: http://bugs.mysql.com/bug.php?id=64624
>
> Any chance we'll see this soon in Percona?

sorry, I spoke too soon, not looking at the bug number. we have already
provided a fix in the latest Percona Server release: 5.5.24-26.0

--
Stewart Smith

Laurynas Biveinis (laurynas-biveinis)
summary: - malloc(): memory corruption with 5.5.18
+ malloc(): memory corruption with replication
Revision history for this message
fengyi (fengyi) wrote :

Master : 5.1.48
Slave : 5.5.18(query cache on)

as http://bugs.mysql.com/bug.php?id=64624 described, I run the slave in valgrind, and got following infomation:

Thread 21:
Conditional jump or move depends on uninitialised value(s)
    at 0x5A6743: Query_cache::send_result_to_client(THD*, char*, unsigned int) (sql_cache.cc:2051)
    by 0x5EDE27: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:5756)
    by 0x800B79: Query_log_event::do_apply_event(Relay_log_info const*, char const*, unsigned int) (log_event.cc:3398)
    by 0x800157: Query_log_event::do_apply_event(Relay_log_info const*) (log_event.cc:3166)
    by 0x572005: Log_event::apply_event(Relay_log_info const*) (log_event.h:1135)
    by 0x56B1A9: apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*) (slave.cc:2351)
    by 0x56B6E9: exec_relay_log_event(THD*, Relay_log_info*) (slave.cc:2511)
    by 0x56D8F6: handle_slave_sql (slave.cc:3329)
    by 0x3A01806D63: start_thread (pthread_create.c:308)

I looked into the souce code, and find that the query buffer allocated in Query_log_event::Query_log_event does not contain the db_len field : sizeof(size_t), but in alloc_query there are statement, length, db_name and flag, therefore result in uninitialized data access in Query_cache::send_result_to_client :

size_t db_len;
    memcpy((char *) &db_len, (sql + query_length + 1), sizeof(size_t));
    if (thd->db_length != db_len)
    {

but I didn't found that this can lead to server crash as the uninitialized data are just junk, but not NULL pointer.

can anyone can tell me why this bug can crash slave?

Revision history for this message
Hui Liu (hickey) wrote :

Hi Fengyi, the slave is crashed due to the memory mistaken of addresses are ruined. Just like this issue, after long time running, mysqld might crashed here and there, though you will see the top backtrace is malloc() or /free().

Revision history for this message
Hui Liu (hickey) wrote :

Instead of alloc a new buf and then copy all content with the correct format in Query_log_event::do_apply_event(), Oracle alloc the extra db_len and then copy the size to get the matched buf in Query_log_event::Query_log_event().

Patch could ref to http://bazaar.launchpad.net/~mysql/mysql-server/5.5/revision/3837

They are opposite though to solve this problem.

Laurynas Biveinis (laurynas-biveinis)
tags: added: upstream
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-535

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.