PXC 5.5.30-23.7.4.404 not able the use a different shared memory allocator library

Bug #1170345 reported by Frederic Descamps
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Fix Released
Medium
Alex Yurchenko
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Fix Released
Undecided
Raghavendra D Prabhu

Bug Description

With the latest version of PXC, it's not possible to define a malloc-lib and start the server using init script or mysqld_safe.

To define the malloc-lib in my.cnf:

[mysqld_safe]
malloc-lib=/usr/lib64/libjemalloc.so.1

Then when you start using init script, this is the only message you get in error log:

30418 07:13:08 mysqld_safe Starting mysqld daemon with databases from /data/mysql
130418 07:13:08 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.AmupX31802
130418 07:13:08 mysqld_safe WSREP: Failed to recover position:

When you start it manually with mysld_safe:

130418 07:22:50 mysqld_safe Adding '/usr/lib64/libjemalloc.so.1' to LD_PRELOAD for mysqld
130418 07:22:50 mysqld_safe Logging to '/data/mysql/mysql-error.log'.
130418 07:22:50 mysqld_safe Starting mysqld daemon with databases from /data/mysql
130418 07:22:50 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.JvKKr11395
/usr/bin/mysqld_safe: line 224: LD_PRELOAD=/usr/lib64/libjemalloc.so.1: No such file or directory
130418 07:22:50 mysqld_safe WSREP: Failed to recover position:

The command launched is : LD_PRELOAD=/usr/lib64/libjemalloc.so.1 nohup /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/data/mysql/mysql-error.log --open-files-limit=8192 --pid-file=/data/mysql/mysql.pid --socket=/data/mysql/mysql.sock --log_error=/tmp/tmp.MXrZjr5440 --wsrep-recover

How to fix ?

on line 224 of /usr/bin/mysqld_safe replace

$mysqld_cmd --log_error=$wr_logfile --wsrep-recover

by:

eval $mysqld_cmd --log_error=$wr_logfile --wsrep-recover

summary: - PXC 5.5.30-23.7.4.404 not able the use a different share memory
- allocatot library
+ PXC 5.5.30-23.7.4.404 not able the use a different shared memory
+ allocator library
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Yes, the wsrep_recovery needs eval_log_error:

diff -u =mysqld_safe /tmp/mysqld_safe
--- /usr/bin/mysqld_safe 2013-04-18 18:48:27.595069642 +0530
+++ /tmp/mysqld_safe 2013-04-18 18:45:55.539099781 +0530
@@ -221,7 +221,7 @@

   log_notice "WSREP: Running position recovery with --log_error=$wr_logfile"

- $mysqld_cmd --log_error=$wr_logfile --wsrep-recover
+ eval_log_error "$mysqld_cmd --log_error=$wr_logfile --wsrep-recover"

   local rp="$(grep 'WSREP: Recovered position:' $wr_logfile)"
   if [ -z "$rp" ]; then

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

I don't think this is a merge regression, this bug has been present ever since wsrep_recover was added to mysqld_safe.

But I presume nobody tried to use malloc-lib with PXC before, so it is failing now.

Another workaround would be to export LD_PRELOAD in the init script.

Changed in percona-xtradb-cluster:
status: New → Confirmed
assignee: nobody → Raghavendra D Prabhu (raghavendra-prabhu)
Changed in codership-mysql:
assignee: nobody → Alex Yurchenko (ayurchen)
importance: Undecided → Medium
milestone: none → 5.5.30-24.8
status: New → Confirmed
Revision history for this message
Frederic Descamps (lefred) wrote :

Was working with Percona-XtraDB-Cluster-server-5.5.29-23.7.1.388

Changed in percona-xtradb-cluster:
status: Confirmed → Fix Committed
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

No, it wasn't.

==========================================================
+ chmod 600 /tmp/tmp.VU380MxLV9
+ log_notice WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ log_generic daemon.notice WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ priority=daemon.notice
+ shift
+ date +%y%m%d %H:%M:%S
+ msg=130418 19:24:35 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ echo 130418 19:24:35 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
130418 19:24:35 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ echo 130418 19:24:35 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.VU380MxLV9
+ LD_PRELOAD=/usr/lib/libjemalloc.so.1 nohup /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/lib/mysql/Buu.err --open-files-limit=65535 --pid-file=/var/lib/mysql/Buu.pid --log_error=/tmp/tmp.VU380MxLV9 --wsrep-recover
/usr/bin/mysqld_safe: 217: /usr/bin/mysqld_safe: LD_PRELOAD=/usr/lib/libjemalloc.so.1: not found
+ grep WSREP: Recovered position: /tmp/tmp.VU380MxLV9
+ rp=
+ [ -z ]
+ grep WSREP /tmp/tmp.VU380MxLV9
+ grep skipping position recovery
+ skipped=
+ [ -z ]
+ cat /tmp/tmp.VU380MxLV9
+ log_error WSREP: Failed to recover position:
+ log_generic daemon.error WSREP: Failed to recover position:
+ priority=daemon.error

=========================================================================

It failed there silently.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Full log for 5.5.29 here http://sprunge.us/ZRTO

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Fix pushed in lp:codership-mysql/5.5-23 r3864

Changed in codership-mysql:
status: Confirmed → Fix Committed
Revision history for this message
Frederic Descamps (lefred) wrote :

Raghu: it wasn't able to perform wsrep-recover but mysqld was able to run

Changed in percona-xtradb-cluster:
status: Fix Committed → Fix Released
Changed in percona-xtradb-cluster:
milestone: none → 5.5.30-23.7.4-2
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

@Fred,

Yes, that was also a bug. It should fail if it was not able to
wsrep-recover, than fail silently and proceed.

Changed in codership-mysql:
milestone: 5.5.30-24.8 → 5.5.31-23.7.4
Changed in codership-mysql:
status: Fix Committed → Fix Released
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1338

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.