Version check crashes innobackupex on CentOS 5

Ville,

It's most likely a Perl or DBD::MySQL bug. Can you get a gdb backtrace from the core file?

Sure. Looks like the culprit is perl-Net-SSLeay or openssl when innobackupex tries to access https://v.percona.com, see attachment. "/usr/bin/GET https://v.percona.com" which uses perl and eventually the same libs doesn't crash so it must be something specific to what the innobackupex version check does. (I wonder why you're not using libwww-perl for the version check BTW...)

i'm having the same issues, is there a fix

The problem is only exists with Percona-Server-shared-compat installed.

rm -rf /tmp/percona-version-check

LD_PRELOAD=/usr/lib64/libmysqlclient.so.15 innobackupex --version-check .
140108 04:52:13 innobackupex: Connected to MySQL server
140108 04:52:13 innobackupex: Executing a version check against the server...
Segmentation fault (core dumped)

But,
I have downloaded libmysqlclient.so.15 from: http://mirror.logol.ru/centos/5.10/os/x86_64//CentOS/mysql-5.0.95-5.el5_9.x86_64.rpm
rpm2cpio mysql-5.0.95-5.el5_9.x86_64.rpm | cpio -ivd ./usr/lib64/mysql/libmysqlclient.so.15.0.0
LD_PRELOAD=./usr/lib64/mysql/libmysqlclient.so.15.0.0 innobackupex --version-check .
140108 04:58:29 innobackupex: completed OK!

The issue and workaround also affects other scripts using SSL and mysqlclient, e.g. pt-agent

same version as Ville and same problem

centos-release-5-10.el5.centos
MySQL-client-5.6.15-1.rhel5
MySQL-devel-5.6.15-1.rhel5
MySQL-server-5.6.15-1.rhel5
MySQL-shared-5.6.15-1.rhel5
MySQL-shared-compat-5.6.15-1.rhel5
percona-xtrabackup-2.1.6-702.rhel5
perl-5.8.8-41.el5
perl-DBD-MySQL-3.0007-2.el5

[root@iduna ~]# innobackupex /tmp/
140119 22:30:25 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup' as 'root' (using password: YES).
140119 22:30:25 innobackupex: Connected to MySQL server
140119 22:30:25 innobackupex: Executing a version check against the server...
Segmentation fault

>> LD_PRELOAD=/usr/lib64/libmysqlclient.so.15 innobackupex --version-check .

part is interesting.

On CentOS6:
 mysql.so from perl-DBD-MySQL links against libmysqlclient.so.16

        ldd /usr/lib64/perl5/auto/DBD/mysql/mysql.so
        libmysqlclient.so.16 => /usr/lib64/libmysqlclient.so.16

However, on CentOS5:

http://pkgs.org/centos-5/centos-x86_64/perl-DBD-MySQL-3.0007-2.el5.x86_64.rpm.html

it links against "libmysqlclient.so.15(libmysqlclient_15)(64bit)"

(also perl-DBD-MySQL on CentOS5 is really really old, you can use
other repos on CentOS5 to get latest DBD though)

So, that explains the CentOS 5 v/s CentOS 6 disparity wrt. this
crash.

The next question is that of libmysqlclient.so.15. In our
packaging, we get this from upstream (Oracle) and package it into
our shared-compat, which also explains why it crashes with
MySQL-shared-compat package as well.

Changing the project to Percona Server, as the root cause seems to be a packaging bug in Percona-Server-shared-compat.

Please note that I and some others affected by this bug don't even have Percona-Server-shared-compat installed (see initial comment and comment 7, we have MySQL-shared-compat from Oracle instead) so the diagnosis in comment 4 is not correct and fixing something in Percona-Server-shared-compat cannot affect us in any way.

libmysqlclient.so.15 in Percona-Server-shared-compat is a binary copy from MySQL-shared-compat. Which means the same problem exists upstream as well.

Comment #4 says that the problem does NOT exist with libmysqlclient.so.15 taken from a CentOS MySQL 5.0 RPM.

So this should be reported against upstream packages. Then we either have to fix it in Percona-Server-shared-compat or wait for an upstream fix.

Yes, the libmysqlclient.so.15 from upstream (which is repackaged
in -shared-compat) is indeed the issue.

As a workaround you may also want to try with latter perl-DBD-MySQL - http://pkgs.org/centos-5/flexbox-x86_64/perl-DBD-MySQL-4.017-1.x86_64.rpm.html But, since this is also dependant on libmysqlclient.so.15 it may crash there.

Also, here are other sources of libmysqlclient.so.15 - http://pkgs.org/download/libmysqlclient.so.15(libmysqlclient_15)(64bit)

Has this been reported to the upstream?

Not yet. As far as I remember I asked somebody to do this last week. Will check later this week.

Encountering this too. Just did a "yum update xtrabackup" today, so presumably it's still a problem upstream.

We are also affected by this bug .

Furthermore, I think --no-version-check should be enabled by default (that is --version-check disabled) .

FWIW, I think so too -- it's not the business of my backup tool to do things like that (at all IMO, but at least not by default). Luckily it can be disabled.

I agree, the decision to enable VC by default is dubious. I marked Percona Toolkit bug #1279502 as affecting PXB as well. Feel free to speak up or use the "Does this bug affect you?" link.

I tested with Percona-Server-shared-compat and with Oracle MySQL shared compat however, i didn't faced any issues with Oracle MySQL shared compat but it few times failed with Percona-Server-shared-compat and sometimes worked during --version-check.

$ cat /etc/*release*
CentOS release 5.9 (Final)

$ rpm -qa | grep -i percona
 Percona-Server-shared-compat-5.5.34-rel32.0.591.rhel5

$ xtrabackup --version
xtrabackup version 2.1.6 for Percona Server 5.1.70 unknown-linux-gnu (x86_64) (revision id: 702)

LD_PRELOAD=/usr/lib64/libmysqlclient.so.15 innobackupex --user=msandbox --password=msandbox --socket=/tmp/mysql_sandbox55528.sock --version-check .
140217 15:26:34 innobackupex: Executing a version check against the server...
Segmentation fault

$ rpm --nodeps -e Percona-Server-shared-compat-5.5.34-rel32.0.591.rhel5

$ rpm -ivh MySQL-shared-compat-5.5.36-1.rhel5.x86_64.rpm
       Preparing... ########################################### [100%]
   1:MySQL-shared-compat ########################################### [100%]

$ LD_PRELOAD=/usr/lib64/libmysqlclient.so.15 innobackupex --user=msandbox --password=msandbox --socket=/tmp/mysql_sandbox55528.sock --version-check .
140217 15:57:02 innobackupex: Executing a version check against the server...
140217 15:57:02 innobackupex: Done.
140217 15:57:08 innobackupex: Connection to database server closed
140217 15:57:08 innobackupex: completed OK!

Oracle MySQL-shared-compat-5.5.36-1.rhel5.x86_64
$ md5sum /usr/lib64/libmysqlclient.so.15 /usr/lib64/libmysqlclient.so.15.0.0
b54366e9aa491f80fba7243e3186e049 /usr/lib64/libmysqlclient.so.15
b54366e9aa491f80fba7243e3186e049 /usr/lib64/libmysqlclient.so.15.0.0

Percona-Server-shared-compat-5.5.34-rel32.0.591.rhel5
$ md5sum /usr/lib64/libmysqlclient.so.15 /usr/lib64/libmysqlclient.so.15.0.0
b54366e9aa491f80fba7243e3186e049 /usr/lib64/libmysqlclient.so.15
b54366e9aa491f80fba7243e3186e049 /usr/lib64/libmysqlclient.so.15.0.0

Note, this doesn't affect PXB in PXC's xtrabackup SST since there it is invoked with --no-version-check (since SST is invoked non-interactively it was deemed unnecessary there). So, PXC shouldn't be affected by this.

Setting to New so that it's reported upstream. Whoever does that, please set back to Confirmed afterwards, thanks.

What is to be reported upstream I wonder? According to https://bugs.launchpad.net/percona-server/+bug/1255451/comments/17 there is no problem with "Oracle MySQL shared compat"...

There is some contradiction between comments #4 and #17 on one hand and #6, #8, #9, #10 on another.

We can keep it as our bug and hopefully its analysis will point out whether there is anything to report upstream.

While trying to take the backup, I am also affected by the "Segmentation fault" bug.

Details :
 xtrabackup -v
xtrabackup version 2.2.3 based on MySQL server 5.6.17 Linux (x86_64) (revision id: )

MySql version:

version | 5.6.17-66.0-log
version_comment | Percona Server (GPL), Release 66.0, Revision 608
version_compile_machine | x86_64

Error details :

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p
140909 13:44:38 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'miketodd' (using password: YES).
140909 13:44:38 innobackupex: Connected to MySQL server
140909 13:44:38 innobackupex: Executing a version check against the server...
Segmentation fault

The same issue exists in redhat. I can use LD_PRELOAD to get around the seg fault on the backup but the restore fails no matter what I set LD_PRELOAD to, percona versions and mysql versions, of libmysqlclient.so.

Using the following versions:

Red Hat Enterprise Linux Server release 5.10 (Tikanga)

MySQL-client.x86_64 5.5.31-2.rhel5 installed
MySQL-devel.x86_64 5.5.31-2.rhel5 installed
MySQL-python.x86_64 1.2.3-0.1.c1.el5 installed
MySQL-server.x86_64 5.5.31-2.rhel5 installed
MySQL-shared.x86_64 5.5.31-2.rhel5 installed
MySQL-shared-compat.x86_64 5.5.31-2.rhel5 installed
perl-DBD-MySQL.x86_64 3.0007-2.el5 installed
percona-xtrabackup.x86_64 2.2.8-5059.el5 installed

Restore command: innobackupex --apply-log --use-memory=75G /data/mysql_data/3306 --defaults-file=/etc/my.cnf

Error on restore:

InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 395505567, file name 81006-db03a-log.001353
InnoDB: Last MySQL binlog file position 0 220080926, file name /data/mysql_logs/3306/81006-db03b-bin.001348
InnoDB: 128 rollback segment(s) are active.
InnoDB: Waiting for purge to start
15:21:07 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x32) [0xa40e5d]
xtrabackup(handle_fatal_signal+0x335) [0x9158b1]
/lib64/libpthread.so.0 [0x30f2e0eca0]
xtrabackup(THD::get_stmt_da()+0xc) [0x7b2cca]
xtrabackup(THD::raise_condition(unsigned int, char const*, Sql_condition::enum_warning_level, char const*)+0x23) [0x902435]
xtrabackup(push_warning(THD*, Sql_condition::enum_warning_level, unsigned int, char const*)+0x3e) [0x92a072]
xtrabackup(push_warning_printf(THD*, Sql_condition::enum_warning_level, unsigned int, char const*, ...)+0x112) [0x92a186]
xtrabackup(ib_warn_row_too_big(dict_table_t const*)+0xb3) [0x698fbf]
xtrabackup(dict_index_add_to_cache(dict_table_t*, dict_index_t*, unsigned long, unsigned long)+0x170) [0x7117ea]
xtrabackup [0x6862cb]
xtrabackup(dict_load_table(char const*, unsigned long, dict_err_ignore_t)+0x4b2) [0x684936]
xtr...

The error on restore in comment #24 is not related to this issue, that is an xtrabackup binary crach, not innobackupex perl crash related to version check. If you continue to see this, please report it as a new bug against percona xtrabackup.

I have tried several combinations of upstream MySQL community 5.5, 5.6 and Percona Server 5.5 and 5.6 with xtrabackup 2.1 and 2.2 on CentOS 5.8 and can not reproduce the issue. Can anyone give me a sequence of packages to install, where from and a specific order from a fresh CentOS 5.x install to reproduce this with?

The initial comment contained a fairly specific list of packages and their origins; MySQL* from Oracle, and obviously percona-xtrabackup from Percona and the rest from CentOS. Note that it also mentioned CentOS 5.10, not 5.8 which you said you were testing with.

Anyway, I no longer have access to any CentOS 5.x boxes nor am I interested in this bug any more, so I'll unsubscribe from it now to stop the mail flood.

https://github.com/percona/percona-xtrabackup/pull/44

Let's keep this bug as version check crashing innobackupex and aborting backup. If you see any other issues with Perl + DBD::MySQL please file separate bug report.

Is the crash going to be fixed or just ignored with this patch? Does the crash leave ugly cores around?

Sent from my iPhone

> On May 7, 2015, at 10:26 AM, Sergei Glushchenko <email address hidden> wrote:
>
> https://github.com/percona/percona-xtrabackup/pull/44
>
> ** Also affects: percona-xtrabackup/2.2
> Importance: Undecided
> Status: New
>
> ** Also affects: percona-xtrabackup/2.3
> Importance: High
> Assignee: George Ormond Lorch III (gl-az)
> Status: Confirmed
>
> ** Changed in: percona-xtrabackup/2.2
> Status: New => Fix Released
>
> ** Changed in: percona-xtrabackup/2.2
> Importance: Undecided => High
>
> ** Changed in: percona-xtrabackup/2.2
> Assignee: (unassigned) => George Ormond Lorch III (gl-az)
>
> ** Changed in: percona-xtrabackup/2.3
> Status: Confirmed => Invalid
>
> ** Changed in: percona-xtrabackup/2.3
> Assignee: George Ormond Lorch III (gl-az) => (unassigned)
>
> ** Changed in: percona-xtrabackup/2.3
> Importance: High => Undecided
>
> ** Changed in: percona-xtrabackup/2.2
> Milestone: None => future-2.2-releases
>
> ** No longer affects: percona-server
>
> ** No longer affects: percona-server/5.5
>
> ** No longer affects: percona-server/5.6
>
> ** Summary changed:
>
> - Version check crashes perl on CentOS 5
> + Version check crashes innobackupex on CentOS 5
>
> --
> You received this bug notification because you are subscribed to Percona
> XtraBackup.
> Matching subscriptions: xtrabackup-bugs
> https://bugs.launchpad.net/bugs/1255451
>
> Title:
> Version check crashes innobackupex on CentOS 5
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/percona-xtrabackup/+bug/1255451/+subscriptions

Alexey,

Any proof that PS IS affected? So far I was able to reproduce only with Oracle's packages (old versions). And it is conflict between libmysqlclient and libcrypto. It look similar to https://bugs.mysql.com/bug.php?id=65055.

There are two workarounds by the way.

First is to LD_PRELOAD libcrypto
Second is to put require IO::Socket::SSL (in innobackupex) before require DBD::mysql.
Both force libcrypto to load before libmysqlclient, so functions needed for SSL will be taken from libcrypto.

If someone however manage to setup xtrabackup to talk with MySQL via SSL, both workarounds will likely stop to work.

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-174