Comment 9 for bug 1368846

Please consider the delay in response as I was waiting for the backup to
complete. We are still hitting with the similar (Incremental Backup)
assertion failure and here is the recent error message from apply log
process.

Backup :

$INNOBACKUPEX_BIN --no-version-check --user=$USER --password=$PASSWORD
--slave-info --incremental --incremental-basedir=$FULLDIR
$BACKUP_DIR/incremental >> $xb_backup_log 2>&1

Apply log:

$INNOBACKUPEX_BIN --no-version-check --apply-log
--incremental-dir=$LAST_BACKUP $($FIND $BACKUP_DIR/full -maxdepth 1
-mindepth 1 -type d -name 20\* | sort | tail -n 1) >> $xb_prepare_log 2>&1

Log:

xtrabackup: starting shutdown with innodb_fast_shutdown = 1
InnoDB: FTS optimize thread exiting.
InnoDB: Starting shutdown...
2014-10-15 07:01:56 4b013940 InnoDB: Assertion failure in thread
1258371392 in file buf0flu.cc line 2507
InnoDB: Failing assertion: UT_LIST_GET_LEN(buf_pool->flush_list) == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:01:56 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x32) [0xa3bc9d]
xtrabackup(handle_fatal_signal+0x335) [0x910c05]
/lib64/libpthread.so.0 [0x36aac0eb10]
/lib64/libc.so.6(gsignal+0x35) [0x36aa030265]
/lib64/libc.so.6(abort+0x110) [0x36aa031d10]
xtrabackup(buf_flush_page_cleaner_thread+0x54e) [0x7680ec]
/lib64/libpthread.so.0 [0x36aac0673d]
/lib64/libc.so.6(clone+0x6d) [0x36aa0d44bd]

On Tue, Oct 14, 2014 at 11:36 AM, Alexey Kopytov <<email address hidden>
> wrote:

> The error log in comment #7 is truncated, so it is not clear whether it
> is the same assertion as in the original report, or a different one.
> >From the stacktrace in comment #7 it looks like it was a different
> assertion. Can you confirm this?
>
> ** Changed in: percona-xtrabackup
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1368846
>
> Title:
> xtrabackup Assertion failure with Apply log process
>
> Status in Percona XtraBackup:
> Incomplete
>
> Bug description:
> While applying the log we are facing an assertion failure issue with
> xtrabackup version 2.1.9 so we decided to upgrade the tool to version
> 2.2.3. But still we are hitting the same error.
>
> The issue is not occurring on every try, last time we found the issue
> when we initiated an incremental backup proceeding to a full backup.
> We have completed the full backup successfully but the incremental
> ended with the same error. Log details are given below.
>
> Version :
> [05:54:02 xxx@xxxx:~]# xtrabackup -v
> xtrabackup version 2.2.3 based on MySQL server 5.6.17 Linux (x86_64)
> (revision id: )
>
> [xxxx@xxxxxx~]$ innobackupex --version
> InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
>
> mysql version: 5.6.17-66.0-log
>
> Log:
>
> InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
> 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
> 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92
> 93 94 95 96 97 98 99
> InnoDB: Apply batch completed
> InnoDB: In a MySQL replication slave the last master binlog file
> InnoDB: position 0 329970412, file name data3-bin.001806
> InnoDB: Last MySQL binlog file position 0 883545871, file name
> data1-bin.002972
> InnoDB: 128 rollback segment(s) are active.
> InnoDB: Waiting for purge to start
> InnoDB: 5.6.17 started; log sequence number 13967966489023
>
> [notice (again)]
> If you use binary log and don't use any hack of group commit,
> the binary log position seems to be:
> InnoDB: Last MySQL binlog file position 0 883545871, file name
> data1-bin.002972
>
> xtrabackup: starting shutdown with innodb_fast_shutdown = 1
> InnoDB: FTS optimize thread exiting.
> InnoDB: Starting shutdown...
> 2014-09-12 05:13:40 4a7ee940 InnoDB: Assertion failure in thread
> 1249831232 in file buf0flu.cc line 2503
> InnoDB: Failing assertion: UT_LIST_GET_LEN(buf_pool->flush_list) == 0
> InnoDB: We intentionally generate a memory trap.
> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
> InnoDB: If you get repeated assertion failures or crashes, even
> InnoDB: immediately after the mysqld startup, there may be
> InnoDB: corruption in the InnoDB tablespace. Please refer to
> InnoDB:
> http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
> InnoDB: about forcing recovery.
> innobackupex: Error:
> innobackupex: ibbackup failed at /usr/bin/innobackupex line 2610.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/percona-xtrabackup/+bug/1368846/+subscriptions
>

--
Regards,

Arunjith A, Consulting Team 3
<email address hidden>
Skype percona.aaravindan
Ph: +91 9539992246 (India), IST (UTC +5:30)