xb_parallel_incremental test sporadically fails

Bug #1081882 reported by Stewart Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
High
Alexey Kopytov
2.0
Fix Released
High
Alexey Kopytov
2.1
Fix Released
High
Alexey Kopytov

Bug Description

from http://jenkins.percona.com/job/percona-xtrabackup-2.0/959/BUILD_TYPE=debug,Host=centos6-32,xtrabackuptarget=innodb51/testReport/junit/%28root%29/t_xb_parallel_incremental/sh/

121121 23:15:12 innobackupex: Starting ibbackup with command: /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/src/xtrabackup_plugin --defaults-file="/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/my.cnf" --defaults-group="mysqld" --prepare --target-dir=/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/full_backup --apply-log-only --incremental-dir=/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/inc_backup

/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/src/xtrabackup_plugin version 2.0.3 for MySQL server 5.1.59 pc-linux-gnu (i686) (revision id: undefined)
incremental backup from 6809259 is enabled.
xtrabackup: cd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/full_backup
xtrabackup: This target seems to be already prepared.
xtrabackup: xtrabackup_logfile detected: size=7716864, start_lsn=(14790201)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/inc_backup
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 7716864
121121 23:15:12 InnoDB: Warning: allocated tablespace 4, old maximum was 0

<SNIP>

InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 6486322, file name ./mysql-bin.000001
121121 23:15:26 InnoDB: Assertion failure in thread 3078010560 in file buf/buf0lru.c line 908
InnoDB: Failing assertion: buf_LRU_old_ratio >= 51
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
innobackupex: Error:
innobackupex: ibbackup failed at /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/innobackupex line 378.
2012-11-21 23:15:26: xb_parallel_incremental.sh: ===> innobackupex failed with exit code 255
2012-11-21 23:15:26: run.sh: Found a leftover mysqld processes with PID 29002, stopping it
------------

Also from http://jenkins.percona.com/job/percona-xtrabackup-2.0/954/BUILD_TYPE=release,Host=centos5-64,xtrabackuptarget=innodb51/testReport/junit/%28root%29/t_xb_parallel_incremental/sh/

121119 05:21:12 innobackupex: Starting ibbackup with command: /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/src/xtrabackup_plugin --defaults-file="/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup --incremental-basedir='/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/full_backup' --parallel=8
innobackupex: Waiting for ibbackup (pid=10057) to suspend
innobackupex: Suspend file '/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/xtrabackup_suspended'

<SNIP>

[05] Copying ./sakila/rental.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/rental.ibd.delta
[04] Copying ./sakila/inventory.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/inventory.ibd.delta
[03] Copying ./sakila/country.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/country.ibd.delta
[07] Copying ./sakila/store.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/store.ibd.delta
[08] Copying ./sakila/category.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/category.ibd.delta
[06] Copying ./sakila/address.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/address.ibd.delta
[07] ...done
[06] ...done
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[04] ...done
[05] xtrabackup: Error: 10 retries resulted in fail. File ./sakila/rental.ibd seems to be corrupted.
[05] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[05] xtrabackup: Error: failed to copy datafile.
innobackupex: Error: ibbackup child process has died at /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/innobackupex line 378.
2012-11-19 05:21:19: xb_parallel_incremental.sh: ===> innobackupex failed with exit code 2
2012-11-19 05:21:19: run.sh: Found a leftover mysqld processes with PID 9048, stopping it
2012-11-19 05:21:19: run.sh: Found a leftover mysqld processes with PID 8844, stopping it
------------

Related branches

Revision history for this message
Stewart Smith (stewart) wrote :

http://jenkins.percona.com/job/percona-xtrabackup-2.0/949/BUILD_TYPE=debug,Host=centos5-64,xtrabackuptarget=innodb51/testReport/junit/%28root%29/t_xb_parallel_incremental/sh/

also shows a failure similar to the second one:

[07] Copying ./sakila/store.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/store.ibd.delta
[03] Copying ./sakila/film.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/film.ibd.delta
[01] Copying ./ibdata1 to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/ibdata1.delta
[02] Copying ./sakila/inventory.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/inventory.ibd.delta
[05] Copying ./sakila/rental.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/rental.ibd.delta
[06] Copying ./sakila/address.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/address.ibd.delta
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[06] ...done
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Error: 10 retries resulted in fail. File ./sakila/store.ibd seems to be corrupted.
[07] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[07] xtrabackup: Error: failed to copy datafile.
innobackupex: Error: ibbackup child process has died at /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/innobackupex line 378.
2012-10-23 22:42:43: xb_parallel_incremental.sh: ===> innobackupex failed with exit code 2
2012-10-23 22:42:43: run.sh: Found a leftover mysqld processes with PID 4214, stopping it
2012-10-23 22:42:43: run.sh: Found a leftover mysqld processes with PID 7601, stopping it

Revision history for this message
Alexey Kopytov (akopytov) wrote :

The "buf_LRU_old_ratio >= 51" is a duplicate of bug #924492. Let's keep this bug as a report about corrupted pages error in xb_parallel_incremental.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-347

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.