lp:~akopytov/percona-xtrabackup/parallel-compression

Created by Alexey Kopytov on 2011-06-29 and last modified on 2012-02-10
Get this branch:
bzr branch lp:~akopytov/percona-xtrabackup/parallel-compression
Only Alexey Kopytov can upload to this branch. If you are Alexey Kopytov please log in for upload directions.

Branch merges

Related bugs

Related blueprints

Branch information

Recent revisions

386. By Alexey Kopytov on 2012-02-10

Rebase of the parallel compression patch on new trunk + post-review
fixes.

Implementation of parallel compression and streaming for XtraBackup.

This revision implements the following changes:

* InnoDB files are now streamed by the xtrabackup binary rather than
innobackupex. As a result, integrity is now verified by xtrabackup and
thus tar4ibd is no longer needed, so it was removed.

* xtrabackup binary now accepts the new '--stream' option which has
exactly the same semantics as the '--stream' option in
innobackupex: it tells xtrabackup to stream all files to the standard
output in the specified format rather than storing them locally.

* The xtrabackup binary can now do parallel compression using the
quicklz library. Two new options were added to xtrabackup to support
this feature:

- '--compress' tells xtrabackup to compress all output data, including
the transaction log file and meta data files, using the specified
compression algorithm. The only currently supported algorithm is
'quicklz'. The resulting files have the qpress archive format,
i.e. every *.qp file produced by xtrabackup is essentially a one-file
qpress archive and can be extracted and uncompressed by the qpress
file archiver (http://www.quicklz.com/).

- '--compress-threads' specifies the number of worker threads used by
xtrabackup for parallel data compression. This option defaults to 1.

Parallel compression ('--compress-threads') can be used together with
parallel file copying ('--parallel'). For example, '--parallel=4
--compress --compress-threads=2' will create 4 IO threads that will
read the data and pipe it to 2 compression threads.

* To support simultaneous compression and streaming, a new custom
streaming format called 'xbstream' was introduced to XtraBackup in
addition to the 'tar' format. That was required to overcome some
limitations of traditional archive formats such as 'tar', 'cpio' and
others that do not allow streaming dynamically generated files, for
example dynamically compressed files. Other advantages of xbstream over
traditional streaming/archive formats include ability to stream multiple
files concurrently (so it is possible to use streaming in the xbstream
format together with the --parallel option) and more compact data
storage.

* To allow streaming and extracting files to/from the xbstream format
produced by xtrabackup, a new utility aptly called 'xbstream' was
added to the XtraBackup distribution. This utility has a tar-like
interface:

- with the '-x' option it extracts files from the stream read from its
standard input to the current directory unless specified otherwise
with the '-C' option.

- with the '-c' option it streams files specified on the command line
to its standard output.

The utility also tries to minimize its impact on the OS page cache by
using the appropriate posix_fadvise() calls when available.

385. By Stewart Smith on 2012-02-10

merge bug fix for bug#917823: log block numbers mismatch

384. By Stewart Smith on 2012-02-10

merge bugfix for bug#733651: include page_size, fast_checksum and log_block_size in backup-my.cnf

383. By Stewart Smith on 2012-02-08

add release date of 1.9.0 in documentation

382. By Stewart Smith on 2012-02-08

merge 1.6.5 release notes

381. By Alexey Kopytov on 2012-02-06

Automerge from 1.6.

380. By Alexey Kopytov on 2012-02-02

Cherrypick of bugfix for bug #907147.

379. By Alexey Kopytov on 2012-02-02

Null-merged all missing 1.6 revisions after cherrypicking from the kewpie branch.

378. By Alexey Kopytov on 2012-02-02

Cherrypick of bugfix for: Bug #891496: tar4ibd crashes on multi-file tablespace files.

377. By Alexey Kopytov on 2012-02-02

Cherrypick of bug826632 fix: parallel option breaks incremental backups

Branch metadata

Branch format:
Branch format 7
Repository format:
Bazaar repository format 2a (needs bzr 1.16 or later)
Stacked on:
lp:percona-xtrabackup/2.0
This branch contains Public information 
Everyone can see this information.