Merge lp:~gl-az/percona-xtrabackup/BT23557-2.1-lp1160788 into lp:percona-xtrabackup/2.1

[George Ormond Lorch III]

Added --decompress and --decrypt options both with functioning --parallel to innobackupex based on lp1160788. These options options will decrypt and/or decompress a backup made with the --compress and/or --encrypt options.

When decrypting, the encryption algorithm and key used when the backup was taken MUST be provided via the --decrypt=ALGORITHM and --encrypt-key=LITERAL-KEY or --encrypt-key-file=KEY-FILE.

For decompression to work, the qpress binary must be present within the path.

--decrypt and --decompress may be used together at the same time to completely normalize a previously compressed and encrypted backup but in some rare instances there may be io buffer overflow issues which would require calling innobackupex twice instead of a combined single call (once for decryption and once for decompression).

The --parallel option will allow multiple files to be decrypted and/or decompressed simultaneously.

Use of these options will remove the original compressed/encrypted files and leave the results in the same location.

test suite cases have been modified to make use of these new options where appropriate:
  test/t/ib_compress_basic.sh --decompress
  test/t/ib_stream_compress.sh --decompress
  test/t/ib_stream_compress_encrypt.sh --decompress
  test/t/xb_compress.sh --decompress
  test/t/xb_compress_encrypt.sh --decrypt --decompress in two individual invocations of innobackupex
  test/t/xb_encrypt.sh --decrypt
  test/t/xb_parallel_compress.sh --decompress --parallel
  test/t/xb_parallel_compress_encrypt.sh --decrypt --decompress --parallel in a single invocation of innobackupex
  test/t/xb_parallel_encrypt.sh --decrypt --parallel

-----
Rebased on trunk and fixed previous MP issue.

jenkins http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.1-param/376/
-----
Rebased on trunk again and fixed more review issues.

jenkins http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.1-param/403/

[George Ormond Lorch III (gl-az)]

23557

jenkins
http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.1-param/325/

[George Ormond Lorch III (gl-az)]

Some metrics from HP Clout instance:

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 1
CPU socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 2
Stepping: 3
CPU MHz: 2666.760
BogoMIPS: 5333.52
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
NUMA node0 CPU(s): 0,1

Database was prepped with sysbench parallel-prepare, 16 tables and 5000000 rows for about ~20GB.

Backup was performed with --compress --encrypt=AES256

Serial, no parallel decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress ./backup

  real 10m50.492s
  user 9m51.688s
  sys 0m53.628s

Parallel with 2 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=2 ./backup

  real 5m38.778s
  user 9m50.381s
  sys 0m56.072s

Parallel with 4 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=4 ./backup

  real 5m38.861s
  user 9m50.900s
  sys 0m56.473s

So we see a linear improvement here with CPU availability as long as i/o can keep up

[George Ormond Lorch III (gl-az)]

Some metrics from some monster machine provided by Vadim:

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Stepping: 7
CPU MHz: 2100.000
BogoMIPS: 4199.43
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-31

Database was prepped with sysbench parallel-prepare, 32 tables and 2500000 rows for about ~20GB.

Backup was performed with --compress --encrypt=AES256

Serial, no parallel decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress ./backup

  real 7m57.206s
  user 8m14.755s
  sys 0m33.522s

Parallel with 16 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=16 ./backup

  real 2m30.460s
  user 9m50.874s
  sys 11m32.019s

Parallel with 24 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=24 ./backup

  real 2m1.898s
  user 9m58.801s
  sys 14m12.701s

Parallel with 32 forks decrypt/decompress:
  $time innobackupex --decrypt=AES256 --encrypt-key=12345678123456781234567812345678 --decompress --parallel=32 ./backup

  real 2m4.559s
  user 10m20.132s
  sys 21m29.176s

Again, we see a nice improvement up to the point we hit i/o limits...

[Stewart Smith (stewart)]

George Ormond Lorch III <email address hidden> writes:
> Some metrics from some monster machine provided by Vadim:

Blog post coming in 5..4..3..2..1 ? :)

--
Stewart Smith

[Vlad Lesin (vlad-lesin)]

Looks good. Two minor remarks:

1) I think there should be test which uses --decrypt and --decompress options in one innobackupex call.

2) Lines 498-500: Why there is "DECRYPT AND DECOMPRESS" log output if only decompression is used?

review: Needs Fixing (g2)

[Raghavendra D Prabhu (raghavendra-prabhu)]

Parallel decompression/decryption are nice. However, are there plans to
add this for streaming? From the tests I see, the decompress/decrypt
are being run on non-streamed backups and since it looks for files with
extensions (qp/xbcrypt), it is not possible to use this on streaming
backups (or earlier streamed backups in case of --encrypt since
decryption needs to be done before xbstream extraction there)

[George Ormond Lorch III (gl-az)]

No because qpress is not stored in a streaming compatible format.
xbcrypt can be handled via stream but not qp.
On 7/12/2013 2:09 AM, Raghavendra D Prabhu wrote:
> Parallel decompression/decryption are nice. However, are there plans to
> add this for streaming? From the tests I see, the decompress/decrypt
> are being run on non-streamed backups and since it looks for files with
> extensions (qp/xbcrypt), it is not possible to use this on streaming
> backups (or earlier streamed backups in case of --encrypt since
> decryption needs to be done before xbstream extraction there)
>

--
George O. Lorch III
Software Engineer, Percona
[phone] x542 US/Arizona (GMT -7)
skype: george.ormond.lorch.iii

[George Ormond Lorch III (gl-az)]

I shouldn't respond until I have had my full coffee...

There are really two problems in doing this right now. One is that
inobackupex --compress only passes the --compress option(s) down into
xtrabackup where only InnoDB files get compressed, all other files
currently go into the archive uncompressed so doing something like
"... xbcrypt -d ... | xbstream -x | qpress -di ..." can't work in a
pure stream since not all files in the stream are .qp. The other is that
qpress does not store the filename/directory structure within the .qp
and as such can't recreate the same output structure as say xbcrypt and
xbstream can, it basically just a block compress/decompresser.

Maybe one day when all of the decryption | decompression | destreaming
is included in a single executable that is aware of the full structure
and format it would be possible. Alexey might have some ideas on an easy
way to pull this off within innobackupex using some of his mad linux skills.

On 7/12/2013 7:37 AM, George Ormond Lorch III wrote:
> No because qpress is not stored in a streaming compatible format.
> xbcrypt can be handled via stream but not qp.
> On 7/12/2013 2:09 AM, Raghavendra D Prabhu wrote:
>> Parallel decompression/decryption are nice. However, are there plans to
>> add this for streaming? From the tests I see, the decompress/decrypt
>> are being run on non-streamed backups and since it looks for files with
>> extensions (qp/xbcrypt), it is not possible to use this on streaming
>> backups (or earlier streamed backups in case of --encrypt since
>> decryption needs to be done before xbstream extraction there)
>>
>

--
George O. Lorch III
Software Engineer, Percona
[phone] x542 US/Arizona (GMT -7)
skype: george.ormond.lorch.iii

[Vlad Lesin (vlad-lesin)]

1) Misprint in 477: "oarallel" instead of "parallel";

2) "DECRYPTING AND/OR DECOMPRESSING" in 524;

3) I have not found checking $option_parallel on positive value.

'parallel=i' => \$option_parallel,

It is supposed $option_parallel is integer. In the case if $option_parallel is less then 0 decrypt_decompress() will not work correctly. It seems there will be infinite loop in:

while ($freepidindex >= $option_parallel)

It should be checked somewhere.

review: Needs Fixing (g2)

[Alexey Kopytov (akopytov)]

review: Approve

=== modified file 'innobackupex'
--- innobackupex	2013-07-08 09:28:05 +0000
+++ innobackupex	2013-07-25 22:07:27 +0000
@@ -89,15 +89,16 @@
 my $option_throttle = '';
 my $option_sleep = '';
 my $option_compress = 999;
+my $option_decompress = '';
 my $option_compress_threads = 1;
 my $option_compress_chunk_size = '';
 my $option_encrypt = '';
+my $option_decrypt = '';
 my $option_encrypt_key = '';
 my $option_encrypt_key_file = '';
 my $encrypt_cmd = '';
 my $option_encrypt_threads = 1;
 my $option_encrypt_chunk_size = '';
-my $option_uncompress = '';
 my $option_export = '';
 my $option_use_memory = '';
 my $option_mysql_password = '';
@@ -315,7 +316,27 @@
 
 my $ibbackup_exit_code = 0;
 
-if ($option_copy_back) {
+if ($option_decompress || $option_decrypt) {
+    # decrypt/decompress files
+    if ($option_decompress && $option_decrypt) {
+      # Call mode 0 first as it is fastest but may cause stream buffer overruns
+      # requiring user invoke innobackupex twice, once for decrypt and again
+      # for decompress. Mode 0 will take .qb.xbcrypt and run it through
+      # xbcrypt | qpress > result. This should get all InnoDB files but any
+      # metadata or MyISAM, etc. will not be .qp.xbcrypt, only .xbcrypt
+      decrypt_decompress(0);
+      # Now call mode 1 to decrypt any uncompressed files, those with only a
+      # .xbcrypt extension
+      decrypt_decompress(1);
+      # Now, just in case there were any compressed files only, run throuh and
+      # decompress any .qp files
+      decrypt_decompress(2);
+    } elsif ($option_decompress) {
+      decrypt_decompress(2);
+    } else {
+      decrypt_decompress(1);
+    }
+} elsif ($option_copy_back) {
     # copy files from backup directory back to their original locations
     copy_back(0);
 } elsif ($option_move_back) {
@@ -409,7 +430,180 @@
     die "$prefix Error: $message";
 }
     
-
+#
+# Finds all files that match the given pattern recursively beneath
+# the given directory. Returns a list of files with their paths relative 
+# to the starting search directory.
+#
+sub find_all_matching_files {
+  my $dir = shift;
+  my $pattern = shift;
+  my $child;
+  my @dirlist;
+  my @retlist;
+ 
+  opendir(FINDDIR, $dir)
+        || Die "Can't open directory '$dir': $!\n";
+
+  while (defined($child = readdir(FINDDIR))) {
+    next if $child eq "." || $child eq "..";
+    if ($child =~ m/$pattern/) { push(@retlist, "$dir/$child"); }
+    elsif ( -d "$dir/$child") { push(@dirlist, "$dir/$child"); }
+  }
+  closedir(FINDDIR);
+
+  foreach $child (@dirlist) {
+    push(@retlist, find_all_matching_files("$child", "$pattern"));
+  }
+  return (@retlist);
+}
+
+#
+# decrypts and/or decompresses an individual file amd removes the source
+# on success.
+# mode is same mode as defined for sub decrypt_decompress()
+sub decrypt_decompress_file {
+  my $mode = shift;
+  my $file = shift;
+  my $file_ext = shift;
+  my $decrypt_opts = shift;
+
+  my $file_cmd;
+  my $dest_file=substr($file, 0, length($file) - length($file_ext));
+
+  if ($mode == 0) {
+    $file_cmd = "xbcrypt --decrypt $decrypt_opts --input=" . $file . " | qpress -dio > " . $dest_file;
+  } elsif ($mode == 1) {
+    $file_cmd = "xbcrypt --decrypt $decrypt_opts --input=" . $file . " --output=" . $dest_file;
+  } elsif ($mode == 2) {
+    $file_cmd = "qpress -do " . $file . " > " . $dest_file;
+  } else {
+    Die("Unknown decrypt_decompress mode : $mode\n"); 
+  }
+  print STDERR "$prefix $file_cmd\n";
+  system("$file_cmd") && Die "$file_cmd failed with $!\n";
+  system("rm -f $file");
+}
+
+#
+# Decrypts and decompresses a backup made with --compress and/or --encrypt
+#
+# mode 0 = decrypt and decompress .qp.xbcrypt files
+# mode 1 = decrypt .xbcrypt files
+# mode 2 = decompress .qp files
+#
+sub decrypt_decompress {
+  my $mode = shift;
+  my $file_ext;
+  my $decrypt_opts = '';
+  my @files;
+
+  # first, build out the decryption opts. They will not be used and option
+  # values may not be specified if doing decompress only.
+  if ($mode != 2) {
+    if ($option_decrypt) {
+      $decrypt_opts = "--encrypt-algo=$option_decrypt"
+    }
+    if ($option_encrypt_key) {
+      $decrypt_opts = $decrypt_opts . " --encrypt-key=$option_encrypt_key";
+    } elsif ($option_encrypt_key_file) {
+      $decrypt_opts = $decrypt_opts . " --encrypt-key-file=$option_encrypt_key_file";
+    }
+  }
+
+  # based on the mode, determine which files we are interested in
+  if ($mode == 0) {
+    $file_ext = ".qp.xbcrypt";
+  } elsif ($mode == 1) {
+    $file_ext = ".xbcrypt";
+  } elsif ($mode == 2) {
+    $file_ext = ".qp";
+  } else {
+    Die("Unknown decrypt_decompress mode : $mode\n"); 
+  }
+
+  # recursively find all files of interest in the backup set
+  @files = find_all_matching_files("$backup_dir", "\\$file_ext\$");
+
+  # runing serially
+  if (!$option_parallel) {
+    my $file;
+    foreach $file (@files) {
+      decrypt_decompress_file($mode, $file, $file_ext, $decrypt_opts);
+    }
+  # running parallel, FUN!
+  } else {
+    my @pids;
+    my @workingfiles;
+    my $file;
+    my $freepidindex;
+    
+    # this will maintain the list of forks so we can wait for completion
+    # and check return status as well as stay within the limits of the 
+    # --parallel value
+    $pids[$option_parallel - 1] = undef;
+
+    # this is the file for which each fork is currently working on. just
+    # used for logging/reporting purposed in case of a failure.
+    $workingfiles[$option_parallel - 1] = undef;
+
+    foreach $file (@files) {
+      # this while loop below tries to find an unused child/fork slot, ie, one
+      # that is not currently working/running. it will loop indefinitely,
+      # iterating the number of slots and watching for a child to finish
+      # and use that now vacant slot.
+      $freepidindex=$option_parallel;
+      while ($freepidindex >= $option_parallel) {
+        for ($freepidindex = 0; $freepidindex < $option_parallel; $freepidindex++) {
+          last if not defined $pids[$freepidindex];
+
+          my $status = waitpid($pids[$freepidindex], WNOHANG);
+          if ($status == $pids[$freepidindex]) {
+            my $childretcode = $? >> 8;
+            if ($childretcode != 0) {
+              Die("Child failed on $workingfiles[$freepidindex] : $childretcode");
+            }
+            last;
+          } elsif ($status == -1) {
+            Die("waitpid failed on $workingfiles[$freepidindex] : $status");
+          }
+        }
+        # couldn't find a free slot yet so sleep for 1/10 th of a second or so.
+        # sleeping for a much longer (even a second) can make the whole process
+        # take longer when there are a lot of smaller files to process as they
+        # can be done very quickly. 1/10th doesn't seem to add any real overhead
+        # to this controlling process when compared to letting it spin freely.
+        if ($freepidindex >= $option_parallel) {
+          usleep(100000);
+        }
+      }
+      # so now we have a slot identified by freepidindex, set up the workingfiles,
+      # fork and do the work.
+      $workingfiles[$freepidindex] = $file;
+      $pids[$freepidindex] = fork();
+      if ($pids[$freepidindex] == 0) {
+        decrypt_decompress_file($mode, $file, $file_ext, $decrypt_opts);
+        exit 0;
+      }
+    }
+    # we've started processing of all files that we know about, now just need
+    # to wait for the last bunch to finish up and check for any errors before
+    # returning.
+    for ($freepidindex = 0; $freepidindex < $option_parallel; $freepidindex++) {
+      next if not defined $pids[$freepidindex];
+      my $status = waitpid($pids[$freepidindex], 0);
+      my $childretcode = $? >> 8;
+      if ($status == $pids[$freepidindex]) {
+        if ($childretcode != 0) {
+          Die("Child failed on $workingfiles[$freepidindex] : $childretcode");
+        }
+      } else {
+        Die("waitpid failed on $workingfiles[$freepidindex] : $status");
+      }
+    }
+  }
+}
+ 
 #
 # backup subroutine makes a backup of InnoDB and MyISAM tables, indexes and 
 # .frm files. It connects to the database server and runs ibbackup as a child
@@ -896,8 +1090,25 @@
                 Die "Backup data file '$backup_dir/$c->{filename}' "
                   . "does not exist, but "
                   . "its compressed copy '$c->{path}.ibz' exists. Check "
-                  . "that you have run "
-                  . "'$innobackup_script --apply-log --uncompress "
+                  . "that you have decompressed "
+                  . " before attempting "
+                  . "'$innobackup_script --copy-back ...'  "
+                  . "or '$innobackup_script --move-back ...' !";
+            } elsif (-e "$backup_dir/$c->{filename}.qp") {
+                Die "Backup data file '$backup_dir/$c->{filename}' "
+                  . "does not exist, but "
+                  . "its compressed copy '$c->{path}.qp' exists. Check "
+                  . "that you have run "
+                  . "'$innobackup_script --decompress "
+                  . "...' before attempting "
+                  . "'$innobackup_script --copy-back ...'  "
+                  . "or '$innobackup_script --move-back ...' !";
+            } elsif (-e "$backup_dir/$c->{filename}.xbcrypt") {
+                Die "Backup data file '$backup_dir/$c->{filename}' "
+                  . "does not exist, but "
+                  . "its compressed copy '$c->{path}.xbcrypt' exists. Check "
+                  . "that you have run "
+                  . "'$innobackup_script --decrypt "
                   . "...' before attempting "
                   . "'$innobackup_script --copy-back ...'  "
                   . "or '$innobackup_script --move-back ...' !";
@@ -990,9 +1201,6 @@
 
     $options = $options . "--prepare --target-dir=$backup_dir";
 
-    if ($option_uncompress) {
-        $options = $options . ' --uncompress';
-    }
     if ($option_export) {
         $options = $options . ' --export';
     }
@@ -1994,8 +2202,11 @@
     my $run = '';
 
     # print some instructions to the user
-    if (!$option_apply_log && !$option_copy_back && !$option_move_back) {
+    if (!$option_apply_log && !$option_copy_back && !$option_move_back
+        && !$option_decrypt && !$option_decompress) {
         $run = 'backup';
+    } elsif ($option_decrypt || $option_decompress) {
+        $run = 'decryption and decompression';
     } elsif ($option_copy_back) {
         $run = 'copy-back';
     } elsif ($option_move_back) {
@@ -2007,7 +2218,8 @@
     print STDERR "           At the end of a successful $run run $innobackup_script\n";
     print STDERR "           prints \"completed OK!\".\n\n";
 
-    if (!$option_copy_back && !$option_move_back) {
+    if (!$option_copy_back && !$option_move_back
+        && !$option_decrypt && !$option_decompress) {
         # we are making a backup or applying log to backup
         if (!$option_apply_log) {
             # we are making a backup, we need mysql server
@@ -2030,19 +2242,22 @@
         }
     }
 
-    # read MySQL options file
-    #read_config_file($config_file, \%config);
-    read_config_file(\%config);
+    if (!$option_decrypt && !$option_decompress) {
+      # read MySQL options file
+      #read_config_file($config_file, \%config);
+      read_config_file(\%config);
 
-    if(!$option_tmpdir) {
-        $option_tmpdir = get_option(\%config, $option_defaults_group, 'tmpdir');
+      if(!$option_tmpdir) {
+          $option_tmpdir = get_option(\%config, $option_defaults_group, 'tmpdir');
+      }
     }
 
     # get innodb log home directory from options file
     #$innodb_log_group_home_dir = 
     #    get_option(\%config, 'mysqld', 'innodb_log_group_home_dir');
 
-    if (!$option_apply_log && !$option_copy_back && !$option_move_back) {
+    if (!$option_apply_log && !$option_copy_back && !$option_move_back
+        && !$option_decrypt && !$option_decompress) {
         # we are making a backup, create a new backup directory
         $backup_dir = File::Spec->rel2abs(make_backup_dir());
         print STDERR "$prefix Created backup directory $backup_dir\n";
@@ -2167,9 +2382,11 @@
 
     # read command line options
     $rcode = GetOptions('compress' => \$option_compress,
+                        'decompress' => \$option_decompress,
                         'compress-threads=i' => \$option_compress_threads,
                         'compress-chunk-size=s' => \$option_compress_chunk_size,
                         'encrypt=s' => \$option_encrypt,
+                        'decrypt=s' => \$option_decrypt,
                         'encrypt-key=s' => \$option_encrypt_key,
                         'encrypt-key-file=s' => \$option_encrypt_key_file,
                         'encrypt-threads=i' => \$option_encrypt_threads,
@@ -2187,7 +2404,6 @@
                         'databases=s' => \$option_databases,
                         'tables-file=s', => \$option_tables_file,
                         'use-memory=s' => \$option_use_memory,
-                        'uncompress' => \$option_uncompress,
                         'export' => \$option_export,
                         'password:s' => \$option_mysql_password,
                         'user=s' => \$option_mysql_user,
@@ -2266,6 +2482,7 @@
         $option_compress = 0;
     }
 
+<<<<<<< TREE
     # validate lock-wait-query-type and kill-long-query-type values
     if (!(grep {$_ eq $option_lock_wait_query_type} qw/all update/)) {
         Die "Wrong value of lock-wait-query-type. ".
@@ -2278,6 +2495,14 @@
             "is specified.";
     }
 
+=======
+    if ($option_parallel && $option_parallel < 1) {
+        print STDERR "$prefix --parallel must be a positive numerical value" .
+            "greater than 0";
+        exit(1);
+    }
+
+>>>>>>> MERGE-SOURCE
     if ($option_stream eq 'tar') {
       $stream_cmd = 'tar chf -';
     } elsif ($option_stream eq 'xbstream') {
@@ -2304,7 +2529,8 @@
         exit(1);
     }
 
-    if (!$option_apply_log && !$option_copy_back && !$option_move_back) {
+    if (!$option_apply_log && !$option_copy_back && !$option_move_back
+        && !$option_decrypt && !$option_decompress) {
         # we are making a backup, get backup root directory
         $option_backup = "1";
         $backup_root = $ARGV[0];
@@ -2335,6 +2561,13 @@
 	exit(1);
     }
 
+    if ($option_decompress) {
+      if (system("which qpress &>/dev/null") >> 8 != 0) {
+        print STDERR "--decompress requires qpress\n";
+	exit(1);
+      }
+    }
+
     print STDERR "\n";
 
     parse_databases_option_value();
@@ -3197,8 +3430,8 @@
 =head1 SYNOPOSIS
 
 innobackupex [--compress] [--compress-threads=NUMBER-OF-THREADS] [--compress-chunk-size=CHUNK-SIZE]
-             [--encrypt=encryption_algorithm] [--encrypt-threads=NUMBER-OF-THREADS] [--encrypt-chunk-size=CHUNK-SIZE]
-             [--encrypt-key=literal_encryption_key] | [--encryption-key-file=MY.KEY]
+             [--encrypt=ENCRYPTION-ALGORITHM] [--encrypt-threads=NUMBER-OF-THREADS] [--encrypt-chunk-size=CHUNK-SIZE]
+             [--encrypt-key=LITERAL-ENCRYPTION-KEY] | [--encryption-key-file=MY.KEY]
              [--include=REGEXP] [--user=NAME]
              [--password=WORD] [--port=PORT] [--socket=SOCKET]
              [--no-timestamp] [--ibbackup=IBBACKUP-BINARY]
@@ -3211,7 +3444,7 @@
              [--compact]     
              BACKUP-ROOT-DIR
 
-innobackupex --apply-log [--use-memory=B] [--uncompress]
+innobackupex --apply-log [--use-memory=B]
              [--defaults-file=MY.CNF]
              [--export] [--redo-only] [--ibbackup=IBBACKUP-BINARY]
              BACKUP-DIR
@@ -3220,6 +3453,10 @@
 
 innobackupex --move-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
 
+innobackupex [--decompress] [--decrypt=ENCRYPTION-ALGORITHM]
+             [--encrypt-key=LITERAL-ENCRYPTION-KEY] | [--encryption-key-file=MY.KEY]
+             [--parallel=NUMBER-OF-FORKS] BACKUP-DIR
+
 =head1 DESCRIPTION
 
 The first command line above makes a hot backup of a MySQL database.
@@ -3254,6 +3491,16 @@
 option removes backup files, it must be used with caution. It may be useful in
 cases when there is not enough free disk space to copy files.
 
+The --decompress --decrypt command will decrypt and/or decompress a backup made
+with the --compress and/or --encrypt options. When decrypting, the encryption
+algorithm and key used when the backup was taken MUST be provided via the
+specified options. --decrypt and --decompress may be used together at the same
+time to completely normalize a previously compressed and encrypted backup. The
+--parallel option will allow multiple files to be decrypted and/or decompressed
+simultaneously. In order to decompress, the qpress utility MUST be installed
+and accessable within the path. This process will remove the original
+compressed/encrypted files and leave the results in the same location.
+
 On success the exit code innobackupex is 0. A non-zero exit code 
 indicates an error.
 
@@ -3296,6 +3543,14 @@
 
 This option specifies the list of databases that innobackupex should back up. The option accepts a string argument or path to file that contains the list of databases to back up. The list is of the form "databasename1[.table_name1] databasename2[.table_name2] . . .". If this option is not specified, all databases containing MyISAM and InnoDB tables will be backed up.  Please make sure that --databases contains all of the InnoDB databases and tables, so that all of the innodb.frm files are also backed up. In case the list is very long, this can be specified in a file, and the full path of the file can be specified instead of the list. (See option --tables-file.)
 
+=item --decompress
+
+Decompresses all files with the .qp extension in a backup previously made with the --compress option.
+
+=item --decrypt=ENCRYPTION-ALGORITHM
+
+Decrypts all files with the .xbcrypt extension in a backup previously made with --encrypt option.
+
 =item --debug-sleep-before-unlock=SECONDS
 
 This is a debug-only option used by the XtraBackup test suite.
@@ -3308,23 +3563,23 @@
 
 This option specifies what extra file to read the default MySQL options from before the standard defaults-file.  The option accepts a string argument. It is also passed directly to xtrabackup's --defaults-extra-file option. See the xtrabackup documentation for details.
 
-=item --encrypt=ENCRYPTION_ALGORITHM
+=item --encrypt=ENCRYPTION-ALGORITHM
 
 This option instructs xtrabackup to encrypt backup copies of InnoDB data
-files using the algorithm specified in the ENCRYPTION_ALGORITHM.
-It is passed directly to the xtrabackup child process.
-Try 'xtrabackup --help' for more details.
-
-=item --encrypt-key=ENCRYPTION_KEY
-
-This option instructs xtrabackup to use the given ENCRYPTION_KEY when using the --encrypt option.
-It is passed directly to the xtrabackup child process.
-Try 'xtrabackup --help' for more details.
-
-=item --encrypt-key-file=ENCRYPTION_KEY_FILE
-
-This option instructs xtrabackup to use the encryption key stored in the given ENCRYPTION_KEY_FILE when using the --encrypt option.
-It is passed directly to the xtrabackup child process.
+files using the algorithm specified in the ENCRYPTION-ALGORITHM.
+It is passed directly to the xtrabackup child process.
+Try 'xtrabackup --help' for more details.
+
+=item --encrypt-key=ENCRYPTION-KEY
+
+This option instructs xtrabackup to use the given ENCRYPTION-KEY when using the --encrypt or --decrypt options.
+During backup it is passed directly to the xtrabackup child process.
+Try 'xtrabackup --help' for more details.
+
+=item --encrypt-key-file=ENCRYPTION-KEY-FILE
+
+This option instructs xtrabackup to use the encryption key stored in the given ENCRYPTION-KEY-FILE when using the --encrypt or --decrypt options.
+
 Try 'xtrabackup --help' for more details.
 
 =item --encrypt-threads
@@ -3426,8 +3681,9 @@
 
 =item --parallel=NUMBER-OF-THREADS
 
-This option specifies the number of threads the xtrabackup child process should use to back up files concurrently.  The option accepts an integer argument. It is passed directly to xtrabackup's --parallel option. See the xtrabackup documentation for details.
-
+On backup, this option specifies the number of threads the xtrabackup child process should use to back up files concurrently.  The option accepts an integer argument. It is passed directly to xtrabackup's --parallel option. See the xtrabackup documentation for details.
+ 
+On --decrypt or --decompress it specifies the number of parallel forks that should be used to process the backup files.
 
 =item --password=WORD
 

=== modified file 'test/inc/ib_stream_common.sh'
--- test/inc/ib_stream_common.sh	2012-10-15 16:18:40 +0000
+++ test/inc/ib_stream_common.sh	2013-07-25 22:07:27 +0000
@@ -29,17 +29,23 @@
 rm -r $mysql_datadir
 
 # Restore sakila
-vlog "Applying log"
+vlog "#####################"
+vlog "# EXTRACTING STREAM #"
+vlog "#####################"
 backup_dir=$topdir/backup
 cd $backup_dir
 run_cmd bash -c "$stream_extract_cmd out"
-test -n "${stream_uncompress_cmd:-""}" && run_cmd bash -c "$stream_uncompress_cmd";
+if [ -n "${stream_uncompress_cmd:=""}" ]; then 
+  vlog "################################"
+  vlog "# DECRYPTING AND DECOMPRESSING #"
+  vlog "################################"
+  run_cmd bash -c "$stream_uncompress_cmd";
+fi
 cd - >/dev/null 2>&1 
 vlog "###########"
 vlog "# PREPARE #"
 vlog "###########"
 innobackupex --apply-log $backup_dir
-vlog "Restoring MySQL datadir"
 mkdir -p $mysql_datadir
 vlog "###########"
 vlog "# RESTORE #"

=== modified file 'test/inc/xb_local.sh'
--- test/inc/xb_local.sh	2013-03-11 19:11:29 +0000
+++ test/inc/xb_local.sh	2013-07-25 22:07:27 +0000
@@ -31,14 +31,18 @@
 # Restore sakila
 vlog "Applying log"
 cd $backup_dir
-test -n "${data_decrypt_cmd:=""}" && run_cmd bash -c "$data_decrypt_cmd"
-test -n "${data_decompress_cmd:-""}" && run_cmd bash -c "$data_decompress_cmd";
+if [ -n "${data_decrypt_cmd:=""}" ] || [ -n "${data_decompress_cmd:=""}" ]; then 
+  vlog "###################################"
+  vlog "# DECRYPTING AND/OR DECOMPRESSING #"
+  vlog "###################################"
+  test -n "${data_decrypt_cmd:=""}" && run_cmd bash -c "$data_decrypt_cmd"
+  test -n "${data_decompress_cmd:-""}" && run_cmd bash -c "$data_decompress_cmd";
+fi
 cd - >/dev/null 2>&1
 vlog "###########"
 vlog "# PREPARE #"
 vlog "###########"
 innobackupex --apply-log $backup_dir
-vlog "Restoring MySQL datadir"
 mkdir -p $mysql_datadir
 vlog "###########"
 vlog "# RESTORE #"

=== modified file 'test/t/ib_compress_basic.sh'
--- test/t/ib_compress_basic.sh	2013-01-07 06:17:40 +0000
+++ test/t/ib_compress_basic.sh	2013-07-25 22:07:27 +0000
@@ -19,8 +19,7 @@
 
 cd $topdir/backup
 
-for i in *.qp;  do qpress -d $i ./; done; \
-for i in sakila/*.qp; do qpress -d $i sakila/; done
+innobackupex --decompress $topdir/backup
 
 innobackupex --apply-log $topdir/backup
 

=== modified file 'test/t/ib_stream_compress.sh'
--- test/t/ib_stream_compress.sh	2013-03-11 19:11:29 +0000
+++ test/t/ib_stream_compress.sh	2013-07-25 22:07:27 +0000
@@ -9,8 +9,7 @@
 
 stream_format=xbstream
 stream_extract_cmd="xbstream -xv <"
-stream_uncompress_cmd="for i in *.qp;  do qpress -d \$i ./; done; \
-for i in sakila/*.qp; do qpress -d \$i sakila/; done"
+stream_uncompress_cmd="innobackupex --decompress ./"
 innobackupex_options="--compress --compress-threads=4 --compress-chunk-size=8K"
 
 . inc/ib_stream_common.sh

=== modified file 'test/t/ib_stream_compress_encrypt.sh'
--- test/t/ib_stream_compress_encrypt.sh	2013-03-11 19:11:29 +0000
+++ test/t/ib_stream_compress_encrypt.sh	2013-07-25 22:07:27 +0000
@@ -11,8 +11,7 @@
 encrypt_key="percona_xtrabackup_is_awesome___"
 stream_format=xbstream
 stream_extract_cmd="(xbcrypt -d -a $encrypt_algo -k $encrypt_key | xbstream -xv) <"
-stream_uncompress_cmd="for i in *.qp;  do qpress -d \$i ./; done; \
-for i in sakila/*.qp; do qpress -d \$i sakila/; done"
+stream_uncompress_cmd="innobackupex --decompress ./"
 innobackupex_options="--compress --compress-threads=4 --compress-chunk-size=8K --encrypt=$encrypt_algo --encrypt-key=$encrypt_key --encrypt-threads=4 --encrypt-chunk-size=8K"
 
 . inc/ib_stream_common.sh

=== modified file 'test/t/xb_compress.sh'
--- test/t/xb_compress.sh	2013-03-11 19:11:29 +0000
+++ test/t/xb_compress.sh	2013-07-25 22:07:27 +0000
@@ -8,7 +8,6 @@
 fi
 
 innobackupex_options="--compress --compress-threads=4 --compress-chunk-size=8K"
-data_decompress_cmd="for i in *.qp;  do qpress -d \$i ./; done; \
-for i in sakila/*.qp; do qpress -d \$i sakila/; done"
+data_decompress_cmd="innobackupex --decompress ./"
 
 . inc/xb_local.sh

=== modified file 'test/t/xb_compress_encrypt.sh'
--- test/t/xb_compress_encrypt.sh	2013-03-11 19:11:29 +0000
+++ test/t/xb_compress_encrypt.sh	2013-07-25 22:07:27 +0000
@@ -13,14 +13,7 @@
 
 innobackupex_options="--compress --compress-threads=4 --compress-chunk-size=8K --encrypt=$encrypt_algo --encrypt-key=$encrypt_key --encrypt-threads=4 --encrypt-chunk-size=8K"
 
-data_decrypt_cmd="for i in *.xbcrypt; do \
-xbcrypt -d -a $encrypt_algo -k $encrypt_key < \$i > \${i:0:\${#i}-8}; \
-rm -f \$i; done; \
-for i in ./sakila/*.xbcrypt; do \
-xbcrypt -d -a $encrypt_algo -k $encrypt_key < \$i > \${i:0:\${#i}-8}; \
-rm -f \$i; done;"
-
-data_decompress_cmd="for i in *.qp;  do qpress -d \$i ./; done; \
-for i in sakila/*.qp; do qpress -d \$i sakila/; done"
+data_decrypt_cmd="innobackupex --decrypt=${encrypt_algo} --encrypt-key=${encrypt_key} ./"
+data_decompress_cmd="innobackupex --decompress ./"
 
 . inc/xb_local.sh

=== modified file 'test/t/xb_encrypt.sh'
--- test/t/xb_encrypt.sh	2013-07-09 15:03:03 +0000
+++ test/t/xb_encrypt.sh	2013-07-25 22:07:27 +0000
@@ -4,6 +4,7 @@
 
 encrypt_algo="AES256"
 encrypt_key="percona_xtrabackup_is_awesome___"
+<<<<<<< TREE
 encrypt_key_file=${TEST_VAR_ROOT}/xb_encrypt.key
 
 echo -n $encrypt_key > $encrypt_key_file
@@ -15,6 +16,13 @@
 for i in ./sakila/*.xbcrypt; do \
 xbcrypt -d -a $encrypt_algo -f $encrypt_key_file < \$i > \${i:0:\${#i}-8}; \
 rm -f \$i; done;"
+=======
+
+echo -n "Percona XtraBackup is awesome!!!" > $encrypt_key_file
+
+innobackupex_options="--encrypt=$encrypt_algo --encrypt-key=$encrypt_key --encrypt-threads=4 --encrypt-chunk-size=8K"
+data_decrypt_cmd="innobackupex --decrypt=${encrypt_algo} --encrypt-key=${encrypt_key} ./"
+>>>>>>> MERGE-SOURCE
 
 . inc/xb_local.sh
 

=== modified file 'test/t/xb_parallel_compress.sh'
--- test/t/xb_parallel_compress.sh	2013-03-11 19:11:29 +0000
+++ test/t/xb_parallel_compress.sh	2013-07-25 22:07:27 +0000
@@ -8,7 +8,6 @@
 fi
 
 innobackupex_options="--parallel=8 --compress --compress-threads=4 --compress-chunk-size=8K"
-data_decompress_cmd="for i in *.qp;  do qpress -d \$i ./; done; \
-for i in sakila/*.qp; do qpress -d \$i sakila/; done"
+data_decompress_cmd="innobackupex --decompress --parallel=8 ./"
 
 . inc/xb_local.sh

=== modified file 'test/t/xb_parallel_compress_encrypt.sh'
--- test/t/xb_parallel_compress_encrypt.sh	2013-03-11 19:11:29 +0000
+++ test/t/xb_parallel_compress_encrypt.sh	2013-07-25 22:07:27 +0000
@@ -12,14 +12,6 @@
 
 innobackupex_options="--parallel=8 --compress --compress-threads=4 --compress-chunk-size=8K --encrypt=$encrypt_algo --encrypt-key=$encrypt_key --encrypt-threads=4 --encrypt-chunk-size=8K"
 
-data_decrypt_cmd="for i in *.xbcrypt; do \
-xbcrypt -d -a $encrypt_algo -k $encrypt_key < \$i > \${i:0:\${#i}-8}; \
-rm -f \$i; done; \
-for i in ./sakila/*.xbcrypt; do \
-xbcrypt -d -a $encrypt_algo -k $encrypt_key < \$i > \${i:0:\${#i}-8}; \
-rm -f \$i; done;"
-
-data_decompress_cmd="for i in *.qp;  do qpress -d \$i ./; done; \
-for i in sakila/*.qp; do qpress -d \$i sakila/; done"
+data_decrypt_cmd="innobackupex --decrypt=${encrypt_algo} --encrypt-key=${encrypt_key} --decompress --parallel=8 ./"
 
 . inc/xb_local.sh

=== modified file 'test/t/xb_parallel_encrypt.sh'
--- test/t/xb_parallel_encrypt.sh	2013-03-11 19:11:29 +0000
+++ test/t/xb_parallel_encrypt.sh	2013-07-25 22:07:27 +0000
@@ -7,11 +7,6 @@
 encrypt_key="percona_xtrabackup_is_awesome___"
 
 innobackupex_options="--parallel=4 --encrypt=$encrypt_algo --encrypt-key=$encrypt_key --encrypt-threads=4 --encrypt-chunk-size=8K"
-data_decrypt_cmd="for i in *.xbcrypt; do \
-xbcrypt -d -a $encrypt_algo -k $encrypt_key < \$i > \${i:0:\${#i}-8}; \
-rm -f \$i; done; \
-for i in ./sakila/*.xbcrypt; do \
-xbcrypt -d -a $encrypt_algo -k $encrypt_key < \$i > \${i:0:\${#i}-8}; \
-rm -f \$i; done;"
+data_decrypt_cmd="innobackupex --decrypt=${encrypt_algo} --encrypt-key=${encrypt_key} --parallel=4 ./"
 
 . inc/xb_local.sh