Merge lp:~hrvojem/percona-xtrabackup/rn-2.1.0-alpha1 into lp:percona-xtrabackup/2.1

[Hrvoje Matijakovic]

[Alexey Kopytov (akopytov)]

A couple of comments:

   - the note about possible CRLF is present twice. Is that intentional?
   - the argument to --encrypt-chunk-size is not measured in kilobytes
     (as originally implemented). It must be specified in bytes now, but
     accepts unit suffixes (i.e. 'K', 'M' or 'G' to indicate kilobytes,
     megabytes and gigabytes, respectively)

I'd also like George to review the encryption docs (technical content and possible grammar?). Adding him to reviewers.

review: Needs Fixing

[George Ormond Lorch III (gl-az)]

As Alexey pointed out, diff line 134 is incorrect, the chunk size specification is in bytes.

Missing is the --compress-chunk-size option that was added as well which performs exactly like --encrypt-chunk-size but for the compression stage.

One thing not mentioned is that encryption can not be used on streaming tar backups.

Everything else looks pretty good.

review: Needs Fixing (g2)

[Alexey Kopytov (akopytov)]

review: Approve

=== modified file 'doc/source/conf.py'
--- doc/source/conf.py	2013-03-19 15:26:59 +0000
+++ doc/source/conf.py	2013-03-29 10:35:27 +0000
@@ -51,9 +51,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '2.1'
+version = '2.1.0'
 # The full version, including alpha/beta/rc tags.
-release = '2.1'
+release = '2.1.0-alpha1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

=== added file 'doc/source/innobackupex/encrypted_backups_innobackupex.rst'
--- doc/source/innobackupex/encrypted_backups_innobackupex.rst	1970-01-01 00:00:00 +0000
+++ doc/source/innobackupex/encrypted_backups_innobackupex.rst	2013-03-29 10:35:27 +0000
@@ -0,0 +1,83 @@
+.. _encrypted_backups_ibk:
+
+===================
+ Encrypted Backups
+===================
+
+|Percona XtraBackup| has implemented support for encrypted backups. This feature was introduced in |Percona XtraBackup| 2.1. It can be used to encrypt/decrypt local or streaming backup with |xbstream| option (streaming tar backups are not supported) in order to add another layer of protection to the backups. Encryption is done with the ``libgcrypt`` library.
+
+Creating Encrypted Backups
+===========================
+
+To make an encrypted backup following options need to be specified (options :option:`--encrypt-key` and :option:`--encrypt-key-file` are mutually exclusive, i.e. just one of them needs to be provided): 
+
+ * :option:`--encryption=ALGORITHM` - currently supported algorithms are: ``AES128``, ``AES192`` and ``AES256``
+
+ * :option:`--encrypt-key=ENCRYPTION_KEY` - proper length encryption key to use. It is not recommended to use this option where there is uncontrolled access to the machine as the command line and thus the key can be viewed as part of the process info. 
+
+ * :option:`--encrypt-key-file=KEYFILE` - the name of a file where the raw key of the appropriate length can be read from. The file must be a simple binary (or text) file that contains exactly the key to be used. 
+
+Both :option:`--encrypt-key` option  and :option:`--encrypt-key-file` option can be used to specify the encryption key.
+
+Using the :option:`--encrypt-key` option
+-----------------------------------------
+Example of the innobackupex command using the :option:`--encrypt-key` should look like this ::
+
+  $ innobackupex --encrypt=AES256 --encrypt-key="secret_key_with_the_length_of_32" /data/backups
+
+
+Using the :option:`--encrypt-key-file` option
+----------------------------------------------
+Example of the innobackupex command using the :option:`--encrypt-key-file` should look like this ::
+
+  $ innobackupex --encrypt=AES256 --encrypt-key-file=/data/backups/keyfile /data/backups
+
+.. note::
+
+  Depending on the text editor used for making the ``KEYFILE``, text file in some cases can contain the CRLF and this will cause the key size to grow and thus making it invalid. Suggested way to do this would be to create the file with: ``echo -n "secret_key_with_the_length_of_32" > /data/backups/keyfile``
+
+
+Both of these examples will create a timestamped directory in :file:`/data/backups` containing the encrypted backup.
+
+.. note:: 
+
+  You can use the :option:`innobackupex --no-timestamp` option to override this behavior and the backup will be created in the given directory.
+
+Optimizing the encryption process
+=================================
+
+Two new options have been introduced with the encrypted backups that can be used to speed up the encryption process. These are :option:`--encrypt-threads` and :option:`--encrypt-chunk-size`. By using the :option:`--encrypt-threads` option multiple threads can be specified to be used for encryption in parallel. Option :option:`--encrypt-chunk-size` can be used to specify the size (in bytes) of the working encryption buffer for each encryption thread (default is 64K).
+
+Decrypting Encrypted Backups
+============================
+
+Backups can be decrypted with :ref:`xbcrypt`. Following one-liner can be used to encrypt the whole folder: ::
+
+  $ for i in `find . -iname "*\.xbcrypt"`; do xbcrypt -d --encrypt-key-file=/root/secret_key --encrypt-algo=AES256 < $i > $(dirname $i)/$(basename $i .xbcrypt) && rm $i; done
+
+When the files have been decrypted backup can be prepared.
+
+Preparing Encrypted Backups
+============================
+
+After the backups have been decrypted, they can be prepared the same way as the standard full backups with the :option:`--apply-logs` option: :: 
+
+  $ innobackupex --apply-log /data/backups/2013-03-25_10-34-04
+
+Restoring Encrypted Backups
+=============================
+
+|innobackupex| has a :option:`--copy-back` option, which performs the restoration of a backup to the server's :term:`datadir` ::
+
+  $ innobackupex --copy-back /path/to/BACKUP-DIR
+
+It will copy all the data-related files back to the server's :term:`datadir`, determined by the server's :file:`my.cnf` configuration file. You should check the last line of the output for a success message::
+
+  innobackupex: Finished copying back files.
+  130201 11:08:13  innobackupex: completed OK!
+
+Other Reading
+=============
+
+* `The Libgcrypt Reference Manual <http://www.gnupg.org/documentation/manuals/gcrypt/>`_
+

=== modified file 'doc/source/innobackupex/innobackupex_option_reference.rst'
--- doc/source/innobackupex/innobackupex_option_reference.rst	2013-01-17 15:11:02 +0000
+++ doc/source/innobackupex/innobackupex_option_reference.rst	2013-03-29 10:35:27 +0000
@@ -26,6 +26,10 @@
 
    This option specifies the number of worker threads that will be used for parallel compression. It is passed directly to the xtrabackup child process. See the :program:`xtrabackup` :doc:`documentation <../xtrabackup_bin/xtrabackup_binary>` for details.
 
+.. option:: --compress-chunk-size
+
+   This option specifies the size of the internal working buffer for each compression thread, measured in bytes. It is passed directly to the xtrabackup child process. See the :program:`xtrabackup` :doc:`documentation <../xtrabackup_bin/xtrabackup_binary>` for details.
+
 .. option:: --copy-back
 
     Copy all the files in a previously made backup from the backup directory to their original locations.
@@ -46,6 +50,26 @@
 
    This option accepts a string argument that specifies the group which should be read from the configuration file. This is needed if you use mysqld_multi.
 
+.. option:: --encrypt=ENCRYPTION_ALGORITHM
+
+   This option instructs xtrabackup to encrypt backup copies of InnoDB data files using the algorithm specified in the ENCRYPTION_ALGORITHM. It is passed directly to the xtrabackup child process. See the :program:`xtrabackup` :doc:`documentation <../xtrabackup_bin/xtrabackup_binary>` for more details.
+
+.. option:: --encrypt-key=ENCRYPTION_KEY
+
+   This option instructs xtrabackup to use the given ENCRYPTION_KEY when using the --encrypt option. It is passed directly to the xtrabackup child process. See the :program:`xtrabackup` :doc:`documentation <../xtrabackup_bin/xtrabackup_binary>` for more details.
+
+.. option:: --encrypt-key-file=ENCRYPTION_KEY_FILE
+
+   This option instructs xtrabackup to use the encryption key stored in the given ENCRYPTION_KEY_FILE when using the --encrypt option. It is passed directly to the xtrabackup child process. See the :program:`xtrabackup` :doc:`documentation <../xtrabackup_bin/xtrabackup_binary>` for more details.
+
+.. option:: --encrypt-threads
+
+   This option specifies the number of worker threads that will be used for parallel encryption. It is passed directly to the xtrabackup child process. See the :program:`xtrabackup` :doc:`documentation <../xtrabackup_bin/xtrabackup_binary>` for more details.
+
+.. option:: --encrypt-chunk-size 
+
+   This option specifies the size of the internal working buffer for each encryption thread, measured in bytes. It is passed directly to the xtrabackup child process. See the :program:`xtrabackup` :doc:`documentation <../xtrabackup_bin/xtrabackup_binary>` for more details.
+
 .. option:: --export
 
    This option is passed directly to :option:`xtrabackup --export` option. It enables exporting individual tables for import into another server. See the |xtrabackup| documentation for details.

=== modified file 'doc/source/innobackupex/innobackupex_script.rst'
--- doc/source/innobackupex/innobackupex_script.rst	2013-02-04 14:09:53 +0000
+++ doc/source/innobackupex/innobackupex_script.rst	2013-03-29 10:35:27 +0000
@@ -36,6 +36,7 @@
    incremental_backups_innobackupex
    partial_backups_innobackupex
    compact_backups_innobackupex
+   encrypted_backups_innobackupex
 
 Proficiency
 ===========

=== modified file 'doc/source/installation/compiling_xtrabackup.rst'
--- doc/source/installation/compiling_xtrabackup.rst	2013-01-17 15:11:02 +0000
+++ doc/source/installation/compiling_xtrabackup.rst	2013-03-29 10:35:27 +0000
@@ -20,7 +20,7 @@
 In Debian-based distributions, you need to: ::
 
   $ apt-get install debhelper autotools-dev libaio-dev wget automake \
-    libtool bison libncurses-dev libz-dev cmake bzr
+    libtool bison libncurses-dev libz-dev cmake bzr libgcrypt11-dev
 
 In ``RPM``-based distributions, you need to: ::
 

=== modified file 'doc/source/intro.rst'
--- doc/source/intro.rst	2012-11-19 02:09:59 +0000
+++ doc/source/intro.rst	2013-03-29 10:35:27 +0000
@@ -54,6 +54,10 @@
 +---------------------------------------+----------------------+-----------------------+
 |Streaming backups                      | Yes                  |                       |      
 +---------------------------------------+----------------------+-----------------------+
+|Compact backups                        | Yes                  |                       |      
++---------------------------------------+----------------------+-----------------------+
+|Encrypted backups                      | Yes                  |                       |      
++---------------------------------------+----------------------+-----------------------+
 |Parallel compression                   | Yes                  |                       |      
 +---------------------------------------+----------------------+-----------------------+
 |LRU backups                            | Yes                  |                       |      

=== modified file 'doc/source/manual.rst'
--- doc/source/manual.rst	2012-11-19 02:09:59 +0000
+++ doc/source/manual.rst	2013-03-29 10:35:27 +0000
@@ -11,9 +11,10 @@
    innobackupex/innobackupex_script
    xtrabackup_bin/xtrabackup_binary
    xbstream/xbstream
+   xbcrypt/xbcrypt
    how_xtrabackup_works
 
-|XtraBackup| is really a set of three tools:
+|XtraBackup| is a set of following tools:
 
 :doc:`innobackupex <innobackupex/innobackupex_script>`
     a wrapper script that provides functionality to backup a whole |MySQL| database instance with :term:`MyISAM`, :term:`InnoDB`, and :term:`XtraDB` tables.
@@ -21,7 +22,10 @@
 :doc:`xtrabackup <xtrabackup_bin/xtrabackup_binary>`
     a compiled *C* binary, which copies only :term:`InnoDB` and :term:`XtraDB` data
     
+:doc:`xbcrypt <xbcrypt/xbcrypt>`
+   utility used for encrypting and decrypting backup files.
+
 :doc:`xbstream <xbstream/xbstream>`
-   new utility that allows streaming and extracting files to/from the :term:`xbstream` format.
+   utility that allows streaming and extracting files to/from the :term:`xbstream` format.
 
 It is possible to use the |xtrabackup| binary alone, however, the recommend way is using it through the |innobackupex| wrapper script and let it execute |xtrabackup| for you. It might be helpful to first learn :doc:`how to use innobackupex <innobackupex/innobackupex_script>`, and then learn  :doc:`how to use xtrabackup <xtrabackup_bin/xtrabackup_binary>` for having a better low-level understanding or control of the tool if needed.

=== modified file 'doc/source/release-notes.rst'
--- doc/source/release-notes.rst	2012-11-19 02:09:59 +0000
+++ doc/source/release-notes.rst	2013-03-29 10:35:27 +0000
@@ -31,3 +31,12 @@
 
    release-notes/2.0/*
 
+Percona |XtraBackup| 2.1 
+=========================
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   release-notes/2.1/*
+

=== added directory 'doc/source/release-notes/2.1'
=== added file 'doc/source/release-notes/2.1/2.1.0.rst'
--- doc/source/release-notes/2.1/2.1.0.rst	1970-01-01 00:00:00 +0000
+++ doc/source/release-notes/2.1/2.1.0.rst	2013-03-29 10:35:27 +0000
@@ -0,0 +1,27 @@
+=======================================
+|Percona XtraBackup| 2.1.0-alpha1
+=======================================
+
+Percona is glad to announce the release of |Percona XtraBackup| 2.1.0-alpha1 on April 2nd 2013. Downloads are available from our download site `here <http://www.percona.com/downloads/XtraBackup/2.1.0/>`_. For this ALPHA release, we will not be making APT and YUM repositories available, just base deb and RPM packages
+
+This is an *ALPHA* quality release and is not intended for production. If you want a high quality, Generally Available release, the current Stable version should be used (currently 2.0.6 in the 2.0 series at the time of writing).
+
+This release contains all of the features and bug fixes in :doc:`Percona XtraBackup 2.0.6 </release-notes/2.0/2.0.6>`, plus the following:
+
+New features
+------------
+
+ |Percona Xtrabackup| now has support for :ref:`compact_backups_ibk`. This feature can be used for taking the backups that will take less amount of disk space.
+
+ |Percona XtraBackup| has implemented :ref:`encrypted_backups_ibk`. This feature can be used to encrypt/decrypt both local and streamed backups in order to add another layer of protection to the backups.
+
+ |innobackupex| now uses Perl's ``DBD::MySQL`` package for server communication instead of spawning the mysql command line client.
+
+ Support for |InnoDB| 5.0 and |InnoDB| 5.1 builtin has been removed from |Percona XtraBackup|.
+
+ After being deprecated in previous version, option :option:`--remote-host` has been completely removed in |Percona XtraBackup| 2.1.
+
+Bugs Fixed
+----------
+
+ |innobackupex| now supports empty arguments in the :option:`--password` option. Bug fixed :bug:`1032667` (*Andrew Gaul*).

=== added directory 'doc/source/xbcrypt'
=== added file 'doc/source/xbcrypt/xbcrypt.rst'
--- doc/source/xbcrypt/xbcrypt.rst	1970-01-01 00:00:00 +0000
+++ doc/source/xbcrypt/xbcrypt.rst	2013-03-29 10:35:27 +0000
@@ -0,0 +1,42 @@
+.. _xbcrypt:
+
+======================
+ The xbcrypt binary
+======================
+
+To support encryption and decryption of the backups, a new tool ``xbcrypt`` was introduced to XtraBackup. 
+
+This utility has been modeled after :ref:`xbstream_binary` to perform encryption and decryption outside of |XtraBackup|. Xbcrypt has following command line options: 
+
+.. option:: -d, --decrypt
+
+   Decrypt data input to output.
+
+.. option::  -i, --input=name
+
+   Optional input file. If not specified, input will be read from standard input.
+
+.. option::  -o, --output=name
+
+   Optional output file. If not specified, output will be written to standard output.
+
+.. option:: -a, --encrypt-algo=name 
+
+   Encryption algorithm.
+
+.. option:: -k, --encrypt-key=name 
+           
+   Encryption key.
+
+.. option:: -f, --encrypt-key-file=name 
+            
+   File which contains encryption key.
+
+.. option:: -s, --encrypt-chunk-size=# 
+
+   Size of working buffer for encryption in bytes. The default value is 64K.
+
+.. option:: -v, --verbose       
+
+   Display verbose status output.
+

=== modified file 'doc/source/xbstream/xbstream.rst'
--- doc/source/xbstream/xbstream.rst	2012-04-21 07:12:28 +0000
+++ doc/source/xbstream/xbstream.rst	2013-03-29 10:35:27 +0000
@@ -1,3 +1,5 @@
+.. _xbstream_binary:
+
 ======================
  The xbstream Binary
 ======================

=== modified file 'innobackupex'
--- innobackupex	2013-03-19 15:06:52 +0000
+++ innobackupex	2013-03-29 10:35:27 +0000
@@ -2823,7 +2823,7 @@
 =item --compress-chunk-size
 
 This option specifies the size of the internal working buffer for each
-compression thread, measured in kilobytes. It is passed directly to the
+compression thread, measured in bytes. It is passed directly to the
 xtrabackup child process. Try 'xtrabackup --help' for more details.
 
 =item --copy-back
@@ -2870,7 +2870,7 @@
 =item --encrypt-chunk-size
 
 This option specifies the size of the internal working buffer for each
-encryption thread, measured in kilobytes. It is passed directly to the
+encryption thread, measured in bytes. It is passed directly to the
 xtrabackup child process. Try 'xtrabackup --help' for more details.
 
 =item --export