Author: Alexander Barkov
Author Date: 2018-10-20 15:51:14 UTC

MDEV-17477 Wrong result for TIME('-2001-01-01 10:20:30') and numerous other str-to-time conversion problems
MDEV-17478 Wrong result for TIME('+100:20:30')

Author: Sergei Golubchik
Author Date: 2018-10-18 17:33:41 UTC

compilation failure on Windows

  sql\sql_acl.cc(3114): error C2220: warning treated as error - no 'object' file generated
  sql\sql_acl.cc(3114): warning C4390: ';': empty controlled statement found; is this the intent?

Author: Sergei Golubchik
Author Date: 2018-10-20 12:05:33 UTC

MDEV-14429 sql_safe_updates in my.cnf not work

add a test case

Author: Vladislav Vaintroub
Author Date: 2018-10-20 11:56:28 UTC

Attempt to fix build by a workaround for a bug in Windows8.1 SDK

Problem description and workaround are here
https://developercommunity.visualstudio.com/content/problem/185399/error-c2760-in-combaseapih-with-windows-sdk-81-and.html

Author: Sergei Golubchik
Author Date: 2018-10-20 11:29:27 UTC

Revert "MDEV-14429 sql_safe_updates in my.cnf not work"

This reverts commit dc3a20b191362227a6f85fadfc3a8b6c76f69ec6.

It requires GET_BIT in include/my_getopt.h, which is available
only in 10.3+

Author: Marko Mäkelä
Author Date: 2018-10-19 16:00:25 UTC

Merge 10.4 into 10.4-mdev-15562

This branch exists just for the record, for preserving the
development history, should it ever be needed.

Author: Marko Mäkelä
Author Date: 2018-10-19 13:49:54 UTC

MDEV-15662 Instant DROP COLUMN or changing the order of columns

Allow ADD COLUMN anywhere in a table, not only adding as the
last column.

Allow instant DROP COLUMN and instant changing the order of columns.

The added columns will always be added last in clustered index records.
In new records, instantly dropped columns will be stored as NULL or
empty when possible.

Information about dropped and reordered columns will be written in
a metadata BLOB (mblob), which is stored before the first 'user' field
in the hidden metadata record at the start of the clustered index.
The presence of mblob is indicated by setting the delete-mark flag in
the metadata record.

The metadata BLOB stores the number of clustered index fields,
followed by an array of column information for each field.
For dropped columns, we store the NOT NULL flag, the fixed length,
and for variable-length columns, whether the maximum length exceeded
255 bytes. For non-dropped columns, we store the column position.

Unlike with MDEV-11369, when a table becomes empty, it cannot
be converted back to the canonical format. The reason for this is
that other threads may hold cached objects such as
row_prebuilt_t::ins_node that could refer to dropped or reordered
index fields.

For instant DROP COLUMN and ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC,
we must store the n_core_null_bytes in the root page, so that the
chain of node pointer records can be followed in order to reach the
leftmost leaf page where the metadata record is located.
If the mblob is present, we will zero-initialize the strings
"infimum" and "supremum" in the root page, and use the last byte of
"supremum" for storing the number of null bytes (which are allocated
but useless on node pointer pages). This is necessary for
btr_cur_instant_init_metadata() to be able to navigate to the mblob.

If the PRIMARY KEY contains any variable-length column and some
nullable columns were instantly dropped, the dict_index_t::n_nullable
in the data dictionary could be smaller than it actually is in the
non-leaf pages. Because of this, the non-leaf pages could use more
bytes for the null flags than the data dictionary expects, and we
could be reading the lengths of the variable-length columns from the
wrong offset, and thus reading the child page number from wrong place.
This is the result of two design mistakes that involve unnecessary
storage of data: First, it is nonsense to store any data fields for
the leftmost node pointer records, because the comparisons would be
resolved by the MIN_REC_FLAG alone. Second, there cannot be any null
fields in the clustered index node pointer fields, but we nevertheless
reserve space for all the null flags.

Limitations (future work):

MDEV-17459 Allow instant ALTER TABLE even if FULLTEXT INDEX exists
MDEV-17468 Avoid table rebuild on operations on generated columns
MDEV-17494 Refuse ALGORITHM=INSTANT when the row size is too large

btr_page_reorganize_low(): Preserve any metadata in the root page.
Call lock_move_reorganize_page() only after restoring the "infimum"
and "supremum" records, to avoid a memcmp() assertion failure.

dict_col_t::DROPPED: Magic value for dict_col_t::ind.

dict_col_t::clear_instant(): Renamed from dict_col_t::remove_instant().
Do not assert that the column was instantly added, because we
sometimes call this unconditionally for all columns.
Convert an instantly added column to a "core column". The old name
remove_instant() could be mistaken to refer to "instant DROP COLUMN".

dict_col_t::is_added(): Rename from dict_col_t::is_instant().

dtype_t::metadata_blob_init(): Initialize the mblob data type.

dtuple_t::is_metadata(), dtuple_t::is_alter_metadata(),
upd_t::is_metadata(), upd_t::is_alter_metadata(): Check if info_bits
refer to a metadata record.

dict_table_t::instant: Metadata about dropped or reordered columns.

dict_table_t::prepare_instant(): Prepare
ha_innobase_inplace_ctx::instant_table for instant ALTER TABLE.
innobase_instant_try() will pass this to dict_table_t::instant_column().
On rollback, dict_table_t::rollback_instant() will be called.

dict_table_t::instant_column(): Renamed from instant_add_column().
Add the parameter col_map so that columns can be reordered.
Copy and adjust v_cols[] as well.

dict_table_t::find(): Find an old column based on a new column number.

dict_table_t::serialise_columns(), dict_table_t::deserialise_columns():
Convert the mblob.

dict_index_t::instant_metadata(): Create the metadata record
for instant ALTER TABLE. Invoke dict_table_t::serialise_columns().

dict_index_t::reconstruct_fields(): Invoked by
dict_table_t::deserialise_columns().

dict_index_t::clear_instant_alter(): Move the fields for the
dropped columns to the end, and sort the surviving index fields
in ascending order of column position.

ha_innobase::check_if_supported_inplace_alter(): Do not allow
adding a FTS_DOC_ID column if a hidden FTS_DOC_ID column exists
due to FULLTEXT INDEX. (This always required ALGORITHM=COPY.)

instant_alter_column_possible(): Add a parameter for InnoDB table,
to check for additional conditions, such as the maximum number of
index fields.

ha_innobase_inplace_ctx::first_alter_pos: The first column whose position
is affected by instant ADD, DROP, or changing the order of columns.

innobase_build_col_map(): Skip added virtual columns.

prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol.
Remove some unnecessary code. Note that the call to
innodb_base_col_setup() should be executed later.

commit_try_norebuild(): If ctx->is_instant(), let the virtual
columns be added or dropped by innobase_instant_try().

innobase_instant_try(): Fill in a zero default value for the
hidden column FTS_DOC_ID (to reduce the work needed in MDEV-17459).
If any columns were dropped or reordered (or added not last),
delete any SYS_COLUMNS records for the following columns, and
insert SYS_COLUMNS records for all subsequent stored columns as well
as for all virtual columns. If any virtual column is dropped, rewrite
all virtual column metadata. Use a shortcut only for adding
virtual columns. This is because innobase_drop_virtual_try()
assumes that the dropped virtual columns still exist in ctx->old_table.

innodb_update_cols(): Renamed from innodb_update_n_cols().

innobase_add_one_virtual(), innobase_insert_sys_virtual(): Change
the return type to bool, and invoke my_error() when detecting an error.

innodb_insert_sys_columns(): Insert a record into SYS_COLUMNS.
Refactored from innobase_add_one_virtual() and innobase_instant_add_col().

innobase_instant_add_col(): Replace the parameter dfield with type.

innobase_instant_drop_cols(): Drop matching columns from SYS_COLUMNS
and all columns from SYS_VIRTUAL.

innobase_add_virtual_try(), innobase_drop_virtual_try(): Let
the caller invoke innodb_update_cols().

innobase_rename_column_try(): Skip dropped columns.

commit_cache_norebuild(): Update table->fts->doc_col.

dict_mem_table_col_rename_low(): Skip dropped columns.

trx_undo_rec_get_partial_row(): Skip dropped columns.

trx_undo_update_rec_get_update(): Handle the metadata BLOB correctly.

trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields.
Log metadata records consistently.
Apparently, the first fields of a clustered index may be updated
in an update_undo vector when the index is ID_IND of SYS_FOREIGN,
as part of renaming the table during ALTER TABLE. Normally, updates of
the PRIMARY KEY should be logged as delete-mark and an insert.

row_undo_mod_parse_undo_rec(), row_purge_parse_undo_rec():
Use trx_undo_metadata.

row_undo_mod_clust_low(): On metadata rollback, roll back the root page too.

row_undo_mod_clust(): Relax an assertion. The delete-mark flag was
repurposed for ALTER TABLE metadata records.

row_rec_to_index_entry_impl(): Add the template parameter mblob
and the optional parameter info_bits for specifying the desired new
info bits. For the metadata tuple, allow conversion between the original
format (ADD COLUMN only) and the generic format (with hidden BLOB).
Add the optional parameter "pad" to determine whether the tuple should
be padded to the index fields (on ALTER TABLE it should), or whether
it should remain at its original size (on rollback).

row_build_index_entry_low(): Clean up the code, removing
redundant variables and conditions. For instantly dropped columns,
generate a dummy value that is NULL, the empty string, or a
fixed length of NUL bytes, depending on the type of the dropped column.

row_upd_clust_rec_by_insert_inherit_func(): On the update of PRIMARY KEY
of a record that contained a dropped column whose value was stored
externally, we will be inserting a dummy NULL or empty string value
to the field of the dropped column. The externally stored column would
eventually be dropped when purge removes the delete-marked record for
the old PRIMARY KEY value.

btr_index_rec_validate(): Recognize the metadata record.

btr_discard_only_page_on_level(): Preserve the generic instant
ALTER TABLE metadata.

btr_set_instant(): Replaces page_set_instant(). This sets a clustered
index root page to the appropriate format, or upgrades from
the MDEV-11369 instant ADD COLUMN to generic ALTER TABLE format.

btr_cur_instant_init_low(): Read and validate the metadata BLOB page
before reconstructing the dictionary information based on it.

btr_cur_instant_init_metadata(): Do not read any lengths from the
metadata record header before reading the BLOB. At this point, we
would not actually know how many nullable fields the metadata record
contains.

btr_cur_instant_root_init(): Initialize n_core_null_bytes in one
of two possible ways.

btr_cur_trim(): Handle the mblob record.

row_metadata_to_tuple(): Convert a metadata record to a data tuple,
based on the new info_bits of the metadata record.

btr_cur_pessimistic_update(): Invoke row_metadata_to_tuple() if needed.
Invoke dtuple_convert_big_rec() for metadata records if the record is
too large, or if the mblob is not yet marked as externally stored.

btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete():
When the last user record is deleted, do not delete the
generic instant ALTER TABLE metadata record. Only delete
MDEV-11369 instant ADD COLUMN metadata records.

btr_cur_optimistic_insert(): Avoid unnecessary computation of rec_size.

btr_pcur_store_position(): Allow a logically empty page to contain
a metadata record for generic ALTER TABLE.

REC_INFO_DEFAULT_ROW_ADD: Renamed from REC_INFO_DEFAULT_ROW.
This is for the old instant ADD COLUMN (MDEV-11369) only.

REC_INFO_DEFAULT_ROW_ALTER: The more generic metadata record,
with additional information for dropped or reordered columns.

rec_info_bits_valid(): Remove. The only case when this would fail
is when the record is the generic ALTER TABLE metadata record.

rec_is_alter_metadata(): Check if a record is the metadata record
for instant ALTER TABLE (other than ADD COLUMN). NOTE: This function
must not be invoked on node pointer records, because the delete-mark
flag in those records may be set (it is garbage), and then a debug
assertion could fail because index->is_instant() does not necessarily
hold.

rec_is_add_metadata(): Check if a record is MDEV-11369 ADD COLUMN metadata
record (not more generic instant ALTER TABLE).

rec_get_converted_size_comp_prefix_low(): Assume that the metadata
field will be stored externally. In dtuple_convert_big_rec() during
the rec_get_converted_size() call, it would not be there yet.

rec_get_converted_size_comp(): Replace status,fields,n_fields with tuple.

rec_init_offsets_comp_ordinary(), rec_get_converted_size_comp_prefix_low(),
rec_convert_dtuple_to_rec_comp(): Add template<bool mblob = false>.
With mblob=true, process a record with a metadata BLOB.

rec_copy_prefix_to_buf(): Assert that no fields beyond the key and
system columns are being copied. Exclude the metadata BLOB field.

rec_convert_dtuple_to_metadata_comp(): Convert an alter metadata tuple
into a record.

row_upd_index_replace_metadata(): Apply an update vector to an
alter_metadata tuple.

row_log_allocate(): Replace dict_index_t::is_instant()
with a more appropriate condition that ignores dict_table_t::instant.
Only a table on which the MDEV-11369 ADD COLUMN was performed
can "lose its instantness" when it becomes empty. After
instant DROP COLUMN or reordering columns, we cannot simply
convert the table to the canonical format, because the data
dictionary cache and all possibly existing references to it
from other client connection threads would have to be adjusted.

row_quiesce_write_index_fields(): Do not crash when the table contains
an instantly dropped column.

Thanks to Thirunarayanan Balathandayuthapani for discussing the design
and implementing an initial prototype of this.
Thanks to Matthias Leich for testing.

Author: Sergey Vojtovich
Author Date: 2018-10-19 15:10:45 UTC

Remove unused code.

Author: Marko Mäkelä
Author Date: 2018-10-19 06:29:28 UTC

MDEV-17466: Remove the debug assertion

This reverts commit 2d4075e1d9dba3dd8503d65fdb145a6dcc056c51
where the debug assertion was added. There seems to be a potential
problem in the purge of indexes that depend on virtual columns.

Ultimately, we should change the InnoDB undo log format so that
all actual secondary index keys are stored there, also for
virtual or spatial indexes. In that way, purge and rollback would
be more straightforward.

Author: varun
Author Date: 2018-09-21 11:01:36 UTC

Tests updated.

Author: Sergey Vojtovich
Author Date: 2018-10-18 18:48:28 UTC

MDEV-14815 - Server crash or AddressSanitizer errors or valgrind warnings
             in thr_lock / has_old_lock upon FLUSH TABLES

Explicit partition access under LOCK TABLES may cause subsequent
statements to crashes the server, deadlock, trigger valgrind warnings or
ASAN errors. Freed memory was being used due to incorrect cleanup.

Some unused code removed.

Author: varun
Author Date: 2018-09-22 18:43:31 UTC

Tests updated.

Author: Larysa Sherepa
Author Date: 2018-10-18 16:46:00 UTC

CTE tests

Author: Dyre Tjeldvoll
Author Date: 2015-04-30 10:56:33 UTC

BUG#19988193: ASSERTION `(*TABLES)->REGINFO.LOCK_TYPE >= TL_READ'
FAILED IN LOCK_EXTERNAL
BUG#21198646: ASSERTION FAILED: (*TABLES)->REGINFO.LOCK_TYPE >=
TL_READ FILE LOCK.CC, LINE 356

This patch addresses two related issues: Calling a procedure which
creates a view from a trigger (BUG#19988193), and creating a function
calling a procedure doing RENAME TABLE (BUG#21198646), could both, in
certain circumstances, trigger an assert.

Root cause was that prelocking of tables with lock_type==TL_IGNORE is
not supported, and so triggers an assert. TL_IGNORE is only used for
source tables in CREATE VIEW statements and the table of a RENAME TABLE
statement. It is very unusual for these statements to be part of
prelocking analysis, as both are implicit commit statements which are
not permitted in triggers and stored functions/procedures. But as the test
cases show; it is possible to have such statements contribute to the
prelocking set, but in both cases the statement is "meaningless", in
the sense that it will trigger an error during execution.

Fix: In mysql_make_view(), avoid adding the backing tables to view_ref
if view_ref->prelocking_placeholder==true and lock_type==TL_IGNORE. In
sp_head::add_used_tables_to_table_list() skip SP_TABLES which have
lock_type=TL_IGNORE.

Test: New test cases added to tablelock.test

Author: Sergei Golubchik
Author Date: 2018-10-17 15:59:42 UTC

update galera test results

Author: Sergei Golubchik
Author Date: 2018-09-27 15:28:01 UTC

wsrep: create a macro for the error: label

that is used by WSREP_TO_ISOLATION_BEGIN and other galera macros,
to avoid the need for wrapping this label in #ifdef WITH_WSREP/#endif

Author: Sachin Setiya
Author Date: 2018-10-14 17:46:53 UTC

MDEV-15919 lower_case_table_names does not behave as expected(nor...
 consistently) on Replication Slave

lower_case_table_names 0 -> 1 replication works, it's safe as long as
 mixed case names mapping to the lower case ones is one-to-one

Author: Bernd Kuhls
Author Date: 2018-10-07 12:25:59 UTC

cmake: fix ucontext detection

On some archs uclibc does not provide the ucontext structure despite
providing ucontext.h, for details see
https://git.buildroot.net/buildroot/commit/?id=f1cbfeea95e6287c7a666aafc182ffa318eff262

This patch improves the detection of ucontext by making sure that
HAVE_UCONTEXT_H is only set when makecontext() was found.

Signed-off-by: Bernd Kuhls <bernd.kuhls@t-online.de>

Author: Sachin Setiya
Author Date: 2018-10-14 17:46:53 UTC

MDEV-15919 lower_case_table_names does not behave as expected(nor...
 consistently) on Replication Slave

lower_case_table_names 0 -> 1 replication works, it's safe as long as
 mixed case names mapping to the lower case ones is one-to-one

Author: Thirunarayanan Balathandayuthapani
Author Date: 2018-10-16 06:08:11 UTC

MDEV-12547: InnoDB FULLTEXT index has too strict innodb_ft_result_cache_limit max limit

- Removed f_n_char from fts_string_t. Instead of that, InnoDB calculate
the number of chars when it is needed.

Author: varun
Author Date: 2018-10-15 18:10:36 UTC

MDEV-14791: Crash with order by expression containing window functions

The issue here is that for a window function in the ORDER BY clause, we were not
creating an extra field in the temporary table for the window function
(which is contained in an expression).
So a call to split_sum_func is added to handle this case.

Author: varun
Author Date: 2018-10-15 16:35:19 UTC

MDEV-17137: Syntax errors with VIEW using MEDIAN

The syntax error happened because we had not implemented a different print for
percentile functions. The syntax is a bit different when we use percentile functions
as window functions in comparision to normal window functions.
Implemented a seperate print function for percentile functions

Author: Andrei Elkin
Author Date: 2018-10-15 12:22:45 UTC

MDEV-17437 Semisync master fires invalid fd value assert

The semisync ack collector hits fd's out-of-bound value assert through
a stack of

/usr/sbin/mysqld(_ZN12Ack_receiver17get_slave_socketsEP6fd_setPj+0x70)[0x7fa3bbe27400]

/usr/sbin/mysqld(_ZN12Ack_receiver3runEv+0x540)[0x7fa3bbe27980]

/usr/sbin/mysqld(ack_receive_handler+0x19)[0x7fa3bbe27a79]

The reason of the failure must be the same as in https://bugs.mysql.com/bug.php?id=79865
whose fixes are applied with minor changes.
The sustain manual testing with `mysqlslap --concurrency > 1024' in "background" while the slave io thread is restarting multiple times.

Author: Vladislav Vaintroub
Author Date: 2018-10-15 10:06:00 UTC

MDEV-14429 sql_safe_updates in my.cnf not work

Author: Sergey Vojtovich
Author Date: 2018-10-14 18:44:14 UTC

MDEV-17441 - InnoDB transition to C++11 atomics

srv_conc_t.n_active and srv_conc_t.n_waiting transition to
simple_atomic_counter.

Author: Vicențiu Ciorbaru
Author Date: 2018-10-14 17:54:52 UTC

Merge branch 'MDEV1732_check_constraint_10.2_v1' of https://github.com/an3l/server into an3l-MDEV1732_check_constraint_10.2_v1

Author: varun
Author Date: 2018-10-14 17:30:23 UTC

MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion

This assert is hit when we do filesort using the priority queue and try to insert elements in
the queue. The compare function used for the priority queue should handle the case for zerolength
sortkey.

Author: Sachin Setiya
Author Date: 2018-10-13 19:12:24 UTC

MDEV-17119 replicate_rewrite_db does not work for single char database name

Fixed issue in logic.

Author: Oleksandr Byelkin
Author Date: 2018-10-12 18:47:22 UTC

MDEV-16207: Sig 11 on RHEL7 on Azure

The problem was that Item_subselect::const_item was not checked correctly and can differ from used_tables result.
Which lead to the situation when Item_direct_view_ref pointing on subselect was constant but the subquery return
FALSE on const_item() request (which was an error).

Now both methods report the same.

Author: Oleksandr Byelkin
Author Date: 2018-10-12 07:07:05 UTC

MDEV-17401: LOAD DATA from very big file into MyISAM table results in EOF error and corrupt index

my_read fixed as in higher versions.
my_pread made as my_read aware of partial read of huge chunks of files
MY_FULL_IO enabled for file operations

Author: Alexander Barkov
Author Date: 2018-10-12 02:15:21 UTC

Adjusting old tests and adding new tests for "MDEV-8765: mysqldump -use utf8mb4 by default"

Author: Vladislav Vaintroub
Author Date: 2018-10-11 18:36:59 UTC

fix test suite after MDEV-15438

Author: Andrei Elkin
Author Date: 2018-10-03 18:45:05 UTC

MDEV-17098 DATE -> DATETIME replication conversion not working, even in ALL_NON_LOSSY mode

Opened up MYSQL_TYPE _DATETIME{,2} <-> _NEWDATE conversions for replication.

Author: Andrei Elkin
Author Date: 2018-10-10 14:36:09 UTC

MDEV-14431 binlog.binlog_flush_binlogs_delete_domain failed in buildbot

The test and also rpl_gtid_delete_domain failed on PPC64 platform
due to an incorrectly specified actual key for searching
in a gtid domain system hash. While the correct size is 32 bits
the supplied value was 8 bytes of long int size on the platform.
The problem became evident thanks to the big endiness which
cut off the *least* significant part of the value field.

Fixed with correcting a dynamic array initialization to hold
now uint32 values as well as the values extraction for
searching in the gtid domain system hash.
A new added test ensures no overflowed values are accepted
for deletion which prevents inadvertent action. Notice though

MariaDB [test]> set @@session.gtid_domain_id=(1 << 32) + 1;
MariaDB [test]> show warnings;
+---------+------+--------------------------------------------------------+
| Level | Code | Message |
+---------+------+--------------------------------------------------------+
| Warning | 1292 | Truncated incorrect gtid_domain_id value: '4294967297' |
+---------+------+--------------------------------------------------------+
MariaDB [test]> select @@session.gtid_domain_id;
+--------------------------+
| @@session.gtid_domain_id |
+--------------------------+
| 4294967295 |
+--------------------------+

Author: Vladislav Vaintroub
Author Date: 2018-10-10 19:33:03 UTC

MDEV-17279 Windows : link C runtime dynamically

Improve previous patch so we do not depend on (tiny) compiler-version
dependent libraries, i.e vcruntime140.dll, and msvcp140.dll

The remaining dependency is Universal CRT, is part of Win10+ OS,
and should already be installed by Windows update on most downlevel systems.

Author: Jan Lindström
Author Date: 2018-10-10 15:30:42 UTC

Fix tests affected by new configuration variable on Galera.

Author: Jan Lindström
Author Date: 2018-10-10 15:25:53 UTC

MDEV-17230: encryption_key_id from alter is ignored by encryption threads

Background: Used encryption key_id is stored to encryption metadata
i.e. crypt_data that is stored on page 0 of the tablespace of the
table. crypt_data is created only if implicit encryption/not encryption
is requested i.e. ENCRYPTED=[YES|NO] table option is used
fil_create_new_single_table_tablespace on fil0fil.cc.

Later if encryption is enabled all tables that use default encryption
mode (i.e. no encryption table option is set) are encrypted with
default encryption key_id that is 1. See fil_crypt_start_encrypting_space on
fil0crypt.cc.

ha_innobase::check_table_options()
 If default encryption is used and encryption is disabled, you may
        not use nondefault encryption_key_id as it is not stored anywhere.

Author: Jan Lindström
Author Date: 2018-10-10 15:12:27 UTC

MDEV-17229: Encryption threads ignore innodb_default_encryption_key_id

Background: Used encryption key_id is stored to encryption metadata
i.e. crypt_data that is stored on page 0 of the tablespace of the
table. crypt_data is created only if implicit encryption/not encryption
is requested i.e. ENCRYPTED=[YES|NO] table option is used
fil_create_new_single_table_tablespace on fil0fil.cc.

innodb_default_encryption_key_id setting should effect only tables created
with ENCRYPTED=YES table option. Similarly ENCRYPTION_KEY_ID table option
should be allowed only for tables using ENCRYPTED=YES table option.

Later if encryption is enabled all tables that use default encryption
mode (i.e. no encryption table option is set) are encrypted with
default encryption key_id that is 1. See fil_crypt_start_encrypting_space on
fil0crypt.cc.

Author: Monty
Author Date: 2018-10-09 15:55:18 UTC

Optimize flush tables with read lock (FTWRL) to not wait for select's

Part of MDEV-5336 Implement LOCK FOR BACKUP

The idea is that instead of waiting in close_cached_tables() for all
tables to be closed, we instead call flush_tables() that does:
- Flush not used objects in table cache to free memory
- Collect all tables that are open
- Call HA_EXTRA_FLUSH on the objects, to get them into "closed state"

The benefit compared to old code is:
- FTWRL doesn't have to wait for long running selects or open HANDLER's

Author: Sachin Setiya
Author Date: 2018-10-10 13:24:39 UTC

dummy commit 2

Author: Igor Babaev
Author Date: 2018-10-09 09:36:09 UTC

MDEV-17096 Pushdown of simple derived tables to storage engines

Interface + Proof of Concept for federatedx.

Author: Thirunarayanan Balathandayuthapani
Author Date: 2018-10-08 16:24:54 UTC

MDEV-16849 Extending indexed VARCHAR column should be instantaneous

- Indexed VARCHAR columns are expanded instantaneously (without index rebuild) for
InnoDB tables using INPLACE algorithm.

Author: Jan Lindström
Author Date: 2018-10-08 15:53:00 UTC

Test change deleting restart parameters.

Author: varun
Author Date: 2018-10-05 15:14:18 UTC

General structures for optimizer tracer designed

Author: Vladislav Vaintroub
Author Date: 2018-10-04 15:06:05 UTC

CMake, Windows - reduce amount of noisy, irrelevant MESSAGE()s

Author: Marko Mäkelä
Author Date: 2018-10-01 06:30:33 UTC

row_build_index_entry(): Make index const

Author: Galina Shalygina
Author Date: 2018-08-15 21:24:52 UTC

MDEV-16188: Use in-memory PK filters built from range index scans

First phase: make optimizer choose to use filter and show it in EXPLAIN.

Author: Vladislav Vaintroub
Author Date: 2018-09-26 18:33:27 UTC

MDEV-17190 : custom build, adding instrumentation of
innodb mutexes and rwlocks

Add locker thread id to mutexes.
Add last shared locker thread id to rwlocks.

Author: Thirunarayanan Balathandayuthapani
Author Date: 2018-09-25 11:23:33 UTC

MDEV-14398 When innodb_encryption_rotate_key_age=0 is set,
  server won't encrypt tablespaces

- For unloaded tablespace, all columns except space id, name
in innodb_tablespaces_encryption table will show NULL value.

Author: Sergei Golubchik
Author Date: 2018-09-24 13:34:23 UTC

update libmariadb

Author: Sergei Golubchik
Author Date: 2018-09-22 20:46:45 UTC

RocksDB: workaround a compiler error on ppc64le

storage/rocksdb/rdb_datadic.cc: In member function 'int myrocks::Rdb_key_def::unpack_integer(myrocks::Rdb_field_packing*, Field*, uchar*, myrocks::Rdb_string_reader*, myrocks::Rdb_string_reader*) const'
storage/rocksdb/rdb_datadic.cc:1781:1: internal compiler error: Segmentation fault
 }

on ppc64le, ubuntu bionic gcc 7.3.0 and debian stretch gcc 6.3.0

The error happens with -ftree-loop-vectorize when trying to vectorize
a particular loop.

Compiler gets confused by __attribute__((optimize("O0")) that comes in
one of the included headers. The intention of this __attribute__ was to
prevent function from being inlined (see ha_rocksdb.cc). Let's use a
more specific attribute that prevents inlining but does not confuse
loop vectorizer.

Author: Sergei Golubchik
Author Date: 2018-09-23 10:58:11 UTC

Merge branch '10.0' into 10.1

Author: Sergei Golubchik
Author Date: 2018-09-22 13:21:20 UTC

tokudb: create and destroy TOKUDB_SHARE::_open_tables_mutex dynamically

to guarantee that it's destroyed after plugin deinit is called

Author: Jan Lindström
Author Date: 2018-09-22 12:24:39 UTC

Adjust galera.query_cache test.

Author: Vicențiu Ciorbaru
Author Date: 2018-09-22 11:01:46 UTC

Small change for feedback debug

Author: Andrei Elkin
Author Date: 2018-07-27 19:55:18 UTC

MDEV-10963 Fragmented BINLOG query

The problem was originally stated in
  http://bugs.mysql.com/bug.php?id=82212
The size of an base64-encoded Rows_log_event exceeds its
vanilla byte representation in 4/3 times.
When a binlogged event size is about 1GB mysqlbinlog generates
a BINLOG query that can't be send out due to its size.

It is fixed with fragmenting the BINLOG argument C-string into
(approximate) halves when the base64 encoded event is over 1GB size.
The mysqlbinlog in such case puts out

    SET @binlog_fragment_0='base64-encoded-fragment_0';
    SET @binlog_fragment_1='base64-encoded-fragment_1';
    BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1);

to represent a big BINLOG 'base64-encoded-"total"'.
Two more statements are composed to promptly release memory
    SET @binlog_fragment_0=NULL;
    SET @binlog_fragment_1=NULL;

The 2 fragments are enough, though the client and server still may
need to tweak their @@max_allowed_packet to satisfy to the fragment
size (which they would have to do anyway with greater number of
fragments, should that be desired).

On the lower level the following changes are made:

Log_event::print_base64()
  remains to call encoder and store the encoded data into a cache but
  now *without* doing any formatting. The latter is left for time
  when the cache is copied to an output file (e.g mysqlbinlog output).
  No formatting behavior is also reflected by the change in the meaning
  of the last argument which specifies whether to cache the encoded data.

my_b_copy_to_file()
  is turned into my_b_copy_to_file_frag()
  which accepts format specifier arguments to build a proper syntax BINLOG
  query in both the fragmented (n_frag > 1) and non-fragmented (n_frag == 1)
  cases.

Rows_log_event::print_helper()
  Takes decision whether to fragment, prepares respective format specifiers
  and invokes the cache-to-file copying function, which is now

copy_cache_frag_to_file_and_reinit()
  replaces original copy_event_cache_to_file_and_reinit() to pass
  extra arguments to my_b_copy_to_file() successor of

my_b_copy_to_file_frag()
  replaces the former pure copier not to also conduct wrapping the encoded
  data per format specifiers.
  With its 'n_frag' argument as 1
  and the rest or args NULL works as the original function.

Author: Alexander Barkov
Author Date: 2018-09-21 14:17:32 UTC

Merge remote-tracking branch 'origin/5.5' into bb-10.0-bar

Author: Sergey Vojtovich
Author Date: 2018-09-21 12:04:16 UTC

MDEV-17200 - pthread_detach called for already detached threads

pthread_detach_this_thread() was intended to be defined to something
meaningful only on some ancient unixes, which don't have
pthread_attr_setdetachstate() defined. Otherwise, on normal unixes,
threads are created detached in the first place.

This was broken in 0f01bf267680244ec488adaf65a42838756ed48e so that
we started calling pthread_detach() for already detached threads.
Intention was to detach aria checkpoint thread.

However in 87007dc2f71634cc460271eb277ad851ec69c04b aria service threads
were made joinable with appropriate handling, which makes breaking
revision unneccessary.

Revert remnants of 0f01bf267680244ec488adaf65a42838756ed48e, so that
pthread_detach_this_thread() is meaningful only on some ancient unixes
again.

Author: Andrei Elkin
Author Date: 2018-09-17 18:40:23 UTC

MDEV-17133 dump thread reads from a past position

Author: Jacob Mathew
Author Date: 2018-09-18 01:39:16 UTC

MDEV-17144: Sample of spider_direct_sql cause crash

The crash occurs when the Spider node server attempts to create an error
message stating that the temporary table is not found. The function to
create the error message is called with incorrect parameters.

I fixed the crash by correcting the incorrect parameter values.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Author: Sachin Setiya
Author Date: 2018-09-17 19:09:32 UTC

MDEV-14716 Wrong binlog entries for a multi-user CREATE USER statement

Problem:-
   Suppose If we have Create User with multiple users.
      CREATE USER
         user1@localhost IDENTIFIED BY 'BsG9#9.cem#!85',
         user2@localhost IDENTIFIED BY PASSWORD 'xxx';
   here on 2nd user we will get error, But user1 will be created.
   But in binlog the whole stmt will be written.
   Actually this might create problem , we can have data in-
   consistency , for example this stmt
   On Master
   INSTALL SONAME "simple_password_check";
   CREATE USER
    user1@localhost IDENTIFIED BY 'BsG9#9.cem#!85',
    user2@localhost IDENTIFIED BY 'bsg9#d.cem#!85';
   we will get error in creation of user2 but in binlog whole
   And on slave we havent installed plugin

Author: Jan Lindström
Author Date: 2018-09-17 09:03:41 UTC

MDEV-17206: Test failure on galera.galera_ist_* in debug builds

Failure is due missing .rdiff files for debug build as tests have
@have_debug combination.

Author: Alexey Botchkov
Author Date: 2018-09-15 21:55:30 UTC

MDEV-16050

Author: Vladislav Vaintroub
Author Date: 2018-09-14 17:53:27 UTC

MDEV-15088 Lighter version of Windows Zip distributions

Add new target win_package, that produces
a) light ZIP (no test suite, no sql-bench, no debugsymbols)
b) debug symbols ZIP

Author: Marko Mäkelä
Author Date: 2018-09-14 15:50:37 UTC

MDEV-17199 Assertion `pos < table->n_v_def' failed after upgrade to 10.2

Before MDEV-5800 in MariaDB 10.2.2, InnoDB knew nothing about
virtual columns. But, they would be present in the TABLE_SHARE.

It appears that the MDEV-12936 fix only worked in the special case
when the virtual columns are the last columns in a table definition.

mysql_fields(): Remove.

build_template_needs_field(): Omit virtual columns if the
InnoDB table definition is known to not contain them, based
on the .frm file version.

build_template_field(): Clean up some code.

ha_innobase::build_template(): Remove redundant computations.
Loop over all MariaDB columns. Skip virtual columns if
InnoDB does not know about them.

calc_row_difference(): Skip not-known-to-InnoDB virtual columns.

wsrep_calc_row_hash(): Skip all virtual columns. This would
fix an out-of-bounds access to dict_table_t::cols[]. Also,
remove some redundant computations and variables.

create_table_check_doc_id_col(): Assert that the .frm file
for a newly created table will be in a format where InnoDB
knows about virtual columns.

Author: Oleksandr Byelkin
Author Date: 2018-09-13 16:30:37 UTC

postmerge

Author: Jan Lindström
Author Date: 2018-09-11 08:21:29 UTC

MDEV-13743: galera_toi_truncate may fail with: query 'reap' succeeded - should have failed with errno 1213

Replace sleep with wait condition and increase the size of the
insert.

Author: Alexey Botchkov
Author Date: 2018-09-11 06:38:46 UTC

MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.

LOCK view WRITE shouldn't block FLUSH view.
So we set the view's mdl_request type to it's tables.

Author: Jan Lindström
Author Date: 2018-09-10 10:43:37 UTC

MDEV-17151: Galera test failure on galera.galera_var_node_address

Add wait_conditions to wait expected node contents.

Author: Vladislav Vaintroub
Author Date: 2018-09-09 04:36:00 UTC

Windows : Fix build with OpenSSL DLLs.

Link with applink shim mentioned in https://www.openssl.org/docs/faq.html,
to avoid "No OpenSSL_Applink" runtime errors.

Author: Sergey Vojtovich
Author Date: 2018-05-29 17:46:31 UTC

C++11 experiment: check if all builders support it

Author: Jan Lindström
Author Date: 2018-09-08 13:57:31 UTC

MDEV-17143: Galera test failure on galera.MW-44

Start general log OFF and then truncate mysql.general_log and
use proper wait timeouts to make sure it is really empty.

Author: Jan Lindström
Author Date: 2018-09-08 05:07:25 UTC

Remove incorrect install command.

Author: Jan Lindström
Author Date: 2018-09-06 16:31:25 UTC

Run #2.

Author: Sergei Golubchik
Author Date: 2018-09-06 18:44:47 UTC

mtr: fix tokuftdump detection in out-of-source builds

Author: Jan Lindström
Author Date: 2018-09-06 10:18:34 UTC

Disable changing output and fix version requirement.

Author: Sachin Setiya
Author Date: 2018-09-06 07:43:52 UTC

Mdev-5628 Assertion `! is_set()' fails on UPDATE on a partitioned table...
with subquery (MySQL:71630)

In mysql_update after prune_partitions(thd, table, conds) we set thd->status
to ok , without considering that thd earlier might have got a error.

Author: Monty
Author Date: 2018-09-05 17:20:16 UTC

Added DBUG_ASSERT_AS_PRINTF to BUILD/..asan-max

Author: Sergey Vojtovich
Author Date: 2018-09-05 10:44:07 UTC

MDEV-16773 - Assertion failed in tdc_remove_table

This assertion fails in thread that removes all table instances for
particular table from table cache (e.g. "DROP TABLE") while another
thread evicts table instance of the same table from table cache
concurrently.

After "MDEV-10296 - Multi-instance table cache" there is a gap in
eviction code of tc_add_table() between removing table from free_tables
and all_tables not protected by any mutexes.

This is now valid table cache state, however assertion wasn't amended
along with original patch. Moved assertion down, after waiting for such
table instances to get closed.

Author: Jan Lindström
Author Date: 2018-09-04 15:50:06 UTC

Galera test case cleanups.

Author: Oleksandr Byelkin
Author Date: 2018-09-03 08:57:02 UTC

Merge branch '5.5' into 10.0

Author: Otto Kekäläinen
Author Date: 2018-09-02 19:36:14 UTC

WIP: test 6

Author: Sergey Petrunia
Author Date: 2018-09-01 15:43:05 UTC

Merge branch '10.2' of github.com:MariaDB/server into bb-10.2-mariarocks-merge

Author: Sergey Petrunia
Author Date: 2018-09-01 12:10:50 UTC

Trigger another BB run

Author: Jan Lindström
Author Date: 2018-08-31 11:38:32 UTC

Disable galera suite until failing test cases are fixed to work.

Author: Monty
Author Date: 2018-08-31 07:47:18 UTC

Fixed several issues with aria_chk

- Made output to be aligned in aria_chk -d
- Aria engine error texts are now written instead of "Undefined error"
- When running with --check --force, tables with wrong TRN's but otherwise
  correct are now zerofilled
- Fixed several bugs in check and recovery related to fulltext
- When doing recovery, store highest found TRID in aria_control_file
  Before this, the

Author: Jan Lindström
Author Date: 2018-08-29 07:31:47 UTC

Merge pull request #615 from grooverdan/5.5-MDEV-13129-apport-filter-wsrep_sst_auth

MDEV-13129: apport - sanitize wsrep_sst_auth from report

Author: Varun Gupta
Author Date: 2018-08-28 20:21:39 UTC

More tests of main suite updated

Author: Jan Lindström
Author Date: 2018-08-27 09:03:02 UTC

MDEV-17062: Test failure on galera.MW-336
MDEV-17058: Test failure on wsrep.variables
MDEV-17060: Test failure on galera.galera_var_slave_threads

Fix incorrect calculation of increased applier (slave) threads.
Note that increase change takes effect "immediately" but we should
use proper wait condition to wait it. Reducing the number of
slave threads is not immediate as thread will only exit after a
replication event.

Author: Jan Lindström
Author Date: 2018-08-27 09:03:02 UTC

MDEV-17062: Test failure on galera.MW-336
MDEV-17058: Test failure on wsrep.variables
MDEV-17060: Test failure on galera.galera_var_slave_threads

Fix incorrect calculation of increased applier (slave) threads.
Note that increase change takes effect "immediately" but we should
use proper wait condition to wait it. Reducing the number of
slave threads is not immediate as thread will only exit after a
replication event.

Author: Sachin Setiya
Author Date: 2018-08-26 05:25:36 UTC

26 aug

Author: Zhang Shuai
Author Date: 2018-03-16 17:22:53 UTC

MDEV-15204: lag/lead function order list mandatory

Author: Anel Husakovic
Author Date: 2018-04-03 12:41:13 UTC

MDEV-14474 information_schema.check_constraints

Implement according to standard SQL specification 2008.
The check_constraints table is used for fetching metadata about
the constraints defined for tables in all databases.

Author: Varun Gupta
Author Date: 2018-08-24 11:30:32 UTC

Fixed ASAN failure for the test main.func_misc

Moved the checks for arguments validation of Item_name_const from the constructor
to Create_func_name_const::create_2_arg
Also reverted the fix bf1c53e9be84437ada32393bb7b4a8ff06dbf369

Author: Jan Lindström
Author Date: 2018-08-23 14:14:13 UTC

Merge remote-tracking branch 'wsrep/10.3_wsrep_api-26' into 10.4

Conflicts:
 mysql-test/include/check-testcase.test
 mysql-test/include/wait_until_connected_again.inc
 mysql-test/suite/galera/disabled.def
 mysql-test/suite/galera/r/MW-416.result
 mysql-test/suite/galera/r/galera#500.result
 mysql-test/suite/galera/r/galera_var_dirty_reads.result
 mysql-test/suite/galera/t/MW-416.test
 mysql-test/suite/galera/t/galera#500.test
 sql/handler.cc
 sql/item_strfunc.cc
 sql/item_strfunc.h
 sql/slave.cc
 sql/sql_class.cc
 sql/sql_class.h
 sql/sql_connect.cc
 sql/sql_insert.cc
 sql/sql_parse.cc
 sql/sql_plugin.cc
 sql/sql_prepare.cc
 sql/wsrep_applier.cc
 sql/wsrep_mysqld.cc
 sql/wsrep_mysqld.h
 sql/wsrep_sst.cc
 sql/wsrep_thd.cc
 storage/innobase/handler/ha_innodb.cc

Author: Alexey Botchkov
Author Date: 2018-08-21 10:47:45 UTC

MDEV-14005 Remove need for Partition Key to be part of Primary Key.

The limitation was removed, so now we check all the partition
for unique key duplicates in these cases.

Author: Jan Lindström
Author Date: 2018-08-20 06:30:34 UTC

Merge branch 'tempesta-tech-sysprg/10.1-MDEV-10754' into 10.1

Author: Varun Gupta
Author Date: 2018-08-16 15:04:38 UTC

Tests result updated for switch optimizer_use_condition_selectivity=3

Author: Oleksandr Byelkin
Author Date: 2018-08-16 11:55:36 UTC

MDEV-16930: Crash when VALUES in derived table contains expressions

Give names to the value constructor columns as in SELECT-list.

Author: Vladislav Vaintroub
Author Date: 2018-08-09 13:50:08 UTC

Fix compiler warning