lp:maria

Author: Marko Mäkelä
Author Date: 2020-10-31 08:10:54 UTC

Try to avoid a shutdown hang

Author: Daniel Black
Author Date: 2020-07-16 06:31:59 UTC

MDEV-22974: mysql_native_password make "invalid" valid

Per b9f3f06857ac, mysql_system_tables_data.sql creates
a mysql_native_password with a salted hash of "invalid" so that `set password`
will detect a native password can be applied:.

SHOW CREATE USER; diligently uses this value in its output
generating the SQL:

   MariaDB [(none)]> show create user;

   +---------------------------------------------------------------------------------------------------+
   | CREATE USER for dan@localhost |
   +---------------------------------------------------------------------------------------------------+
   | CREATE USER `dan`@`localhost` IDENTIFIED VIA mysql_native_password USING 'invalid' OR unix_socket |
   +---------------------------------------------------------------------------------------------------+

Attempting to execute this before this patch results in:

  MariaDB [(none)]> CREATE USER `dan2`@`localhost` IDENTIFIED VIA mysql_native_password USING 'invalid' OR unix_socket;
  ERROR 1372 (HY000): Password hash should be a 41-digit hexadecimal number

As such, deep the implementation of mysql_native_password we make "invalid" valid (pun intended)
such that the above create user will succeed. We do this by storing
"*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE" (credit: Oracle MySQL), that is of an INCORRECT
length for a scramble.

In native_password_authenticate we check the length of this cached value
and immediately fail if it is anything other than the scramble length.

native_password_get_salt is only called in the context of set_user_salt, so all setting of native
passwords to hashed content of 'invalid', quite literally create an invalid password.

So other forms of "invalid" are valid SQL in creating invalid passwords:

   MariaDB [(none)]> set password = 'invalid';
   Query OK, 0 rows affected (0.001 sec)

   MariaDB [(none)]> alter user dan@localhost IDENTIFIED BY PASSWORD 'invalid';
   Query OK, 0 rows affected (0.000 sec)

closes #1628

Reviewer: serg@mariadb.com

Author: Sergey Petrunia
Author Date: 2020-10-30 18:25:25 UTC

Add dbug_print_sel_arg() debugging help function

Author: Marko Mäkelä
Author Date: 2020-10-30 17:06:50 UTC

MDEV-24054 Assertion in_LRU_list failed in buf_flush_try_neighbors()

buf_flush_try_neighbors(): Before invoking buf_page_t::ready_for_flush(),
check that the freshly looked up buf_pool.page_hash entry actually is
a buffer page and not a buf_pool.watch[] sentinel for purge buffering.

This race condition was introduced in MDEV-15053
(commit b1ab211dee599eabd9a5b886fafa3adea29ae041).
It is rather hard to hit this bug, because
buf_flush_check_neighbors() already checked the condition.
The problem exists if buf_pool.watch_set() was invoked for
a page in the range after the check in buf_flush_check_neighbor()
had been finished.

Author: Marko Mäkelä
Author Date: 2020-10-30 14:48:37 UTC

WIP MDEV-22387

Author: Monty
Author Date: 2020-10-30 13:33:18 UTC

MDEV-23824 SIGSEGV in end_io_cache on REPAIR LOCAL TABLE for Aria table

Bugs fixed:
- prepare_for_repair() didn't close all open files if table opened failed
  because of out-of-memory
- If dd_recreate_table() failed, the data file was not properly restored
  from it's temporary name
- Aria repair initializing code didn't properly clear all used structs
  before calling error, which caused crashed in memory-free calls.
- maria_delete_table() didn't register if table open failed. This could
  calls my_error() to be called without returning 1 to the caller, which
  cased failures in my_ok()
-

Author: Oleksandr "Sanja" Byelkin
Author Date: 2020-10-30 13:01:24 UTC

Merge branch 'bb-10.2-release' into bb-10.3-release

Author: Sergei Golubchik
Author Date: 2020-10-30 11:02:48 UTC

MDEV-22694 New -compat RPM package breaks certain upgrades

only provide/obsolete mysql-libs up to 5.6.99

5.7 comes with libmysqlclient.so.20

Author: Sergei Golubchik
Author Date: 2020-10-30 11:11:43 UTC

w1

Author: Marko Mäkelä
Author Date: 2020-10-30 10:58:16 UTC

MDEV-23991 fixup: Initialize the memory

This regression was introduced in
commit afc9d00c66db946c8240fe1fa6b345a3a8b6fec1.
This is a partial backport of
commit 199863d72b7cccaa4c75641c50c45a83b568ab8c from 10.4.

Author: Sergei Golubchik
Author Date: 2020-10-29 21:29:50 UTC

Fix RPM packaging on cmake 3.18+

cmake has caught up and since version 3.18 it started supporting
CPACK_RPM_POST_TRANS_SCRIPT_FILE, something we've supported for
two years and cmake 2.8.11. Both implementation add %posttrans tag
and rpmbuild gets confused.

Disable our implementation for cmake 3.18+

Author: sjaakola
Author Date: 2020-09-24 12:22:07 UTC

MDEV-23328 Server hang due to Galera lock conflict resolution

Problem

Whenever Galera BF (brute force) transaction decides to abort conflicting
transaction it will kill that thread using thd::awake()

User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()

Whenever one of these actions is executed we will hold number of InnoDB
internal mutexes and thd mutexes. Sometimes these mutexes are taken in
different order causing mutex deadlock.

Lets create victim transaction that will take record locks and is on
QUERY_IDLE state i.e. no write-sets are replicated to other nodes in
a Galera cluster. We will call this as victim_thread.

--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2
--connection node_2a
CREATE TABLE t1(a int not null primary key auto_increment,b int) engine=InnoDB;
INSERT INTO t1 VALUES (1,0);
BEGIN;
UPDATE t1 SET b=2 WHERE a=1;

From same node lets take thread_id for victim_thread

--connection node_2
--let $k_thread = `SELECT ID FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'root' AND COMMAND = 'Sleep' LIMIT 1`

Lets create transaction that will be run on applier at node_2 as brute
force (BF). Certification of this transaction naturally passes on
node_1 and write-set is send to node_2 to be executed. We call this as
bf_kill thread.

--connection node_1
update t1 set b= 5 where a=1;

Applier will naturally try to take record locks for updated rows.
However, as victim_thread transaction already has same record locks
granted we have a problem. This BF transaction should wait for its
turn to get record locks, but we should not wait. Thus we need to
abort local victim_thread transaction holding conflicting record
locks.

We find this at lock_rec_other_has_conflicting(). We are holding
lock_sys->mutex and when the conflicting record lock granted to
victim_thread is found we take victim_trx->mutex. Exectution of
this victim bf kill continue to wsrep_innobase_kill_one_trx and
we take victim_thread->LOCK_thd_data mutex before we take any
actions.

This means that bf_kill always take these mutexes in order:
  lock_sys->mutex -> victim_trx->mutex -> victim_thread->LOCK_thd_data

Now let's consider other way to kill our victim_thread. This is user
SQL-statement KILL and we will call it kill.

--connection node_2
--send_eval KILL $k_thread

This execution starts from kill_one_thread() and we will try to find
thread with same id as user has provided in find_thread_by_id() and
when we find it we take victim_thread->LOCK_thd_data mutex. Execution
continues to victim_thread->awake() call and that will call
innobase_kill_query(). There we take lock_sys->mutex and
victim_trx->mutex before victim transaction record locks are canceled.

This means that kill always takes these mutexes in order:
   victim_thread->LOCK_thd_data -> lock_sys->mutex -> victim_trx->mutex

Because nothing really restricts concurrent scheduling of both bf_kill
and kill and we take mutexes in reverse order we could end up a mutex
deadlock where both bf_kill and kill wait each other.

One possible scheduling of bf_kill thread and kill thread is execution
where bf_kill is executed until it has acquired lock_sys->mutex and
victim_trx->mutex and then we start execution of kill thread. There we
take victim_thread->LOCK_thd_data mutex but we cant continue exectuion
to innobase_kill_query() as we can't acquire lock_sys->mutex hold by
bf_kill. So if we then continue bf_kill thread execution it will try
to take victim_thread->LOCK_thd_data mutex but it can't have it as it
is hold by kill thread. Thus, we have reached point in exection where
both bf_kill thread is waiting for kill thread and kill thread is
waiting for bf_kill thread. And this deadlock is possible because of
opposite order of mutex acquisition.

Fix

The fix in this commit adds THD::wsrep_aborter flag to synchronize who
can kill the victim. This flag is set both when BF is called for from
innodb and by KILL command. Either path of victim killing will bail
out if victim's wsrep_killed is already set to avoid mutex conflicts
with the other aborter execution. THD::wsrep_aborter records the
aborter THD's ID. This is needed to preserve the right to kill the
victim from different locations for the same aborter thread. It is
also good error logging, to see who is reponsible for the abort.

Changed manual KILL command to perform first a round to mark the victim
for aborting. Victim marking will be initiated from inside innodb,
and by holding the same mutexes (lock sys and trx mutex) as high
priority thread (e.g. replication applier) would have when preparing
for BF abort.

With this victim marking, the victim may or may not be marked for abort
by the KILL command issuer. This does not matter, the point in this PR
is to guarantee that victim marking will happen through same mutex
locking protocol, both by BF aborting and manual killing use cases.
This should guarantee that these two different ways to abort nother
query will not end up in mutex deadlock.

The actual kill is processed after this victim marking round, and will
happen in same way as earlier. However, only manual KILL or BF aborting
will complete the victim killing.

Reviewed and cleaned-by: Jan Lindström <jan.lindstrom@mariadb.com>

Author: Elena Stepanova
Author Date: 2020-10-30 00:47:21 UTC

List of unstable tests for 10.1.48 release

Author: Sergei Golubchik
Author Date: 2020-10-29 21:29:50 UTC

Fix RPM packaging on cmake 3.18+

cmake has caught up and since version 3.18 it started supporting
CPACK_RPM_POST_TRANS_SCRIPT_FILE, something we've supported for
two years and cmake 2.8.11. Both implementation add %posttrans tag
and rpmbuild gets confused.

Disable our implementation for cmake 3.18+

Author: varun
Author Date: 2020-10-29 16:08:06 UTC

 MDEV-24033: SIGSEGV in __memcmp_avx2_movbe from queue_insert | SIGSEGV in __memcmp_avx2_movbe from native_compare

The issue here was the system variable max_sort_length was being applied
to decimals and it was truncating the value for decimals to the number
of bytes set by max_sort_length.
This was leading to a buffer overflow as the values were written
to the buffer without truncation and then we moved the offset to
the number of bytes(set by max_sort_length), that are needed for comparison.

The fix is to not apply max_sort_length for fixed size types like INT,
DECIMALS and only apply max_sort_length for CHAR, VARCHARS, TEXT and
BLOBS.

Author: varun
Author Date: 2020-10-29 16:08:06 UTC

 MDEV-24033: SIGSEGV in __memcmp_avx2_movbe from queue_insert | SIGSEGV in __memcmp_avx2_movbe from native_compare

The issue here was the system variable max_sort_length was being applied
to decimals and it was truncating the value for decimals to the number
of bytes set by max_sort_length.
This was leading to a buffer overflow as the values were written
to the buffer without truncation and then we moved the offset to
the number of bytes(set by max_sort_length), that are needed for comparison.

The fix is to not apply max_sort_length for fixed size types like INT,
DECIMALS and only apply max_sort_length for CHAR, VARCHARS, TEXT and
BLOBS.

Author: Monty
Author Date: 2020-10-29 16:34:26 UTC

MDEV-23222 SIGSEG in maria_create() because of double free

The crash happens because a double free in the case CREATE TABLE fails
because there is a conflicting tables on disk.

Fixed by ensuring that the double free can't happen.

Author: Marko Mäkelä
Author Date: 2020-10-29 16:25:40 UTC

Merge 10.4 into 10.5

Author: Larysa Sherepa
Author Date: 2020-10-29 14:20:48 UTC

MDEV-23103 rpl.rpl_gtid_delete_domain failed in buildbot

Author: Sergei Golubchik
Author Date: 2020-10-29 12:03:08 UTC

Merge commit 'c05e571e7d53ba11a3422490cff0d93e6691ea62' into HEAD

Author: Marko Mäkelä
Author Date: 2020-10-29 10:11:26 UTC

MDEV-22805 post-fix

Author: midenok
Author Date: 2020-10-29 09:14:19 UTC

MDEV-22805 SIGSEGV in check_fields on UPDATE

Additional case for PS protocol: UPDATE is converted to multi-update
in mysql_multi_update_prepare().

Author: varun
Author Date: 2020-10-29 09:40:08 UTC

Fixing some test failures

Author: Sergei Golubchik
Author Date: 2020-10-27 23:58:38 UTC

update columnstore

Author: Vicențiu Ciorbaru
Author Date: 2020-10-29 06:49:03 UTC

MYSQL_JSON: Update test case to omit .so or .dll extension

Author: Andrei Elkin
Author Date: 2020-10-28 19:52:27 UTC

multi-engine, safe to truncate, memory release in binlog truncated error branches; fixes to tests.

Author: Teemu Ollakka
Author Date: 2020-10-25 09:17:27 UTC

MDEV-22681 EXECUTE IMMEDIATE crashes server if wsrep is on.

A wsrep transaction was started for EXECUTE IMMEDIATE, which
caused assertion failure when the executed statement was
CREATE TABLE which should be executed in TOI mode.

As a fix, don't start wsrep transaction for EXECUTE IMMEDIATE
to let the wsrep state logic to be handled from inside stored
procedure codepath.

Author: Vladislav Vaintroub
Author Date: 2020-10-27 19:31:16 UTC

MDEV-24040 Named pipe permission issue

Tighten access control - deny FILE_CREATE_PIPE_INSTANCE permission to
everyone except current user (the one that runs mysqld)

Author: Vladislav Vaintroub
Author Date: 2020-10-27 19:54:03 UTC

MDEV-24037 Use NtFlushBuffersFileEx(FLUSH_FLAGS_FILE_DATA_SYNC_ONLY) on Windows

This avoids flushing file metadata on NTFS , and writing to <drive>:\$Log
file. With heavy write workload this can consume up to 1/3 of the
server's IO bandwidth.

Author: varun
Author Date: 2020-10-26 06:35:47 UTC

MDEV-24015: SQL Error (1038): Out of sort memory when enough memory for the sort buffer is provided

For a correlated subquery filesort is executed multiple times.
During each execution, sortlength() computed total sort key length in
Sort_keys::sort_length, without resetting it first.

Eventually Sort_keys::sort_length got larger than @@sort_buffer_size, which
caused filesort() to be aborted with error.

Fixed by making sortlength() to compute lengths only during the first
invocation. Subsequent invocations return pre-computed values.

Author: Sachin Setiya
Author Date: 2020-10-27 18:57:49 UTC

xyz 2

Author: Vlad Lesin
Author Date: 2020-10-27 17:03:41 UTC

MDEV-24026: InnoDB: Failing assertion: os_total_large_mem_allocated >= size upon incremental backup

Preliminary fix without test case.

Author: Eugene
Author Date: 2020-10-27 09:24:55 UTC

MDEV-23991 dict_table_stats_lock() has unnecessarily long scope

Patch removes dict_index_t::stats_latch. Table/index statistics now
protected with dict_sys->mutex. That way statistics computation can
happen in parallel in several threads and dict_sys->mutex will be locked
only for a short period of time.

This patch is a joint work with Marko Mäkelä

dict_index_t::lock: make mutable which allows to pass const pointer
when only lock is touched in an object

btr_height_get()
btr_get_size(): make index argument const for better type safety

btr_estimate_number_of_different_key_vals(): now returns computed values
instead of setting fields in dict_index_t directly

remove everything related to dict_index_t::stats_latch

dict_stats_index_set_n_diff(): now returns computed values instead
of setting fields in dict_index_t directly

dict_stats_analyze_index(): now returns computed values instead
of setting fields in dict_index_t directly

Reviewed by: Marko Mäkelä

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-10-21 16:03:29 UTC

MDEV-23693 Failing assertion: my_atomic_load32_explicit(&lock->lock_word, MY_MEMORY_ORDER_RELAXED) == X_LOCK_DECR

InnoDB frees the block lock during buffer pool shrinking when other
thread is yet to release the block lock. While shrinking the
buffer pool, InnoDB allows the page to be freed unless it is buffer
fixed. In some cases, InnoDB releases the latch after unfixing the
block.

Fix:
====
- InnoDB should unfix the block after releases the latch.

- Add more assertion to check buffer fix while accessing the page.

- Introduced block_hint structure to store buf_block_t pointer
and allow accessing the buf_block_t pointer only by passing a
functor. It returns original buf_block_t* pointer if it is valid
or nullptr if the pointer become stale.

- Replace buf_block_is_uncompressed() with
buf_pool_t::is_block_pointer()

This change is motivated by a change in mysql-5.6.50:
mysql/mysql-server@46e60de444a8fbd876cc6778a7e64a1d3426a48d
Bug #31036301 ASSERTION FAILURE: SYNC0RW.IC:429:LOCK->LOCK_WORD

Author: Anel Husakovic
Author Date: 2020-10-27 11:16:53 UTC

MDEV-24018: SIGSEGV in Item_func_nextval::update_table on SELECT SETVAL

Author: Vladislav Vaintroub
Author Date: 2020-10-27 10:49:17 UTC

MDEV-24034 Policy CMP0075 is not set during compile

The policy is not set for 10.2
If it is set, CMake would complain about bundled zlib for which the policy
is not set.

Fix:
- Set policy for 10.2 for the top level project.
For 10.3+ it was already set

- Cleanup zlib to remove unneeded stuff. It is an internal static library,
it needs none of PROJECT, library versioning, RC file on Windows.
The name of the library on Unix does not make any difference, since it is
static and compiled in.

Author: mkaruza
Author Date: 2020-10-21 07:48:52 UTC

MDEV-22707: galera got stuck after flush tables

Deadlock is possible between applier thread and local committing thread with active FLUSH TABLE.
Applier thread should skip table share checks and locks when opening table.

Author: Anel Husakovic
Author Date: 2020-10-27 01:19:05 UTC

MDEV-23769: MTR can abort before it prints the test result summary

- Patch is solving generating report on warning
    To repeat the error run single worker:
    ```
    ./mtr --mysqld=--lock-wait-timeout=-xx 1st 1st --force --parallel 1
    ```
    or `N` workers with `N+1` tests with failures and `force`
    ```
    ./mtr --mysqld=--lock-wait-timeout=-xx 1st 1st grant5 --force --parallel 2
    ```
- Patch is doing cosmetic fix of `current_test` log file which holds the old log value of test `CURRENT TEST:..` in `mark_log()` in case of `unknown option` and as such
  the logic which is using it's content doesn't output valid log content and doesn't generate valid `$test->{'comment'}` message.asdf

Author: Alexey Botchkov
Author Date: 2020-10-26 20:30:39 UTC

MDEV-22524 SIGABRT in safe_mutex_unlock with
session_track_system_variables and max_relay_log_size.

lock LOCK_global_system_variables around the get_one_variable() call
in the Session_sysvars_tracker::store_variable().

Author: midenok
Author Date: 2020-10-26 09:31:14 UTC

MDEV-18734 blob mem_root for each m_ordered_rec_buffer element

Processing infinite amount of rows will drain out RAM via
m_ordered_root. The patch solves the problem by introducing
per-element memory pools. So each element in m_ordered_rec_buffer has
its personal mem_root. Before the record is stored into
m_ordered_rec_buffer element its mem_root is freed by returning memory
to pool.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-10-23 15:20:03 UTC

MDEV-23693 Failing assertion: my_atomic_load32_explicit(&lock->lock_word, MY_MEMORY_ORDER_RELAXED) == X_LOCK_DECR

InnoDB frees the block lock during buffer pool shrinking when other
thread is yet to release the block lock. While shrinking the
buffer pool, InnoDB allows the page to be freed unless it is buffer
fixed. In some cases, InnoDB releases the latch after unfixing the
block.

Fix:
====
- InnoDB should unfix the block after releases the latch.

- Add more assertion to check buffer fix while accessing the page.

- Introduced block_hint structure to store buf_block_t pointer
and allow accessing the buf_block_t pointer only by passing a
functor. It returns original buf_block_t* pointer if it is valid
or nullptr if the pointer become stale.

- Replace buf_block_is_uncompressed() with buf_pointer_is_block_field_instance()

This change is motivated by a change in mysql-5.6.50:
46e60de444a8fbd876cc6778a7e64a1d3426a48d
Bug #31036301 ASSERTION FAILURE: SYNC0RW.IC:429:LOCK->LOCK_WORD

Author: Daniel Black
Author Date: 2020-10-26 08:21:00 UTC

MDEV-23326: mysql_install_db - fix Debian patch

Author: Marko Mäkelä
Author Date: 2020-10-26 08:00:39 UTC

Amend 32ab5e10295b058a9a94aa8612393a9970e9812a

Always wait log_flush_task to finish on shutdown.

Author: varun
Author Date: 2020-10-26 07:40:44 UTC

Followup fix

Author: Dmitry Shulga
Author Date: 2020-10-26 05:28:01 UTC

MDEV-22805: SIGSEGV in check_fields on UPDATE

For debug build of MariaDB server running of the following test case
will hit the assert `thd->lex->sql_command == SQLCOM_UPDATE' in the function
check_fields() on attempt to execute the UPDATE statement.

  CREATE TABLE t1 (a INT);
  UPDATE t1 FOR PORTION OF APPTIME FROM (SELECT 1 FROM t1) TO 2 SET a = 1;

Stack trace to the fired assert statement
  DBUG_ASSERT(thd->lex->sql_command == SQLCOM_UPDATE)
listed below:
  mysql_execute_command() ->
    mysql_multi_update_prepare() -->
      Multiupdate_prelocking_strategy::handle_end() -->
        check_fiels()

It's worth to note that this stack trace looks like a multi update
statement is being executed. The fired assert is checked inside the
function check_fields() in case table->has_period() returns the value
true that in turns happens when temporal period specified in the UPDATE
statement. Condition specified in the DEBUG_ASSERT statement returns
the false value since the data member thd->lex->sql_command have the
value SQLCOM_UPDATE_MULTI. So, the main question is why a program control
flow go to the path prescribed for handling MULTI update statement
despite of the fact that the ordinary UPDATE statement being executed.

The answer is a way that SQL grammar rules written.

When the statement
  UPDATE t1 FOR PORTION OF APPTIME FROM (SELECT 1 FROM t1) TO 2 SET a = 1;
being parsed an action for the rule 'table_primary_ident' (part of this action
is listed below to simplify description) is invoked to handle the table
name 't1' specified in the clause 'SELECT 1 FROM t1'.

table_primary_ident:
  table_ident opt_use_partition opt_for_system_time_clause
  opt_table_alias_clause opt_key_definition
  {
    SELECT_LEX *sel= Select;
    sel->table_join_options= 0;
    if (!($$= Select->add_table_to_list(thd, $1, $4,

This action calls the method st_select_lex::add_table_to_list()
to add the table name 't1' to the list of tables being used by the statement.

Later, an action for the following grammar rule
update_table_list:
  table_ident opt_use_partition for_portion_of_time_clause
  opt_table_alias_clause opt_key_definition
  {
    SELECT_LEX *sel= Select;
    sel->table_join_options= 0;
    if (!($$= Select->add_table_to_list(thd, $1, $4,

is invoked to handle the clause 't1 FOR PORTION OF APPTIME FROM ... TO 2'.
This action also calls the method st_select_lex::add_table_to_list()
to add the table name 't1' to the list of tables being used by the statement.

In result the table name 't1' contained twice in this list.

Presence of duplicate names for the table 't1' in a list of table used by
a statement leads to the fact that the function unique_table() called
from the function mysql_update() returns the value true that forces
implementation of the function mysql_update() to return the value 2 as
a signal to fall through the case boundary of the switch statement placed
in the function mysql_execute_statement() and start handling of the case
for sql_command SQLCOM_UPDATE_MULTI. The compound statement block for the
case SQLCOM_UPDATE_MULTI invokes the function mysql_multi_update_prepare()
that executes the statement
  set thd->lex->sql_command= SQLCOM_UPDATE_MULTI;
and after that calls the method
  Multiupdate_prelocking_strategy::handle_end(). Finally, this method
invokes the check_field() function and assert is fired.

The above analysis shows that update for a table that simultaneously specified
both as a destination table of UPDATE statement and as a table taking part in
subquery is actually treated by MariaDB server as multi-update statement.
Taking into account that multi-update statement for temporal period
table is not supported yet by MariaDB, correct way to fix the bug is to return
the error ER_NOT_SUPPORTED_YET for this case.

Author: Nikita Malyavin
Author Date: 2019-03-05 11:12:54 UTC

MDEV-17891 Assertion failure upon attempt to replace into a full table

Problem: Assertion `transactional_table || !changed ||
thd->transaction.stmt.modified_non_trans_table' failed due REPLACE into a
versioned table.

It is not specific to system versioning/pertitioning/heap, but this
combination makes it much easier to reproduce.

The thing is to make first ha_update_row call succeed to make
info->deleted != 0. And then make REPLACE fail by any reason.

In this scenario we overflow versioned partition, so next ha_update_row
succeeds, but corresponding ha_write_row fails to insert history record.

Fix: modified_non_trans_table is set in one missed place

Author: Nikita Malyavin
Author Date: 2020-10-14 05:40:46 UTC

handler: move row change start signal down after the checks

Author: Vicențiu Ciorbaru
Author Date: 2020-10-23 15:09:01 UTC

MDEV-14945 possible buffer overflow in stack resolver

According to https://stackoverflow.com/questions/22827510/how-to-avoid-bad-fd-set-buffer-overflow-crash
it seems that using select instead of poll can cause additional memory
allocations. As we are in a crashed state, we must prevent allocating
any memory (if possible). Thus, switch select call to poll.

Author: Sergey Vojtovich
Author Date: 2020-04-28 10:25:42 UTC

MDEV-19508 - Issue with: #define SI_KERNEL

Reverted 07e9b1389, use setsid() instead.

Conventional use of SIGHUP to dump daemon information or flush caches
assumes preceeding setsid() call.

Ideally we should not setup SIGHUP handler in bootstrap and (probably)
debug_gdb modes.

Additional information on SysV style and systemd style daemons:
https://www.freedesktop.org/software/systemd/man/daemon.html

closes #1392

Author: Daniel Black
Author Date: 2020-10-24 10:01:07 UTC

MDEV-22974: mysql_native_password make '*40chars' invalid if nonhex

Porting a show create user from mysql may have the hashed password
string: *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE.

Obviously this isn't a hash because it contains non-hex characters.

We do however recognise the *{40 chars not all are hex} as a pattern
for an invalid password and processes that internally. This was
determined to be the general pattern that MySQL used.

We use use same trick with "invalid" so that someone doesn't work out
what this hash this invalid string does correspond to (because
non-hex characters are accepted by the get_salt_from_password
password).

As a consequence of being a little more lax, we've also made it strict
that the 41 character has to begin with a '*'.

Author: Daniel Black
Author Date: 2020-08-30 00:53:20 UTC

MDEV-23630: mysqldump logically dump system table information

Add --system={all, users, plugins, udfs, servers, stats, timezones}

This will dump system information from the server in
a logical form like:
* CREATE USER
* GRANT
* SET DEFAULT ROLE
* CREATE ROLE
* CREATE SERVER
* INSTALL PLUGIN
* CREATE FUNCTION

"stats" is the innodb statistics tables or EITS and
these are dumped as INSERT/REPLACE INTO statements
without recreating the table.

"timezones" is the collection of timezone tables
which are important to transfer to generate identical
results on restoration.

Two other options have an effect on the SQL generated by
--system=all. These are mutually exclusive of each other.
* --replace
* --insert-ignore

--replace will include "OR REPLACE" into the logical form
like:
* CREATE OR REPLACE USER ...
* DROP ROLE IF EXISTS (MySQL-8.0+)
* CREATE OR REPLACE ROLE ...
* UNINSTALL PLUGIN IF EXISTS (10.4+) ... (before INSTALL PLUGIN)
* DROP FUNCTION IF EXISTS (MySQL-5.7+)
* CREATE OR REPLACE [AGGREGATE] FUNCTION
* CREATE OR REPLACE SERVER

--insert-ignore uses the construct " IF NOT EXISTS" where
supported in the logical syntax.

'CREATE OR REPLACE USER' includes protection against
being run as the same user that is importing the mysqldump.

Includes experimental support for dumping mysql-5.7/8.0
system tables and exporting logical SQL compatible with MySQL.

Updates mysqldump man page, including this information and
(removing obsolute bug reference)

Author: Sujatha Sivakumar
Author Date: 2020-04-09 15:15:45 UTC

MDEV-21117: --tc-heuristic-recover=rollback is not replication safe

Problem:
=======
When run after master server crash --tc-heuristic-recover=rollback produces
inconsistent server state with binlog still containing transactions that were
rolled back by the option. Such way recovered server may not be used for
replication.

Fix:
===
A new --tc-heuristic-recover=BINLOG_TRUNCATE value is added to the
server option. It works as ROLLBACK for engines but makes its best to
synchronize binlog to truncate it to the last committed transaction.

The BINLOG_TRUNCATE algorithm starts scanning possibly a sequences of
binlog files starting from the binlog checkpoint file to find a
truncate position candidate. The candidate truncate position
corresponds to the beginning of the first GTID event whose transaction
is not found to have been committed. There must be no committed
transactions beyond this position *but* non-transactional events are
possible. This is also checked to warn when the case.
Rollback and possibly completion of partial commits (in the multi-engine case)
is done regardless, as well as the binlog GTID state is adjusted
accordingly. Also a STOP binlog event is written at the end of
truncated binary log.

Author: varun
Author Date: 2020-10-23 12:21:59 UTC

WIP

Author: Sergei Golubchik
Author Date: 2020-08-01 11:12:50 UTC

MDEV-21201 No records produced in information_schema query, depending on projection

Reimplement MDEV-14275 Improving memory utilization for information schema

Postpone temp table instantiation until after setup_fields().

Replace all unused (not marked in read_set) columns in an I_S table
with CHAR(0). This can drastically reduce the footprint of a MEMORY
table (a TABLE_CATALOG alone is 1538 bytes per row).

This does not change the engine. If the table was decided to be Aria
(because of, say, blobs) then after optimization it'll stay Aria
even if all blobs were removed.

Note 1: when transforming table structure, share->blob_fields is
preserved, otherwise Aria might switch from DYNAMIC to STATIC row format
and expect a special field for a deleted mark, which create_tmp_tabe
didn't provide.

Note 2: optimizer was doing handler::info() (to know the number of rows)
before the temp table is populated. That didn't make much sense. Now
it's done before the table is even instantiated. Preserve the old
behavior and report 0 rows.

This reverts e2664ee8362 and a8458a2345e

Author: Alexey Botchkov
Author Date: 2020-10-23 08:20:17 UTC

MDEV-20744 SET GLOBAL `replicate_do_db` = DEFAULT causes crash.

DEFAULT for the replicate_do_db is the "" as our documentation states.

Author: Rucha Deodhar
Author Date: 2020-08-06 11:53:49 UTC

MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING

Anaylsis: When we have INSERT/REPLACE...RETURNING and have qualified asterisk,
table_name is not NULL and context->table_list is either NULL or has
incorrect reference because context->table_list has tables from the FROM clause.
context->table_list has incorrect reference (has table from the FROM clause
instead of table we are inserting into) for INSERT/REPLACE...SELECT...RETURNING
because we have a FROM clause from the SELECT statement.
For INSERT/REPLACE...RETURNING it is NULL because there is no FROM clause.
Fix: If table_name is not null, check if we have INSERT/REPLACE...RETURNING.
If so, the reference should be the table we are inserting into and not
the table in the FROM clause.

Author: Anel Husakovic
Author Date: 2020-08-31 14:36:32 UTC

MDEV-21786 mysqldump will forget sequence definition details on --no-data dump

- Original patch was contributed by Jani Tolonen <jani.k.tolonen@gmail.com>
https://github.com/an3l/server/commits/bb-10.3-anel-MDEV-21786-dump-sequence
which distinguishes data structure (linked list) of sequences from
tables.
- Added standard sql output to prevent future changes
of sequences and disabled locks for sequences.
- Added test case for `MDEV-20070: mysqldump won't work correct on
sequences` where table column depends on sequence value.
- Restore sequence last value in the following way:
  - Find `next_not_cached_value` and use it to `setval()`
  - We just need for logical restore, so don't execute `setval()`
  - `setval()` should be showed also in case of `--no-data` option.

Reviewed-by: daniel@mariadb.com

Author: Stepan Patryshev
Author Date: 2020-10-22 11:40:11 UTC

MDEV-23659 Update Galera disabled.def file

Author: Sujatha Sivakumar
Author Date: 2020-10-22 02:04:39 UTC

MDEV-10149: 10.5 changes

Author: Sujatha Sivakumar
Author Date: 2020-10-22 02:04:39 UTC

MDEV-10149: 10.4 changes

Author: Sujatha Sivakumar
Author Date: 2020-10-22 02:04:39 UTC

MDEV-10149: 10.3 changes

Author: Sujatha Sivakumar
Author Date: 2020-10-22 02:04:39 UTC

MDEV-10149: 10.2 changes

Author: Sujatha Sivakumar
Author Date: 2020-10-22 01:46:29 UTC

MDEV-10149: sys_vars.rpl_init_slave_func fails sporadically in buildbot

problem:
========
mysqltest: In included file "./include/assert.inc":
included from mysql-test/suite/sys_vars/t/rpl_init_slave_func.test at line 69:
Assertion text: '@@global.max_connections = @start_max_connections'
Assertion result: '0'

mysqltest: In included file "./include/assert.inc":
included from mysql-test/suite/sys_vars/t/rpl_init_slave_func.test at line 86:
Assertion text: '@@global.max_connections = @start_max_connections + 1'
Assertion result: '0'

Analysis:
=========
A slave SQL thread sets its Running state to Yes very early in its
initialisation, before the majority of initialisation actions, including
executing the init_slave command, are done. Thus the testcase has a race
condition where the initial replication setup might finish executing later
than the testcase SET GLOBAL init_slave, making the testcase see its effect
where it checks for its absence.

Fix:
===
Include 'sync_slave_sql_with_master.inc' at the beginning of the test to
ensure that slave applier has completed the execution of 'init_slave' command
and proceeded to event application. Replace the apparently needless RESET
MASTER / RESET SLAVE etc.

Patch is based on:
https://github.com/percona/percona-server/pull/1464/commits/b91e2e6f90611aa299c302929fb8b068e8ac0dee
Author: laurynas-biveinis

Author: Daniel Black
Author Date: 2020-10-22 00:48:18 UTC

getopt: make --help option for SETS readable

Author: Monty
Author Date: 2020-10-14 23:25:57 UTC

MDEV-23842 Atomic RENAME TABLE

- Major rewrite of ddl_log.cc and ddl_log.h
  - ddl_log.cc described in the beginning how the recovery works.
  - ddl_log.log has unique signature and is dynamic. It's easy to
    add more information to the header and other ddl blocks while still
    being able to execute old ddl entries.
  - IO_SIZE for ddl blocks is now dynamic. Can be changed without affecting
    recovery of old logs
  - Code is more modular and is now usable outside of partition handling.
- Added ddl_log_entry_phase[], number of phases for each DDL action,
  which allowed me to greatly simply set_global_from_ddl_log_entry()
- Changed how strings are stored in log entries, which allows us to
  store much more information in a log entry.
- ddl log is now always created at start and deleted on normal shutdown.
  This simplices things notable.
- Added probes debug_crash_here() and debug_simulate_error() to simply
  crash testing and allow crash after a given number of times a probe
  is executed. See comments in debug_sync.cc and rename_table.test for
  how this can be used.
- Reverting failed table and view renames is done trough the ddl log.
  This ensures that the ddl log is tested also outside of recovery.
- Added helper function 'handler::needs_lower_case_filenames()'
- Extend binary log with Q_XID events. ddl log handling is using this
  to check if a ddl log entry was logged to the binary log (if yes,
  it will be deleted from the log during ddl_log_close_binlogged_events()

mysqltest.cc changes:
- --die will now replace $variables with their values
- $error will contain the error of the last failed statement

storage engine changes:
- maria_rename() was changed to be more robust against crashes during
  rename.

Author: Kentoku SHIBA
Author Date: 2020-04-15 15:44:20 UTC

MDEV-20100 MariaDB 13.3.9 Crash "[ERROR] mysqld got signal 11 ;"

Some functions on ha_partition call functions on all partitions, but handler->reset() is only called that pruned by m_partitions_to_reset. So Spider didn't clear pointer on unpruned partitions, if the unpruned partitions are used by next query, Spider reference the pointer that is already freed.

Author: Roman Nozdrin
Author Date: 2020-10-21 19:12:54 UTC

Fixed compilation issues with debug build and install test issues for
Ubuntu Xenial

Author: Dmitry Shulga
Author Date: 2020-10-21 17:42:10 UTC

MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS

Compiler warnings like one listed below are generated during server build on MacOS:

[88%] Building C object plugin/auth_pam/CMakeFiles/pam_user_map.dir/mapper/pam_user_map.c.o
mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:87:41: error: passing
'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types
with different sign [-Werror,-Wpointer-sign]
if (getgrouplist(user, user_group_id, loc_groups, &ng) < 0)
   ^~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/unistd.h:650:43: note:
passing argument to parameter here
int getgrouplist(const char *, int, int *, int *);
    ^

In case MariaDB server is build with -DCMAKE_BUILD_TYPE=Debug it results in
build error.

The reason of compiler warnings is that declaration of the Posix C API function
getgrouplist() on MacOS differs from declaration of getgrouplist() proposed
by Posix.

To suppress this compiler warning cmake configure was adapted to detect what
kind of getgrouplist() function is declared on the build platform and
set the macros HAVE_POSIX_GETGROUPLIST in case the building platform supports
Posix compatible interface for the getgrouplist() function. Depending on
whether this macros is set the compatible type of arguments is used to pass
parameter values to the function.

Author: Eugene
Author Date: 2020-10-20 17:10:40 UTC

MDEV-23356 InnoDB: Failing assertion: field->col->mtype == type, crash or ASAN failures in row_sel_convert_mysql_key_to_innobase, InnoDB indexes are inconsistent after INDEX changes

innobase_rename_indexes_cache(): fix corruption of index cache. Index ids
help distinguish indexes when their names clash.

innobase_rename_indexes_cache(): fix corruption of index statistics table.
Use unique temporary names to avoid names clashing.

Reviewed by: Marko Mäkelä

Author: midenok
Author Date: 2020-10-21 17:04:19 UTC

x.test

Author: midenok
Author Date: 2018-10-31 12:51:50 UTC

MDEV-17573 Assertion in federatedx on multi-update

Cause: shared federatedx_io cannot store table-specific data.

Fix: move current row reference `federatedx_io_mysql::current` to
ha_federatedx.

FederatedX connection (represented by federatedx_io) is stored into
federatedx_txn::txn_list of per-server connections (see
federatedx_txn::acquire()). federatedx_txn object is stored into THD
(see ha_federatedx::external_lock()). When multiple handlers acquire
FederatedX connection they get single federatedx_io instance. Multiple
handlers do their operation via federatedx_io_mysql::mark_position()
and federatedx_io_mysql::fetch_row() in arbitrarty manner. They access
the same federatedx_io_mysql instance and same MYSQL_ROWS *current
pointer, so one handler disrupts the work of the other.

Related to "MDEV-14551 Can't find record in table on multi-table update
with ORDER BY".

Author: Jan Lindström
Author Date: 2020-10-21 08:39:24 UTC

MDEV-21514 : Galera test failure on galera.galera_wan_restart_sst

Replace unnecessary sleeps with real wait_conditions to make
sure correct cluster sizes.

Author: Kentoku SHIBA
Author Date: 2020-04-15 15:44:20 UTC

MDEV-20100 MariaDB 13.3.9 Crash "[ERROR] mysqld got signal 11 ;"

Some functions on ha_partition call functions on all partitions, but handler->reset() is only called that pruned by m_partitions_to_reset. So Spider didn't clear pointer on unpruned partitions, if the unpruned partitions are used by next query, Spider reference the pointer that is already freed.

Author: Monty
Author Date: 2020-10-19 19:04:45 UTC

MDEV-23648 s3.partition_move 'innodb' test failure - object deleted before copy attempted

Updated submodule libmarias3 which fixes the problem
Enabled disable S3 tests.

Author: Kentoku SHIBA
Author Date: 2020-04-15 15:44:20 UTC

MDEV-20100 MariaDB 13.3.9 Crash "[ERROR] mysqld got signal 11 ;"

Some functions on ha_partition call functions on all partitions, but handler->reset() is only called that pruned by m_partitions_to_reset. So Spider didn't clear pointer on unpruned partitions, if the unpruned partitions are used by next query, Spider reference the pointer that is already freed.

Author: Kentoku SHIBA
Author Date: 2020-04-15 15:44:20 UTC

MDEV-20100 MariaDB 13.3.9 Crash "[ERROR] mysqld got signal 11 ;"

Some functions on ha_partition call functions on all partitions, but handler->reset() is only called that pruned by m_partitions_to_reset. So Spider didn't clear pointer on unpruned partitions, if the unpruned partitions are used by next query, Spider reference the pointer that is already freed.

Author: Roman Nozdrin
Author Date: 2020-10-20 13:08:16 UTC

MCS engine ref update

Author: Daniele Sciascia
Author Date: 2020-10-15 07:24:01 UTC

MDEV-23081 Stray XA transactions at startup, with wsrep_on=OFF

Change xarecover_handlerton so that transaction with WSREP prefixed
xids are rolled back when Galera is disabled.

Author: Julius Goryavsky
Author Date: 2020-08-11 12:03:02 UTC

MDEV-21039: Server fails to start with unknown mysqld_safe options

Adding any unknown option to the "[mysqld_safe]" section makes
mysqld impossible to start with mysqld_multi. For example, after
adding the unknown option "numa_interleave" to the "[mysqld_safe]"
section, mysqld_multi exits with the following diagnostics:

[ERROR] /usr/local/mysql/bin/mysqld: unknown option '--numa_interleave'

To get rid of this behavior, this patch by default adds the "--loose-"
prefix to all unknown (for mysqld_safe) options. This behavior can be
enabled explicitly with the --ignore-unknown option and disabled with
the --no-ignore-unknown option.

Author: Julius Goryavsky
Author Date: 2020-08-11 12:03:02 UTC

MDEV-21039: Server fails to start with unknown mysqld_safe options

Adding any unknown option to the "[mysqld_safe]" section makes
mysqld impossible to start with mysqld_multi. For example, after
adding the unknown option "numa_interleave" to the "[mysqld_safe]"
section, mysqld_multi exits with the following diagnostics:

[ERROR] /usr/local/mysql/bin/mysqld: unknown option '--numa_interleave'

To get rid of this behavior, this patch by default adds the "--loose-"
prefix to all unknown (for mysqld_safe) options. This behavior can be
enabled explicitly with the --ignore-unknown option and disabled with
the --no-ignore-unknown option.

Author: Rucha Deodhar
Author Date: 2020-10-16 14:49:09 UTC

MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in
Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK)

Analysis: Assertion failure happens because we don't ignore kill query
even if 'OK' is already sent.
Fix: Check if we already sent 'OK'. If sent, ignore kill query.

Author: Julius Goryavsky
Author Date: 2020-03-24 13:55:07 UTC

MDEV-21951: mariabackup SST fail if data-directory have lost+found directory

To fix this, it is necessary to add an option to exclude the
database with the name "lost+found" from processing (the database
name will be checked by the check_if_skip_database_by_path() or
by the check_if_skip_database() function, and as a result
"lost+found" will be skipped).

In addition, it is necessary to slightly modify the verification
logic in the check_if_skip_database() function.

Also added a new test galera_sst_mariabackup_lost_found.test

Author: Anel Husakovic
Author Date: 2020-10-13 12:36:25 UTC

MDEV-22552: mytop packaging

- Adding socket check for MariaDB/Mysql driver
- Don't use `COMPONENT client` for RPM based on 0d05c9f8699d1ac5
Reviewed by: Sergei Golubchik

Author: varun
Author Date: 2020-10-16 06:13:56 UTC

MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison

Allow materialization strategy when collations on the
inner and outer sides of an IN subquery are the same and the
character set of the inner side is a proper subset of the character
set on the outer side.
This allows conversion from utf8mb3 to utf8mb4
as the former is a subset of the later.

Reviewed by Sergei Petrunia

Author: Roman Nozdrin
Author Date: 2020-10-16 15:41:54 UTC

Update the MCS engine ref

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-10-16 06:39:41 UTC

MDEV-23966 btr_search_sys->hash_tables accessed without taking proper ahi latch

In btr_search_update_hash_on_insert(), btr_search_sys->hash_tables is
being accessed without taking proper ahi latch. During buffer pool
resizing, btr_get_search_table() is being accessed and it leads to
segmentation fault. Taking ahi lock before btr_search_sys->hash_tables.

Author: Daniel Black
Author Date: 2019-09-26 22:48:27 UTC

MDEV-19508: signal handler SI_KERNEL not defined on Solaris/AIX

Solaris and AIX doesn't have a SI_KERNEL code for its siginfo code
structure. But they define sigwaitinfo. For these we can use
SI_USER and values <= aren't kernel.

SI_USER is defined on AIX (since AIX 6.1 at least) as:
/usr/include/sys/signal.h
    #define SI_USER 0 /* signal sent by another process with kill */

So its effectively not kernel.

Linux also uses SI_USER as its highest user code.

FreeBSD-12.1 available defines are:

  #if __POSIX_VISIBLE || __XSI_VISIBLE
  #define SI_NOINFO 0 /* No signal info besides si_signo. */
  #define SI_USER 0x10001 /* Signal sent by kill(). */
  #define SI_QUEUE 0x10002 /* Signal sent by the sigqueue(). */
  #define SI_TIMER 0x10003 /* Signal generated by expiration of */
                                          /* a timer set by timer_settime(). */
  #define SI_ASYNCIO 0x10004 /* Signal generated by completion of */
                                          /* an asynchronous I/O request.*/
  #define SI_MESGQ 0x10005 /* Signal generated by arrival of a */
                                          /* message on an empty message queue. */
  #define SI_KERNEL 0x10006
  #define SI_LWP 0x10007 /* Signal sent by thr_kill */
  #endif

Original commit 07e9b1389857 showed an explicit desire to catch user
signals aka kill(SI_USER). So for FreeBSD != SI_KERNEL is too broad and
should be <= SI_USER too.

ref: https://docs.oracle.com/cd/E36784_01/html/E36873/siginfo.h-3head.html
ref: https://github.com/omniosorg/illumos-omnios/blob/master/usr/src/uts/common/sys/siginfo.h#L130

Author: Sergey Vojtovich
Author Date: 2020-04-28 10:25:42 UTC

MDEV-19508 - Issue with: #define SI_KERNEL

Reverted 07e9b1389, use setsid() instead.

Conventional use of SIGHUP to dump daemon information or flush caches
assumes preceeding setsid() call.

Ideally we should not setup SIGHUP handler in bootstrap and (probably)
debug_gdb modes.

Additional information on SysV style and systemd style daemons:
https://www.freedesktop.org/software/systemd/man/daemon.html

Author: Sergey Petrunia
Author Date: 2020-10-09 14:48:22 UTC

MDEV-23938: innodb row_search_idx_cond_check handle ICP_ABORTED_BY_USER

- row_search_mvcc() should return DB_INTERRUPTED when it got killed.
- Add a syncpoint for the ICP check.
- Add test coverage for killed-during-ICP-check scenario

Backport of MDEV-22761 fixes for ICP from 10.4 commits:
* a6f956488c712bef3b13660584d1b905e0c676cc
* c03885cd9ceb1ede7f49a9e218022b401b3a1e28

XtraDB was fixed in deb3b9a17498

Reviewer: Daniel Black

Author: Julius Goryavsky
Author Date: 2020-10-13 11:37:33 UTC

MDEV-23659: Update Galera disabled.def file

This fix removes from the list of disabled tests all
tests that were fixed by the patch for MDEV-21770.

Author: Daniel Black
Author Date: 2020-10-09 21:28:53 UTC

MDEV-22761: innodb row_search_idx_cond_check handle CHECK_ABORTED_BY_USER

Part #3:
Two more cases within row_search_mvcc need to handle the CHECK_ABORTED_BY_USER
and process this as a DB_INTERRUPTED.

Author: Nikita Malyavin
Author Date: 2019-10-09 06:48:50 UTC

MDEV-19130 Assertion failed in handler::update_auto_increment

add store/restore_auto_increment in period portion insert/update functions

Author: Marko Mäkelä
Author Date: 2020-10-14 06:21:10 UTC

MDEV-23955 main.multi_update_big times out

Copying generated data from the sequence engine should be faster
than copying from MyISAM to MyISAM. Let us hope that this fixes
the test failure on Travis CI ARMv8.

Author: Igor Babaev
Author Date: 2020-10-13 19:25:09 UTC

MDEV-18511: CTE support for UPDATE and DELETE statements
MDEV-23552: Merge mergeable derived tables used at the top level of UPDATE
            statements

A complement for the previous patch that allows to run successfully all
tests from
  derived_update.test, cte_update.test, view_update.test,
  derived_update_multi.test, cte_update_multi.test, view_update_multi.test
and from
  view.test, derived.test, derived_view.test,
  cte_nonrecursive, cte recursive
with --ps-protocol.

Some tests from view.test and derived_view were changed though. Later this
will be fixed.

Author: Julius Goryavsky
Author Date: 2020-10-13 11:37:33 UTC

MDEV-23659: Update Galera disabled.def file

This fix removes from the list of disabled tests all
tests that were fixed by the patch for MDEV-21770.

Author: Anel Husakovic
Author Date: 2020-10-06 15:43:13 UTC

MDEV-16676 Using malloc-lib=jemalloc in MariaDB 10.2 causes non-critical error about missing mysql_config on startup

- This commit is based on patch 84fe9720a4d2483ff67b6a and suggestion of
Jean Weisbuch to use `ldconfig -p`
- Format of ldconfig output:
"libjemalloc.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libjemalloc.so.1"

```
$ ./scripts/mysqld_safe
201013 13:36:50 mysqld_safe Adding '/usr/lib/x86_64-linux-gnu/libjemalloc.so.1' to LD_PRELOAD for mysqld
201013 13:36:50 mysqld_safe Logging to '/home/anel/builds/data-10.5/mysqld_safe_anel.err'.
201013 13:36:50 mysqld_safe Starting mysqld daemon with databases from /home/anel/builds/data-10.5
```
Reviewed by: Daniel Black, Faustin Lammler

Author: Julius Goryavsky
Author Date: 2020-10-13 11:37:33 UTC

MDEV-23659: Update Galera disabled.def file

This fix removes from the list of disabled tests all
tests that were fixed by the patch for MDEV-21770.

Author: midenok
Author Date: 2020-09-21 13:06:13 UTC

MDEV-21652 Foreign key migration from old version

== fk_check_legacy_storage(), fk_upgrade_legacy_storage() ==

fk_check_legacy_storage() checks whether upgrade is required for the
given table name by looking SYS_FOREIGN table for corresponding record
existence.

fk_upgrade_legacy_storage() does the upgrade routine which includes
getting the foreign keys from SYS_FOREIGN[_COLS], updating the
foreign/referenced shares as well as their FRM files, deleting the
records from SYS_FOREIGN[_COLS] tables.

Both routines utilize the internal SQL for SYS_FOREIGN[_COLS]
processing.

== Upgrade foreign keys via backoff action ==

When table is opened fk_check_legacy_storage() detects whether upgrade
is required and HA_ERR_FK_UPGRADE is returned to SQL layer which then
handles this error by backoff action from Open_table_context where
table is opened again with HA_OPEN_FOR_REPAIR flag which indicates
that fk_upgrade_legacy_storage() is required. After
fk_upgrade_legacy_storage() is done fk_check_legacy_storage() is
checked again to ensure that SYS_FOREIGN[_COLS] are empty for the
given table.

== Check foreign/referenced indexes existence ==

fk_upgrade_legacy_storage() via fk_upgrade_push_fk() fails if there
are no indexes in foreign/referenced tables for the given data
acquired from SYS_FOREIGN[_COLS].

== Internal SQL: select into both func and vars extension ==

fk_upgrade_legacy_storage() utilizes syntax extension in internal SQL:

  FETCH c INTO fk_upgrade_create_fk() fk_id, unused;

Thus the data is fetched into both fk_upgrade_create_fk() function and
fk_id variable.

== Rename table, rename column, drop table, drop column handling ==

When foreign table is opened it is automatically upgraded by backoff
action. But if the referenced table is altered or dropped first there
is no chance for the foreign table to get the correct data. So the
SYS_FOREIGN_[COLS] must be kept in sync with the above DDL operations
in respect of referenced names. DROP TABLE for the referenced table is
disabled as usual. DROP TABLE, DROP COLUMN relied in 10.5 on
dict_foreign_t cached data for the referenced tables. Now there is no
such possibility for the legacy data so we have to look at
SYS_FOREIGN_[COLS] directly.

== Reverted some SYS_FOREIGN(_COLS) routines ==

Rename table and rename column handling was done in sync with
SYS_FOREIGN_[COLS] in 10.5. To retain the above DDL consistency for
the referenced tables we still use that old synchronization code.

== ALGORITHM=COPY handling ==

Since we cannot faingrain ALGORITHM=COPY in innobase handler it is
disabled for the referenced tables unless the foreign tables are upgraded.

The check is done in create_table_info_t::create_table() and is
equivalent to DROP TABLE check as we are actually dropping the old
table after the copy routine is done.

== WITH_INNODB_LEGACY_FOREIGN_STORAGE macro ==

Every SYS_FOREIGN_[COLS] handling is wrapped inside
WITH_INNODB_LEGACY_FOREIGN_STORAGE compilation macro. When this macro
is disabled the foreign key upgrade is not possible. Future versions
will obsolete the upgrade completely.

== innodb_eval_sql debug interface ==

Test cases must fill SYS_FOREIGN_[COLS] with data. This is done with
setting the new innodb_eval_sql debug variable. The value of that
variable is processed by que_eval_sql().

== Some syntax error-friendly parser handling ==

que_eval_sql() was unfriendly on syntax errors: it just failed with
SIGABRT exception. To keep the server alive some frequent syntax
errors are now returned as DB_ERROR from que_eval_sql().

Author: midenok
Author Date: 2020-09-21 13:06:13 UTC

MDEV-21652 Foreign key migration from old version

== fk_check_legacy_storage(), fk_upgrade_legacy_storage() ==

fk_check_legacy_storage() checks whether upgrade is required for the
given table name by looking SYS_FOREIGN table for corresponding record
existence.

fk_upgrade_legacy_storage() does the upgrade routine which includes
getting the foreign keys from SYS_FOREIGN[_COLS], updating the
foreign/referenced shares as well as their FRM files, deleting the
records from SYS_FOREIGN[_COLS] tables.

Both routines utilize the internal SQL for SYS_FOREIGN[_COLS]
processing.

== Upgrade foreign keys via backoff action ==

When table is opened fk_check_legacy_storage() detects whether upgrade
is required and HA_ERR_FK_UPGRADE is returned to SQL layer which then
handles this error by backoff action from Open_table_context where
table is opened again with HA_OPEN_FOR_REPAIR flag which indicates
that fk_upgrade_legacy_storage() is required. After
fk_upgrade_legacy_storage() is done fk_check_legacy_storage() is
checked again to ensure that SYS_FOREIGN[_COLS] are empty for the
given table.

== Check foreign/referenced indexes existence ==

fk_upgrade_legacy_storage() via fk_upgrade_push_fk() fails if there
are no indexes in foreign/referenced tables for the given data
acquired from SYS_FOREIGN[_COLS].

== Internal SQL: select into both func and vars extension ==

fk_upgrade_legacy_storage() utilizes syntax extension in internal SQL:

  FETCH c INTO fk_upgrade_create_fk() fk_id, unused;

Thus the data is fetched into both fk_upgrade_create_fk() function and
fk_id variable.

== Rename table, rename column, drop table, drop column handling ==

When foreign table is opened it is automatically upgraded by backoff
action. But if the referenced table is altered or dropped first there
is no chance for the foreign table to get the correct data. So the
SYS_FOREIGN_[COLS] must be kept in sync with the above DDL operations
in respect of referenced names. DROP TABLE for the referenced table is
disabled as usual. DROP TABLE, DROP COLUMN relied in 10.5 on
dict_foreign_t cached data for the referenced tables. Now there is no
such possibility for the legacy data so we have to look at
SYS_FOREIGN_[COLS] directly.

== Reverted some SYS_FOREIGN(_COLS) routines ==

Rename table and rename column handling was done in sync with
SYS_FOREIGN_[COLS] in 10.5. To retain the above DDL consistency for
the referenced tables we still use that old synchronization code.

== ALGORITHM=COPY handling ==

Since we cannot faingrain ALGORITHM=COPY in innobase handler it is
disabled for the referenced tables unless the foreign tables are upgraded.

The check is done in create_table_info_t::create_table() and is
equivalent to DROP TABLE check as we are actually dropping the old
table after the copy routine is done.

== WITH_INNODB_LEGACY_FOREIGN_STORAGE macro ==

Every SYS_FOREIGN_[COLS] handling is wrapped inside
WITH_INNODB_LEGACY_FOREIGN_STORAGE compilation macro. When this macro
is disabled the foreign key upgrade is not possible. Future versions
will obsolete the upgrade completely.

== innodb_eval_sql debug interface ==

Test cases must fill SYS_FOREIGN_[COLS] with data. This is done with
setting the new innodb_eval_sql debug variable. The value of that
variable is processed by que_eval_sql().

== Some syntax error-friendly parser handling ==

que_eval_sql() was unfriendly on syntax errors: it just failed with
SIGABRT exception. To keep the server alive some frequent syntax
errors are now returned as DB_ERROR from que_eval_sql().

Author: Vladislav Vaintroub
Author Date: 2020-09-22 16:12:58 UTC

MDEV-19237 Do not resend unchanged result set metadata for prepared statements