Author: varun
Author Date: 2020-05-26 12:27:07 UTC

MDEV-17606: Query returns wrong results (while using CHARACTER SET utf8)

The issue here was that the left expr and right expr of the ANY subquery
had different character sets, so we were converting the left expr to utf8 character set.
So when this conversion was happening we were actually converting the item inside the cache,
it looked like <cache>(convert(t1.l1 using utf8)), which is incorrect.
To fix this problem we are going to store the reference of the left expr and convert that
to utf8 character set, it would look like convert(<cache>(`test`.`t1`.`l1`) using utf8)

Author: Sergei Golubchik
Author Date: 2020-05-26 12:11:41 UTC

update C/C

Author: Julius Goryavsky
Author Date: 2020-05-26 12:02:48 UTC

Merge branch 'codership-10.4-MDEV-22666-v2' into bb-10.4-MDEV-22666

Author: Julius Goryavsky
Author Date: 2020-05-26 12:01:13 UTC

Forgotten include files were added to check the necessary conditions for running the test

Author: midenok
Author Date: 2018-09-10 08:48:12 UTC

MDEV-16937 Strict SQL with system versioned tables causes issues

Respect system fields in NO_ZERO_DATE mode.

This is the subject for refactoring in MDEV-19597

Author: midenok
Author Date: 2020-05-26 10:14:47 UTC

MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field

update_virtual_field() is called as part of index rebuild in
ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish.

Assertion in update_virtual_field() was put as part of MDEV-16222
because update_virtual_field() returns in_use->is_error(). The idea:
wrongly mixed semantics of error status before update_virtual_field()
and the status returned by update_virtual_field(). The former can
falsely influence the latter.

Author: Marko Mäkelä
Author Date: 2020-05-26 09:44:05 UTC

Merge 5.5 into 10.1

Author: Marko Mäkelä
Author Date: 2020-05-26 09:23:20 UTC

MDEV-22513 main.processlist_notembedded Timeout in wait_until_count_sessions

Do not blindly disconnect the connection that is in WAIT_FOR
because it could happen that neither the disconnect nor the
SIGNAL would be processed before RESET would discard the signal.

Author: Marko Mäkelä
Author Date: 2020-05-26 08:54:55 UTC

Merge 10.3 into 10.4

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-05-25 16:12:26 UTC

MDEV-22637 Rollback of insert fails when column reorder happens

- During column reorder table rebuild, rollback of insert fails.
Reason is that InnoDB tries to fetch the column position from
new clustered index and it exceeds default column value tuple fields.
So InnoDB should use the table column position while searching for
defaults column value.

Author: Eugene Kosov
Author Date: 2020-05-25 19:41:59 UTC

MDEV-8069 DROP or rebuild of a large table may lock up InnoDB

Now all data file removal in InnoDB happens inside tpool.
tpool task iterates all files in data directry and removes trash.
Yes, in will iterate a lot of files if a lot of tables/databases
exist. No, code is not optimized to handle 300K DELETE TABLE RPS.

.trash: new InnoDB common file extension

os_unique_file_name(): creates unique file name

os_file_exists(): convenience function

file_remove_task: tpool task which removes data files

file_remove_callback(): traverses all elements in MariaDB data path
and removes all .trash files

fil_delete_tablespace(): not removes but renames file to unique name
and asks tpool to actually remove it.

fil_delete_file(): not removes but renames file to unique name and asks
tpool to actually remove it.

srv_start(): asks tpool to remove all .trash files that could left from
previous server run

Author: Andrei Elkin
Author Date: 2020-05-05 17:32:32 UTC

MDEV-15152 Optimistic parallel slave doesnt cope well with START SLAVE UNTIL

The immediate bug was caused by a failure to recognize a correct
position to stop the slave applier run in optimistic parallel mode.
There were the following set of issues that the analysis unveil.
1 incorrect estimate for the event binlog position passed to
  is_until_satisfied
2 wait for workers to complete by the driver thread did not account non-group events
  that could be left unprocessed and thus to mix up the last executed
  binlog group's file and position:
  the file remained old and the position related to the new rotated file
3 incorrect 'slave reached file:pos' by the parallel slave report in the error log
4 relay log UNTIL missed out the parallel slave branch in
  is_until_satisfied.

The patch addresses all of them to simplify logics of log change
notification in either the master and relay-log until case.
P.1 is addressed with passing the event into is_until_satisfied()
for proper analisis by the function.
P.2 is fixed by changes in handle_queued_pos_update().
P.4 required removing relay-log change notification by workers.
Instead the driver thread updates the notion of the current relay-log
fully itself with aid of introduced
bool Relay_log_info::until_relay_log_names_defer.

An extra print out of the requested until file:pos is arranged
with --log-warning=3.

Author: Andrei Elkin
Author Date: 2020-05-05 17:32:32 UTC

MDEV-15152 Optimistic parallel slave doesnt cope well with START SLAVE UNTIL

The immediate bug was caused by a failure to recognize a correct
position to stop the slave applier run in optimistic parallel mode.
There were the following set of issues that the analysis unveil.
1 incorrect estimate for the event binlog position passed to
  is_until_satisfied
2 wait for workers to complete by the driver thread did not account non-group events
  that could be left unprocessed and thus to mix up the last executed
  binlog group's file and position:
  the file remained old and the position related to the new rotated file
3 incorrect 'slave reached file:pos' by the parallel slave report in the error log
4 relay log UNTIL missed out the parallel slave branch in
  is_until_satisfied.

The patch addresses all of them to simplify logics of log change
notification in either the master and relay-log until case.
P.1 is addressed with passing the event into is_until_satisfied()
for proper analisis by the function.
P.2 is fixed by changes in handle_queued_pos_update().
P.4 required removing relay-log change notification by workers.
Instead the driver thread updates the notion of the current relay-log
fully itself with aid of introduced
bool Relay_log_info::until_relay_log_names_defer.

An extra print out of the requested until file:pos is arranged
with --log-warning=3.

Author: Andrei Elkin
Author Date: 2020-05-05 17:32:32 UTC

MDEV-15152 Optimistic parallel slave doesnt cope well with START SLAVE UNTIL

The immediate bug was caused by a failure to recognize a correct
position to stop the slave applier run in optimistic parallel mode.
There were the following set of issues that the analysis unveil.
1 incorrect estimate for the event binlog position passed to
  is_until_satisfied
2 wait for workers to complete by the driver thread did not account non-group events
  that could be left unprocessed and thus to mix up the last executed
  binlog group's file and position:
  the file remained old and the position related to the new rotated file
3 incorrect 'slave reached file:pos' by the parallel slave report in the error log
4 relay log UNTIL missed out the parallel slave branch in
  is_until_satisfied.

The patch addresses all of them to simplify logics of log change
notification in either the master and relay-log until case.
P.1 is addressed with passing the event into is_until_satisfied()
for proper analisis by the function.
P.2 is fixed by changes in handle_queued_pos_update().
P.4 required removing relay-log change notification by workers.
Instead the driver thread updates the notion of the current relay-log
fully itself with aid of introduced
bool Relay_log_info::until_relay_log_names_defer.

An extra print out of the requested until file:pos is arranged
with --log-warning=3.

Author: Sergey Vojtovich
Author Date: 2020-05-25 18:25:03 UTC

MDEV-22697 - InnoDB: remove trx->no

trx->no is duplicated by trx->rw_trx_hash_element->no for no good reason.
The latter is preferred for performance reasons: allows to avoid taking
trx->rw_trx_hash_element->mutex when snapshotting.

Author: midenok
Author Date: 2020-05-25 14:51:51 UTC

MDEV-10962 Deadlock with 3 concurrent DELETEs by unique key

Deadlock on concurrent acquisition of multiple types is fixed by
prioritization over waiting locks if transaction already has
non-waiting lock on same (block, heap_no).

Deadlock happens by this scheme:

1. TXA acquires L1 (index "a", X|NOT_GAP);
2. TXB waits L2 (index "a", X|NOT_GAP) for L1;
3. TXA waits L3 (index "a", X) for L2.

TXB waits TXA, TXA waits TXB: DeadlockChecker wants a victim.

Since L3 is weaker than L1 we can not apply optimization from
MDEV-18706. But the problem of this loop is silly queuing. TXA already
acquired non-gap lock L1 and wants to expand it into next-key. TXB from
parallel thread does the same. Deadlock detection processes the
requests in FIFO order so the above order fails. But if L3 is acquired
before L2 there is no deadlock.

We can make queue smarter: if there are already lock seized on same
resource (block, heap_no), no matter what type it is, we priritize all
further locks on that transaction. There is no treat to fair
scheduling as this is basically lock promotion in 2PL scheme.

Author: Sachin Setiya
Author Date: 2020-05-25 13:51:56 UTC

MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails

If Blob(or parent) field read_set is not set then Do not calc hash in
that case.

Author: Marko Mäkelä
Author Date: 2020-05-25 16:41:58 UTC

Merge 10.2 into 10.3

Author: Oleksandr Byelkin
Author Date: 2019-05-22 12:59:00 UTC

MDEV-5924: MariaDB could crash after changing the query_cache size

The real problem was that attempt to roll back cahnes after end of memory in QC was made incorrectly and lead to using uninitialized memory.
(bug has nothing to do with resize operation, it is just lack of resources erro processed incorrectly)

Author: Vladislav Vaintroub
Author Date: 2020-05-23 11:33:26 UTC

allow thread_pool_size_basic on Windows, in generic threadpool mode

Author: Sachin Setiya
Author Date: 2020-05-22 17:14:37 UTC

MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at /data/src/10.4-bug/sql/rpl_parallel.cc, line 470 upon shutdown during FTWRL

Problem:- When we issue FTWRL with shutdown in parallel, there is race between
FTWRL and shutdown. Shutdown might destroy the mutex (pool->LOCK_rpl_thread_pool)
before FTWRL can lock it. So we can get crash on FTWRL thread

Solution:- mysql_mutex_destroy(pool->LOCK_rpl_thread_pool) should wait for
FTWRL thread to complete its work , and then destroy.
So slave_prepare_for_shutdown will just deactivate the pool, and mutex is destroyed
later in end_slave()

Author: varun
Author Date: 2020-05-20 13:29:52 UTC

MDEV-22360: Sufficient conditions for accurate calculation of join cardinality

WIP

Author: Sachin Setiya
Author Date: 2020-04-17 11:02:51 UTC

MDEV-22179 rr(record and replay) support for mtr

This feature adds the support for rr in mtr. These 2 options are added
--rr run the mysqld in rr record mode
--rr= run the rr with custom record option, for multiple
            options use --rr= for each option.
            For example
              ./mtr main.view --rr=-h --rr=-u --rr=-c=23
--boot-rr run the mysqld performing bootstrap in rr record mode

Recording are stored in mysql-test/var/rr folder.
To run recording please run
rr replay var/rr/mysql-X

Limitations
Restart will create a new recording.
Repeat will work on same recording , So might be harder to debug.
If test create the multiple instance of mariadb all will be stored in var/rr

Author: Andrei Elkin
Author Date: 2020-05-24 13:52:02 UTC

Affected by binlog first redoing tests

Author: Monty
Author Date: 2020-05-23 11:36:33 UTC

Fixed compiler failure on windows

Author: Monty
Author Date: 2020-05-23 09:27:05 UTC

Fixed failure in sysvars_server_embedded,32bit

Author: varun
Author Date: 2020-05-22 15:45:17 UTC

WIP for printing ranges for non-indexed columns

Author: Roman Nozdrin
Author Date: 2020-05-22 17:59:44 UTC

Clean up for debian packaging and updates for systemd service units.

Author: Marko Mäkelä
Author Date: 2020-05-22 17:44:09 UTC

Merge 10.5

FIXME: Race conditions on buf_fix_count! Maybe we need buf_block.mutex
to protect it?

Failing tests:
encryption.corrupted_during_recovery
innodb.innodb_bulk_create_index
innodb.innodb_wl6326_big

Author: Marko Mäkelä
Author Date: 2020-05-22 15:36:01 UTC

MDEV-22669 InnoDB lacks CRC-32C acceleration on IA-32

In mysql/mysql-server@17e497bdb793bc6b8360aa1c626dcd8bb5cfad1b
MySQL 5.6.3 introduced innodb_checksum_algorithm=crc32 and
implemented it for AMD64 using the SSE 4.2 instructions
(incorrectly advertised as "SSE2" in a startup message).
It was not implemented on IA-32 or on Windows.

Since MariaDB 10.2.11 commit 2401d14e6b4449dd6181f728e160eb7c8f23bd40
we make use of the SSE4.2 CRC-32C instructions on Windows on both IA-32
and AMD64.

Let us be consistent and implement CRC-32C for IA-32 on all
available platforms. GCC 4.8.2 and GCC 4.8.5 complain
"error: PIC register clobbered by 'ebx' in 'asm'"
so we will only enable this code for IA-32 starting with GCC 5.

Also, we will clean up the implementation further after
commit 1312b4ebb652bf5eb9e97292ca62f0de3b0ff243.

has_sse4_2(): Replaces ut_cpuid().

ut_crc32c_8(): Replaces ut_crc32_8_hw().

ut_crc32c_64(): Replaces ut_crc32_64_low_hw(), ut_crc32_64_hw().

ut_crc32_hw(): Rewrite.

ut_crc32c_8_sw(): Replaces ut_crc32_8_sw().

ut_crc32c_64_sw(): Replaces ut_crc32_64_low_sw(), ut_crc32_64_sw().

ut_crc32_sw(): Rewrite. Avoid code bloat and do not unroll the
ut_crc32c_64_sw() loop, because no benefit has been demonstrated.

ut_crc32_init(): Only invoke ut_crc32_slice8_table_init()
if no acceleration is available.

Author: Sachin Setiya
Author Date: 2020-03-10 15:56:14 UTC

MDEV-11675 Lag Free Alter On Slave

Desc:- This will split Alter into 2 different commits. START ALTER and COMMIT
/ROLLBACK ALTER , Start Alter will be written in binlog as soon as we get the
locks for the table, alter will proceeds as usual and at the time of writing
binlog if alter is successful we will write COMMIT Alter other wise ROLLBACK
Alter.

Usage:- Using this feature is quite simple.
  1. On master you have to turn on `BINLOG_SPLIT_ALTER` dynamic variable.
  2. Slave must be using parallel replication.

So alter is divided like this.
1. ALTER TABLE ... EXECUTE = UNTIL COMMIT thread_id (Internally called Start Alter)
(Query_log_event with FL_START_ALTER_E1 flag)
2. ALTER TABLE ... EXECUTE = COMMIT thread_id / (Internally called Commit Alter)
(Query_log_event with FL_COMMIT_ALTER_E1 flag)
2. ALTER TABLE ... EXECUTE = ROLLBACK thread_id (Internally called Rollback Alter)
(Query_log_event with FL_ROLLBACK_ALTER_E1 flag)

identifier is thread_id.

BINLOG_SPLIT_ALTER :- new global/session variable on the master to allow splitting of alter
in binlog.

Architecture:-

Working at Slave Side
There are 3 main functions called from Sql_cmd_alter_table::execute
process_start_alter :- If normal cases when |W| > |SA| it will simple return
                       START_ALTER_PARSE, but when reserved_start_alter_thread is true
                       we simple binlog and return START_ALTER_SKIP.
process_commit_alter :- We will iterate into mi->start_alter_list and set the
                        start_alter_info->state = COMMIT_ALTER, And wait till it
                        is committed Start alter worker
process_rollback_alter :- We will iterate into mi->start_alter_list and set the
                        start_alter_info->state = ROLLBACK_ALTER, And wait till it
                        is committed Start alter worker

mysql_inplace_alter_table.
Master Side:-
There is not much change into master side.
So after getting exclusive metadata lock on table SA(START ALTER) is written on binlog
ALTER TABLE ... EXECUTE = UNTIL COMMIT thread_id
And at the time of write_bin_log we write
ALTER TABLE ... EXECUTE = COMMIT thread_id
If there is error we write
ALTER TABLE ... EXECUTE = ROLLBACK thread_id
So alter statement will be divided into 2 , hence 2 gtid.

Slave Side:-
We will add a instance of start_alter_info into mi->start_alter_list and wait for
its state to changed from ::REGISTERED to either COMMIT_ALTER or ROLLBACK_ALTER.
And after writing the binlog we change the state to COMMITTED

Scheduling for split alter on Slave
START ALTER
rpl_parallel_thread has this new parameter current_start_alter_id its value will be
zero in the case of Non start alter events or commit/rollback alter.
So when ::do_event() receives a FL_START_ALTER_E1 we will call choose thread normally
But we will avoid the already assigned rpl_threads[x] which have non zero
current_start_alter_id. get_split_alter_thread_id_part_1 takes care of this part.
But we cant assign the current_start_alter_id yet because Gtid_log_event does not have it.
So when the next Query_log_event appears we will assign the thread_id to
rpt->current_start_alter_id.

COMMIT/ROLLBACK Alter :- Processing this event is similar to normal event processing, apart
from that we will also reset current_start_alter_id of corresponding SA.

At any point of time at max we will have Number of workers - 1 Start Alters, If we get more
concurrent alter then we set reserved_start_alter_thread to 1. So that process_start_alter
just binary log it and no processing.

For keeping |W| > |SA| we have counter in rpl_parallel_thread_pool , current_start_alters
This should be inc/dec depending on current running SAs

Shutdown cases:-

Master:- If master shutdown in between SA and CA ,and tries the alter again we
         can have binlog like SA(t1) Crash SA(t1) CA(t1). To handle this slave
         calls check_and_remove_duplicate_alter and if it finds same SA then it
         sends ROLLBACK signal to old SA.
Slave:- In the case of Stop slave issued in between processing of
         SA SA {stop slave} CA CA, There are 2 changes 1. We wont wait for SA
         workers in wait_for_done, 2. After wait_for_done is completed we send the
         rollback signal to loner SAs

mdev-11675

Desc:- This will split Alter into 2 different commits. START ALTER and COMMIT
/ROLLBACK ALTER , Start Alter will be written in binlog as soon as we get the
locks for the table, alter will proceeds as usual and at the time of writing
binlog if alter is successful we will write COMMIT Alter other wise ROLLBACK
Alter.

Usage:- Using this feature is quite simple.
  1. On master you have to turn on `BINLOG_SPLIT_ALTER` dynamic variable.
  2. Slave must be using parallel replication.

So alter is divided like this.
1. ALTER TABLE ... EXECUTE = UNTIL COMMIT thread_id (Internally called Start Alter)
(Query_log_event with FL_START_ALTER_E1 flag)
2. ALTER TABLE ... EXECUTE = COMMIT thread_id / (Internally called Commit Alter)
(Query_log_event with FL_COMMIT_ALTER_E1 flag)
2. ALTER TABLE ... EXECUTE = ROLLBACK thread_id (Internally called Rollback Alter)
(Query_log_event with FL_ROLLBACK_ALTER_E1 flag)

identifier is thread_id.

BINLOG_SPLIT_ALTER :- new global/session variable on the master to allow splitting of alter
in binlog.

Architecture:-

Working at Slave Side
There are 3 main functions called from Sql_cmd_alter_table::execute
process_start_alter :- If normal cases when |W| > |SA| it will simple return
                       START_ALTER_PARSE, but when reserved_start_alter_thread is true
                       we simple binlog and return START_ALTER_SKIP.
process_commit_alter :- We will iterate into mi->start_alter_list and set the
                        start_alter_info->state = COMMIT_ALTER, And wait till it
                        is committed Start alter worker
process_rollback_alter :- We will iterate into mi->start_alter_list and set the
                        start_alter_info->state = ROLLBACK_ALTER, And wait till it
                        is committed Start alter worker

mysql_inplace_alter_table.
Master Side:-
There is not much change into master side.
So after getting exclusive metadata lock on table SA(START ALTER) is written on binlog
ALTER TABLE ... EXECUTE = UNTIL COMMIT thread_id
And at the time of write_bin_log we write
ALTER TABLE ... EXECUTE = COMMIT thread_id
If there is error we write
ALTER TABLE ... EXECUTE = ROLLBACK thread_id
So alter statement will be divided into 2 , hence 2 gtid.

Slave Side:-
We will add a instance of start_alter_info into mi->start_alter_list and wait for
its state to changed from ::REGISTERED to either COMMIT_ALTER or ROLLBACK_ALTER.
And after writing the binlog we change the state to COMMITTED

Scheduling for split alter on Slave
START ALTER
rpl_parallel_thread has this new parameter current_start_alter_id its value will be
zero in the case of Non start alter events or commit/rollback alter.
So when ::do_event() receives a FL_START_ALTER_E1 we will call choose thread normally
But we will avoid the already assigned rpl_threads[x] which have non zero
current_start_alter_id. get_split_alter_thread_id_part_1 takes care of this part.
But we cant assign the current_start_alter_id yet because Gtid_log_event does not have it.
So when the next Query_log_event appears we will assign the thread_id to
rpt->current_start_alter_id.

COMMIT/ROLLBACK Alter :- Processing this event is similar to normal event processing, apart
from that we will also reset current_start_alter_id of corresponding SA.

At any point of time at max we will have Number of workers - 1 Start Alters, If we get more
concurrent alter then we set reserved_start_alter_thread to 1. So that process_start_alter
just binary log it and no processing.

For keeping |W| > |SA| we have counter in rpl_parallel_thread_pool , current_start_alters
This should be inc/dec depending on current running SAs

Shutdown cases:-

Master:- If master shutdown in between SA and CA ,and tries the alter again we
         can have binlog like SA(t1) Crash SA(t1) CA(t1). To handle this slave
         calls check_and_remove_duplicate_alter and if it finds same SA then it
         sends ROLLBACK signal to old SA.
Slave:- In the case of Stop slave issued in between processing of
         SA SA {stop slave} CA CA, There are 2 changes 1. We wont wait for SA
         workers in wait_for_done, 2. After wait_for_done is completed we send the
         rollback signal to loner SAs

Author: Oleksandr Byelkin
Author Date: 2020-05-20 11:34:51 UTC

MDEV-22545: my_vsnprintf behaves not as in C standard

Added parameter %T for string which should be visibly truncated.

Author: Oleksandr Byelkin
Author Date: 2020-05-20 11:34:51 UTC

MDEV-22545: my_vsnprintf behaves not as in C standard

Added parameter %T for string which should be visibly truncated.

Author: Alexander Barkov
Author Date: 2020-05-21 14:16:27 UTC

MDEV-22111 ERROR 1064 & 1033 and SIGSEGV on CREATE TABLE w/ various charsets on 10.4/5 optimized builds | Assertion `(uint) (table_check_constraints - share->check_constraints) == (uint) (share->table_check_constraints - share->field_check_constraints)' failed

The code incorrectly assumed in multiple places that TYPELIB
values cannot have 0x00 bytes inside. In fact they can:

  CREATE TABLE t1 (a ENUM(0x61, 0x0062) CHARACTER SET BINARY);

Note, the TYPELIB value encoding used in FRM is ambiguous about 0x00.

So this fix is partial.

It fixes 0x00 bytes in many (but not all) places:

- In the middle or in the end of a value:
    CREATE TABLE t1 (a ENUM(0x6100) ...);
    CREATE TABLE t1 (a ENUM(0x610062) ...);

- In the beginning of the first value:
    CREATE TABLE t1 (a ENUM(0x0061));
    CREATE TABLE t1 (a ENUM(0x0061), b ENUM('b'));

- In the beginning of the second (and following) value of the *last* ENUM/SET
  in the table:

    CREATE TABLE t1 (a ENUM('a',0x0061));
    CREATE TABLE t1 (a ENUM('a'), b ENUM('b',0x0061));

However, it does not fix 0x00 when:

- 0x00 byte is in the beginning of a value of a non-last ENUM/SET
  causes an error:

   CREATE TABLE t1 (a ENUM('a',0x0061), b ENUM('b'));
   ERROR 1033 (HY000): Incorrect information in file: './test/t1.frm'

  This is an ambuguous case and will be fixed separately.
  We need a new TYPELIB encoding to fix this.

Details:

- unireg.cc

  The function pack_header() incorrectly used strlen() to detect
  a TYPELIB value length. Adding a new function typelib_values_packed_length()
  which uses TYPELIB::type_lengths[n] to detect the n-th value length,
  and reusing the new function in pack_header() and packed_fields_length()

- table.cc
  fix_type_pointers() assumed in multiple places that values cannot have
  0x00 inside and used strlen(TYPELIB::type_names[n]) to set
  the corresponding TYPELIB::type_lengths[n].

  Also, fix_type_pointers() did not check the encoded data for consistency.

  Rewriting fix_type_pointers() code to populate TYPELIB::type_names[n] and
  TYPELIB::type_lengths[n] at the same time, so no additional loop
  with strlen() is needed any more.

  Adding many data consistency tests.

  Fixing the main loop in fix_type_pointers() to use memchr() instead of
  strchr() to handle 0x00 properly.

  Fixing create_key_infos() to return the result in a LEX_STRING rather
  that in a char*.

Author: Eugene
Author Date: 2020-05-17 13:24:40 UTC

intrusive::list fixes

namespace intrusive: removed

split class into two: ilist<T> and sized_ilist<T> which has a size field.

ilist<T> no more NULLify pointers to bring a slignly better performance.
As a consequence, fil_space_t::is_in_unflushed_spaces and
fil_space_t::is_in_rotation_list boolean members are needed now.

Author: Anel Husakovic
Author Date: 2020-03-05 15:40:29 UTC

MDEV-21367: mysqld_safe log don't log to err.log

Introduced with 6b7918d524d5 in `10.2` just check for helper and handle
it if exist.

Author: Rasmus Johansson
Author Date: 2020-05-20 13:21:16 UTC

MDEV-22569 Startup scripts in 10.5 still start "mysqld" instead of "mariadbd"

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-05-20 12:18:20 UTC

MDEV-22637 Rollback of insert fails when prefix index primary key DDL happens

- During primary key prefix index rebuild, rollback of insert fails.
Reason is that InnoDB tries to fetch the column position from
new clustered index and it exceeds default column value tuple fields.
So InnoDB should use the table column position while searching for
defaults column value.

Author: Rasmus Johansson
Author Date: 2020-05-19 09:10:24 UTC

MDEV-22631 some test causes MTR interruption without generating summary and XML

Author: midenok
Author Date: 2018-09-10 08:48:12 UTC

MDEV-16937 Strict SQL with system versioned tables causes issues

Vers SQL: respect system fields in NO_ZERO_DATE mode.

This is the subject for refactoring in MDEV-19597

Author: Oleksandr Byelkin
Author Date: 2020-05-19 08:36:25 UTC

check MDEV-22173

Author: Sujatha Sivakumar
Author Date: 2020-05-19 08:26:06 UTC

MDEV-22451: 10.4 changes.

Author: Sujatha Sivakumar
Author Date: 2020-05-19 07:51:10 UTC

MDEV-22451-10.3 Testing

Author: Sujatha Sivakumar
Author Date: 2020-05-19 07:40:39 UTC

MDEV-22451-10.2 Merge testing

Author: Jan Lindström
Author Date: 2020-05-19 07:10:30 UTC

MDEV-18838 : galera.galera_toi_truncate: Test failure: mysqltest: query 'reap' succeeded - should have failed with errno 1213

Test cleanup.

Author: midenok
Author Date: 2019-07-22 12:38:37 UTC

MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field

Preserve and restore statement DA.

update_virtual_field() is called as part of index rebuild in
ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish.

Assertion in update_virtual_field() was put as part of MDEV-16222
because update_virtual_field() returns in_use->is_error(). The idea:
wrongly mixed semantics of error status before update_virtual_field()
and the status returned by update_virtual_field(). The former can
falsely influence the latter.

Preserve global error status and run update_virtual_field() with clear
DA since no matter how SQL command is finished it must update the
index after bulk INSERT.

Author: midenok
Author Date: 2019-07-25 17:50:54 UTC

MDEV-18706 InnoDB locking code debug helpers

Author: Vladislav Vaintroub
Author Date: 2020-05-15 22:19:00 UTC

MDEV-7021 allow config file parameter for mysql_install_db.exe

The new parameter is called --config

The config file is copied to the data directory as my.ini, and possibly
modified.

Bootstrap will now use my.ini to create tables. In case of non-standard
directory/file placements, this will ensure correct permissions for the
service user.

mysql_install_db_win test is extended to test --config with non-trivial
config file.

Author: Vlad Lesin
Author Date: 2020-05-13 11:32:12 UTC

MDEV-22554: "mariabackup --prepare" exits with code 0 even though innodb
error is logged

The fix is to set flag in ib::error::~error() and check it in
mariabackup.

ib::error::error() is replaced to ib::warn::warn() in
AIO::linux_create_io_ctx() because of two reasons:

1) if we leave it as is, then mariabackup MTR tests with --mem MTR option will
fail, :because Linux AIO can not be used on tmpfs,

2) when Linux AIO can not be initialized, InnoDB falls back to simulated
AIO, so such sutiation is not fatal error, it should be treated as warning.

Author: Sergei Golubchik
Author Date: 2020-05-17 16:44:23 UTC

MDEV-18496 Crash when Aria encryption is enabled but plugin not available

fix uninitialized struct member

Author: Sujatha Sivakumar
Author Date: 2020-05-18 11:21:22 UTC

MDEV-22451: SIGSEGV in __memmove_avx_unaligned_erms/memcpy from _my_b_write on CREATE after RESET MASTER

Analysis:
========
RESET MASTER TO # command deletes all binary log files listed in the index
file, resets the binary log index file to be empty, and creates a new binary
log with number #. When the user provided binary log number is greater than
the max allowed value '2147483647' server fails to generate a new binary log.
The RESET MASTER statement marks the binlog closure status as
'LOG_CLOSE_TO_BE_OPENED' and exits. Statements which follow RESET MASTER
try to write to binary log they find the log_state != LOG_CLOSED and
proceed to write to binary log cache and it results in crash.

Fix:
===
During MYSQL_BIN_LOG open, if generation of new binary log name fails then the
"log_state" needs to be marked as "LOG_CLOSED". With this further statements
will find binary log as closed and they will skip writing to the binary log.

Author: Marko Mäkelä
Author Date: 2020-05-18 11:04:31 UTC

MDEV-22611 Assertion btr_search_enabled failed during buffer pool resizing

In commit ad6171b91cac33e70bb28fa6865488b2c65e858c (MDEV-22456)
we removed the acquisition of the adaptive hash index latch
from the caller of btr_search_update_hash_ref().
The tests innodb.innodb_buffer_pool_resize_with_chunks
and innodb.innodb_buffer_pool_resize
would occasionally fail starting with 10.3,
due to MDEV-12288 causing more purge activity during the test.

btr_search_update_hash_ref(): After acquiring the adaptive hash index
latch, check that the adaptive hash index is still enabled on the page.

Author: Kentoku SHIBA
Author Date: 2019-11-01 07:40:26 UTC

MENT-458 MTR Big test "spider/bugfix.sql_mode_mariadb & myself" are both failing on Azure MTR pipeline

Support the dash number of MariaDB versions by Spider's tests

Author: varun
Author Date: 2020-05-18 05:01:17 UTC

MDEV-22509: Server crashes in Field_inet6::store_inet6_null_with_warn / Field::maybe_null

For field with type INET, during EITS collection the min and max values are store in text
representation in the statistical table.
While retrieving the value from the statistical table, the value is stored back in the original
field using binary form instead of text and this was resulting in the crash.

Introduced 2 functions in the Field structure:
1) store_to_statistical_minmax_field
2) store_from_statistical_minmax_field

Author: Alexander Barkov
Author Date: 2020-05-18 07:29:43 UTC

MDEV-21995 Server crashes in Item_field::real_type_handler with table value constructor

1. Code simplification:

Item_default_value handled all these values:
a. DEFAULT(field)
b. DEFAULT
c. IGNORE
and had various conditions to distinguish (a) from (b) and from (c).

Introducing a new abstract class Item_contextually_typed_value_specification,
to handle (b) and (c), so the hierarchy now looks as follows:

Item
  Item_result_field
    Item_ident
      Item_field
        Item_default_value - DEFAULT(field)
  Item_contextually_typed_value_specification
    Item_default_specification - DEFAULT
    Item_ignore_specification - IGNORE

2. Introducing a new virtual method is_evaluable_expression() to
determine if an Item is:
- a normal expression, so its val_xxx()/get_date() methods can be called
- or a just an expression substitute, whose value methods cannot be called.

3. Disallowing Items that are not evalualble expressions in table value
   constructors.

Author: Jan Lindström
Author Date: 2020-05-08 08:35:15 UTC

MDEV-21483 : Galera MTR tests failed: galera.MW-328A galera.MW-328B

Enable tests with additional galera output to find out actual
reason for test failures.

Author: varun
Author Date: 2020-05-18 01:47:49 UTC

MDEV-22583: Selectivity for BIT columns in filtered column for EXPLAIN is incorrect

For BIT columns when EITS is collected, we store the integral value in text representation
in the min and max fields of the statistical table (that is mysql.column_stats)
When this value is retrieved from the statistical table to original table field
then we try to store the text representation in the original field which is INCORRECT.
The value that is retrieved should be converted to integral type and that value should
be stored back in the original field. This would get us the correct estimate for
selectivity of the predicate.

Author: varun
Author Date: 2020-05-18 05:01:17 UTC

MDEV-22509: Server crashes in Field_inet6::store_inet6_null_with_warn / Field::maybe_null

Author: Vladislav Vaintroub
Author Date: 2020-05-15 22:19:00 UTC

MDEV-7021 allow config file parameter for mysql_install_db.exe

The new parameter is called --config

The config file is copied to the data directory as my.ini, and possibly
modified.

Bootstrap will now use my.ini to create tables. In case of non-standard
directory/file placements, this will ensure correct permissions for the
service user.

mysql_install_db_win test is extended to test --config with non-trivial
config file.

Author: Marko Mäkelä
Author Date: 2020-05-16 04:54:09 UTC

Merge 10.3 into 10.4

Author: Julius Goryavsky
Author Date: 2020-05-15 16:35:19 UTC

MDEV-22554: galera_sst_mariabackup fails with "Failed to start mysqld.2"

The problem is caused by the operation of netcat streamer and does
not appear on systems where socat is installed. We need to add the
"-N" option for netcat to call shutdown() on the socket when receiving
EOF from STDIN.

Author: Julius Goryavsky
Author Date: 2020-05-15 16:35:19 UTC

MDEV-22554: galera_sst_mariabackup fails with "Failed to start mysqld.2"

The problem is caused by the operation of netcat streamer and does
not appear on systems where socat is installed. We need to add the
"-N" option for netcat to call shutdown() on the socket when receiving
EOF from STDIN.

Author: Eugene
Author Date: 2020-05-15 15:39:05 UTC

MDEV-19749 MDL scalability regression after backup locks

MDL_lock::Ticket_list::remove_ticket(): reduce algoritmic
complexity from O(N) to O(1)

MDL_lock::Ticket_list::clear_bit_if_not_in_list(): removed

MDL_lock::Ticket_list::m_type_counters: a map of ticket type
to count. Initialization is memset(0) which takes time.

Author: Anel Husakovic
Author Date: 2020-05-12 14:16:05 UTC

MDEV-22312: Bad error message for SET DEFAULT ROLE when user account is not granted the role

- `SET DEFAULT ROLE xxx [FOR yyy]` should say:
  "User yyy has not been granted a role xxx" if:
    - The current user (not the user `yyy` in the FOR clause) can see the
    role xxx. It can see the role if:
      * role exists in `mysql.roles_mappings` (traverse the graph),
      * If the current user has read access on `mysql.user` table - in
    that case, it can see all roles, granted or not.
    - Otherwise it should be "Invalid role specification".

In other words, it should not be possible to use `SET DEFAULT ROLE` to discover whether a specific role exist or not.

Author: Alexey Botchkov
Author Date: 2020-05-15 11:25:28 UTC

MDEV-17399 Add support for JSON_TABLE.

Syntax for JSON_TABLE added.
The ha_json_table handler added. Executing the JSON_TABLE we
create the temporary table of the ha_json_table, add dependencies
of other tables and sometimes conditions to WHERE.

Author: Monty
Author Date: 2020-05-15 08:51:31 UTC

Fixed access to uninitalized memory found by valgrind

Author: Vlad Lesin
Author Date: 2019-11-27 22:16:22 UTC

Add some extra logging to catch the case when page is written with wrong offset.

Author: varun
Author Date: 2020-05-14 21:07:16 UTC

MDEV-22556: Incorrect result for window function when using encrypt-tmp-files=ON

The issue here is that end_of_file for encrypted temporary IO_CACHE (used by filesort) is updated
using lseek.
Encryption adds storage overhead and hides it from the caller by recalculating offsets and lengths.
Two different IO_CACHE cannot possibly modify the same file
because the encryption key is randomly generated and stored in the IO_CACHE.
So when the tempfiles are encrypted DO NOT use lseek to change end_of_file.

Further observations about updating end_of_file using lseek
1) The end_of_file update is only used for binlog index files
2) The whole point is to update file length when the file was modified via a different file descriptor.
3) The temporary IO_CACHE files can never be modified via a different file descriptor.
4) For encrypted temporary IO_CACHE, end_of_file should not be updated with lseek

Author: varun
Author Date: 2020-05-14 13:08:49 UTC

MDEV-18100: User defined aggregate functions not working correctly when the schema is changed

The issue here was that when the schema was changed the value for the THD::server_status
is ored with SERVER_SESSION_STATE_CHANGED.
For custom aggregate functions, currently we check if the server_status is equal to
SERVER_STATUS_LAST_ROW_SENT then we should terminate the execution of the custom
aggregate function as there are no more rows to fetch.
So the check should be that if the server status has the bit set for
SERVER_STATUS_LAST_ROW_SENT then we should terminate the execution of the
custom aggregate function.

Author: Jan Lindström
Author Date: 2020-05-14 06:17:14 UTC

MDEV-22494 : Galera assertion lock_sys.mutex.is_owned() at lock_trx_handle_wait_low

Problem was that trx->lock.was_chosen_as_wsrep_victim variable was
not set back to false after it was set true.

wsrep_thd_bf_abort
 Add assertions for correct mutex status and take necessary
 mutexes before calling thd->awake_no_mutex().

innobase_rollback_trx()
 Reset trx->lock.was_chosen_as_wsrep_victim

wsrep_abort_slave_trx()
 Removed unused function.

wsrep_innobase_kill_one_trx()
 Added function comment, removed unnecessary parameters
 and added debug assertions to enforce correct usage. Added
 more debug output to help out on error analysis.

wsrep_abort_transaction()
 Added debug assertions and removed unused variables.

trx0trx.h
 Removed assert_trx_is_free macro and replaced it with
 assert_freed() member function.

trx_create()
 Use above assert_free() and initialize wsrep variables.

trx_free()
 Use assert_free()

trx_t::commit_in_memory()
 Reset lock.was_chosen_as_wsrep_victim

trx_rollback_for_mysql()
 Reset trx->lock.was_chosen_as_wsrep_victim

Add test case galera_bf_kill

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-05-12 17:19:00 UTC

- Review comments

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-05-12 17:19:00 UTC

- Review comments

Author: Alexander Barkov
Author Date: 2020-05-12 15:45:21 UTC

MDEV-20261 NULL passed to String::eq, SEGV, server crash, regression in 10.4

Type_handler_xxx::Item_const_eq() can handle only non-NULL values.
The code in Item_basic_value::eq() did not take this into account.

Adding a test to detect three different combinations:
- Both values are NULLs, return true.
- Only one value is NULL, return false.
- Both values are not NULL, call Type_handler::Item_const_eq()
  to check equality.

Author: Vlad Lesin
Author Date: 2020-05-12 10:57:09 UTC

MDEV-22398: mariabackup.innodb_xa_rollback fails on repeat

Flush LSN to system tablespace on innodb shutdown if XA is rolled back by
mariabackup.

Author: Julius Goryavsky
Author Date: 2020-05-12 11:29:17 UTC

MDEV-20401: Server incorrectly auto-sets lower_case_file_system value

Server auto-sets lower_case_file_system value based on default
datadir's behavior instead of instead of using the directory specified
by the user through the configuration file or command line options.

This patch fixes this problem.

Author: Sachin Setiya
Author Date: 2020-05-12 11:30:26 UTC

completed

Author: Julius Goryavsky
Author Date: 2020-05-12 11:29:17 UTC

MDEV-20401: Server incorrectly auto-sets lower_case_file_system value

Server auto-sets lower_case_file_system value based on default
datadir's behavior instead of instead of using the directory specified
by the user through the configuration file or command line options.

This patch fixes this problem.

Author: Monty
Author Date: 2020-05-11 04:58:31 UTC

Temporary commit

- Split thd::has_transactions() to thd::has_transactions() and
  thd::has_transactions_and_rollback()

Author: varun
Author Date: 2020-05-11 14:26:14 UTC

MDEV-22498: SIGSEGV in Bitmap<64u>::merge on SELECT

For the case when the optimizer does the IN-EXISTS transformation,
the equality condition is injected in the WHERE OR HAVING clause of
the subquery. If the select list of the subquery has a reference to
the parent select make sure to use the reference and not the original
item.

Author: Alexey Botchkov
Author Date: 2020-05-11 12:27:22 UTC

The customized server_audit plugin to test if it affects.

Author: Sergei Golubchik
Author Date: 2020-05-08 07:14:34 UTC

cleanup: foreign-keys.test vs foreign_key.test

Author: Sergey Vojtovich
Author Date: 2020-05-07 19:07:03 UTC

Wait for stats in the LOADING state

Fixes both stats and histograms.

To be fixed up into "ae6a37e Thread safe statistics loading".

Author: Alexander Barkov
Author Date: 2020-05-07 15:20:17 UTC

MDEV-22043 Special character leads to assertion in my_wc_to_printable_generic on 10.5.2 (debug)

The code did not take into account that:
- U+005C (backslash) can occupy more than mbminlen characters (e.g. in sjis)
- Some character sets do not have a code for U+005C (e.g. swe7)

Adding a new function my_wc_to_printable into MY_CHARSET_HANDLER to
cover all special cases easier.

Author: Otto Kekäläinen
Author Date: 2020-05-04 17:45:18 UTC

Deb: Touch migrated-from-my.cnf-settings.conf

Remove once buildbot.askmonty.org has been
updated not to expect this file.

Author: Vladislav Vaintroub
Author Date: 2020-05-07 11:25:56 UTC

Fix WolfSSL build

- Do not build with TLSv1.0,it stopped working,at least with SChannel client
- Disable a test that depends on TLSv1.0
- define FP_MAX_BITS always, to fix 32bit builds.
- Increase MAX_AES_CTX_SIZE, to fix build on Linux

Author: Sergey Petrunia
Author Date: 2020-05-06 19:35:43 UTC

MDEV-21794: Optimizer flag rowid_filter leads to long query, Part#2

Update TokuDB to use the new names of Index Condition Pushdown's
datatypes and constants

Author: Jan Lindström
Author Date: 2020-05-06 14:23:49 UTC

MDEV-21515 : Galera test sporadic failure on galera.galera_wsrep_new_cluster: Result content mismatch

Test starts two servers and we do not know order they really start,
thus wsrep_local_index can be 1 or 2.

Author: varun
Author Date: 2020-04-26 10:12:57 UTC

MDEV-7317: Make an index ignorable to the optimizer

This feature adds the functionality of ignorability for indexes.
Indexes are not ignored be default.

To control index ignorability explicitly for a new index,
use IGNORE or NOT IGNORE as part of the index definition for
CREATE TABLE, CREATE INDEX, or ALTER TABLE.

Primary keys (explicit or implicit) cannot be made ignorable.

The table INFORMATION_SCHEMA.STATISTICS get a new column named IGNORE that
would store whether an index needs to be ignored or not.

An optimizer switch is introduced named ignore_indexes, which if enabled
would make the optimzier not to consider all the ignorable indexes.

Author: varun
Author Date: 2020-04-26 10:12:57 UTC

MDEV-7317: Make an index ignorable to the optimizer

Author: Sergey Vojtovich
Author Date: 2020-05-03 20:15:00 UTC

Thread_cache::unregistred_connections()

There is a gap between connection spawn time and connection becoming
killable time (registration in server_threads). If shutdown is initiated
when such connection is being executed through this gap, connection may
access destroyed data or cause shutdown hang.

Fixed by adding a counter of connections going through this gap and
waiting until this counter drops down to 0 on shutdown.

Part of
MDEV-18353 - Shutdown may miss to wait for connection thread

Author: Monty
Author Date: 2020-05-06 08:51:03 UTC

Pushed faulty test

Author: Jan Lindström
Author Date: 2020-03-12 13:34:50 UTC

MDEV-21910 : KIlling thread on Galera could cause mutex deadlock

Following issues here:

Whenever Galera BF (brute force) transaction decides to abort conflicting
transaction it will kill that thread using thd::awake()

User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()

Whenever one of these actions is executed we will hold number of InnoDB
internal mutexes and thd mutexes. Sometimes these mutexes are taken in
different order causing mutex deadlock.

Lets consider a example starting from Galera BF transaction deciding to abort
conflicting transaction (let's call this thread1):

thread1:
lock_rec_other_has_conflicting
  here we hold both lock_sys mutex and trx mutex for conflicting lock transaction.
wsrep_innobase_kill_one_trx
  For victim we call wsrep_thd_awake

Next thread2 we assume user to execute KILL QUERY to some other executing
query (note it can't be BF query).

thread2:
sql_kill()
kill_one_thread
find_thread_by_id
  takes LOCK_thd_kill
thd->awake_no_mutex()

thread1:
thd->awake(KILL_QUERY)
  Tries to have LOCK_thd_kill but it is hold by thread2 so we wait (note that
  we still hold lock_sys mutex and trx mutex).

thread2:
ha_kill_query()
kill_handlerton
innobase_kill_query
lock_trx_handle_wait
  lock_mutex_enter() must wait as thread1 is holding it

Thus thread1 lock_sys, trx_mutex waits -> thread2 LOCK_thd_kill waits lock_sys -> thread1
==> thread1 waits -> thread2 waits -> thread1 ==> mutex deadlock.

In this patch we will fix Galera BF and user kill cases so that we enqueue
victim thread to a list while we hold InnoDB mutexes and we then release them.
A new background thread will pick victim thread from this new list and uses
thd::awake() with no InnoDB mutexes. Idea is similar to replication background
kill. This fix enforces that we take LOCK_thd_data -> lock sys mutex -> trx mutex
always in this order.

wsrep_mysqld.cc
 Here we introduce a list where victim threads are stored,
 condition variable to be used to wake up background thread
 and mutex to protect list.

wsrep_thd.cc
 Create a new background thread to handle victim thread
 abort. We may take wsrep_thd_LOCK mutex here but not any
 InnoDB mutexes.

wsrep_innobase_kill_one_trx
 Remove all the wsrep code that was moved to wsrep_thd.cc
 We just enqueue required information to background kill
 list and cancel victim trx lock wait if there is such.
 Here we have InnoDB lock sys mutex and trx mutex so here
 we can't take wsrep_thd_LOCK mutex.

wsrep_abort_transaction
 Cleanup only.

Author: Jan Lindström
Author Date: 2020-03-12 13:34:50 UTC

MDEV-21910 : KIlling thread on Galera could cause mutex deadlock

Following issues here:

Whenever Galera BF (brute force) transaction decides to abort conflicting
transaction it will kill that thread using thd::awake()

User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()

Whenever one of these actions is executed we will hold number of InnoDB
internal mutexes and thd mutexes. Sometimes these mutexes are taken in
different order causing mutex deadlock.

Lets consider a example starting from Galera BF transaction deciding to abort
conflicting transaction (let's call this thread1):

thread1:
lock_rec_other_has_conflicting
  here we hold both lock_sys mutex and trx mutex for conflicting lock transaction.
wsrep_innobase_kill_one_trx
  For victim we call wsrep_thd_awake

Next thread2 we assume user to execute KILL QUERY to some other executing
query (note it can't be BF query).

thread2:
sql_kill()
kill_one_thread
find_thread_by_id
  takes LOCK_thd_kill
thd->awake_no_mutex()

thread1:
thd->awake(KILL_QUERY)
  Tries to have LOCK_thd_kill but it is hold by thread2 so we wait (note that
  we still hold lock_sys mutex and trx mutex).

thread2:
ha_kill_query()
kill_handlerton
innobase_kill_query
lock_trx_handle_wait
  lock_mutex_enter() must wait as thread1 is holding it

Thus thread1 lock_sys, trx_mutex waits -> thread2 LOCK_thd_kill waits lock_sys -> thread1
==> thread1 waits -> thread2 waits -> thread1 ==> mutex deadlock.

In this patch we will fix Galera BF and user kill cases so that we enqueue
victim thread to a list while we hold InnoDB mutexes and we then release them.
A new background thread will pick victim thread from this new list and uses
thd::awake() with no InnoDB mutexes. Idea is similar to replication background
kill. This fix enforces that we take LOCK_thd_data -> lock sys mutex -> trx mutex
always in this order.

wsrep_mysqld.cc
 Here we introduce a list where victim threads are stored,
 condition variable to be used to wake up background thread
 and mutex to protect list.

wsrep_thd.cc
 Create a new background thread to handle victim thread
 abort. We may take wsrep_thd_LOCK mutex here but not any
 InnoDB mutexes.

wsrep_innobase_kill_one_trx
 Remove all the wsrep code that was moved to wsrep_thd.cc
 We just enqueue required information to background kill
 list and cancel victim trx lock wait if there is such.
 Here we have InnoDB lock sys mutex and trx mutex so here
 we can't take wsrep_thd_LOCK mutex.

wsrep_abort_transaction
 Cleanup only.

Author: Jan Lindström
Author Date: 2020-03-12 13:34:50 UTC

MDEV-21910 : KIlling thread on Galera could cause mutex deadlock

Following issues here:

Whenever Galera BF (brute force) transaction decides to abort conflicting
transaction it will kill that thread using thd::awake()

User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()

Whenever one of these actions is executed we will hold number of InnoDB
internal mutexes and thd mutexes. Sometimes these mutexes are taken in
different order causing mutex deadlock.

Lets consider a example starting from Galera BF transaction deciding to abort
conflicting transaction (let's call this thread1):

thread1:
lock_rec_other_has_conflicting
  here we hold both lock_sys mutex and trx mutex for conflicting lock transaction.
wsrep_innobase_kill_one_trx
  For victim we call wsrep_thd_awake

Next thread2 we assume user to execute KILL QUERY to some other executing
query (note it can't be BF query).

thread2:
sql_kill()
kill_one_thread
find_thread_by_id
  takes LOCK_thd_kill
thd->awake_no_mutex()

thread1:
thd->awake(KILL_QUERY)
  Tries to have LOCK_thd_kill but it is hold by thread2 so we wait (note that
  we still hold lock_sys mutex and trx mutex).

thread2:
ha_kill_query()
kill_handlerton
innobase_kill_query
lock_trx_handle_wait
  lock_mutex_enter() must wait as thread1 is holding it

Thus thread1 lock_sys, trx_mutex waits -> thread2 LOCK_thd_kill waits lock_sys -> thread1
==> thread1 waits -> thread2 waits -> thread1 ==> mutex deadlock.

In this patch we will fix Galera BF and user kill cases so that we enqueue
victim thread to a list while we hold InnoDB mutexes and we then release them.
A new background thread will pick victim thread from this new list and uses
thd::awake() with no InnoDB mutexes. Idea is similar to replication background
kill. This fix enforces that we take LOCK_thd_data -> lock sys mutex -> trx mutex
always in this order.

wsrep_mysqld.cc
 Here we introduce a list where victim threads are stored,
 condition variable to be used to wake up background thread
 and mutex to protect list.

wsrep_thd.cc
 Create a new background thread to handle victim thread
 abort. We may take wsrep_thd_LOCK mutex here but not any
 InnoDB mutexes.

wsrep_innobase_kill_one_trx
 Remove all the wsrep code that was moved to wsrep_thd.cc
 We just enqueue required information to background kill
 list and cancel victim trx lock wait if there is such.
 Here we have InnoDB lock sys mutex and trx mutex so here
 we can't take wsrep_thd_LOCK mutex.

wsrep_abort_transaction
 Cleanup only.

Author: Oleksandr Byelkin
Author Date: 2020-05-06 06:44:50 UTC

fix for the old compiler.

Author: Jan Lindström
Author Date: 2020-04-29 11:46:52 UTC

Analysis from 10.4

BF abort:
wsrep_kill_victim (lock_sys->mutex, trx->mutex) => wsrep_kill_one_trx (thd->LOCK_thd_data)

KILL QUERY:
find_thread_by_id(thd->LOCK_thd_data , thd->LOCK_thd_kill) => thd->awake_no_mutex() => innobase_kill_query() => lock_trx_handle_wait( lock_sys->mutex, trx->mutex if not wsrep victim)

From these only lock_sys->mutex is global mutex. All others are either thread internal (thd) or transaction internal (trx). Therefore, mutex deadlock is possible if and only if thread we have selected as victim for BF abort and thread user is trying to kill are exactly the same one. While BF abort also will call thd->awake() it will have trx->lock.was_chosen_as_wsrep_victim=true and then we do not take lock_sys->mutex or trx->mutex at lock_trx_handle_wait(). Second problem is possible when in wsrep_kill_one_trx() we set trx->lock.was_chosen_as_wsrep_victim=true and release thd->LOCK_thd_data mutex. Now if we have a schedule where KILL is executed there is possibility that we either try to take lock_sys->mutex and have to wait or if this is same thread as victim no InnoDB mutexes are taken. Furthermore, in bf_abort() we might take thd->LOCK_thd_data again in wsrep-lib.

In this fix candidate we release lock_sys->mutex and trx->mutex before
we call wsrep_innobase_kill_one_trx(). Thus, it is safe to take
thd->LOCK_thd_data similarly as KILL does. We can remove
was_chosen_as_wsrep_victim variable from trx as it is not anymore
needed.

Author: varun
Author Date: 2020-05-05 15:14:43 UTC

MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed.

A temporary table is needed for window function computation but if only a NAMED WINDOW SPEC
is used and there is no window function, then there is no need to create a temporary
table as there is no stage to compute WINDOW FUNCTION

Author: Kentoku SHIBA
Author Date: 2020-05-05 13:41:10 UTC

MDEV-20502 Queries against spider tables return wrong values for columns following constant declarations.

Add test cases.

Author: Sergei Golubchik
Author Date: 2020-05-04 21:25:20 UTC

MDEV-22123 On RHEL8 mariadb-server is not provided

setup alternative name for rhel8/centos8

use correct C/C version when providing mariadb-connector-c

Author: Sujatha Sivakumar
Author Date: 2020-05-05 10:13:33 UTC

MDEV:20220: Add status variable that gets incremented if slave's parallel worker thread is idle due to full queue

Fix: Added a new status variable named 'Slave_idle_parallel_worker_count'.
The variable gets incremented whenever a worker enters idle state. ie
Workers state in SHOW PROCESSLIST becomes "Waiting for work from SQL thread".
'Slave_idle_parallel_worker_count' variable gets cleared during START SLAVE.