lp:maria

Author: Eugene
Author Date: 2020-03-28 21:41:57 UTC

MDEV-22074 UBSAN: applying zero offset to null pointer in hash.c

The fix: return fast when no work should be performed.

Author: varun
Author Date: 2020-03-28 21:06:32 UTC

Making EITS use packed keys in Unique class

Author: Marko Mäkelä
Author Date: 2020-03-28 19:05:55 UTC

MDEV-20377: Enable MemorySanitizer user-poisoning

For ENGINE=Aria, we work around bugs in various tests that catch
writes of uninitialized bytes from the Aria page cache.
(Why do we even write anything on DROP TABLE?)

Author: midenok
Author Date: 2020-03-28 18:49:20 UTC

MENT-670 fix inconsistency between INSERT .. VALUES and INSERT .. SELECT

Author: Rasmus Johansson
Author Date: 2020-03-28 06:48:56 UTC

MDEV-17812 Use MariaDB in error messages instead of MySQL

Changed wording in error messages from MySQL to MariaDB.

Tests that have these errors recorded were updated.

Author: Rasmus Johansson
Author Date: 2020-03-27 15:03:03 UTC

MDEV-17812 Use MariaDB in error messages instead of MySQL

Changed wording in error messages from MySQL to MariaDB.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-03-27 11:02:52 UTC

MDEV-21832 FORCE all partition to rebuild if any one of the
  partition does rebuild

- In ha_innobase::commit_inplace_alter_table() assumes that all partition
should do the same kind of alter operations. During DDL, if one partition
requires table rebuild and other partition doesn't need rebuild
then all partition should be forced to rebuild.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-03-27 11:02:34 UTC

MDEV-21832 FORCE all partition to rebuild if any one of the
  partition does rebuild

- In ha_innobase::commit_inplace_alter_table() assumes that all partition
should do the same kind of alter operations. During DDL, if one partition
requires table rebuild and other partition doesn't need rebuild
then all partition should be forced to rebuild.

Author: Sergey Vojtovich
Author Date: 2020-03-26 22:24:49 UTC

num_worker_threads my_atomic to Atomic_counter

Author: Alexander Barkov
Author Date: 2020-03-27 07:32:41 UTC

MDEV-22057 REPLICATION MASTER ADMIN is missing in root account after upgrade

Author: Monty
Author Date: 2020-03-26 17:39:51 UTC

Make mysql.innodb_mysql_lock deterministic

MDEV-7861 main.innodb_mysql_lock fails sporadically in buildbot

Author: Eugene
Author Date: 2020-03-22 23:08:01 UTC

MDEV-22005 UBSAN: applying non-zero offset 2 to null pointer in my_charpos_mb()

Empty comment has a correct length.

Author: Sachin Setiya
Author Date: 2020-03-26 14:52:54 UTC

Patch version 3

Author: Monty
Author Date: 2020-03-24 11:34:03 UTC

More cost fixes:

- heap::read_time() and heap::keyread_time() adjusted to not add +1.
  This was to ensure that handler::keyread_time() doesn't give
  higher cost for heap tables than for normal tables. One effect of
  this is that heap and derived tables stored in heap will prefer
  key access as this is now regarded as cheap.
- Changed cost for index read in sql_select.cc to match
  multi_range_read_info_const(). All index cost calculation is now
  done trough one function.
- 'ref' will now use quick_cost for keys if it exists. This is done
  so that for '=' ranges, 'ref' is prefered over 'range'.
- scan_time() now takes avg_io_costs() into account
- get_delayed_table_estimates() uses block_size and avg_io_cost()
- Removed default argument to test_if_order_by_key(); simplifies code

Author: Anel Husakovic
Author Date: 2020-03-26 10:00:45 UTC

PR 900 update for debian

Author: Marko Mäkelä
Author Date: 2020-03-25 16:22:20 UTC

Merge 10.5

Author: Oleksandr Byelkin
Author Date: 2020-03-19 14:02:09 UTC

MDEV-22037: Add ability to skip content of some tables (work around for MDEV-20939)

--ignore-table-data parameter added.

Author: Jan Lindström
Author Date: 2020-03-12 13:34:50 UTC

MDEV-21910 : KIlling thread on Galera could cause mutex deadlock

Following issues here:

Whenever Galera BF (brute force) transaction decides to abort conflicting
transaction it will kill that thread using thd::awake()

User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()

Whenever one of these actions is executed we will hold number of InnoDB
internal mutexes and thd mutexes. Sometimes these mutexes are taken in
different order causing mutex deadlock.

Lets consider a example starting from Galera BF transaction deciding to abort
conflicting transaction (let's call this thread1):

thread1:
lock_rec_other_has_conflicting
  here we hold both lock_sys mutex and trx mutex for conflicting lock transaction.
wsrep_innobase_kill_one_trx
  For victim we call wsrep_thd_awake

Next thread2 we assume user to execute KILL QUERY to some other executing
query (note it can't be BF query).

thread2:
sql_kill()
kill_one_thread
find_thread_by_id
  takes LOCK_thd_kill
thd->awake_no_mutex()

thread1:
thd->awake(KILL_QUERY)
  Tries to have LOCK_thd_kill but it is hold by thread2 so we wait (note that
  we still hold lock_sys mutex and trx mutex).

thread2:
ha_kill_query()
kill_handlerton
innobase_kill_query
lock_trx_handle_wait
  lock_mutex_enter() must wait as thread1 is holding it

Thus thread1 lock_sys, trx_mutex waits -> thread2 LOCK_thd_kill waits lock_sys -> thread1
==> thread1 waits -> thread2 waits -> thread1 ==> mutex deadlock.

In this patch we will fix Galera BF and user kill cases so that we enqueue
victim thread to a list while we hold InnoDB mutexes and we then release them.
A new background thread will pick victim thread from this new list and uses
thd::awake() with no InnoDB mutexes. Idea is similar to replication background
kill. This fix enforces that we take LOCK_thd_data -> lock sys mutex -> trx mutex
always in this order.

wsrep_mysqld.cc
 Here we introduce a list where victim threads are stored,
 condition variable to be used to wake up background thread
 and mutex to protect list.

wsrep_thd.cc
 Create a new background thread to handle victim thread
 abort. We may take wsrep_thd_LOCK mutex here but not any
 InnoDB mutexes.

wsrep_innobase_kill_one_trx
 Remove all the wsrep code that was moved to wsrep_thd.cc
 We just enqueue required information to background kill
 list and cancel victim trx lock wait if there is such.
 Here we have InnoDB lock sys mutex and trx mutex so here
 we can't take wsrep_thd_LOCK mutex.

wsrep_abort_transaction
 Cleanup only.

Author: Jan Lindström
Author Date: 2020-03-12 13:34:50 UTC

MDEV-21910 : KIlling thread on Galera could cause mutex deadlock

Following issues here:

Whenever Galera BF (brute force) transaction decides to abort conflicting
transaction it will kill that thread using thd::awake()

User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()

Whenever one of these actions is executed we will hold number of InnoDB
internal mutexes and thd mutexes. Sometimes these mutexes are taken in
different order causing mutex deadlock.

Lets consider a example starting from Galera BF transaction deciding to abort
conflicting transaction (let's call this thread1):

thread1:
lock_rec_other_has_conflicting
  here we hold both lock_sys mutex and trx mutex for conflicting lock transaction.
wsrep_innobase_kill_one_trx
  For victim we call wsrep_thd_awake

Next thread2 we assume user to execute KILL QUERY to some other executing
query (note it can't be BF query).

thread2:
sql_kill()
kill_one_thread
find_thread_by_id
  takes LOCK_thd_kill
thd->awake_no_mutex()

thread1:
thd->awake(KILL_QUERY)
  Tries to have LOCK_thd_kill but it is hold by thread2 so we wait (note that
  we still hold lock_sys mutex and trx mutex).

thread2:
ha_kill_query()
kill_handlerton
innobase_kill_query
lock_trx_handle_wait
  lock_mutex_enter() must wait as thread1 is holding it

Thus thread1 lock_sys, trx_mutex waits -> thread2 LOCK_thd_kill waits lock_sys -> thread1
==> thread1 waits -> thread2 waits -> thread1 ==> mutex deadlock.

In this patch we will fix Galera BF and user kill cases so that we enqueue
victim thread to a list while we hold InnoDB mutexes and we then release them.
A new background thread will pick victim thread from this new list and uses
thd::awake() with no InnoDB mutexes. Idea is similar to replication background
kill. This fix enforces that we take LOCK_thd_data -> lock sys mutex -> trx mutex
always in this order.

wsrep_mysqld.cc
 Here we introduce a list where victim threads are stored,
 condition variable to be used to wake up background thread
 and mutex to protect list.

wsrep_thd.cc
 Create a new background thread to handle victim thread
 abort. We may take wsrep_thd_LOCK mutex here but not any
 InnoDB mutexes.

wsrep_innobase_kill_one_trx
 Remove all the wsrep code that was moved to wsrep_thd.cc
 We just enqueue required information to background kill
 list and cancel victim trx lock wait if there is such.
 Here we have InnoDB lock sys mutex and trx mutex so here
 we can't take wsrep_thd_LOCK mutex.

wsrep_abort_transaction
 Cleanup only.

Author: Oleksandr Byelkin
Author Date: 2020-03-22 10:33:53 UTC

MDEV-19273: Server crash in MDL_ticket::has_stronger_or_equal_type or Assertion `thd->mdl_context.is_lock_owner(MDL_key::TABLE, table->db.str, table->table_name.str, MDL_SHARED)' failed in mysql_rm_table_no_locks

Early report error in case of DROP SEQUENCE <non-sequence>

Do not use error variable for other purposes except error.

Author: Vladislav Vaintroub
Author Date: 2020-03-24 22:30:40 UTC

MDEV-22032 update HeidiSQL to version 11

Author: Nikita Malyavin
Author Date: 2020-03-16 16:59:11 UTC

forbid REPLACE/ODKU on tables containing WITHOUT OVERLAPS

Author: Julius Goryavsky
Author Date: 2020-03-24 13:55:07 UTC

MDEV-21951: mariabackup SST fail if data-directory have lost+found directory

To fix this, it is necessary to add an option to exclude the
database with the name "lost+found" from processing (the database
name will be checked by the check_if_skip_database_by_path() or
by the check_if_skip_database() function, and as a result
"lost+found" will be skipped).

In addition, it is necessary to slightly modify the verification
logic in the check_if_skip_database() function.

Author: Monty
Author Date: 2020-03-23 18:13:24 UTC

Use truncate instead of delete to make stat_tables.inc more repeatable

Author: Oleksandr Byelkin
Author Date: 2020-03-16 15:53:10 UTC

MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error

Added indication of truncated string for "s" and "M" formats

Author: Oleksandr Byelkin
Author Date: 2020-03-16 15:53:10 UTC

MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error

Added indication of truncated string for "s" and "M" formats

Author: Sachin Setiya
Author Date: 2020-03-24 07:17:03 UTC

Empty commit 1

Author: Rasmus Johansson
Author Date: 2020-03-24 05:28:47 UTC

MDEV-22003 mysql_config disappeared from 10.5 (e.g. binary tarball or source build)

Author: varun
Author Date: 2020-03-22 22:14:24 UTC

MDEV-22011: DISTINCT with JSON_ARRAYAGG gives wrong results

For JSON_ARRAYAGG with DISTINCT also store the null bytes for the record
inside the Unique object

Author: midenok
Author Date: 2020-03-23 23:51:45 UTC

MDEV-20515 multi-update tries to position updated table by null reference

Cause

Join tmp table inserts null row because of OUTER JOIN, that's
expected. Since `multi_update::prepare2()` converted
`Item_temptable_rowid` into `Item_field` (28dbdf3)
`multi_update::send_data()` accesses join tmp record directly and
treats it as a normal row ignoring null status of ref field. NULL ref
field is then treated as normal in `multi_update::do_updates()` which
tries to position updated table by reference 0.

Note that reference 0 may be valid reference and the first row of
table can be wrongly updated (see multi_update.test).

Fix

Do not add row into multi-update tmp table in case of null ref
field. Join tmp table does not have null_row status at this time (as
well as `STATUS_NULL_ROW`) and cannot be skipped by these properties
(see first comment in multi_update::send_data()). But it has all null
fields (including the ref field).

Author: Vicențiu Ciorbaru
Author Date: 2020-03-23 19:20:14 UTC

Include the correct path to wsrep_sst_common

Author: Marko Mäkelä
Author Date: 2020-03-23 17:11:43 UTC

MDEV-21452 Default MUTEXTYPE=sys for improved performance

On POWER and ARMv8, the native mutex implementation of the standard
library on GNU/Linux seems to perform noticeably better than
the current default MUTEXTYPE=event or the MUTEXTYPE=futex.

On x86, the difference is smaller, and the MUTEXTYPE=event might be
slightly faster than the system mutex, but not by much.

Let us default to MUTEXTYPE=sys, that is, sys_mutex_t
(pthread_mutex_t or CRITICAL_SECTION) wrapped by
OSMutex and OSTrackMutex. Later, we might remove the
other implementations altogether.

Author: Sergei Golubchik
Author Date: 2020-03-18 18:45:25 UTC

remove redundant info on rpl test failure

these three SHOW statements (and more) are issued from
include/analyze-sync_with_master.test too.
no need to do it twice.

Author: Jan Lindström
Author Date: 2020-03-23 10:58:57 UTC

Merge branch 'codership-10.5-MDEV-21988' into 10.5

Author: Anel Husakovic
Author Date: 2020-03-23 10:05:56 UTC

Don't compile plugins if server is not compiled

Author: Larysa Sherepa
Author Date: 2020-03-19 17:55:57 UTC

MDEV-21360 restore debud_dbug through a session variable instead of '-d,..'

Author: Eugene
Author Date: 2020-03-22 21:58:51 UTC

MDEV-21990 Issue a message on changing deprecated innodb_log_files_in_group

innodb_init_params(): make sure that srv_n_log_files == 1 always

Author: Sachin Setiya
Author Date: 2020-03-22 07:04:21 UTC

final commit

Author: Sergei Golubchik
Author Date: 2020-03-21 19:21:58 UTC

cleanup: remove unused PLUGIN_INIT_SKIP_DYNAMIC_LOADING

Author: Jan Lindström
Author Date: 2020-03-20 13:47:07 UTC

Merge branch 'codership-10.4-MDEV-21675' into 10.4

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-03-19 17:56:25 UTC

MDEV-21572 innodb.leaf_page_corrupted_during_recovery failed in buildbot,
  server crashed

- InnoDB unnecessarily reads the page even though it has fully initialized
buffered redo log records. Allow the page initialization redo log to
apply for the page in buf_page_get_gen().

Author: Vladislav Vaintroub
Author Date: 2020-03-18 23:29:44 UTC

MDEV-21930 RocksDB does not compile anymore, with Visual Studio

Update submodule, change the source file list
rename CACHE_LINE_SIZE in ut0counter, so it does not conflics with
rocksdb sources, which also defines this constant now.

Author: Alexander Barkov
Author Date: 2020-03-20 11:24:06 UTC

MDEV-21977 main.func_math fails due to undefined behaviour

The problem happened in these line:

uval0= (ulonglong) (val0_negative ? -val0 : val0);
uval1= (ulonglong) (val1_negative ? -val1 : val1);

return check_integer_overflow(val0_negative ? -(longlong) res : res,
                              !val0_negative);

when unary minus was performed on -9223372036854775808.
This behavior is undefined in C/C++.

Author: Alexey Bychko
Author Date: 2020-03-19 14:34:05 UTC

MENT-663 build & test Xpand RPMs and the .so for bintars

added xpand plugin to debian build

Author: Marko Mäkelä
Author Date: 2020-03-19 15:30:26 UTC

WIP (crashes/hangs!) MDEV-15053 Reduce buf_pool_t::mutex contention

TODO: Carefully review and document the changed latching rules,
make ASAN happy, and possibly make more use of std::atomic
and make more things private in buf_pool_t to control access.

Author: Jan Lindström
Author Date: 2020-03-19 10:39:26 UTC

Fix test case name and add debug sync include.

Author: Daniel Black
Author Date: 2020-03-17 04:10:46 UTC

Deb: libcurl4 in bionic+/buster+

Otherwise fall back to libcurl3

Author: Otto Kekäläinen
Author Date: 2020-03-17 21:52:01 UTC

Travis-CI: Ignore clang/GCC 6 that permanently fails on 10.5 branch

The purpose of a CI system is to validate if a change is acceptable or not.
Thus tests that always fail should be ignored, otherwise the result of the
CI (green/red) is not reliable and actionable.

Author: midenok
Author Date: 2020-03-18 19:49:05 UTC

MDEV-21941 RENAME doesn't work for system time or period fields

- Ignore system-invisible fields (as well as for setting default value);
- Handle rename of system time and period fields.

Author: Alexey Botchkov
Author Date: 2020-03-18 09:55:21 UTC

MDEV-14057 InnoDB GIS tests fail.

Tests fixed.
MBR::Within() function fixed.

Author: Monty
Author Date: 2020-03-17 19:39:04 UTC

Debug push to find problem with rpl.rpl_skip_replication

Author: Andrew Hutchings
Author Date: 2019-09-30 16:23:46 UTC

MDEV-20329 Fix S3 engine OpenSSL race

With OpenSSL < 1.1 there is a potential for a race condition to occur.
This can cause the S3 engine to crash. The workaround is to add locking
callbacks to OpenSSL so that this doesn't happen.

https://curl.haxx.se/libcurl/c/threadsafe.html

There is a fix in libMariaS3 for this which when a certain flag is set
(HAVE_CURL_OPENSSL_UNSAFE) will add the required locks.

This patch adds CMake support so that the flag is set if it is found
that Curl is compiled with an unsafe OpenSSL version. For example Ubuntu
16.04 with libcurl4-openssl-dev.

Author: Monty
Author Date: 2020-03-17 16:09:49 UTC

Merge remote-tracking branch 'origin/10.4' into 10.5

Author: Marko Mäkelä
Author Date: 2020-03-17 09:24:59 UTC

Rename buf_pool->mutex back from buf_pool->LRU_list_mutex

Author: Alexey Botchkov
Author Date: 2020-03-17 08:48:22 UTC

MDEV-14057 InnoDB GIS tests fail.

Tests fixed and reenabled.

Author: Igor Babaev
Author Date: 2020-03-13 13:55:32 UTC

MDEV-21932 A fast plan with ROR index-merge is ignored when
           'index_merge_sort_union=off'

When index_merge_sort_union is set to 'off' and index_merge_union is set
to 'on' then any evaluated index merge scan must consist only of ROR scans.
The cheapest out of such index merges must be chosen. This index merge
might not be the cheapest index merge.

Author: Monty
Author Date: 2020-03-06 12:10:53 UTC

Make JT_EQ_REF and JT_REF have equal calculated cost formula

Author: Sergey Petrunia
Author Date: 2020-03-16 19:43:23 UTC

MDEV-21955: Alternative way of doing packed-sort keys

Use "Variable-Length Space Padded Encoding" that MyRocks uses.
The value comparison function is memcmp().

A very crude just to evaluate the performance of this.

Author: Marko Mäkelä
Author Date: 2020-03-16 15:51:09 UTC

MDEV-20590: Extend innodb_file_format for 10.4

* mapped (default): Like 'append', but allow the metadata record to
store a column map, to support instant add/drop/reorder (MDEV-15562)

* strict_mapped: Like 'mapped', but on ALTER TABLE, if a table
(or partition) is not in the 10.4-compatible format in some future
version of the server, force the table to be rebuilt.

Author: Marko Mäkelä
Author Date: 2020-03-16 13:35:37 UTC

MDEV-20590 Introduce a file format constraint to ALTER TABLE

If a table is altered using the MDEV-11369/MDEV-15562/MDEV-13134
ALGORITHM=INSTANT, it can force the table to use a non-canonical
format:

* A hidden metadata record at the start of the clustered index
is used to store each column's DEFAULT value. This makes it possible
to add new columns that have default values without rebuilding the table.

* Starting with MDEV-15562 in MariaDB Server 10.4, a BLOB in the
hidden metadata record is used to store column mappings. This makes
it possible to drop or reorder columns without rebuilding the table.
This also makes it possible to add columns to any position or drop
columns from any position in the table without rebuilding the table.

If a column is dropped without rebuilding the table, old records
will contain garbage in that column's former position, and new records
will be written with NULL values, empty strings, or dummy values.

This is generally not a problem. However, there may be cases where
users may want to avoid putting a table into this format.
For example, users may want to ensure that future UPDATE operations
after an ADD COLUMN will be performed in-place, to reduce write
amplification. (Instantly added columns are essentially always
variable-length.) Users might also want to avoid bugs similar to
MDEV-19916, or they may want to be able to export tables to
older versions of the server.

We will revive the previously deprecated option innodb_file_format,
with the following values:

* barracuda (0): do not allow instant add/drop/reorder,
to maintain format compatibility with MariaDB 10.x and MySQL 5.x

* strict_barracuda: Like 'barracuda', but on ALTER TABLE, if a table
(or partition) is not in the canonical format, force a table rebuild.

* append (default in 10.3): Store a hidden metadata record that
allows columns to be appended to the table (MDEV-11369);
compatible with MariaDB 10.3

* strict_append: Like 'append', but on ALTER TABLE, if a table
(or partition) is not in the 10.3-compatible format,
force the table to be rebuilt. (On 10.3, this is the same as 'append'.)

Starting with 10.4:

* mapped (default): Like 'append', but allow the metadata record to
store a column map, to support instant add/drop/reorder (MDEV-15562)
* strict_mapped: Like 'mapped', but on ALTER TABLE, if a table
(or partition) is not in the 10.4-compatible format,
force the table to be rebuilt. (On 10.4, this is the same as 'mapped'.)

Author: Elena Stepanova
Author Date: 2020-03-15 17:15:11 UTC

Testing buildbot changes for skipping upgrade on old branches

Author: Andrei Elkin
Author Date: 2020-03-08 19:52:30 UTC

MDEV-742: read-only, pure myisam, binlog-*, @@skip_log_bin corner cases

(Pushed to 10.5)
Are addressed along the following policies.
Prepared read-only, or on non-transactional engines, or
binlog-* filter in binlog, or skipped to binlog XA:s remains in the xid cache after disconnect.
But their consequent completion with Commit or Rollback differs.

1. The read-only at reconnect marks XID to fail for future
completion with ER_XA_RBROLLBACK.

2. `binlog-*` filtered XA when it changes engine data is regarded
as loggable even when nothing got cached for binlog.
An empty XA-prepare group is recorded. Consequent Commit-or-Rollback
succeeds in the Engine(s) as well as recorded into binlog.

3. The same applies to the non-transactional engine XA.

4. @@skip_log_bin=OFF does not record anything at XA-prepare (obviously),
but the completion event is recorded into binlog to admit
inconsistency with slave.

The following actions are taken by the patch.

At XA-prepare:
  when empty binlog cache - don't do anything to binlog if RO,
  otherwise write empty XA_prepare (assert(binlog-filter case)).
At Disconnect:
  when Prepared && RO (=> no binlogging was done)
    set Xid_cache_element::error := ER_XA_RBROLLBACK
    *keep* XID in the cache, and rollback the transaction.
At XA-"complete":
   Discover the error, if any don't binlog the "complete", return the error to the user.

RO patch review notes.

Author: varun
Author Date: 2020-03-12 09:11:03 UTC

MDEV-21922: Allow packing addon fields even if they don't honour max_length_for_sort_data

Addon fields will be packed if the length of addon fields is greater
than max_length_for_sort_data.

Author: Vladislav Vaintroub
Author Date: 2020-03-14 19:52:17 UTC

dynamic nr of Innodb io threads

Author: Sergei Golubchik
Author Date: 2020-03-13 17:15:57 UTC

MDEV-21599 plugins.server_audit fails sporadically in buildbot

Author: midenok
Author Date: 2020-03-13 12:31:55 UTC

MDEV-21471 ER_CRASHED_ON_USAGE upon UPDATE FOR PORTION on Aria table

Turn read cache off for periodic update.

Like 498a96a4 says:

Aria with row_format=fixed uses IO_CACHE of type READ_CACHE for
sequential read in update loop. When history row is inserted inside
this loop the cache misses it and fails with error.

This applicable to any additional row inserts on UPDATE. In this case
it was initiated by UPDATE FOR PORTION.

Related to MDEV-20441.

Author: Jan Lindström
Author Date: 2020-03-12 13:34:50 UTC

MDEV-21910 : KIlling thread on Galera could cause mutex deadlock

Following issues here:

Whenever Galera BF (brute force) transaction decides to abort conflicting
transaction it will kill that thread using thd::awake()

User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()

Whenever one of these actions is executed we will hold number of InnoDB
internal mutexes and thd mutexes. Sometimes these mutexes are taken in
different order causing mutex deadlock.

Lets consider a example starting from Galera BF transaction deciding to abort
conflicting transaction (let's call this thread1):

thread1:
lock_rec_other_has_conflicting
  here we hold both lock_sys mutex and trx mutex for conflicting lock transaction.
wsrep_innobase_kill_one_trx
  For victim we call wsrep_thd_awake

Next thread2 we assume user to execute KILL QUERY to some other executing
query (note it can't be BF query).

thread2:
sql_kill()
kill_one_thread
find_thread_by_id
  takes LOCK_thd_kill
thd->awake_no_mutex()

thread1:
thd->awake(KILL_QUERY)
  Tries to have LOCK_thd_kill but it is hold by thread2 so we wait (note that
  we still hold lock_sys mutex and trx mutex).

thread2:
ha_kill_query()
kill_handlerton
innobase_kill_query
lock_trx_handle_wait
  lock_mutex_enter() must wait as thread1 is holding it

Thus thread1 lock_sys, trx_mutex waits -> thread2 LOCK_thd_kill waits lock_sys -> thread1
==> thread1 waits -> thread2 waits -> thread1 ==> mutex deadlock.

In this patch we will fix Galera BF and user kill cases so that we enqueue
victim thread to a list while we hold InnoDB mutexes and we then release them.
A new background thread will pick victim thread from this new list and uses
thd::awake() with no InnoDB mutexes. Idea is similar to replication background
kill. This fix enforces that we take LOCK_thd_data -> lock sys mutex -> trx mutex
always in this order.

wsrep_mysqld.cc
 Here we introduce a list where victim threads are stored,
 condition variable to be used to wake up background thread
 and mutex to protect list.

wsrep_thd.cc
 Create a new background thread to handle victim thread
 abort. We may take wsrep_thd_LOCK mutex here but not any
 InnoDB mutexes.

wsrep_innobase_kill_one_trx
 Remove all the wsrep code that was moved to wsrep_thd.cc
 We just enqueue required information to background kill
 list and cancel victim trx lock wait if there is such.
 Here we have InnoDB lock sys mutex and trx mutex so here
 we can't take wsrep_thd_LOCK mutex.

wsrep_abort_transaction
 Cleanup only.

Author: Jan Lindström
Author Date: 2020-03-12 13:34:50 UTC

MDEV-21910 : KIlling thread on Galera could cause mutex deadlock

Following issues here:

Whenever Galera BF (brute force) transaction decides to abort conflicting
transaction it will kill that thread using thd::awake()

User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()

Whenever one of these actions is executed we will hold number of InnoDB
internal mutexes and thd mutexes. Sometimes these mutexes are taken in
different order causing mutex deadlock.

Lets consider a example starting from Galera BF transaction deciding to abort
conflicting transaction (let's call this thread1):

thread1:
lock_rec_other_has_conflicting
  here we hold both lock_sys mutex and trx mutex for conflicting lock transaction.
wsrep_innobase_kill_one_trx
  For victim we call wsrep_thd_awake

Next thread2 we assume user to execute KILL QUERY to some other executing
query (note it can't be BF query).

thread2:
sql_kill()
kill_one_thread
find_thread_by_id
  takes LOCK_thd_kill
thd->awake_no_mutex()

thread1:
thd->awake(KILL_QUERY)
  Tries to have LOCK_thd_kill but it is hold by thread2 so we wait (note that
  we still hold lock_sys mutex and trx mutex).

thread2:
ha_kill_query()
kill_handlerton
innobase_kill_query
lock_trx_handle_wait
  lock_mutex_enter() must wait as thread1 is holding it

Thus thread1 lock_sys, trx_mutex waits -> thread2 LOCK_thd_kill waits lock_sys -> thread1
==> thread1 waits -> thread2 waits -> thread1 ==> mutex deadlock.

In this patch we will fix Galera BF and user kill cases so that we enqueue
victim thread to a list while we hold InnoDB mutexes and we then release them.
A new background thread will pick victim thread from this new list and uses
thd::awake() with no InnoDB mutexes. Idea is similar to replication background
kill. This fix enforces that we take LOCK_thd_data -> lock sys mutex -> trx mutex
always in this order.

wsrep_mysqld.cc
 Here we introduce a list where victim threads are stored,
 condition variable to be used to wake up background thread
 and mutex to protect list.

wsrep_thd.cc
 Create a new background thread to handle victim thread
 abort. We may take wsrep_thd_LOCK mutex here but not any
 InnoDB mutexes.

wsrep_innobase_kill_one_trx
 Remove all the wsrep code that was moved to wsrep_thd.cc
 We just enqueue required information to background kill
 list and cancel victim trx lock wait if there is such.
 Here we have InnoDB lock sys mutex and trx mutex so here
 we can't take wsrep_thd_LOCK mutex.

wsrep_abort_transaction
 Cleanup only.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2020-03-12 17:54:47 UTC

- Fixing the fts failure hang during shutdown. Basically,
fts_optimize_callback() should free the fts_optimize_wq variable
and make it as null. It allows the subsequent fts_optimize_callback()
to return earlier.

Author: Sujatha Sivakumar
Author Date: 2020-03-12 12:36:27 UTC

MDEV-10047: Upmerge Testing.

Author: Sujatha Sivakumar
Author Date: 2020-03-12 12:36:27 UTC

MDEV-10047: Upmerge Testing.

Author: Sujatha Sivakumar
Author Date: 2020-03-12 12:17:54 UTC

MDEV-10047: table-based master info repository

Problem:
=======
When we upgrade from "mysql" to "mariadb" if slave is using repositories as
tables their data is completely ignored and no warning is issued in error log.

Fix:
===
"mysql_upgrade" test should check for the presence of data in
"mysql.slave_master_info" and "mysql.slave_relay_log_info" tables. When tables
have some data the upgrade script should report a warning which hints users
that the data in repository tables will be ignored.

Author: Eugene
Author Date: 2020-03-11 13:40:34 UTC

MDEV-21918 improve page_zip_verify_checksum()

actually, page_zip_verify_checksum() generally allows all-zeroes
checksums because our CRC32 checksum is something like
crc_1 ^ crc_2 ^ crc_3

Also, all zeroes page is considered correct.

As a side effect fix nasty reinterpret_cast<> UB

Also, since c0f47a4a5842 innodb_checksum_algorithm=full_crc32
exists which computes CRC32 in one go (without bitwise arithmetic)

Author: Oleksandr Byelkin
Author Date: 2020-03-11 13:28:24 UTC

Merge branch '10.3' into 10.4

Author: varun
Author Date: 2020-03-11 11:23:35 UTC

Pack addon fields always if we can save more than 8 bytes

Author: Julius Goryavsky
Author Date: 2020-03-11 11:07:06 UTC

Merge branch '10.5' of https://github.com/MariaDB/server into bb-10.5-MDEV-13362

Author: Julius Goryavsky
Author Date: 2020-03-02 22:46:07 UTC

MDEV-13362: implement --require_secure_transport option

Currently, if a user wants to require TLS for every connection made
over the network, then every user account on the system needs to be
created with "REQUIRE SSL" or one of the other TLS options.

Implementing a require_secure_transport system varuable (which,
in particular, can be set using the --require_secure_transport=ON
command line option) in the MariaDB Server would make it a lot
easier to require TLS (or other secure transport) system-wide.

This patch implements this new system variable, adds the ability
to set it with SQL statements, from the command line and from the
configuration file, and also contains improvements for mtr that allow
the user to establish non-secure TCP/IP connections (for example,
to verify the operation of the new option).

Author: Sachin Setiya
Author Date: 2020-03-10 15:47:34 UTC

States reduced

Author: Nikita Malyavin
Author Date: 2020-03-10 14:46:24 UTC

support NULL fields in key

Author: Sujatha Sivakumar
Author Date: 2020-03-10 10:24:55 UTC

MDEV-10047: table-based master info repository

Problem:
=======
When we upgrade from "mysql" to "mariadb" if slave is using repositories as
tables their data is completely ignored and no warning is issued in error log.

Fix:
===
"mysql_upgrade" test should check for the presence of data in
"mysql.slave_master_info" and "mysql.slave_relay_log_info" tables. When tables
have some data the upgrade script should report a warning which hints users
that the data in repository tables will be ignored.

Author: Sergey Petrunia
Author Date: 2020-03-10 08:23:43 UTC

Merge XPand Storage Engine (rebased)

Author: Sergey Petrunia
Author Date: 2020-03-09 19:43:59 UTC

XPand PR#73: Better behavior with offline cluster

Patches from Michael Erickson:
remove my_error and my_printf_error
make table discovery work
add more debug
attempt to fix dup key error

Author: Vladislav Vaintroub
Author Date: 2020-03-09 19:44:56 UTC

Fix comments in buf_read_page

Author: Andrei Elkin
Author Date: 2020-03-02 15:12:35 UTC

MDEV-21659 XA rollback foreign_xid is allowed inside active XA
MDEV-21854 xa commit `xid` one phase for already prepared transaction must always error out

Added state and one-phase option checks to XA "external" commit/rollback
branches. While the XA standard does not prohibit it,
Commit and Rollback of an XA external to the current ongoing transaction
is not allowed; after all the current transaction may rollback
to not being able to revert that decision.

Author: midenok
Author Date: 2018-02-20 12:35:09 UTC

MDEV-15724 - Possible crash in parser

Parser: uninitialized Lex->create_last_non_select_table under
mysql_unpack_partition() fix.

Tested with main, parts suites.

Author: Jan Lindström
Author Date: 2020-03-09 07:00:14 UTC

MDEV-21758 : Events switched randomly to SLAVESIDE_DISABLED

Change events only on Galera environment where idea is that
event is enabled only on one node of the cluster and nodes
are identified by server_id.

Author: Sergey Petrunia
Author Date: 2020-03-08 21:29:48 UTC

MDEV-21784: Performance testing for packed sort keys

A piece of cumulative patch for

  MDEV-21580 Allow packed sort keys in sort buffer

which just adds some virtual functions (without any calls to them)

Author: Vicențiu Ciorbaru
Author Date: 2020-03-08 14:19:43 UTC

Correctly link mysqlclient.pc to mariadb.pc under multi-arch support

Author: Vicențiu Ciorbaru
Author Date: 2020-03-07 17:21:40 UTC

Check for CPU_COUNT macro within my_getncpus

* Small refactor of my_getncpus function to compile for very old glibc < 2.6.
* Cleanup code to eliminate duplication.

Author: Anel Husakovic
Author Date: 2020-03-05 15:40:29 UTC

MDEV-21367: mysqld_safe log don't log to err.log

Introduced with 6b7918d524d5 in `10.2` just check for helper and handle
it if exist.

Author: Marko Mäkelä
Author Date: 2020-03-06 13:21:57 UTC

WIP clean up log upgrade

FIXME: innodb.log_corruption fails with result diff,
and fails to delete ib_logfile1. But it no longer crashes!

Author: Monty
Author Date: 2020-03-06 12:21:20 UTC

Second stage of optimizer_trace optimizations

- Move testing of my_writer to inline functions to avoid calls
- Made more functions inline. Especially thd->thread_started()
  is now very optimized!
- Moved Opt_trace_stmt classe to opt_trace_context.h to get critical
  functions inline

Author: Oleksandr Byelkin
Author Date: 2020-03-06 10:06:48 UTC

Merge branch '5.5' into 10.1

Author: Otto Kekäläinen
Author Date: 2020-03-04 17:09:16 UTC

Fix syntax error mysql-test-run in man page

This fixes errors like:

  $ LC_ALL=en_US.UTF-8 MANROFFSEQ='' MANWIDTH=80 \
    man --warnings -E UTF-8 -l -Tutf8 -Z mysql-test-run.pl.1 > /dev/null

  troff: <standard input>:246: warning [p 2, 6.0i, div '3tbd1,1', 0.3i]: can't break line
  troff: <standard input>:275: warning [p 2, 6.0i, div '3tbd6,1', 0.8i]: can't break line

Author: Otto Kekäläinen
Author Date: 2020-03-04 16:30:08 UTC

Fix various spelling errors

e.g.
- dont -> don't
- occurence -> occurrence
- succesfully -> successfully

Also remove trailing space in selected files.

These changes span:
- server core
- Connect and Innobase storage engine code
- OQgraph, Sphinx and TokuDB storage engines

Author: Daniel Black
Author Date: 2019-09-26 22:43:10 UTC

auth_gssapi: fix include path for Solaris

tested on:
$ uname -a
SunOS openindiana 5.11 illumos-b97b1727bc i86pc i386 i86pc

Author: Sujatha Sivakumar
Author Date: 2020-02-10 07:13:26 UTC

MDEV-21469: Implement crash-safe logging of the user XA

Description: Make XA PREPARE, XA COMMIT and XA ROLLBACK statements crash-safe.

Implementation:
In order to ensure consistent replication XA statements like XA PREPARE, XA
COMMIT and XA ROLLBACK are firstly written into the binary log and then to
storage engine. In a case server crashes after writing to binary log but not
in storage engine it will lead to inconsistent state.

In order to make both binary log and engine to be consistent crash recovery
needs to be initiated. During crash recovery binary log needs to be parsed to
identify the transactions which are present only in binary log and not present
in engine. These transaction are resubmitted to engine to make it consistent
along with binary log.

Author: varun
Author Date: 2020-03-03 08:13:59 UTC

Small correction to analyze output

Author: Vladislav Vaintroub
Author Date: 2020-02-07 21:12:35 UTC

MDEV-21534 - Improve innodb redo log group commit performance

Introduce special synchronization primitive group_commit_lock
for more efficient synchronization of redo log writing and flushing.

The goal is to reduce CPU consumption on log_write_up_to, to reduce
the spurious wakeups, and improve the throughput in write-intensive
benchmarks.

Author: Roman Nozdrin
Author Date: 2020-02-28 15:44:56 UTC

MDEV-21841 CONV() function doesn't truncate its output to 21 when uses default charset.