Author: Angelique Sklavounos
Author Date: 2022-12-02 01:46:45 UTC

MDEV-16951: This was caused by a race condition where the log rotation occurs before the Binlog_checkpoint is written to the binary log indicated in the error message. Fixed by ensuring Binlog_checkpoint is written before the binary log rotation.

Author: Alexey Botchkov
Author Date: 2022-11-29 11:10:51 UTC

m_mysql removed.

Author: midenok
Author Date: 2022-12-01 13:34:17 UTC

MDEV-30112 ASAN errors in Item_ident::print / generate_partition_syntax

Like in MDEV-16110 we must release items allocated on thd->mem_root by
reopening the table.

MDEV-16290 relocated MDEV-16110 fix in 10.5 so it works for MDEV-28576
as well. 10.3 without MDEV-16290 now duplicates this fix.

Author: Vicențiu Ciorbaru
Author Date: 2022-11-24 17:50:14 UTC

MDEV-30023 Revoking Privilege on the Column Yields the Error

The change from MDEV-29465 exposed a flaw in replace_column_table
where again we were not properly updating the column-level bits.

replace_table_table was changed in MDEV-29465 to properly update
grant_table->init_cols, however replace_column_table still only
modified grant_column->rights when the GRANT_COLUMN already existed.

This lead to a missmatch between GRANT_COLUMN::init_rights and
GRANT_COLUMN::rights, *if* the GRANT_COLUMN already existed.

As an example:

GRANT SELECT (col1) ...
Here:
For col1
GRANT_COLUMN::init_rights and GRANT_COLUMN::rights are set to 1 (SELECT) in
replace_column_table.

GRANT INSERT (col1) ...
Here, without this patch GRANT_COLUMN::init_rights is still 1 and
GRANT_COLUMN::rights is 3 (SELECT_PRIV | INSERT_PRIV)

Finally, if before this patch, one does:

REVOKE SELECT (col1) ...

replace_table_table will see that init_rights loses bit 1 thus it
considers there are no more rights granted on that particular table.

This prompts the whole GRANT_TABLE to be removed via the first revoke,
when the GRANT_COLUMN corresponding to it should still have init_rights == 2.

By also updating replace_column_table to keep init_rights in sync
properly, the issue is resolved.

Reviewed by <serg@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2022-11-30 18:59:38 UTC

MDEV-29981: Replica stops with 'Found invalid event in binary log'

Prototype commit for a primary to print certain event's
headers before sending to replica to facilitate debugging
a release build. Specifically, it will output the header
of a fake rotate event, the following format description
event, and the next event.

Author: Nikita Malyavin
Author Date: 2022-11-29 19:40:07 UTC

MDEV-29069 follow-up: improve DEFAULT rules

previously, fields with DEFAULTs were allowed just when expression is
deterministic. In case of online alter, we should recursively check that
underlying fields of expression also either have explicit values, or
have DEFAULT following this validity rule.

Author: Daniele Sciascia
Author Date: 2022-11-11 12:08:20 UTC

MDEV-29878 Galera test failure on MDEV-26575

Test MDEV-26575 fails when it runs after MDEV-25389. This is because
the latter simulates a failure while an applier thread is
created in `start_wsrep_THD()`. The failure was not handled correctly
and would not cleanup the created THD from the global
`server_threads`. A subsequent shutdown would hang and eventually fail
trying to close this THD.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Author: sjaakola
Author Date: 2022-11-08 14:36:34 UTC

MDEV-29512 deadlock between commit monitor and THD::LOCK_thd_data mutex

This commit contains only a mtr test for reproducing the issue in MDEV-29512
The actual fix will be pushed in wsrep-lib repository

The hanging in MDEV-29512 happens when binlog purging is attempted, and there is
one local BF aborted transaction waiting for commit monitor.

The test will launch two node cluster and enable binlogging with expire log days,
to force binlog purging to happen.
A local transaction is executed so that will become BF abort victim, and has advanced
to replication stage waiting for commit monitor for final cleanup (to mark position in innodb)
after that, applier is released to complete the BF abort and due to binlog configuration,
starting the binlog purging. This is where the hanging would occur, if code is buggy

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Author: Yuchen Pei
Author Date: 2022-11-29 05:46:53 UTC

MDEV-28891 Spider: remove #ifdef SPIDER_FIELD_FIELDPTR_REQUIRES_THDPTR

unifdef -DSPIDER_FIELD_FIELDPTR_REQUIRES_THDPTR -m
storage/spider/*.{cc,h}

and removing the definition manually.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>
Reviewed-by: Nayuta Yanagisawa <nayuta.yanagisawa@mariadb.com>

Author: Yuchen Pei
Author Date: 2022-11-29 01:08:27 UTC

MDEV-29269 Spider: remove #ifdef ITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC

Produced using the following command

unifdef -UITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC -m storage/spider/spd_*

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>
Reviewed-by: nayuta.yanagisawa@mariadb.com

Author: Monty
Author Date: 2022-11-28 16:37:55 UTC

Fixed warning from innodb.create_isl_with_direct if have_symlink is disabled

Author: Lena Startseva
Author Date: 2022-11-21 09:59:48 UTC

MDEV-29390: Improve coverage for UPDATE and DELETE statements in MTR test suites

For the update_use_source.test tests, data recovery in the table has been changed
from a rollback transaction to a complete delete and re-insert of the data with
optimaze table. Cases are now being checked on three engines.

Author: Julius Goryavsky
Author Date: 2022-11-22 11:11:49 UTC

hashicorp vault plugin: added statistics variables

Author: Sergey Petrunia
Author Date: 2022-11-21 14:28:43 UTC

MDEV-30059: Optimizer Trace: plan_prefix should be a comma-separated-list

Author: Sergey Petrunia
Author Date: 2022-11-21 14:28:43 UTC

MDEV-30059: Optimizer Trace: plan_prefix should be a comma-separated-list

Author: Monty
Author Date: 2022-11-14 15:08:09 UTC

Fixed bug in Aria with aria_log files that are exactly 8K

In the case one has an old Aria log file that ands with a Aria checkpoint
and the server restarts after next recovery, just after created a
new Aria log file (of 8K), the Aria recovery code would abort.
If one would try to delete all Aria log files after this (but not the
aria_control_file), the server would crash during recovery.

The problem was that translog_get_last_page_addr() would regard a log file
of exactly 8K as illegal and the rest of the code could not handle this
case.

Another issue was that if there was a crash directly after the log file
head was written to the next page, the code in translog_get_next_chunk()
would crash.

This patch fixes most of the issues, but not all. For Sanja to look at!

Things fixed:
- Added code to ignore 8K log files.
- Removed ASSERT in translog_get_next_chunk() that checks if page only
  contains the log page header.

Author: Monty
Author Date: 2022-11-15 16:55:39 UTC

fixed test result

Author: Marko Mäkelä
Author Date: 2022-11-15 06:59:30 UTC

Merge 10.6

Interesting test failures:

./mtr --parallel=auto --mysqld=--debug-dbug=d,ib_log_checkpoint_avoid \
--max-test-fail=0 --big-test --force \
binlog{,_encryption}.binlog_incident rpl.rpl_skip_incident \
binlog.binlog_killed_simulate rpl.rpl_loaddata_fatal

Author: Nayuta Yanagisawa
Author Date: 2022-11-11 16:47:23 UTC

MDEV-29947 Spider doesn't return all rows when doing a join of two tables with no usable keys

The present commit only has a MTR test case because the bug has been
fixed by reverting the commits that caused it.

Author: Marko Mäkelä
Author Date: 2022-11-08 11:49:55 UTC

MDEV-29504/MDEV-29849 TRUNCATE breaks FOREIGN KEY locking

ha_innobase::referenced_by_foreign_key(): Protect the check with
dict_sys.freeze(), to prevent races with TRUNCATE TABLE.
The test innodb.instant_alter_crash has been adjusted for this
additional locking.

dict_table_is_referenced_by_foreign_key(): Removed (merged to
the only caller).

create_table_info_t::create_table(): Ignore missing indexes for
FOREIGN KEY constraints if foreign_key_checks=0.

create_table_info_t::create_table_update_dict(): Rewritten as
a static function. Do not return any error.

ha_innobase::create(): When trx!=nullptr and we are operating
on a persistent table, do not rollback, commit, or release the
data dictionary latch.

ha_innobase::truncate(): Protect the entire critical section
with an exclusive dict_sys.latch, so that
ha_innobase::referenced_by_foreign_key() on referenced tables
will return a consistent result. In case of a failure,
invoke dict_load_foreigns() to restore also any FOREIGN KEY
constraints.

ha_innobase::free_foreign_key_create_info(): Define inline.

lock_release(): Disregard innodb_evict_tables_on_commit_debug=ON
when dict_sys.locked() holds. It would hold when fts_load_stopword()
is invoked by create_table_info_t::create_table_update_dict().

dict_sys_t::locked(): Return whether the current thread is holding
the exclusive dict_sys.latch.

dict_sys_t::frozen_not_locked(): Return whether any thread is
holding a shared dict_sys.latch.

Author: Angelique Sklavounos
Author Date: 2022-11-07 07:21:29 UTC

MDEV-25611: Dummy commit to test again

Author: Daniel Black
Author Date: 2022-11-07 03:13:46 UTC

service_encryption.h HAVE_ALLOC_H was global

Rather than global include for header files, we use
cmake to make service_encryption.h a generated file
and populate it with the discovery of ALLOCA_H or not.

Fixes: #2289
Closes: #2321

Author: Georg Richter
Author Date: 2022-11-06 14:03:04 UTC

Updated libmariadb

Author: midenok
Author Date: 2022-10-30 23:03:34 UTC

MDEV-29831 Galera crashes when running CoR for a locked table after
    setting the minimum memory for a user session.

Failure happens when finalize_atomic_replace() was already finished
and we removed the table from locked tables list.
finalize_locked_tables() doesn't know about that, it doesn't add back
last deleted lock because operation_failed == true.
reopen_tables() doesn't reopen table and as a result we have NULL in
pos_in_locked_tables->table.

The fix adds the knowledge that the locked_tables_count changed since
the start of the command. And if that happened we
add_back_last_deleted_lock(). That makes MDEV-29544 fix with
locked_tables_decremented deprecated.

Alternative fix would add atomic_replace_finished to Atomic_info and
updated it on successful finalize_atomic_replace(). Then the condition
would look like this:

  if (atomic_replace_finished || !operation_failed)
  {
    /*
      Add back the deleted table and re-created table as a locked table
      This should always work as we have a meta lock on the table.
    */
    thd->locked_tables_list.add_back_last_deleted_lock(pos_in_locked_tables);
  }

Author: Marko Mäkelä
Author Date: 2022-10-31 07:38:59 UTC

MDEV-16232 preparation: Disable row_prebuilt_t::fetch_cache

The minimal change to disable the use of the fetch_cache is the 2 hunks
#if 1/#else/#endif in row_search_mvcc(). The rest is disabling dead code
and data.

Author: Vlad Lesin
Author Date: 2022-10-27 20:35:34 UTC

MDEV-28709 unexpected X lock on Supremum in READ COMMITTED

Post-push fix. The flag of transaction which indicates that it's necessary
to forbid gap lock inheritance after XA PREPARE could be inverted if
lock_release_on_prepare_try() is invoked several times. The fix is to
toggle it on lock_release_on_prepare() exit.

Author: Alexey Botchkov
Author Date: 2022-10-25 11:39:03 UTC

MDEV-29800 MDEV-29800 ERROR 1062 (23000): Duplicate entry 'test-t-0' for key 'PRIMARY' on CoR...

The CREATE TABLE command can appear in the ha_spider::rename_table(),
so should be handled.

Author: Daniel Black
Author Date: 2022-10-27 06:29:11 UTC

MDEV-24377: Fixed review nits

Author: Alexander Barkov
Author Date: 2022-10-26 10:26:21 UTC

MDEV-9777 MyISAM and InnoDB work differently when comparing a TIME column to an empty string

This problem was fixed earlier (presumably by MDEV-9604 in 10.1).
Adding MTR tests only.

Author: Igor Babaev
Author Date: 2022-10-26 01:07:27 UTC

MDEV-28846 Poor performance when rowid filter contains no elements

When a range rowid filter was used with an index ref access the cost of
accessing the index entries for the records rejected by the filter was not
taken into account. For a ref access by an index with big average number
of records per key this led to poor execution plans if selectivity of the
used filter was high.
The patch resolves this problem. It also introduces a minor optimization
that skips look-ups into a filter that turns out to be empty.
With this patch the output of ANALYZE stmt reports the number of look-ups
into used rowid filters.

The test cases that were supposed to use rowid filters have been adjusted
in order to use similar execution plans after this fix.

This patch prepared for 10.5 differs from the patch resolving the problem
for 10.4 because he code that calculates the cost of index only access has
been changed for 10.5 making usage of rowid filters more favorable.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Alexey Botchkov
Author Date: 2022-10-25 11:39:03 UTC

MDEV-29800 MDEV-29800 ERROR 1062 (23000): Duplicate entry 'test-t-0' for key 'PRIMARY' on CoR...

The CREATE TABLE command can appear in the ha_spider::rename_table(),
so should be handled.

Author: Daniel Black
Author Date: 2022-10-24 23:45:57 UTC

MDEV-XXX make my_free mark memory as MEM_UNDEFINED

Allow MSAN/Valgrind to find memory that is reused without
initialization.

Author: Nayuta Yanagisawa
Author Date: 2022-09-27 06:22:57 UTC

MDEV-29644 a potential bug of null pointer dereference in spider_db_mbase::print_warnings()

The function spider_db_mbase::print_warnings() can potentially result
in a null pointer dereference.

Remove the null pointer dereference by cleaning up the function.

Author: Oleg Smirnov
Author Date: 2022-10-23 13:48:43 UTC

MDEV-29640 FederatedX does not properly handle pushdown in case of difference in local and remote table names

FederatedX table on the local server may refer to a table having
another name on the remote server. The remote table may even reside
in a different database. For example:

-- Remote server
CREATE TABLE t1 (id int(32));

-- Local server
CREATE TABLE t2 ENGINE="FEDERATEDX"
CONNECTION="mysql://joe:joespass@192.168.1.111:9308/federatedx/t1";

It's not a problem while the federated_pushdown is disabled 'cause
the CONNECTION strings are being parsed for every table during
the execution, so the table names are translated from local to remote.
But in case of the federated_pushdown the whole query is pushed down
to the engine without any translation, so the remote server may try
to select data from a nonexistent table (for example, query
"SELECT * FROM t2" will try to retrieve data from nonexistent "t2").

Solution: do not allow pushing down queries with tables having
different names on local and remote names.

Author: Daniel Black
Author Date: 2022-10-19 07:07:17 UTC

Deb: remove lsb_release dep and use os-release

If nothing is there, assume its sid, because the
base debian configuration is written for the latest
sid.

Author: Igor Babaev
Author Date: 2022-10-19 02:11:47 UTC

MDEV-7487 Semi-join optimization for single-table update/delete statements

This patch allows to use semi-join optimization at the top level of
single-table update and delete statements.
The problem of supporting such optimization became easy to resolve after
processing a single-table update/delete statement started using JOIN
structure. This allowed to use JOIN::prepare() not only for multi-table
updates/deletes but for single-table ones as well. This was done in the
patch for mdev-28883:
Re-design the upper level of handling UPDATE and DELETE statements.

Note that JOIN::prepare() detects all subqueries that can be considered
as candidates for semi-join optimization. The code added by this patch
looks for such candidates at the top level and if such candidates are found
in the processed single-table update/delete the statement is handled in
the same way as a multi-table update/delete.

    Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Igor Babaev
Author Date: 2022-10-19 00:19:21 UTC

MDEV-7487 Semi-join optimization for single-table update/delete statements

This patch allows to use semi-join optimization at the top level of
single-table update and delete statements.
The problem of supporting such optimization became easy to resolve after
processing a single-table update/delete statement started using JOIN
structure. This allowed to use JOIN::prepare() not only for multi-table
updates/deletes but for single-table ones as well. This was done in the
patch for mdev-28883:
Re-design the upper level of handling UPDATE and DELETE statements.

Note that JOIN::prepare() detects all subqueries that can be considered
as candidates for semi-join optimization. The code added by this patch
looks for such candidates at the top level and if such candidates are found
in the processed single-table update/delete the statement is handled in
the same way as a multi-table update/delete.

    Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Monty
Author Date: 2022-10-18 16:57:23 UTC

Disable valgrind for rpl_start_alter_restart_master

This is needed as the test does restarts, which doesn't work that well
with valgrind.

Author: Julius Goryavsky
Author Date: 2022-10-18 03:42:43 UTC

Revert "MDEV-27682: bundled wsrep_notify.sh causes mariadbd to freeze during start"

This reverts commit 19f0b96d53dec47d7b8680c44997afba2ed7431e.

Author: midenok
Author Date: 2022-10-17 21:48:10 UTC

ddl_log_link_chains() improved

ddl_log_link_chains() now can operate on empty master chain. If the
master chain is empty it first writes empty execute entry to
it. Usable when you need to link the chains before start filling the
master chain. ddl_log_execute_recovery() now can skip empty execute
entries.

Author: midenok
Author Date: 2022-10-17 00:15:19 UTC

Fix

Author: Nikita Malyavin
Author Date: 2022-10-13 15:08:48 UTC

few rgi assertions. this can proof that rgi is always present

Author: Vlad Lesin
Author Date: 2022-10-13 13:00:58 UTC

MDEV-28709 unexpected X lock on Supremum in READ COMMITTED

Forbid gap lock inheritance if XA was prepared.

Use the most significant bit of trx_t::n_ref to indicate that gap lock
inheritance is forbidden.

Based on mysql/mysql-server@b063e52a8367dc9d5ed418e7f6d96400867e9f43

Author: Sergei Golubchik
Author Date: 2022-10-13 21:06:17 UTC

fixup! MDEV-28943

MDEV-29056 Replica SQL thread stops with 1846 error on ALTER ONLINE after LOCK WRITE

every ALTER TABLE variant that succeeds without LOCK TABLE
must succeed under LOCK TABLE

every ALTER TABLE variant that fails without LOCK TABLE
must fail under LOCK TABLE

Author: Daniel Black
Author Date: 2022-10-11 23:42:54 UTC

MDEV-29678 Valgrind/MSAN uninitialised value errors upon PS with ALTER under ONLY_FULL_GROUP_BY

st_select_lex::init_query is called in the exectuion of EXECUTE
IMMEDIATE 'alter table ...'. so reset the initialization at the
same point we set join= 0.

Author: Dmitry Shulga
Author Date: 2022-10-11 11:22:40 UTC

MDEV-16128: Server crash in Item_func::print_op on 2nd execution of PS

For some queries that involve tables with different but convertible
character sets for columns taking part in the query, repeatable
execution of such queries in PS mode or as part of a stored routine
would result in server abnormal termination.

For example,
  CREATE TABLE t1 (a2 varchar(10));
  CREATE TABLE t2 (u1 varchar(10) CHARACTER SET utf8);
  CREATE TABLE t3 (u2 varchar(10) CHARACTER SET utf8);
  PREPARE stmt FROM
    "SELECT t1.* FROM (t1 JOIN t2 ON (t2.u1 = t1.a2))
     WHERE (EXISTS (SELECT 1 FROM t3 WHERE t3.u2 = t1.a2))";

  EXECUTE stmt;
  EXECUTE stmt; <== Running this prepared statement the second time
                    results in server crash.

The reason of server crash is that an instance of the class
Item_func_conv_charset, that created for conversion of a column
from one character set to another, is allocated on execution
memory root but pointer to this instance is stored in an item
placed on prepared statement memory root. Below is calls trace to
the place where an instance of the class Item_func_conv_charset
is created.

setup_conds
 Item_func::fix_fields
  Item_bool_rowready_func2::fix_length_and_dec
   Item_func::setup_args_and_comparator
    Item_func_or_sum::agg_arg_charsets_for_comparison
     Item_func_or_sum::agg_arg_charsets
      Item_func_or_sum::agg_item_set_converter
       Item::safe_charset_converter

And the following trace shows the place where a pointer to
the instance of the class Item_func_conv_charset is passed
to the class Item_func_eq, that is created on a memory root of
the prepared statement.

Prepared_statement::execute
 mysql_execute_command
  execute_sqlcom_select
   handle_select
    mysql_select
     JOIN::optimize
      JOIN::optimize_inner
       convert_join_subqueries_to_semijoins
        convert_subq_to_sj

To fix the issue, switch to the Prepared Statement memory root
before calling the method Item_func::setup_args_and_comparator
in order to place any created Items on permanent memory root.
It may seem that such approach would result in a memory
leakage in case the parameter marker '?' is used in the query
as in the following example
  PREPARE stmt FROM
    "SELECT t1.* FROM (t1 JOIN t2 ON (t2.u1 = t1.a2))
     WHERE (EXISTS (SELECT 1 FROM t3 WHERE t3.u2 = ?))";
  EXECUTE stmt USING convert('A' using latin1);
but it wouldn't since for such case any of the parameter markers
is treated as a constant and no subquery to semijoin optimization
is performed.

Author: Jan Lindström
Author Date: 2022-10-11 06:39:33 UTC

MDEV-29602 : Galera debug build crashes when the spider plugin is enabled

Spider system tables should be created so that wsrep_on=OFF.

Author: Nayuta Yanagisawa
Author Date: 2022-04-25 09:29:01 UTC

MDEV-27233 Server hangs when using --init-file which loads Spider and creates a Spider table

Spider waits until the server initialization has been completed
(i.e., mysqld_server_started has been set to 1) before creating the
Spider system tables. Spider also wait until the system tables have
been created before instantiating ha_spider.

These waits leads to dead lock when a non-system Spider table is
created by --in-file. This is because queries passed by --in-file
are executed during the server initialization and creating the
non-system Spider causes an instantiation of ha_spider.

In the first place, there seems to be no need for Spider to do such
a synchronization. Thus, we simply remove the synchronization.

Author: Vladislav Vaintroub
Author Date: 2022-09-30 13:43:36 UTC

MDEV-29251 Set output directories during compilation

..to create s directory layout that closely resembles zip/tarball/MSI
install.

If we call ${CMAKE_BINARY_DIR} rsp CMAKE_BINARY_DIR/$<CONFIG>
$build_dir , then the compilation will place executables into
$build_dir/bin, and plugins into $build_dir/lib/plugins

Immediate benefits are
- searching, copying or symlinking of plugins in mtr is no more necessary
- searching for executables in mtr is simplified.
- Windows mariadbd.exe will to automatically set correct default
plugin directory (it had always been relative to location of the exe itself)

Other changes
"initial_database" target will create datadir $buildroot/data
directory, instead of $buildroot/sql/data, to still allow
server on Windows to start by just typing <path-to-mysqld> --console in
the shell.

Author: Monty
Author Date: 2022-10-04 13:16:06 UTC

Added test cases for preceding test

This includes all test changes from
"Changing all cost calculation to be given in milliseconds"
and forwards

Author: Sergey Petrunia
Author Date: 2022-10-04 09:12:20 UTC

Fix compile warning-as-error on amd64-ubuntu-1804-clang10-asan

Author: Lena Startseva
Author Date: 2022-10-03 16:35:49 UTC

    MDEV-29390: Improve coverage for UPDATE and DELETE statements in MTR test suites

    Added "explain" for queries and checks for engines

Author: Vlad Lesin
Author Date: 2022-09-30 15:13:21 UTC

MDEV-29575 Access to innodb_trx, innodb_locks and innodb_lock_waits along with detached XA's can cause SIGSEGV

trx->mysql_thd can be zeroed-out between thd_get_thread_id() and
thd_query_safe() calls in fill_trx_row(). trx_disconnect_prepared()
zeroes out trx->mysql_thd. And this can cause null pointer dereferencing
in fill_trx_row().

fill_trx_row() is invoked from fetch_data_into_cache(), which, in turns,
iterates transactions with trx_sys.trx_list.for_each() function, which
holds trx_sys.trx_list.mutex during the iteration.

Bug fix is in reseting trx_t::mysql_thd in trx_disconnect_prepared()
under trx_sys.trx_list.mutex lock.

MTR test case can't be created for the fix, as we need to wait for
trx_t::mysql_thd reseting in fill_trx_row() after trx_t::mysql_thd was
checked for null while trx_sys.trx_list.mutex is held. But
trx_t::mysql_thd must be reset in trx_disconnect_prepared() under
trx_sys.trx_list.mutex. There will be deadlock.

Author: Oleg Smirnov
Author Date: 2022-10-01 15:37:25 UTC

MDEV-29624 Fix memory leak on pushdown of derived table

Deallocation of TABLE_LIST::dt_handler and TABLE_LIST::pushdown_derived
was performed in multiple places if code. This not only made the code
more difficult to maintain but also lead to memory leaks when
dt_handler and pushdown_derived were allocated but actual pushdown
of the derived table did not take place due to the whole embedding
unit pushdown.
This commit puts deallocation of TABLE_LIST::dt_handler and
TABLE_LIST::pushdown_derived to the single point - JOIN::cleanup().

Author: Sergey Petrunia
Author Date: 2022-10-01 12:41:04 UTC

MDEV-29677: Wrong result with join query and innodb fulltext search

InnoDB FTS scan was used by a subquery. A subquery execution may start
a table read and continue until it finds the first matching record
combination. This can happen before the table read returns EOF.

The next time the subquery is executed, it will start another table read.
InnoDB FTS table read fails to re-initialize its data structures in this
scenario and will try to continue the scan started at the first execution.

Fixed by ha_innobase::ft_init() to stop the FTS scan if there is one.

Additional complication is the testcase: The original testcase used
the query plan with DEPENDENT SUBQUERY, but then Exists-to-IN optimization
was pushed which changed the test query's query plan to use MATERIALIZED.

The query plan with MATERIALIZED didn't hit the bug. I've adjusted the
testcase to use the QEP with DEPENDENT SUBQUERY instead.

Author: Brandon Nesterenko
Author Date: 2022-09-29 19:40:51 UTC

MDEV-18798: Remove trailing whitespaces

Author: Nayuta Yanagisawa
Author Date: 2022-09-06 13:35:03 UTC

MDEV-29421 Thread (10.6+) and server hangs (10.4/10.5) in 'Opening tables' (on optimized builds) and SIGABRT in safe_mutex_lock (on debug) on I_S read when using Spider

In spider_free_share(), Spider acquires the mutex, spider_open_tables,
and then makes statistics of the target table persistent.

When table_open_cache is small, the statistics persistence can lead to
another call of spider_free_share() and this results in the double
lock on the mutex.

The mutex spider_tbl_mutex is acquired unnecessarily long. Shortening
the locking period resolves the problem.

Author: Julius Goryavsky
Author Date: 2022-09-20 12:24:05 UTC

MDEV-28669: processing for recursive subdirectories

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-07-04 13:23:01 UTC

MDEV-29022 add_slave destroy child list and has dead code

Nowdays subquery in a UNION's ORDER BY placed correctly in fake select,
the only problem was incorrect Name_resolution_contect is fixed by this
patch in parsing, so we do not need scanning/reseting of ORDER BY of
a union.

(in 10.4 all work in parser correctly)

Author: Nayuta Yanagisawa
Author Date: 2022-09-26 14:17:55 UTC

MDEV-26852 Spider: -Werror=maybe-uninitialized raises on spd_sys_table.cc and ha_spider.cc

The warning "maybe-uninitialized" raises on spd_sys_table.cc and
ha_spider.cc when the sever is compiled with CMAKE_CXX_FLAGS=-Og.

Author: Nayuta Yanagisawa
Author Date: 2022-09-20 12:28:19 UTC

MDEV-24283 Assertion `bitmap_is_set(&m_part_info->read_partitions, m_part_spec.start_part)' failed in ha_partition::handle_ordered_index_scan

On the optimization of MAX/MIN clauses by opt_sum_query(), the
optimizer tries to access a partitioned table, even though all the
partitions of the table have been pruned away in a earlier stage.
This leads to the assertion error.

We stop the optimization by opt_sum_query() if all the partitions
were pruned way.

Author: Oleg Smirnov
Author Date: 2022-09-19 14:36:09 UTC

MDEV-29104 Optimize queries to INFORMATION_SCHEMA.PARAMETERS/ROUTINES

For queries like
    "SELECT * FROM INFORMATION_SCHEMA.PARAMETERS
      WHERE SPECIFIC_NAME='proc_name'"
and
    "SELECT * FROM INFORMATION_SCHEMA.ROUTINES
      WHERE ROUTINE_NAME='proc_name'"
there is a possibility to avoid loading a stored procedure code and
parsing it to retrieve parameters.
If the name of the procedure/function is specified explicitly then
it is possible to filter out routines that do not match at
an early stage.

Author: Dmitry Shulga
Author Date: 2022-09-20 14:44:41 UTC

MDEV-5816: Stored programs: validation of stored program statements

This is the prerequisite patch introducing the class sp_lex_instr
that encapsulates access to an instance of the class sp_lex_keeper.
Every SP instruction that need to get access to a LEX object on its
processing should inherit this class and implement two abstract methods:
  is_invalid(),
  invalidate().

These methods will be used in subsequent patches to implement recompilation of
SP instructions on failure.

Currently, the following instructions are derived from the class sp_lex_instr:
  sp_instr_stmt,
  sp_instr_set,
  sp_instr_set_trigger_field,
  sp_instr_jump_if_not,
  sp_instr_freturn,
  sp_instr_cpush,
  sp_instr_cursor_copy_struct,
  sp_instr_set_case_expr

Additionally, this patch converts the class sp_instr_opt_meta
to base abstract class (that is, not inherited from the class
sp_instr). Since this class originally was designed to provide a way
for opimizer to update a destination address for jump SP-instructions,
the only useful method at the interface of this class is set_destination
and therefore inheritance from the class sp_instr is meaningless.

Every jump SP instruction now must be inhereted directly from
the class sp_instr_opt_meta and additionally from either the class
sp_lex_instr or sp_instr depending on whether this SP instruction
need to get access to a LEX object or not.

Moreover, the class sp_cursor doesn't own a data member of
the class sp_lex_keeper any more. Instead, the virtual method
get_lex_keeper() has been added to the class sp_cursor() that
returns nullptr and this method is overridden in the derived class
sp_instr_cpush to provide a pointer to a real instance of
the class sp_lex_keeper. Doing this way we exclude duplication
of a data member of the type sp_lex_keeper at the class sp_instr_cpush
since it is derived both from sp_lex_instr and sp_cursor, and sp_lex_instr
already encapsulates a data member of the class sp_lex_keeper.

Author: Vicențiu Ciorbaru
Author Date: 2022-09-12 13:37:52 UTC

MDEV-14443: Ignore denies privilege now takes effect

Author: Marko Mäkelä
Author Date: 2022-09-19 12:30:09 UTC

WIP MDEV-22718: purge_sys.low_limit_no() is not protected

purge_sys_t::low_limit_no(): Adjust a comment. Actually, this
is protected after all.

TrxUndoRsegsIterator::set_next(): Reduce the critical section
of purge_sys.rseg->latch. Some purge_sys fields are accessed
only by the purge coordinator task.

Author: Nayuta Yanagisawa
Author Date: 2022-09-19 05:58:55 UTC

MDEV-28522 Delete constant SPIDER_SQL_TYPE_*_HS

The HandlerSocket support of Spider has been deleted by MDEV-26858.
Thus, the constants, SPIDER_SQL_TYPE_*_HS, are no longer necessary.

Author: Daniel Black
Author Date: 2022-09-19 00:37:26 UTC

MDEV-16194 stack not resolve (addr2line)

Remove the offset when invoking addr2line as tested by
Sergey Vojtovich.

Author: Igor Babaev
Author Date: 2022-09-16 21:59:56 UTC

MDEV-7487 Semi-join optimization for single-table update/delete statements

This patch allows to use semi-join optimization at the top level of
single-table update and delete statements.
The problem of supporting such optimization became easy to resolve after
processing a single-table update/delete statement started using JOIN
structure. This allowed to use JOIN::prepare() not only for multi-table
updates/deletes but for single-table ones as well. This was done in the
patch for mdev-28883:
Re-design the upper level of handling UPDATE and DELETE statements.

Note that JOIN::prepare() detects all subqueries that can be considered
as candidates for semi-join optimization. The code added by this patch
looks for such candidates at the top level and if such candidates are found
in the processed single-table update/delete the statement is handled in
the same way as a multi-table update/delete.

    Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Igor Babaev
Author Date: 2022-09-17 05:36:22 UTC

Another fix after the latest rebase of commits for MDEV-28883

Author: Oleg Smirnov
Author Date: 2022-09-14 11:11:19 UTC

MDEV-29535 Cleanup error handling in ha_federatedx

1. Class ha_federatedx_derived_handler overrides virtual function
derived_handler::print_error() with an empty implementation which
causes problems with error handling. Errors are being simply ignored
and occur during later processing.

2. Federatedx_handler_base::share can be NULL on unsuccessful attempt
of initialization, this must be checked.

Author: Vladislav Vaintroub
Author Date: 2022-08-25 22:38:00 UTC

MDEV-26715 Fix mysql_install_db_win tests.

It looks like C/C reconnect retains the database. It is correct behavior
However there is a problem that mysql_install_db.exe does not create
'test' database, thus using mysql_install_db to create datadir,
then restarting server and reconnecting does not succeed, if 'test' was
current db before reconnect.

The problem became apparent with MDEV-26715. Why it worked before,
with a simpler 'root' definition is a mystery.

Fix is to switch current connection to 'mysql' database, prior to
reconnect to the instance created via mysql_install_db.exe

Author: Lawrin Novitsky
Author Date: 2022-09-13 14:18:29 UTC

MDEV-29514 Extending plugins sql service with mysql_options,

mysql_fetch_lenghts, mysql_set_character_set, mysql_num_fields and
mysql_select_db.
sql service version has been changed to 0x0101
ABI files updated.

Author: Marko Mäkelä
Author Date: 2022-09-13 05:40:05 UTC

Merge 10.5 into 10.6

Author: Angelique Sklavounos
Author Date: 2022-09-12 23:17:52 UTC

MDEV-26633, MDEV-27950: intermediate commit to test changes

Author: Lena Startseva
Author Date: 2022-06-09 03:32:51 UTC

MDEV-27691: make working view-protocol

Tests with checking metadata or that cannot be run with
the view-protocol are excluded from --view-protocol.
For tests that do not allow the use of an additional connection,
the util connection is disabled with "--disable_service_connection".
Also cases with bugs for --view-protocol are disabled.

Author: Michael Widenius
Author Date: 2022-08-21 02:53:54 UTC

Added optimizer_costs.h which includes all optimizer costs

This makes it easier to see how costs changes over commits

Author: Michael Widenius
Author Date: 2022-08-21 02:53:54 UTC

Added optimizer_costs.h which includes all optimizer costs

This makes it easier to see how costs changes over commits

Author: Andrei
Author Date: 2022-08-16 17:03:15 UTC

MDEV-24660 MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0'

The shutdown time assert was caused by untimely deactivation of
the binlog background thread and related structs destruction.
It could specifically occur when a transaction is replication unsafe
and has to be completed with a ROLLBACK event in binlog.

This gets fixed with the binlog background thread stop relocation
to a point and user transactions have been completed.
A test case is added to binlog.binlog_checkpoint which
also receives as a bonus a minor correction to reactivate a MDEV-4322 test
case that originally required a shutdown phase (that ceased to do).

Author: Alexander Barkov
Author Date: 2022-09-07 06:50:40 UTC

More UCA collation performance improvements

Author: Vlad Lesin
Author Date: 2022-09-05 17:03:00 UTC

Log waiting and waited transaction info if there was lock wait timeout
and the waiting transaction is in prepared state and executed by slave
thread.

The commit is for custom build. If you want to push it to trunk, wrap
the code with "#ifdef UNIV_DEBUG". The same is true for the previous
commit.

Author: Marko Mäkelä
Author Date: 2022-09-06 09:44:58 UTC

MDEV-29440 InnoDB instant ALTER TABLE recovery must use READ UNCOMMITTED

In commit 8f8ba758559e473f643baa0a0601d321c42517b9 (MDEV-27234)
the data dictionary recovery was changed to use READ COMMITTED
so that table-rebuild operations (OPTIMIZE TABLE, TRUNCATE TABLE,
some forms of ALTER TABLE) would be recovered correctly.

However, for operations that avoid a table rebuild thanks to
being able to instantly ADD, DROP or reorder columns, recovery
must use the READ UNCOMMITTED isolation level so that changes to
the hidden metadata record can be rolled back.

We will detect instant operations by detecting uncommitted changes
to SYS_COLUMNS in case there is no uncommitted change of SYS_TABLES.ID
for the table. In any table-rebuilding DDL operation, the SYS_TABLES.ID
(and likely also the table name) will be updated.

As part of rolling back the instant ALTER TABLE operation, after the
operation on the hidden metadata record has been rolled back, a rollback
of an INSERT into SYS_COLUMNS in row_undo_ins_remove_clust_rec() will
invoke trx_t::evict_table() to discard the READ UNCOMMITTED definition
of the table. After that, subsequent recovery steps will load and use
the correct table definition.

Author: tanruixiang
Author Date: 2022-06-16 07:05:35 UTC

MDEV-26182: Implement JSON_INTERSECT()

First we introduce the compare_whole parameter. The intersection results of
different types of values vary according to compare_whole.
If compare_whole is true (When the array or object or scalar is not at the
outermost layer, compare_whole is true.):
  it means that the two jsons being compared, whether they are scalars, arrays
  or objects, must be exactly equal to have an intersection. Different types
  indicate that there is no intersection between jsons. If two scalars are
  compared, there is an intersection between them, which means that the two
  scalars are exactly the same. If two arrays are compared, there is an
  intersection between them, which means that the values at all indices of the
  two arrays are equal. If two objects are compared, there is an intersection
  between them, which means that all KV pairs of the two objects are equal.
if compare_whole is false:
  When a value is a subset of another value, there is an intersection. For
  example, taking the intersection of two objects is taking their common KV
  pairs. The intersection of array and array is to take their common elements.
  An object and a scalar take an intersection with an array. As long as the
  object or scalar exists in the array, the intersection is the object or the
  scalar.

We put a json into the hash, scan another json, and efficiently get the
intersection by reading and updating the hash information.

Author: Dmitry Shulga
Author Date: 2022-09-01 09:52:41 UTC

MDEV-5816: Stored programs: validation of stored program statements

Follow-up patch that fixes existing mtr tests.

Author: Vlad Lesin
Author Date: 2022-09-01 06:55:49 UTC

MDEV-29433 innodb.lock_delete_updated is unstable

Use suspend thread syncpoint instead of include/wait_condition.inc to
make sure DELETE created waiting lock before the next UPDATE begins
locking.

Author: Vlad Lesin
Author Date: 2022-09-01 06:55:49 UTC

MDEV-29433 innodb.lock_delete_updated is unstable

Use suspend thread syncpoint instead of include/wait_condition.inc to
make sure DELETE created waiting lock before the next UPDATE begins
locking.

Author: Sergey Petrunia
Author Date: 2022-08-29 20:12:27 UTC

MDEV-29382: Query returns wrong number of records

The issue occurred when DuplicateWeedout optimization was applied
together with Join Buffer, and also there was an SJ-Materialization-lookup
table in the join prefix.

DuplicateWeedout optimization includes the rowid of SJ-Materialization
temptable into its Duplicate checks.
However, Join Buffering code did not save/restore the rowids of
SJ-Materialization temptables. This meant the check could miss some
duplicates.

The fix makes the Join Buffering code to save/restore rowids for
SJ-Materialization temp.tables.

Author: Marko Mäkelä
Author Date: 2022-08-29 12:14:59 UTC

MDEV-27983: InnoDB hangs after loading a ROW_FORMAT=COMPRESSED page

If multiple threads invoke buf_page_get_low() on a ROW_FORMAT=COMPRESSED
page that does not reside in the buffer pool, then one of the threads
will end up acquiring an exclusive page latch (the "if" statement
right before the new wait_for_unzip: label) and other threads will
end up waiting for a shared latch while holding a buffer-fix.
The exclusive latch holder would then wait for the buffer-fixes to
be released while the buffer-fix holders are waiting for the shared latch.

buf_page_get_low(): Prevent the hang that was introduced
in commit 9436c778c3adba7c29dab5649668433d71e086f2 (MDEV-27058),
by releasing the buffer-fix, sleeping some time, and retrying the
page lookup.

Author: Elena Stepanova
Author Date: 2022-08-28 18:23:28 UTC

MDEV-29403 innodb.innodb_sys_semaphore_waits fails with wrong errno 5014

take into account C/C specific CR_ERR_NET_WRITE error

Author: Marko Mäkelä
Author Date: 2022-08-26 12:18:11 UTC

MDEV-28800 WIP: Avoid crashes on memory allocation failure

FIXME: Allocate locks upfront for page split or reorganize,
so that the operation can gracefully fail before any irreversible
persistent changes are performed. This affects lock_move_reorganize_page(),
lock_move_rec_list_end(), lock_move_rec_list_start(),
btr_root_raise_and_insert(), btr_insert_into_right_sibling(),
btr_page_split_and_insert().

buf_block_alloc(): Remove. This was an alias of
buf_LRU_get_free_block(false). Let us call that function directly.

buf_LRU_get_free_block(), buf_buddy_alloc_low(), buf_buddy_alloc():
If there is no free block in the buffer pool, return nullptr.

recv_sys_t::recover_low(), recv_sys_t::recover(): Return an error code,
which may be DB_OUT_OF_MEMORY.

lock_rec_create_low(): Return nullptr if the lock table is full.
This will be the only caller of buf_pool.running_out().

btr_search_check_free_space_in_heap(): Replaced with
btr_search_lock_and_alloc().

Author: Daniel Black
Author Date: 2022-07-29 14:11:08 UTC

MDEV-23607 MariaBackup - align required GRANTS to cmd options

Since the 10.5 split of the privileges, the required GRANTs
for various mariabackup operations has changed.

In the addition of tests, a number of mappings where incorrect:

The option --lock-ddl-per-table didn't require connection admin.

The option --safe-slave-backup requires SLAVE MONITOR even without
the --no-lock option.

Change MySQL server -> MariaDB server in code/information messages.

Author: Alexander Barkov
Author Date: 2022-08-02 12:23:08 UTC

Backporting MDEV-29159 from 10.7 to 10.6

MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability

1. Store assignment failures on incompatible data types now raise errors if:
- STRICT_ALL_TABLES or STRICT_TRANS_TABLES sql_mode is used, and
- IGNORE is not used

Otherwise, only a warning is raised and the statement continues.

2. Changing the error/warning test as follows:

-ERROR HY000: Illegal parameter data types inet6 and int for operation 'SET'
+ERROR HY000: Cannot cast 'int' as 'inet6' in assignment of `db`.`t`.`col`

so in case of a big table it's easier to see which column has the problem.
The new error text is also applied to SP variables.

Author: Jan Lindström
Author Date: 2022-08-25 06:31:08 UTC

Instrumentation

Author: Vicențiu Ciorbaru
Author Date: 2022-08-24 13:57:02 UTC

Cleanup mysqlbinlog

Author: Brandon Nesterenko
Author Date: 2022-04-27 01:51:42 UTC

MDEV-28294: set default role bypasses Replicate_Wild_Ignore_Table: mysql.%

Problem:
========
When replicating SET DEFAULT ROLE, the pre-update check (i.e. that
in set_var_default_role::check()) tries to validate the existence of
the given rules/user even when the targeted tables are ignored. When
previously issued CREATE USER/ROLE commands are ignored by the
replica because of the replication filtering rules, this results in
an error because the targeted data does not exist.

Solution:
========
Before checking that the given roles/user exist of a SET DEFAULT
ROLE command, first ensure that the mysql.user and
mysql.roles_mapping tables are not excluded by replication filters.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
Sergei Golubchik <serg@mariadb.com>

Author: Nikita Malyavin
Author Date: 2022-07-21 23:18:21 UTC

MDEV-29068 Cascade foreign key updates do not apply in online alter

Foreign key cascade operations are still implemented through innodb.

Add support for online logging in row_ins_foreign_check_on_constraint.

upd_node_t::sql_is_online_alter is added to reuse cascade->row and
cascade->upd_row.

Author: Nayuta Yanagisawa
Author Date: 2022-08-23 10:53:59 UTC

Fix wrong diff introduced by merge commit

Many Spider tests were broken by the merge commit, 36d173e.

Author: Marko Mäkelä
Author Date: 2022-08-22 13:24:10 UTC

Merge 10.6

Author: Marko Mäkelä
Author Date: 2022-08-18 14:39:46 UTC

MDEV-29043 mariabackup --compress hangs

compress_write(): Fix a race condition between threads that would
use the same worker thread object. Make thd->data_avail contain the
thread identifier of the submitter, and add thd->avail_cond to
notify other compress_write() threads that are waiting for a slot.

Author: Igor Babaev
Author Date: 2022-08-17 04:43:40 UTC

Correction after rebase of bb-10.10-MDEV-28883 against the latest 10.10.

Author: Monty
Author Date: 2022-08-11 10:05:23 UTC

TEMPORARY PUSH: Changing all cost calculation to be given in ms

- Added tests/check_costs.pl, a tool to verify optimizer cost calculations.
  - Most costs has been found with this program. All steps to calculate
    the new costs are documented in Docs/optimizer.costs
- User optimizer_cost variables are given in usec (as individual
  costs can be very small). Internally they are stored in ms.
- Changed DISK_READ_COST (was DISK_SEEK_BASE_COST) from a hard disk cost
  (9 ms) to common SSD cost (400MB/sec).
- Changed the following handler functions to return IO_AND_CPU_COST.
  This makes it easy to apply different cost modifiers in ha_..time()
  functions for io and cpu costs.
  - scan_time()
  - rndpos_time()
  - keyread_time()
- Enhanched keyread_time() to calculate the full cost of reading of a set
  of keys with a given number of ranges and optionall number of blocks that
  need to be accessed.
- Removed read_time() as keyread_time() + rndpos_time() is the same thing.
- Added the following new optimizer_variables:
  - optimizer_scan_lookup_cost
  - optimizer_row_lookup_cost
  - optimizer_index_lookup_cost
  - optimizer_disk_read_cost
- Added include/my_tracker.h ; Useful include file to quickly test costs
  of a function.
- Tuned sequence and heap engine costs (rest will be done in an updated
  commit)