Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-05-14 15:27:58 UTC

MDEV-34156 InnoDB fails to apply the redo log for compressed tablespace

Problem:
=======
- During recovery, InnoDB fails to apply the redo log
for compressed tablespace. The reason is that
InnoDB assumes that pages has been freed while applying
the redo log for it. InnoDB does multiple scan due to
small buffer pool size. Problematic page has been freed and
reinitialize multiple times. InnoDB stores the freed page
information before it ran out of memory.
But InnoDB assigns the freed page ranges to tablespace in
recv_init_crash_recovery_spaces() even though
InnoDB doesn't have complete freed range information.

Solution:
=========
rec_init_crash_recovery_spaces(): Don't add the freed ranges
to tablespace if InnoDB has to read the redo log again.

Author: Monty
Author Date: 2024-04-18 13:29:19 UTC

MDEV-33938 Analyze table on sequences should be prohibited

Author: Oleg Smirnov
Author Date: 2024-03-10 10:03:04 UTC

Parser for optimizer hints

Author: Julius Goryavsky
Author Date: 2024-05-14 02:52:53 UTC

MDEV-32633: additional post-merge changes for 10.5+

Author: Sergei Golubchik
Author Date: 2024-05-14 11:31:53 UTC

cleanup: compile with -fno-operator-names in maintainer mode

Author: Sergei Golubchik
Author Date: 2024-05-14 11:31:53 UTC

cleanup: compile with -fno-operator-names in maintainer mode

Author: Julius Goryavsky
Author Date: 2024-05-14 11:18:33 UTC

galera: wsrep-lib submodule update

Author: Denis Protivensky
Author Date: 2024-05-07 13:42:13 UTC

MDEV-33952: Fix flaky galera_create_table_as_select test with debug sync

The test that triggers multi-master conflict between two CTAS commands
uses LOCK/UNLOCK TABLES to block local CTAS from progress. It could
result in a race when UNLOCK TABLES command is issued a bit earlier
then needed, causing local CTAS to run further and change wsrep
transaction state, so that a different code path is taken later and
the original error gets overridden, causing the test to fail.
The solution is to replace LOCK/UNLOCK TABLES with debug sync points.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Lena Startseva
Author Date: 2023-08-30 05:22:07 UTC

MDEV-31933: Make working view-protocol + ps-protocol (running two protocols together)

Fix for v. 10.5

Author: VladislavVaintroub
Author Date: 2024-05-14 08:23:05 UTC

MDEV-33627 refactor threading in mariadb-import

Use threadpool, instead of one-thread-and-connection-per-table

Author: midenok
Author Date: 2024-05-13 17:58:51 UTC

Error codes

Author: Yuchen Pei
Author Date: 2024-05-07 03:49:17 UTC

MDEV-28105 Return error in ha_spider::write_row() if info(HA_STATUS_AUTO) fails

Spider calls info with HA_STATUS_AUTO to update auto increment info,
which may attempt to connect the data node. If the connection fails,
it may emit an error and return the same error. This error should not
be of lower priority than any possible error from the later call to
handler::update_auto_increment().

Without this change, certain errors from update_auto_increment() such
as HA_ERR_AUTOINC_ERANGE may get ignored, causing my_insert() to call
my_ok(), which fails the assertion because the error was emitted in
the info() call (Diagnostics_area::is_set() returns true).

Author: Dave Gosselin
Author Date: 2024-04-03 17:16:25 UTC

MDEV-32397, MDEV-32403 Crashes during join processing.

Queries having the following form may cause a crash
  SELECT t1.a FROM ( SELECT a AS a1 FROM t1 ) dt
  JOIN t1 ON a1 LIKE EXISTS ( SELECT a + RAND () FROM t1 UNION SELECT a FROM t1);
because the table t1 has some JOIN cleanup operations performed prematurely
during the subselect.

In this particular case, the presence of RAND() makes the subquery
uncacheable, necessitating the need to execute the subquery multiple
times during join record evaluation. Each time the subquery runs, it
creates its own JOIN structure which has references to the table t1.
When the subquery completes, JOIN::cleanup and functions called by it
result in ha_end_keyread() being called on table t1. However, we are
not done with table t1 because the upper level `select t1.a from...`
query requires table t1 to be open. To solve this, we make the executor
aware of when we're in subqueries like this and delay JOIN cleanup
until the end of the query.

Author: Denis Protivensky
Author Date: 2024-03-25 13:40:55 UTC

MDEV-32633: Fix Galera cluster <-> native replication interaction

It's possible to establish Galera multi-cluster setups connected
through the native replication when every Galera cluster is configured
to have a separate domain ID.
For this setup to work, we need to replace domain ID values in generated
GTID events when they are written at transaction commit to the values
configured by Wsrep replication.

At the same time, it's possible that the GTID event already contains
a correct domain ID if it comes through the native replication from
another Galera cluster.
In this case, when such an event is applied either through a native
replication slave thread or through Wsrep applier, we write GTID event
on transaction start and avoid writing it during transaction commit.

The code contained multiple problems that were fixed:
- applying GTID events didn't work because it's applied without a
running server transaction and Wsrep transaction was not started
- GTID event generation on transaction start didn't contain proper
"standalone" and "is_transactional" flags that the original applied
GTID event contained
- condition determining that GTID event is written on transaction start
to avoid writing it on commit relied on the fact that the GTID event
is the first found in transaction/statement caches, which wasn't the
case and resulted in duplicate GTID events written
- instead of relying on the caches to find a GTID event, a simple check
is introduced that follows the exact rules for checking if event is
written at transaction start as described above
- the test case is improved to check that exact GTID events are
applied after two Galera clusters have synced.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2024-05-13 20:31:45 UTC

MDEV-21322: report slave progress to the master

This patch extends the command `SHOW REPLICA HOSTS` with three
columns:

 1) Gtid_State_Sent. This represents that latest GTIDs sent to the
replica in each domain. It will always be populated, regardless of
the semi-sync status (i.e. asynchronous connections will still
update this column with the latest GTID state sent to the replica).

 2) Gtid_State_Ack. For semi-synchronous connections (only), this
column represents the last GTID in each domain that the replica has
acknowledged.

 3) Sync_Status. This value represents the synchronization status of
the replica, and is used to help determine how to interpret the
Gtid_State_Ack column. There are four possible values:

   3.1) Initializing. This means the binlog dump thread is still
initializing, and has not yet determined the synchronization status
of the replica.

   3.2) Asynchronous: This means the replica is not configured for
semi-sync replication, and thereby, Gtid_State_Ack should always be
empty.

   3.3) Semi-sync Stale: This means the replica is configured for
semi-sync replication, however, connected using an old state, and is
not readily able to send ACKs for new transactions. Functionally,
this means that the primary will try to catch the replica up-to-date
by sending transactions which will not be ACKed. Additionally, the
value shown by Gtid_State_Ack will be empty until the replica
catches up and ACKs its first transaction.

   3.4) Semi-sync Active: This means the replica is configured for
semi-sync replication, and is readily sending ACKs for new
transactions it receives. It is possible for Gtid_State_Ack to be
empty while Sync_Status is "Semi-sync Active" if no new transactions
have been executed on the primary since the replica has connected.

Additionally, this patch creates a new semantic for the
configuration rpl_semi_sync_master_timeout=0. That is, now when 0,
1) new transactions will not attempt to wait for an ACK before
completing, and 2) the primary will still request ACKs from the
replica for new transactions. This means that Gtid_State_Ack will be
updated for each ACK from the replica and Sync_Status will read as
"Semi-sync Active". Effectively, this creates a mode to mimic the
asynchronous connection behavior, while allowing one to monitor the
progress at which the primary is sending transactions to the replica
via the new columns Gtid_State_Sent and Gtid_State_Ack.

Also note that a new error message was added to account for the case
that Gtid_State_(Sent/Ack) represents a binary log file that was
purged/cannot be found.

The overall implementation is rather simple. It leverages the
existing semi-sync framework, where the replica uses binlog file:pos
to ACK transactions, in order to infer GTID state by performing a
binlog lookup at the time `SHOW REPLICA HOSTS` is executed. In
particular, the Slave_info struct is extended to store 1) the binlog
file:pos pair of the transaction which was last sent to the replica,
2) the binlog file:pos pair that was last ACKed by the replica, and
3) and enum to represent the Sync_Status.

This patch was initially started by @JackSlateur in PR#1427, where
it was then transferred to @an3l who buffed it out in PR#2374, and
final touches were put on by @bnestere.

Reviewed By:
============
<TODO>

Author: Dave Gosselin
Author Date: 2024-05-13 14:36:11 UTC

MDEV-34129 mariadb-install-db appears to hang on macOS

A bug in signal_handler thread initialization led to an overcomplicated
implementation of wait_for_signal_handler_to_end, namely that we could
fail to initialize the signal handler but mark it as active anyway.
This meant that we could wait for it to terminate when it didn't exist
in the first place.

Additionally, let's immediately close down the signal handler loop when
we decide to break connections--it's the start of process termination
anyway, and there's no need to wait once we've invoked break_connections.

Author: Sergey Petrunia
Author Date: 2024-04-16 09:40:47 UTC

MDEV-33752: EXPLAIN UPDATE acquires locks on table rows

Variant#2: modify thd_sql_command().

Make EXPLAIN commands behave like SELECTs when it comes to locking:
- In the SQL parser, make EXPLAIN DELETE and EXPLAIN UPDATE not to
  request write MDL or table locks.
- Same as above in multi_delete_set_locks_and_link_aux_tables().
- InnoDB analyzes the return value of thd_sql_command(). Make it return
  SQLCOM_SELECT for EXPLAINs.

Note that before this patch, EXPLAIN UPDATE over InnoDB table would read
rows using READ-COMMITTED, like UPDATE does. After this patch, EXPLAIN
UPDATE over InnoDB table would read rows from transaction's snapshot.

Author: Dmitry Shulga
Author Date: 2024-05-06 13:10:06 UTC

MDEV-33769: Memory leak found in the test main.rownum run with --ps-protocol against a server built with the option -DWITH_PROTECT_STATEMENT_MEMROOT

A memory leak happens on the second execution of a query that run in PS mode
and uses the function ROWNUM().

A memory leak took place on allocation of an instance of the class Item_int
for storing a limit value that is performed at the function set_limit_for_unit
indirectly called from JOIN::optimize_inner. Typical trace to the place where
the memory leak occurred is below:
 JOIN::optimize_inner
  optimize_rownum
   process_direct_rownum_comparison
    set_limit_for_unit
     new (thd->mem_root) Item_int(thd, lim, MAX_BIGINT_WIDTH);

To fix this memory leak, calling of the function optimize_rownum()
has to be performed only once on first execution and never called
after that. To control it, the new data member
  first_rownum_optimization
added into the structure st_select_lex.

Author: Dmitry Shulga
Author Date: 2024-05-06 13:10:06 UTC

MDEV-33769: Memory leak found in the test main.rownum run with --ps-protocol against a server built with the option -DWITH_PROTECT_STATEMENT_MEMROOT

A memory leak happens on the second execution of a query that run in PS mode
and uses the function ROWNUM().

A memory leak took place on allocation of an instance of the class Item_int
for storing a limit value that is performed at the function set_limit_for_unit
indirectly called from JOIN::optimize_inner. Typical trace to the place where
the memory leak occurred is below:
 JOIN::optimize_inner
  optimize_rownum
   process_direct_rownum_comparison
    set_limit_for_unit
     new (thd->mem_root) Item_int(thd, lim, MAX_BIGINT_WIDTH);

To fix this memory leak, calling of the function optimize_rownum()
has to be performed only once on first execution and never called
after that. To control it, the new data member
  first_rownum_optimization
added into the structure st_select_lex.

Author: Sergei Golubchik
Author Date: 2024-05-13 09:09:55 UTC

main.alter_table_online fails in --view

disable view protocol (DROP VIEW doesn't work very well with transactions)
and cosmetic cleanups

Author: Sergei Golubchik
Author Date: 2024-05-13 09:09:55 UTC

main.alter_table_online fails in --view

disable view protocol (DROP VIEW doesn't work very well with transactions)
and cosmetic cleanups

Author: Sergey Petrunia
Author Date: 2024-05-03 10:40:06 UTC

MDEV-33533: Crash at execution of DELETE when trying to use rowid filter

(Based on original patch by Oleksandr Byelkin)

Multi-table DELETE can execute via "buffered" mode: at phase #1 it collects
rowids of rows to be deleted, then at phase #2 in multi_delete::do_deletes()
it calls handler->rnd_pos() to read rows to be deleted and deletes them.

The problem occurred when phase #1 used Rowid Filter on the table that
phase #2 would be deleting from.
In InnoDB, h->rnd_init(scan=false) and h->rnd_pos() is an index scan over PK
under the hood. So, at phase #2 ha_innobase::rnd_init() would try to use the
Rowid Filter and hit an assertion inside ha_innobase::rnd_init().

Note that multi-table UPDATE works similarly but was not affected, because
patch for MDEV-7487 added code to disable rowid filter for phase #2 in
multi_update::do_updates().

This patch changes the approach:
- It makes InnoDB not use Rowid Filter in rnd_pos() scans: it is disabled in
  ha_innobase::rnd_init() and enabled back in ha_innobase::rnd_end().
- multi_update::do_updates() no longer disables Rowid Filter for phase#2 as
  it is no longer necessary.

Author: Sergey Petrunia
Author Date: 2024-05-03 10:40:06 UTC

MDEV-33533: Crash at execution of DELETE when trying to use rowid filter

(Based on original patch by Oleksandr Byelkin)

Multi-table DELETE can execute via "buffered" mode: at phase #1 it collects
rowids of rows to be deleted, then at phase #2 in multi_delete::do_deletes()
it calls handler->rnd_pos() to read rows to be deleted and deletes them.

The problem occurred when phase #1 used Rowid Filter on the table that
phase #2 would be deleting from.
In InnoDB, h->rnd_init(scan=false) and h->rnd_pos() is an index scan over PK
under the hood. So, at phase #2 ha_innobase::rnd_init() would try to use the
Rowid Filter and hit an assertion inside ha_innobase::rnd_init().

Note that multi-table UPDATE works similarly but was not affected, because
patch for MDEV-7487 added code to disable rowid filter for phase #2 in
multi_update::do_updates().

This patch changes the approach:
- It makes InnoDB not use Rowid Filter in rnd_pos() scans: it is disabled in
  ha_innobase::rnd_init() and enabled back in ha_innobase::rnd_end().
- multi_update::do_updates() no longer disables Rowid Filter for phase#2 as
  it is no longer necessary.

Author: Yuchen Pei
Author Date: 2024-05-13 07:22:20 UTC

MDEV-32627 Spider: use CONNECTION string in SQLDriverConnect

Author: Yuchen Pei
Author Date: 2024-05-12 23:15:14 UTC

MDEV-32487 Check plugin is ready when resolving storage engine

This handles the situation when one thread is still initiating a
storage engine plugin, while another is creating a table using it.

Author: Sergey Petrunia
Author Date: 2024-05-12 16:10:47 UTC

MDEV-34125: ANALYZE FORMAT=JSON: r_engine_stats.pages_read_time_ms has wrong scale

Use the right scale. Also, change the comments.

Author: Sergei Golubchik
Author Date: 2024-05-12 09:25:32 UTC

sporadic failures of galera.galera_sst_mariabackup

the test failed almost always in release (but not in debug) builds with

--- galera_sst_mariabackup.result
+++ galera_sst_mariabackup.reject
@@ -5,7 +5,7 @@
 connection node_1;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces
-0
+3
 connection node_2;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces

and

[Warning] InnoDB: Cannot change innodb_undo_tablespaces=0 because previous shutdown was not with innodb_fast_shutdown=0

because mariadbd *before this test* wasn't using innodb_fast_shutdown=0

Fix the bootstrap to use innodb_fast_shutdown=0 (and the bootstrap
creates a starting point for any test that uses a .cnf file)

followup for cac0fc97cca4

also, remove redundant include/have_innodb.inc

Author: Sergei Golubchik
Author Date: 2024-05-12 09:25:32 UTC

sporadic failures of galera.galera_sst_mariabackup

the test failed almost always in release (but not in debug) builds with

--- galera_sst_mariabackup.result
+++ galera_sst_mariabackup.reject
@@ -5,7 +5,7 @@
 connection node_1;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces
-0
+3
 connection node_2;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces

and

[Warning] InnoDB: Cannot change innodb_undo_tablespaces=0 because previous shutdown was not with innodb_fast_shutdown=0

because mariadbd *before this test* wasn't using innodb_fast_shutdown=0

Fix the bootstrap to use innodb_fast_shutdown=0 (and the bootstrap
creates a starting point for any test that uses a .cnf file)

followup for cac0fc97cca4

also, remove redundant include/have_innodb.inc

Author: Sergei Golubchik
Author Date: 2024-05-12 09:25:32 UTC

sporadic failures of galera.galera_sst_mariabackup

the test failed almost always in release (but not in debug) builds with

--- galera_sst_mariabackup.result
+++ galera_sst_mariabackup.reject
@@ -5,7 +5,7 @@
 connection node_1;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces
-0
+3
 connection node_2;
 select @@innodb_undo_tablespaces;
 @@innodb_undo_tablespaces

and

2024-05-11 14:35:36 0 [Warning] InnoDB: Cannot change innodb_undo_tablespaces=0 because previous shutdown was not with innodb_fast_shutdown=0

because mariadbd *before this test* wasn't using innodb_fast_shutdown=0

Fix the bootstrap to use innodb_fast_shutdown=0 (and the bootstrap
creates a starting point for any test that uses a .cnf file)

followup for cac0fc97cca4

also, remove redundant include/have_innodb.inc

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-05-11 16:31:47 UTC

MDEV-34057 Inconsistent FTS state in concurrent scenarios

Problem:
=======
- This commit is a merge of mysql commit 129ee47ef994652081a11ee9040c0488e5275b14.
InnoDB FTS can be in inconsistent state when sync operation
terminates the server before committing the operation. This
could lead to incorrect synced doc id and incorrect query results.

Solution:
========
- During sync commit operation, InnoDB should pass
the sync transaction to update the max doc id
in the config table.

Author: Sergei Golubchik
Author Date: 2024-05-09 18:37:40 UTC

MDEV-33852 start the server after deb installation

and fix installation of mysql.service on buster

followup for ec09c034d84f

Author: VladislavVaintroub
Author Date: 2024-05-10 15:30:51 UTC

MDEV-33748 : use C++ thread_local, also for mysys TLS variable.

This avoids compiler-specific definitions, or C11 requirement.

Author: Yuchen Pei
Author Date: 2024-05-10 00:01:15 UTC

MDEV-33490 MENT-1504 Fix some english strings in spider.

Author: Rex Johnston
Author Date: 2024-03-18 20:50:19 UTC

MDEV-32583 UUID() should be treated as stochastic for the purposes of forcing query materialization

RAND() and UUID() are treated differently with respect to subquery
materialization both should be marked as uncacheable, forcing materialization.
Altered Create_func_uuid(_short)::create_builder()

Author: Rex Johnston
Author Date: 2024-05-09 18:30:26 UTC

MDEV-28621 group by optimization incorrectly removing subquery where subject buried in a function

Do not remove the unit from the query graph. Group by item removal happens during prepare phase.
Revert change after execution.

Author: Vicențiu Ciorbaru
Author Date: 2024-02-17 15:03:30 UTC

Initial HNSW implementation

This commit includes the work done in collaboration with Hugo Wen from
Amazon:

    MDEV-33408 Alter HNSW graph storage and fix memory leak

    This commit changes the way HNSW graph information is stored in the
    second table. Instead of storing connections as separate records, it now
    stores neighbors for each node, leading to significant performance
    improvements and storage savings.

    Comparing with the previous approach, the insert speed is 5 times faster,
    search speed improves by 23%, and storage usage is reduced by 73%, based
    on ann-benchmark tests with random-xs-20-euclidean and
    random-s-100-euclidean datasets.

    Additionally, in previous code, vector objects were not released after
    use, resulting in excessive memory consumption (over 20GB for building
    the index with 90,000 records), preventing tests with large datasets.
    Now ensure that vectors are released appropriately during the insert and
    search functions. Note there are still some vectors that need to be
    cleaned up after search query completion. Needs to be addressed in a
    future commit.

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

As well as the commit:

    Introduce session variables to manage HNSW index parameters

    Three variables:

    hnsw_max_connection_per_layer
    hnsw_ef_constructor
    hnsw_ef_search

    ann-benchmark tool is also updated to support these variables in commit
    https://github.com/HugoWenTD/ann-benchmarks/commit/e09784e for branch
    https://github.com/HugoWenTD/ann-benchmarks/tree/mariadb-configurable

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

Author: Vicențiu Ciorbaru
Author Date: 2024-02-17 15:03:30 UTC

Initial HNSW implementation

This commit includes the work done in collaboration with Hugo Wen from
Amazon:

    MDEV-33408 Alter HNSW graph storage and fix memory leak

    This commit changes the way HNSW graph information is stored in the
    second table. Instead of storing connections as separate records, it now
    stores neighbors for each node, leading to significant performance
    improvements and storage savings.

    Comparing with the previous approach, the insert speed is 5 times faster,
    search speed improves by 23%, and storage usage is reduced by 73%, based
    on ann-benchmark tests with random-xs-20-euclidean and
    random-s-100-euclidean datasets.

    Additionally, in previous code, vector objects were not released after
    use, resulting in excessive memory consumption (over 20GB for building
    the index with 90,000 records), preventing tests with large datasets.
    Now ensure that vectors are released appropriately during the insert and
    search functions. Note there are still some vectors that need to be
    cleaned up after search query completion. Needs to be addressed in a
    future commit.

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

As well as the commit:

    Introduce session variables to manage HNSW index parameters

    Three variables:

    hnsw_max_connection_per_layer
    hnsw_ef_constructor
    hnsw_ef_search

    ann-benchmark tool is also updated to support these variables in commit
    https://github.com/HugoWenTD/ann-benchmarks/commit/e09784e for branch
    https://github.com/HugoWenTD/ann-benchmarks/tree/mariadb-configurable

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

Author: Vicențiu Ciorbaru
Author Date: 2024-02-17 15:03:30 UTC

Initial HNSW implementation

This commit includes the work done in collaboration with Hugo Wen from
Amazon:

    MDEV-33408 Alter HNSW graph storage and fix memory leak

    This commit changes the way HNSW graph information is stored in the
    second table. Instead of storing connections as separate records, it now
    stores neighbors for each node, leading to significant performance
    improvements and storage savings.

    Comparing with the previous approach, the insert speed is 5 times faster,
    search speed improves by 23%, and storage usage is reduced by 73%, based
    on ann-benchmark tests with random-xs-20-euclidean and
    random-s-100-euclidean datasets.

    Additionally, in previous code, vector objects were not released after
    use, resulting in excessive memory consumption (over 20GB for building
    the index with 90,000 records), preventing tests with large datasets.
    Now ensure that vectors are released appropriately during the insert and
    search functions. Note there are still some vectors that need to be
    cleaned up after search query completion. Needs to be addressed in a
    future commit.

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

As well as the commit:

    Introduce session variables to manage HNSW index parameters

    Three variables:

    hnsw_max_connection_per_layer
    hnsw_ef_constructor
    hnsw_ef_search

    ann-benchmark tool is also updated to support these variables in commit
    https://github.com/HugoWenTD/ann-benchmarks/commit/e09784e for branch
    https://github.com/HugoWenTD/ann-benchmarks/tree/mariadb-configurable

    All new code of the whole pull request, including one or several files
    that are either new files or modified ones, are contributed under the
    BSD-new license. I am contributing on behalf of my employer Amazon Web
    Services, Inc.

Author: Yuchen Pei
Author Date: 2024-05-09 03:56:26 UTC

MDEV-32180 [demo] drop functions in spider deinit

This will not work because of MDEV-32796: if spider is used in any
way (e.g. create table ... engine=Spider), any attempt to uninstall it
will be deferred to server shutdown, by which time it would be
impossible to send queries over local sql service, because
close_connection() sets thd->killed to KILL_CONNECTION

Author: Yuchen Pei
Author Date: 2024-05-09 01:16:12 UTC

MDEV-34064 [demo] return error from ha_commit_trans() in rpl_slave_state::truncate_state_table(THD *thd)

just a demo - a similar fix to MDEV-28105 and MDEV-27186.

Whether the 12607 error makes sense is perhaps a different matter:

mysqltest: At line 7: query 'CHANGE MASTER TO master_host='a',master_port=1,master_user='a',master_demote_to_slave=1' failed: <Unknown> (12607): This xid is not exist

Author: Yuchen Pei
Author Date: 2024-05-09 01:08:14 UTC

MDEV-29027 ASAN errors in spider_db_free_result after partition DDL

Spider calls ha_spider::close() at least twice on ALTER TABLE ... ADD
PARTITION. The first call frees wide_handler and the second call
accesses wide_handler->trx->thd (heap-use-after-free).

In general, there seems to be no problem with using THD obtained by
the macro current_thd() except in background threads. Thus, we simply
replace wide_handler->trx->thd with current_thd().

Original author: Nayuta Yanagasawa

Author: Yuchen Pei
Author Date: 2024-05-09 01:02:35 UTC

MDEV-27186 spider/partition: Report error on info() failure

Like MDEV-28105, spider may attempt to connect to remote server in
info(), and it may emit an error upon failure to connect. In this
case, the downstream caller ha_partition::open() should return the
error to avoid inconsistency.

This fixes MDEV-27186, MDEV-27237, MDEV-27334, MDEV-28241, MDEV-34101.

Author: Yuchen Pei
Author Date: 2024-05-09 00:28:55 UTC

MDEV-27186 spider/partition: Report error on info() failure

Like MDEV-28105, spider may attempt to connect to remote server in
info(), and it may emit an error upon failure to connect. In this
case, the downstream caller ha_partition::open() should return the
error to avoid inconsistency.

This fixes MDEV-27186, MDEV-27237, MDEV-27334, MDEV-28241, MDEV-34101.

Author: Julius Goryavsky
Author Date: 2024-05-09 00:18:16 UTC

MDEV-33677 : Port Performance Schema for Galera from ES

Added performance schema tables for Galera.
This is a port of the code from ES 10.6.

Refs:
MENT-346 commit 86c06851
MENT-346 commit 10d0fcaf
MENT-1333 commit 2005adc3
MENT-347 commit aaca43b4
MENT-1370 commit 1f460e6b

Author: Igor Babaev
Author Date: 2024-05-08 22:30:15 UTC

MDEV-30073 Wrong result on 2nd execution of PS for query with NOT EXISTS

This is a preliminary patch to check possible failures in buildbot.
Variant 7

Author: Sergei Golubchik
Author Date: 2024-05-08 22:04:20 UTC

columnstore 6.4.8-2

Author: Rex Johnston
Author Date: 2024-04-29 23:39:20 UTC

MDEV-19520 pushdown_from_having_into_where() not calling fix_fields() correctly

fix_fields() called in pushdown_from_having_into_where() assuming that the
item is immutable.
normalize_cond() extended to include (not f)=>(f == 0) to avoid the
issue where NOT(field) was being transformed to (field == 0) during
this fix_fields call.

Author: Sergei Golubchik
Author Date: 2024-05-08 07:31:10 UTC

test needs to cleanup after itself

Author: Sergey Petrunia
Author Date: 2024-05-03 10:40:06 UTC

MDEV-33533: Crash at execution of DELETE when trying to use rowid filter

(Based on original patch by Oleksandr Byelkin)

Multi-table DELETE can execute via "buffered" mode: at phase #1 it collects
rowids of rows to be deleted, then at phase #2 in multi_delete::do_deletes()
it calls handler->rnd_pos() to read rows to be deleted and deletes them.

The problem occurred when phase #1 used Rowid Filter on the table that
phase #2 would be deleting from.
In InnoDB, h->rnd_init(scan=false) and h->rnd_pos() is an index scan over PK
under the hood. So, at phase #2 ha_innobase::rnd_init() would try to use the
Rowid Filter and hit an assertion inside ha_innobase::rnd_init().

Note that multi-table UPDATE works similarly but was not affected, because
patch for MDEV-7487 added code to disable rowid filter for phase #2 in
multi_update::do_updates().

This patch changes the approach:
- It makes InnoDB not use Rowid Filter in rnd_pos() scans: it is disabled in
  ha_innobase::rnd_init() and enabled back in ha_innobase::rnd_end().
- multi_update::do_updates() no longer disables Rowid Filter for phase#2 as
  it is no longer necessary.

Author: Sergei Golubchik
Author Date: 2024-05-08 17:01:38 UTC

Merge branch 'bb-10.5-serg' into bb-10.6-serg

Author: Dave Gosselin
Author Date: 2023-11-30 16:18:17 UTC

MDEV-28509 Dereferenced null pointer of type 'struct JOIN_TAB' in add_key_field

This patch fixes a crash when calculating join statistics during query
optimization for queries with dangling WINDOW references. Put another way,
the system may crash when a query defines a WINDOW but doesn't then refer to
it.

Item::marker is overloaded for different uses, the most typical refer to it as
a bit field. However, the setup_group function uses it to mark that a field
was found when traversing a GROUP BY. Originally, this marking set the
Item::marker field to 1 to indicate that it was found. Later on in setup_group
(and only when SQL mode ONLY_FULL_GROUP_BY is enabled), we would skip any such
marked fields when checking that fields only referenced those found in the
GROUP BY; otherwise, it would be silly to find fields of the GROUP BY within
the GROUP BY field itself. Setting Item::marker to 1 seemed mostly harmless
at that point in time. But later, in git sha 4d143a6ff6, we introduced
several changes: (1) the value of marker in setup_group was changed from 1 to
UNDEF_POS, (2) Item::marker was changed from uint8 to int8 (which has since
been changed to an int), and (3) UNDEF_POS which is defined to be -1 was also
added.

Queries that define WINDOWs internally will setup groups and orders as part
of query processing via the setup_group function. Consequently because of
the behavior described earlier above, such queries may have items with markers
as UNDEF_POS (-1, or 0xffffffff) which is the same as marking all of the flag
bits as set. This is disastrous for those users of Item::marker which refer
to it as a bit field as some of those bits are mutually exclusive in meaning,
and in many places we don't mask the bits we're interested in, we just
compare the value of the field as a whole to some flag value with direct
comparison. In particular, the method
Item_direct_view_ref::grouping_field_transformer_for_where would
incorrectly see that the ref's marker was set for substitution when it was
actually -1, all bits set, taking the wrong execution path leading to the
crash.

Upon return from the setup_group function, we set the value of the marker
flag to zero if we set it to -1. This preserves the marker behavior for
'full group by' (if configured) while not otherwise allowing the marker
flag state to leak outside of this function.

Author: Dave Gosselin
Author Date: 2024-02-02 19:29:08 UTC

MDEV-32424 Crash in subselect engine

According to MDEV-29363, recompute the with_subquery() attribute
in the pushdown code to avoid writing to freed memory.

Author: Dave Gosselin
Author Date: 2024-03-18 17:03:33 UTC

MDEV-32777 DDL operations very slow under macOS

macOS is a development platform for our product but calls to
fcntl(F_FULLFSYNC) are too slow for developer needs. macOS offers
at least three different disk flush calls, making tradeoffs between
reliability and performance (see analysis notes on this MDEV).
On macOS only and as a compromise, we will back out the change to
call fcntl(F_FULLFSYNC), sacrificing a full flush in favor of more
performant writes.

Author: Dave Gosselin
Author Date: 2024-04-15 19:17:30 UTC

MDEV-33616 Tests failing on macOS

These tests rely on THR_KEY_mysys but it is not initialized. On
Linux, the corresponding thread variable is null, but on macOS it has a
nonzero value. In all cases, initialize the variable explicitly. We could
call MY_INIT, however some memory is leaked on test program exit in that
case. Instead, factor just enough to be able to initialize THR_KEY_mysys
from the test code (as well as from the library code).

Author: Dave Gosselin
Author Date: 2024-04-26 16:13:31 UTC

MDEV-33971 NAME_CONST in WHERE clause replaced by inner item

Improve performance of queries like
  SELECT * FROM t1 WHERE field = NAME_CONST('a', 4);
by, in this example, replacing the WHERE clause with field = 4
in the case of ref access.

The rewrite is done during fix_fields and we disambiguate this
case from other cases of NAME_CONST by inspecting where we are
in parsing. We rely on THD::where to accomplish this. To
improve performance there, we change the type of THD::where to
be an enumeration, so we can avoid string comparisons during
Item_name_const::fix_fields. Consequently, this patch also
changes all usages of THD::where to conform likewise.

Author: VladislavVaintroub
Author Date: 2024-05-04 08:06:11 UTC

MDEV-29955 post-fix - add CMake policy CMP0074, so ZLIB_ROOT is not ignored

Author: Monty
Author Date: 2024-05-07 19:03:38 UTC

fixup! 9831521fc852f72317024778c723e28d121635e6

Author: Yuchen Pei
Author Date: 2024-05-08 04:03:08 UTC

MDEV-28105 Return error in ha_spider::write_row() if info(HA_STATUS_AUTO) fails

Spider calls info with HA_STATUS_AUTO to update auto increment info,
which may attempt to connect the data node. If the connection fails,
it may emit an error and return the same error. This error should not
be of lower priority than any possible error from the later call to
handler::update_auto_increment().

Without this change, certain errors from update_auto_increment() such
as HA_ERR_AUTOINC_ERANGE may get ignored, causing my_insert() to call
my_ok(), which fails the assertion because the error was emitted in
the info() call (Diagnostics_area::is_set() returns true).

Author: Monty
Author Date: 2024-05-07 21:38:59 UTC

fixup! f551fa75e13dd57b057117c570f2e2414ec6d500

Author: Monty
Author Date: 2024-05-07 19:03:38 UTC

fixup! 9831521fc852f72317024778c723e28d121635e6

Author: Monty
Author Date: 2024-05-07 21:38:59 UTC

fixup! f551fa75e13dd57b057117c570f2e2414ec6d500

Author: Monty
Author Date: 2024-05-07 14:21:01 UTC

MDEV-34055 Assertion '...' failure or corruption errors upon REPAIR on Aria tables

The problem was two fold:
- REPAIR TABLE t1 USE_FRM did not work for transactional
  Aria tables (Table was thought to be repaired, which it was not) which
  caused issues in later usage of the table.
- When swapping tmp_data file to data file, sort_info files where not
  updated. This caused problems if there was several unique keys and
  there was a duplicate for the second key.

Author: Sergey Petrunia
Author Date: 2024-05-07 11:10:35 UTC

MDEV-28621: group by optimization incorrectly removing subquery where subject buried in a function

Workaround patch: Do not remove GROUP BY clause when it has
subquer(ies) in it.

remove_redundant_subquery_clauses() removes redundant GROUP BY clause
from queries in form:
  expr IN (SELECT no_aggregates GROUP BY ...)
  expr {CMP} {ALL|ANY|SOME} (SELECT no_aggregates GROUP BY ...)
This hits problems when the GROUP BY clause itself has subquer(y/ies).

This patch is just a workaround: it disables removal of GROUP BY clause
if the clause has one or more subqueries in it.

Tests:
- subselect_elimination.test has all known crashing cases.
- subselect4.result, insert_select.result are updated.
Note that in some cases results of SELECT are changed too (not just
EXPLAINs). These are caused by non-deterministic SQL: when running a
query like:

  x > ANY( SELECT col1 FROM t1 GROUP BY constant_expression)

without removing the GROUP BY, the executor is free to pick the value
of t1.col1 from any row in the GROUP BY group (denote it $COL1_VAL).
Then, it computes x > ANY(SELECT $COL1_VAL).

When running the same query and removing the GROUP BY:

   x > ANY( SELECT col1 FROM t1)

the executor will actually check all rows of t1.

Author: Galina Shalygina
Author Date: 2024-04-30 19:31:47 UTC

MDEV-23878 Wrong result with semi-join and splittable derived table

Due to this bug a wrong result might be expected from queries with
an IN subquery predicate in the WHERE clause and a derived table in the
FROM clause to which split optimization could be applied.

The function JOIN::fix_all_splittings_in_plan() used the value of the
bitmap JOIN::sjm_lookup_tables() such as it had been left after the
search for the best plan for the select containing the splittable
derived table. That value could not be guaranteed to be correct. So the
recalculation of this bitmap is needed to exclude the plans with key
accesses from SJM lookup tables.

Approved by Igor Babaev <igor@maridb.com>

Author: Monty
Author Date: 2024-05-07 07:57:43 UTC

fixup! 9c20f04ef024cd4cb42e884fecae0b540453f4dc

Author: Monty
Author Date: 2024-05-07 07:45:27 UTC

fixup! 3a7a3e9b3be3504e1bfc32c931e14ed47a317d9b

Author: Sergei Golubchik
Author Date: 2023-11-04 09:11:51 UTC

don't disable lto in DEB builds

Author: Kristian Nielsen
Author Date: 2024-05-07 06:45:31 UTC

Merge 10.6 to 10.11

Author: Kristian Nielsen
Author Date: 2024-05-07 06:45:31 UTC

Merge 10.6 to 10.11

Author: Yuchen Pei
Author Date: 2024-05-06 04:46:18 UTC

MDEV-32640 Reset thd->lex->mi.connection_name.str towards the end of mysql_execute_command

Reset the connection_name to contain a null string, if the pointer
points to the same space as that of the system variable
default_master_connection.

We do this because the system variable may be updated which could free
the pointer and create a new one, causing use-after-free for
re-execution of prepared statements and stored procedures where the
LEX may be reused.

This allows connection_name to be set again be to the system variable
pointer in the next call of this function (see earlier in this
function), after any possible updates to the system variable.

Author: =?utf-8?q?Jan_Lindstr=C3=B6m?=
Author Date: 2024-04-12 12:40:11 UTC

MDEV-33898 : Galera test failure on galera.MW-369

Additional changes for the galera_vote_rejoin_ddl test (for 10.5+).

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Yuchen Pei
Author Date: 2024-05-07 00:16:38 UTC

MDEV-34098 source start_slave.inc in spider suites

The spider suite should --source include/start_slave.inc to make sure
the slave is up before proceeding.

Author: Yuchen Pei
Author Date: 2024-05-06 02:46:31 UTC

MDEV-30929 spider.spider_fixes_part: wait and restart slave

In the absence of insight of the cause of spider.spider_fixes_part
failure as described in MDEV-30929, This is a workaround, which could
help narrow the possibility down to whether slave SQL thread attempts
to read from file that maybe not yet on disk. It does not otherwise
affect the coverage of the test.

I have pushed this commit 4 times, but have yet to encounter the
failure as described in MDEV-30929, so it could also fix the test and
stop the CI pollution.

Also replaced START SLAVE; with --source include/start_slave.inc
inside the slave_test_init.inc files.

Author: Alexander Barkov
Author Date: 2024-05-06 11:01:32 UTC

MDEV-34085 Server crash ASAN used-after-poison upon 2nd execution of PS with erroneous timestamp conversion

The optimization code replacing DATETIME comparison to TIMESTAMP comparison
in conditions like:
- WHERE timestamp_col=const_expr
- WHERE const_expr IN (SELECT timestamp_column FROM t1)
worked as follows:

- Install an internal condition handler (suppressing and counting warnings).
- Convert const_expr from its data type to TIMESTAMP
- Check the warning count collected by the internal condition handler:
  * If any warnings happened during the constant conversion,
    then continue with DATETIME comparison.
  * Otherwise, go to the next stage of switching to TIMESTAMP comparison.

This scenario did not take into account that in some cases warnings
are escalated to errors. Errors were not caught by the internal handler,
so Type_handler_datetime_common::convert_item_for_comparison()
returned with an SQL error in the diagnostics area.
The calling code did not expect this.

Fixing the code to suppress and count both errors and warnings, to make sure
Type_handler_datetime_common::convert_item_for_comparison() returns without
adding any errors to DA if the conversion to TIMESTAMP fails and it decides
to go with DATETIME comparison.

Author: Denis Protivensky
Author Date: 2024-03-05 08:51:04 UTC

MDEV-32633: Fix Galera cluster <-> native replication interaction

It's possible to establish Galera multi-cluster setups connected
through the native replication when every Galera cluster is configured
to have a separate domain ID.
For this setup to work, we need to replace domain ID values in generated
GTID events when they are written at transaction commit to the values
configured by Wsrep replication.

At the same time, it's possible that the GTID event already contains
a correct domain ID if it comes through the native replication from
another Galera cluster.
In this case, when such an event is applied either through a native
replication slave thread or through Wsrep applier, we write GTID event
on transaction start and avoid writing it during transaction commit.

The code contained multiple problems that were fixed:
- applying GTID events didn't work because it's applied without a
running server transaction and Wsrep transaction was not started
- GTID event generation on transaction start didn't contain proper
"standalone" and "is_transactional" flags that the original applied
GTID event contained
- condition determining that GTID event is written on transaction start
to avoid writing it on commit relied on the fact that the GTID event
is the first found in transaction/statement caches, which wasn't the
case and resulted in duplicate GTID events written
- instead of relying on the caches to find a GTID event, a simple check
is introduced that follows the exact rules for checking if event is
written at transaction start as described above
- the test case is improved to check that exact GTID events are
applied after two Galera clusters have synced.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-05-06 14:19:11 UTC

MDEV-33980 mariadb-backup --backup is missing retry logic for undo tablespaces

- This is a merge of commit f378e7643459f0baa6c4149a795463f2f46b6871
from 10.4 to 10.5.

Author: Sergei Golubchik
Author Date: 2024-05-06 14:14:11 UTC

update C/C 3.1

Author: Sergei Golubchik
Author Date: 2024-05-06 14:14:11 UTC

update C/C 3.1

Author: Georg Richter
Author Date: 2024-05-06 12:35:03 UTC

Merge branch 'bb-11.4-georg' of https://github.com/mariadb/server into bb-11.4-georg

Author: Julius Goryavsky
Author Date: 2024-05-06 11:55:42 UTC

Merge branch 10.4 into 10.5

Author: Lena Startseva
Author Date: 2024-05-06 10:53:24 UTC

Removing MDEV-27871 from tastcases because it is not a bug
Fix for v.11.2

Author: Sergey Petrunia
Author Date: 2024-05-06 10:34:39 UTC

MDEV-33781: rocksdb.locking_issues_case5_rc fails windows ... : Disable it

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-05-06 07:23:42 UTC

MDEV-33087 ALTER TABLE...ALGORITHM=COPY should build indexes more efficiently

- During copy algorithm, InnoDB should use bulk insert operation
for row by row insert operation. By doing this, copy algorithm
can effectively build indexes. This optimization is disabled
for temporary table, versioning table and table which has
foreign key relation.

Introduced the variable innodb_alter_copy_bulk to allow
the bulk insert operation for copy alter operation
inside InnoDB. This is enabled by default

ha_innobase::extra(): HA_EXTRA_END_ALTER_COPY mode tries to apply
the buffered bulk insert operation, updates the non-persistent
table stats.

row_merge_bulk_t::write_to_index(): Update stat_n_rows after
applying the bulk insert operation

row_ins_clust_index_entry_low(): In case of copy algorithm,
switch to bulk insert operation.

copy_data_error_ignore(): Handles the error while copying
the data from source to target file.

Author: Yuchen Pei
Author Date: 2024-05-06 04:46:18 UTC

MDEV-32640 Reset thd->lex->mi.connection_name.str towards the end of mysql_execute_command

Reset the connection_name to contain a null string, if the pointer
points to the same space as that of the system variable
default_master_connection.

We do this because the system variable may be updated which could free
the pointer and create a new one, causing use-after-free for
re-execution of prepared statements and stored procedures where the
LEX may be reused.

This allows connection_name to be set again be to the system variable
pointer in the next call of this function (see earlier in this
function), after any possible updates to the system variable.

Author: Alexander Barkov
Author Date: 2024-05-01 18:04:56 UTC

MDEV-34037 DATETIME <-> TIMESTAMP conversion in a virtual column corrups the table on @@time_zone change

This patch:
- Adds deterministic ways to perform DATETIME->TIMESTAMP and
  TIMESTAMP->DATETIME conversion which do not depend on
  the @@time_zone system variable:

    UNIX_TIMESTAMP(datetime_expr, 'tz')
    CAST(timestamp_expr AT TIME ZONE 'tz' AS DATETIME[(N)])

- Adds a warning if a stored or an indexed virtual column depends
  on @@time_zone, with a suggestion to use either of the above
  expressions. In a later version (11.6?) the warning will be changed
  to an error.

Details:

- Adding a new class Session_env_dependency to pull an Item's dependencies
  on various system variables to the upper level.
  Adding a new virtual method Item::value_depends_on_session_env().
  For now it can watch dependencies on @@time_zone:
    SYS_VAR_TIME_ZONE_GMT_SEC_TO_TIME - TIMESTAMP->DATETIME conversion
    SYS_VAR_TIME_ZONE_TIME_TO_GMT_SEC - DATETIME->TIMESTAMP conversion
  Later it will also handle other system variables which make expressions
  non-deterministic (e.g. @@div_prec_increment, @@max_allowed_packet).

  The new class and the new method are implemented in a very similar
  way with Sql_mode_dependency and Item::value_depends_on_sql_mode(),
  which were earlier added to handle sql_mode flag dependencies.

- A generated Field now checks if its GENERATED ALWAYS clause
  depends on system variables and returns a warning with a proposed
  workaround, e.g.:

  > Function or expression 'col1' cannot be used in the GENERATED ... of 'col2'
  > DATETIME to TIMESTAMP conversion depends on @@time_zone.
  > For safe conversion use: unix_timestamp(datetime_expr, 'tz')

- New syntax was added to support deterministic data type conversion,
  idependent from @@time_zone:

  * For DATETIME->TIMESTAMP conversion:

    UNIX_TIMESTAMP(datetime_expr, 'tz')

    A new function overload with two arguments:
    - The first argument must be strictly of the DATETIME data type.
    - The second argument must be a string literal with a valid time zone name
      or an '+01:00' style offset.
    In case if the arguments do not meet these criteria, an error is raised.

  * For TIMESTAMP->DATETIME conversion:

    CAST(timestamp_expr AT TIME ZONE 'tz' AS DATETIME[(N)])

    This syntax uses the SQL standard AT TIME ZONE clause.
    For now, AT TIME ZONE is possible only inside CAST(.. AS DATETIME..).
    'tz' must be a valid time zone name or a '+01:00' style offset.

    The CAST argument must be strictly of the TIMESTAMP data type.
    In case of a different data type, and error is raised.

- New internal data type "unix_timestamp" was added using
  two new data type handlers:
  * Type_handler_unix_timestamp deriving from Type_handler_longlong
    (for unix timestamps without fractional seconds)

  * Type_handler_unix_timestampf deriving from Type_handler_newdecimal
    (for unix timestamps with fractional seconds)

  All variants of UNIX_TIMESTAMP() now return this new data type.
  Also, the new data type is preserved in hybrid type expressions:
    e.g. CASE .. THEN, COALESCE(), LEAST(), GREATEST(), etc
  and in rounding functions:
    ROUND(), TRUNCATE(), CEILING(), FLOOR().

  So for example,
    ROUND(COALESCE(UNIX_TIMESTAMP(dt1, 'tz'),
                   UNIX_TIMESTAMP(dt2, 'tz'))
  still has the "unix_timestamp" data type.

- An expression of the data type can now be used to set a TIMESTAMP
  column directly, without having to wrap into FROM_UNIXTIME():

    INSERT INTO t1 (timestamp_column) VALUES (UNIX_TIMESTAMP(...));
    UPDATE t1 SET timestamp_column= UNIX_TIMESTAMP(...);

  This change allows to avoid unnecessary time zone conversion
  which happens inside FROM_UNIXTIME(), and thus transfer
  the exact timeval value up to a TIMESTAMP column.

  This is an incompatible change, but it should be safe, because before
  the change an attempt to write UNIX_TIMESTAMP() directly to a
  TIMESTAMP column raised an error anyway.

- In all other contexts, the new data type "unix_timestamp" looks
  like BIGINT or DECIMAL.

Author: Julius Goryavsky
Author Date: 2024-05-03 13:16:25 UTC

MDEV-34071: Failure during the galera_3nodes_sr.GCF-336 test

This commit fixes sporadic failures in galera_3nodes_sr.GCF-336
test. The following changes have been made here:

1) A small addition to the test itself which should make
   it more deterministic by waiting for non-primary state
   before COMMIT;
2) More careful handling of the wsrep_ready variable in
   the server code (it should always be protected with mutex).

No additional tests are required.

Author: Yuchen Pei
Author Date: 2024-04-30 04:00:43 UTC

MDEV-34036 Reset spider_hton_ptr in spider_db_done()

Otherwise spider_direct_sql may still think the spider plugin is
available even after spider_db_done() was called.

Author: Sergei Golubchik
Author Date: 2024-05-05 15:13:20 UTC

MDEV-30727 Check spider_hton_ptr in spider udfs

UDF isn't supposed to use my_error(), it should return
the error message in the provided error message buffer

Fixes valgrind:

==93993== Conditional jump or move depends on uninitialised value(s)
==93993== at 0x484ECCD: strnlen (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==93993== by 0x1AD2B2C: process_str_arg (my_vsnprintf.c:259)
==93993== by 0x1AD47E0: my_vsnprintf_ex (my_vsnprintf.c:696)
==93993== by 0x1A3B91E: my_error (my_error.c:120)
==93993== by 0xF87BE8: udf_handler::fix_fields(THD*, Item_func_or_sum*, unsigned int, Item**) (item_func.cc:3638)

followup for 267dd5a993d5

Author: Sergei Golubchik
Author Date: 2024-04-25 10:47:23 UTC

sporadic failures of rpl.rpl_parallel_sbm

the test waits for the event to get stuck on MASTER_DELAY,
but on a slow/overloaded slave the event might pass MASTER_DELAY
before the test starts waiting.

Wait for the event to get stuck on the LOCK TABLES (after MASTER_DELAY),
the event cannot avoid that,

Author: Monty
Author Date: 2024-04-18 08:45:21 UTC

squash! 8372d62a863008ea6faa1547db5c275a5863f167

MDEV-33881 Userstat skips system tables inconsistently

Fixed that all no tables in 'mysql' schema are not included in userstat.
A beneif of this is that the server is not reading statistics tables
if mysql.proc or other tables in mysql is accessed.

Author: Kristian Nielsen
Author Date: 2024-05-04 09:09:37 UTC

Possible fix for stray deadlock kill after parallel replication of XA PREPARE, using somewhat undesirable global mutex in trx_disconnect_prepared()

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Kristian Nielsen
Author Date: 2024-05-04 09:02:41 UTC

MDEV-34042: Deadlock kill of XA PREPARE can break replication / rpl.rpl_parallel_multi_domain_xa sporadic failure

Refinement of the original patch.

Move the code to reset the kill up into the parent class
Xid_apply_log_event, to also fix the similar issue for XA COMMIT.

Increase the number of slave retries in the test case
rpl.rpl_parallel_multi_domain_xa to fix some sporadic failures. The test
generates massive amounts of conflicting transactions in multiple
independent domains, which can cause multiple rollback+retry for a
transaction as it conflicts with transactions in other domains one-by-one.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Kristian Nielsen
Author Date: 2024-05-04 09:09:37 UTC

Possible fix for stray deadlock kill after parallel replication of XA PREPARE, using somewhat undesirable global mutex in trx_disconnect_prepared()

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Kristian Nielsen
Author Date: 2024-05-04 09:02:41 UTC

MDEV-34042: Deadlock kill of XA PREPARE can break replication / rpl.rpl_parallel_multi_domain_xa sporadic failure

Refinement of the original patch.

Move the code to reset the kill up into the parent class
Xid_apply_log_event, to also fix the similar issue for XA COMMIT.

Increase the number of slave retries in the test case
rpl.rpl_parallel_multi_domain_xa to fix some sporadic failures. The test
generates massive amounts of conflicting transactions in multiple
independent domains, which can cause multiple rollback+retry for a
transaction as it conflicts with transactions in other domains one-by-one.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Alexander Barkov
Author Date: 2024-05-05 11:28:37 UTC

MDEV-34088 The TIMESTAMP value of '1970-01-01 00:00:00' can be indirectly inserted in strict mode

In strict mode a timestamp(0) column could be directly assigned from
another timestamp(N>0) column with the value '1970-01-01 00:00:00.1'
(at time zone '+00:00'), or with any other value '1970-01-01 00:00:00.XXXXXX'
with non-zero microsecond value XXXXXX.
This assignment happened silently without warnings or errors.

It worked as follows:

- The value {tv_sec=0, tv_usec=100000}, which is '1970-01-01 00:00:00.1'
  was rounded to {tv_sec=0, tv_usec=0}, which is '1970-01-01 00:00:00.0'

- Then {tv_sec=0, tv_usec=0} was silently re-interpreted as zero datetime.

After the fix this assignment always raises a warning,
which in case of the strict mode is escalated to an error.

The problem in this scenario is that '1970-01-01 00:00:00' cannot be stored,
because its timeval value {tv_sec=0, tv_usec=0} is reserved for zero datetimes.
Thus the warning should be raised no matter if sql_mode allows or disallows
zero dates.

Author: Alexander Barkov
Author Date: 2024-05-04 19:41:55 UTC

MDEV-34061 unix_timestamp(coalesce(timestamp_column)) returns NULL on '1970-01-01 00:00:00.000001'

Field_timestampf::val_native() checked only the
first four bytes to detect zero dates.
That was not enough. Fixing the code to check all packed_length()
bytes to detect zero dates.

Author: Alexander Barkov
Author Date: 2024-05-04 18:34:14 UTC

MDEV-34069 Zero datetime reinterprets as '1970-01-01 00:00:00' on field_datetime=field_timestamp

The code in Field_timestamp::save_in_field() did not catch
zero datetime and stored it to the other field like a usual value
using store_timestamp_dec(), which knows nothing about zero date and
treats {tv_sec=0, tv_usec=0} as a normal timeval value corresponding to
'1970-01-01 00:00:00 +00:00'.

Fixing the code to catch the special combination (ts==0 && sec_pat==0) and
store it using store_time_dec() with a zero datetime passed as an argument.

Author: Oleg Smirnov
Author Date: 2024-05-04 12:50:55 UTC

MDEV-34041 Display additional information for materialized subqueries in EXPLAIN/ANALYZE FORMAT=JSON

Author: Sergei Golubchik
Author Date: 2024-05-04 09:00:23 UTC

typo fixed

Author: Nikita Malyavin
Author Date: 2024-03-05 01:48:14 UTC

fix race in the MDEV-32614 test

Sometimes 'continue' signal could be missed.