Author: Sergei Golubchik
Author Date: 2023-05-25 09:42:25 UTC

columnstore warnings with gcc 12.x and 13.x

Author: Monty
Author Date: 2023-05-27 12:35:12 UTC

MDEV-31237 Assertion `!(tab->select && tab->select->quick)' failed in make_join_readinfo

The problem was a wrong assert. I changed it to match the code in
best_access_path().

The given test case was a bit tricky for the optimizer, which first
decided on using a index scan (because of force index), but then
test_if_skip_sort_order() decided to use range anyway to handle
distinct.

Author: Monty
Author Date: 2023-05-27 12:35:12 UTC

MDEV-31237 Assertion `!(tab->select && tab->select->quick)' failed in make_join_readinfo

The problem was a wrong assert. I changed it to match the code in
best_access_path().

The given test case was a bit tricky for the optimizer, which first
decided on using a index scan (because of force index), but then
test_if_skip_sort_order() decided to use range anyway to handle
distinct.

Author: Monty
Author Date: 2023-05-27 13:31:22 UTC

MDEV-31226 Server crash or assertion failure with row size close to join_buffer_size

The problem was that JOIN_CACHE::alloc_buffer() did not check if the
given join_buffer_value is less than the query require.

Added a check for this and disabled join cache if it cannot be used.

Author: Monty
Author Date: 2023-05-27 13:31:22 UTC

MDEV-31226 Server crash or assertion failure with row size close to join_buffer_size

The problem was that JOIN_CACHE::alloc_buffer() did not check if the
given join_buffer_value is less than the query require.

Added a check for this and disabled join cache if it cannot be used.

Author: Monty
Author Date: 2023-05-27 12:35:12 UTC

MDEV-31237 Assertion `!(tab->select && tab->select->quick)' failed in make_join_readinfo

The problem was a wrong assert. I changed it to match the code in
best_access_path().

The given test case was a bit tricky for the optimizer, which first
decided on using a index scan (because of force index), but then
test_if_skip_sort_order() decided to use range anyway to handle
distinct.

Author: Monty
Author Date: 2023-05-26 14:26:42 UTC

MDEV-31356: Range cost cost calculations does not take into account join_buffer

Cost changes caused by this change:
- range queries with join buffer now have a notable smaller cost.
- range ranges are bit more expensive as the MULTI_RANGE_COST is now
  properly applied to it in all cases (this extra cost is equal to a
  key lookup).
- table scan cost is slight smaller as we now assume data is cached in
  the engine after the first scan pass. (We did this before for range
  scans and other access methods).
- partition tables had wrong values for max_row_blocks and
  max_index_blocks. Correcting this, causes range access on
  partitioned tables to have slightly higher cost because of the
  increased estimated IO.
- Using first match + join buffer caused 'filtered' to be calcualted
  wrong. (Only affected EXPLAIN, not query costs).
- Added cost_without_join_buffer to optimizer_trace.
- check_quick_select() adjusted the number of rows according to persistent
  statistics, but did not adjust cost. Now fixed.

The big change in the patch are:

- In best_access_path(), where we now are using storing the cost in
  'ALL_READ_COST cost' and only converting it to a double at the end.
   This allows us to more exactly calculate the effect of the join_cache.
- In JOIN_TAB::estimate_scan_time(), store the cost also in a
  ALL_READ_COST object.

One of effect if this change is that when joining very small tables:

t1 some_access_method
t2 range
t3 ALL Use join buffer

This is swiched to

t1 some_access_method
t3 ALL
t2 range use join buffer

Both plans has the same cost, but as table scan in this case has less
cost than rang, the table scan will be considered first and thus have
precidence.

Test case changes:
- optimizer_trace - Addition of cost_without_join_buffer
- subselect_mat_cost_bugs - Small tables and scan versus range
- range & range_mrr_icp - Range + join_cache is faster than ref
- optimizer_trace - cost_without_join_buffer, smaller scan cost,
                             range setup cost.
- mrr - range+join_buffer used as smaller cost

Author: Marko Mäkelä
Author Date: 2023-05-26 13:42:22 UTC

Merge

Author: Marko Mäkelä
Author Date: 2023-05-26 13:40:07 UTC

MDEV-31350: Hang in innodb.recovery_memory

buf_flush_page_cleaner(): Whenever buf_pool.ran_out(), invoke
buf_pool.get_oldest_modification(0) so that all clean blocks
will be removed from buf_pool.flush_list and buf_flush_LRU_list_batch()
will be able to evict some pages.

This fixes a regression that was likely caused by
commit a55b951e6082a4ce9a1f2ed5ee176ea7dbbaf1f2 (MDEV-26827).

Author: Alexander Barkov
Author Date: 2023-05-26 12:04:02 UTC

Extra tests for MDEV-30483 After upgrade to 10.6 from Mysql 5.7 seeing "InnoDB: Column last_update in table mysql.innodb_table_stats is BINARY(4) NOT NULL but should be INT UNSIGNED NOT NULL"

Adding tests demonstrating that columns:
- mysql.innodb_table_stats.last_update
- mysql.innodb_index_stats.last_update

contain sane values close to NOW() rathar than a garbage.

Tests cover these three underlying TIMESTAMP data formats:

- MariaDB Field_timestamp0 - UINT4 based
  Like in a MariaDB native installation running with mysql56_temporal_format=0

- MariaDB Field_timestampf - BINARY(4) based, with UNSIGNED_FLAG
  Like in a MariaDB native installation running with mysql56_temporal_format=1

- MySQL-alike Field_timestampf - BINARY(4) based, without UNSIGNED_FLAG
  Like with a MariaDB server running over a MySQL-5.6 directory
  (e.g. during a migragion).

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-05-26 12:04:41 UTC

- Remove the change buffer entries for the freeing page if the block
exist in buffer pool

Author: Yuchen Pei
Author Date: 2023-05-26 01:00:49 UTC

MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs

We introduce a new set of synchronisation primitives ("query ready")
that are signaled when the server is ready to take queries, and make
spider initialisation use them instead of the primitives associated
with server start.

During spider initialisation, if the server is query ready, proceed
with the initialisation synchronously (requiring the sql
service). Otherwise, leave the init to the first sts bg thread as
usual.

When creating a spider handler before the spider init is done, if the
server is not query ready, just give up and return NULL. If the server
is query ready, then wait for a specific amount of time (5s) for the
spider init to be done, and on timeout give up and return NULL.

How this fixes / prevents various init bugs:

MDEV-27233 (spider installed and used in init file). Before -
spider_create_handler() called by read_init_file() waits for bg thread
to finish init, whereas the bg thread is waiting for server
start. After - spider init is run synchronously because the server is
query ready when read_init_file(). No funny business with concurrency.

MDEV-29904 (spider loaded with plugin-load-add). Before (before the
original patch to MDEV-27233 was reverted) - fully synchronous spider
init fails due to dependencies on Aria / acl init. After - spider init
is run async, and as soon as the server is query ready, the init
proceeds and succeeds.

MDEV-22979 (mysqld --bootstrap --plugin-load-add=ha_spider). Before -
bootstrap queries causes call to spider_create_handler(), which waits
for bg thread to finish init, whereas bg thread waits for server
start. After - spider init is run async, waiting for server to be
query ready. spider_create_hander() notices bg thread still trying to
init spider, and server is not query ready so it gives up.

MDEV-28218 (spider is loaded in a query after server start,
immediately before another query dropping a spider table). Before - bg
thread is created to init spider async, while the server proceeds to
the next query that tries to drop a spider system table. race
condition causes the situation where the server holds the mdl lock to
the table and waits for the bg thread in spider_create_handler(), and
the bg thread waiting for the mdl lock to create the table. After -
since server has already started, spider init is run synchronously,
thus no concurrency issues. Same applies to the variant of MDEV-28218
represented by mdev_28218_init_file where the queries are in an init
file.

Another variant of MDEV-28218 represented by mdev_28218_mixed loads
spider with plugin-load-add and executes the drop table query in an
init_file. Spider init is async because the server was not query
ready. As soon as the the server becomes query ready the server tries
to execute the drop table query. It waits on the init in
spider_create_handler(), but because of the deadlock it times out
after 5 seconds. The drop table query did not find any table, but
spider init then proceeds as it finally acquires the mdl lock, and
creates the table. The moral of the story is to advise users not to
mix plugin-load-add=ha_spider with init_file containing queries
related to spider system tables. Queries using spider engine to create
tables are still ok because there's no deadlock.

MDEV-30370 (mysqld --wsrep-recover --plugin-load-add=spider). The
timed wait in the patch for MDEV-30370 still applies, so it still
passes.

More on the design:

Compared to the server start primitives, the query ready primitives
are signaled earlier ("query ready point") to avoid unnecessary wait
and deadlocks.

The spider initialisation does not only require other plugins like
Aria to be initialised first, but also acl init (due to its creation
of udfs) as well as the creation of the `mysql` database (due to its
system tables being in this database). Thus the point where it can go
ahead with the initialisation should be after the bootstrap.

By the time the server can accept queries by reading from init files,
it can definitely accept all spider init queries. So the point of
spider initialisation can be before this point.

Looking at the lines between boostrap and reading init file, the most
natural position is just before the call to read_init_file(), where we
can also give the primitives a most meaningful name "query ready".

Author: Angelique Sklavounos
Author Date: 2023-05-25 23:10:53 UTC

MDEV-30214: Generalize log filename in IO Error message

Author: Brandon Nesterenko
Author Date: 2023-05-25 22:04:23 UTC

MDEV-13915: Fix binlog_encrypion.rpl_parallel

Author: Rucha Deodhar
Author Date: 2023-05-25 10:15:43 UTC

MDEV-30145: JSON_TABLE: allow to retrieve the key when iterating on JSON
objects

Idea behind implementation:
We get the json object specified by the json path. Then, transform it into
key-value pairs by going over the json. Get each key-value pair
one-by-one and return the result.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-05-25 12:49:51 UTC

MDEV-31264 Purge trying to access freed secondary index page

- InnoDB purge tries to access aborted secondary index and access
the freed secondary index root page.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-25 10:53:43 UTC

Merge branch '10.11' into bb-10.11-release

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-25 10:02:34 UTC

Merge branch '10.10' into bb-10.10-release

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-25 10:02:34 UTC

Merge branch '10.10' into bb-10.10-release

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-25 09:35:05 UTC

Merge branch '10.9' into bb-10.9-release

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-25 09:35:05 UTC

Merge branch '10.9' into bb-10.9-release

Author: Marko Mäkelä
Author Date: 2023-05-25 08:28:02 UTC

MDEV-31234 InnoDB does not free UNDO after the fix of MDEV-30671

trx_purge_truncate_rseg_history(): Add a parameter to specify if
the entire rollback segment is safe to be freed. If not, we may
still be able to invoke trx_undo_truncate_start() and free some pages,
but we will avoid leaking undo log pages that are not yet ready
to be processed. This fixes a regression that was introduced in
MDEV-30671.

trx_purge_free_segment(), trx_purge_truncate_rseg_history():
Replace some unreachable code with debug assertions.
A buffer-fix does prevent pages from being evicted
from the buffer pool; see buf_page_t::can_relocate().

trx_sys_t::any_active_transactions(): Separately count XA PREPARE
transactions.

srv_purge_should_exit(): Terminate slow shutdown if the history size
does not change and XA PREPARE transactions exist in the system.
This will avoid a hang of the test innodb.recovery_shutdown.

Author: Alexander Barkov
Author Date: 2023-05-24 05:18:14 UTC

Basic support for timestamps with time zone.

Author: Yuchen Pei
Author Date: 2023-05-25 02:52:38 UTC

MDEV-30435 MDEV-30981 Fix ubsan errors w.r.t. memcpy in spd_trx.cc

Extract the indexed string memcopy pattern in spd_trx.cc to a static
inline function.

Also updated the ubsan check in mdev_26541.test (h/t roel).

Author: Yuchen Pei
Author Date: 2023-05-25 02:52:38 UTC

MDEV-30435 MDEV-30981 Fix ubsan errors w.r.t. memcpy in spd_trx.cc

Extract the indexed string memcopy pattern in spd_trx.cc to a static
inline function.

Also updated the ubsan check in mdev_26541.test (h/t roel).

Author: Monty
Author Date: 2023-05-24 14:32:19 UTC

Fixed typo in xtrabackup.c

Author: Sergei Golubchik
Author Date: 2023-05-24 10:33:35 UTC

fix federatedx select pushdown after rebase

Author: Monty
Author Date: 2023-05-23 12:32:50 UTC

MDEV-7389 Request: log warnings into SQL_ERROR_LOG

Changes:
- Audit_null records and displays warning count
- sql_error_log prints warnings

Reviewer: Alexey Botchkov <holyfoot@askmonty.org>

Author: Monty
Author Date: 2023-05-23 12:32:50 UTC

MDEV-7389 Request: log warnings into SQL_ERROR_LOG

Changes:
- Audit_null records and displays warning count
- sql_error_log prints warnings

Reviewer: Alexey Botchkov <holyfoot@askmonty.org>

Author: Dmitry Shulga
Author Date: 2023-05-24 08:29:09 UTC

MDEV-14959: Control over memory allocated for SP/PS

Use sp_head::executed_counter only for debug output. The decision
to mark sp_head's mem_root as read only is performed based on
the fact that every SP instruction has been executed at least once.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-24 07:54:30 UTC

Merge branch '10.11' into 11.0

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-24 07:38:18 UTC

Merge branch '10.6' into 10.8

Author: Yuchen Pei
Author Date: 2023-05-23 05:42:35 UTC

MDEV-31269 Fixing exists2in 2nd ps execution segfault

In the exists2in transformation, if 1) we are in a ps/sp execution and
2) the equalities picked out for the transformation intersect with the
temporary query arena free list, skip the transformation.

Further analysis:

Consider the following example:

--8<---------------cut here---------------start------------->8---
set optimizer_switch=default;
CREATE TABLE t1 (a INT);
CREATE TABLE t2 (b INT);
PREPARE st FROM "
SELECT * FROM t2
HAVING 0 IN (
  SELECT a FROM t1
  WHERE EXISTS (
    SELECT a FROM t1
    WHERE b = a
  )
)
";
EXECUTE st;
EXECUTE st;
drop table t1, t2;
--8<---------------cut here---------------end--------------->8---

1. During the JOIN::prepare() of the first execution, the Item_field referring to t2.b is replaced by an Item_ref
2. The item_ref is moved around in the exists2in transformation and subsequent optimizations
3. At the end of the 1st ps execution, the Item_ref is freed due to its creation in the ps execution as part of the cleanup
4. During the JOIN::prepare() of the second execution, it tries to restore the optimized version of the statement, in doing so it does a sanity check and causes a sigsegv when trying to access the freed Item_ref.

Why the problem does not occur with the transformation disabled:

- during step 1 above, the replacement also adds an item to the change_list
- during step 2 above, the optimizations cause the equality {{b = a}} to be moved around as a whole, thereby moving its {{args}} which is an {{Item **}} instead of the {{item *}}'s as a whole
- during step 3 above, the change is rolled back, so the {{Item **}} that is the args of the {{Item_func_eq}} goes from {{b (Item_ref) = a}} becomes {{b (Item_field) = a}}
- during step 4 above, there's no use-after-free in the restored optimized statment because it automatically contains the {{b (Item_field) = a}}.

Why we can't do the same with the transformation enabled:

- during step 2, we break the {{b = a}} into the outer_expr (b) and local_field (a), add the outer_expr to left_expr and local_field to the inner select item list. we cannot move the equality as a whole during this transformation.

Author: Yuchen Pei
Author Date: 2023-05-23 05:42:35 UTC

MDEV-31269 Fixing exists2in 2nd ps execution segfault

In the exists2in transformation, if 1) we are in a ps/sp execution and
2) the equalities picked out for the transformation intersect with the
temporary query arena free list, skip the transformation.

Further analysis:

Consider the following example:

--8<---------------cut here---------------start------------->8---
set optimizer_switch=default;
CREATE TABLE t1 (a INT);
CREATE TABLE t2 (b INT);
PREPARE st FROM "
SELECT * FROM t2
HAVING 0 IN (
  SELECT a FROM t1
  WHERE EXISTS (
    SELECT a FROM t1
    WHERE b = a
  )
)
";
EXECUTE st;
EXECUTE st;
drop table t1, t2;
--8<---------------cut here---------------end--------------->8---

1. During the JOIN::prepare() of the first execution, the Item_field referring to t2.b is replaced by an Item_ref
2. The item_ref is moved around in the exists2in transformation and subsequent optimizations
3. At the end of the 1st ps execution, the Item_ref is freed due to its creation in the ps execution as part of the cleanup
4. During the JOIN::prepare() of the second execution, it tries to restore the optimized version of the statement, in doing so it does a sanity check and causes a sigsegv when trying to access the freed Item_ref.

Why the problem does not occur with the transformation disabled:

- during step 1 above, the replacement also adds an item to the change_list
- during step 2 above, the optimizations cause the equality {{b = a}} to be moved around as a whole, thereby moving its {{args}} which is an {{Item **}} instead of the {{item *}}'s as a whole
- during step 3 above, the change is rolled back, so the {{Item **}} that is the args of the {{Item_func_eq}} goes from {{b (Item_ref) = a}} becomes {{b (Item_field) = a}}
- during step 4 above, there's no use-after-free in the restored optimized statment because it automatically contains the {{b (Item_field) = a}}.

Why we can't do the same with the transformation enabled:

- during step 2, we break the {{b = a}} into the outer_expr (b) and local_field (a), add the outer_expr to left_expr and local_field to the inner select item list. we cannot move the equality as a whole during this transformation.

Author: Marko Mäkelä
Author Date: 2023-05-24 05:36:30 UTC

Merge bb-10.5-release into bb-10.6-release

Author: Marko Mäkelä
Author Date: 2023-05-24 05:28:20 UTC

Merge bb-10.5-release into 10.5

Author: Marko Mäkelä
Author Date: 2023-05-24 05:25:26 UTC

MDEV-31234 fixup: Free some UNDO pages earlier

trx_purge_truncate_rseg_history(): Add a parameter to specify if
the entire rollback segment is safe to be freed. If not, we may
still be able to invoke trx_undo_truncate_start() and free some pages.

Author: Marko Mäkelä
Author Date: 2023-05-23 14:20:35 UTC

Experiment: Remove handler::unlock_row() and friends

Failing test(s): innodb.innodb main.partition_innodb_semi_consistent innodb.innodb-semi-consistent innodb.innodb_lock_wait_timeout_1 main.concurrent_innodb_unsafelog main.unsafe_binlog_innodb innodb.create_table_insert_skip_locked rpl.rpl_unsafe_statements innodb.partition_locking innodb.skip_locked_nowait innodb.deadlock_wait_lock_race

Author: Marko Mäkelä
Author Date: 2023-05-23 12:32:20 UTC

Remove row_prebuilt_t::fetch_cache[]

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-05-23 10:45:43 UTC

- Handle the undo log truncation recover code

Author: midenok
Author Date: 2023-05-22 16:11:21 UTC

MDEV-25495 AUTO argument to specify max number of partitions

Example:

  create table t1 (x int) with system versioning
  partition by system_time interval 1 hour auto 25;

Keeps 24 history partitions (and 1 current partition).

Note, MAX_SYM cannot be used for syntax expansion as this is function
name (sql_functions[] in lex.h)

fast_alter_partition_table() changes:

Auto-add like auto-drop now goes through REBUILD branch. MDL_EXCLUSIVE
is taken before mysql_change_partitions() like ALTER_PARTITION_ADD
branch does.

ha_partition::change_partitions() changes:

ADD PARTITION and DROP PARTITION in one command (as well as any other
alter partition operations) is not supported by the original code. The
patch partially solves this problem: ha_partition::change_partitions()
now can accept both PART_TO_BE_DROPPED and PART_TO_BE_ADDED states in
the partitions list. Originally change_partitions() was not used for
arbitrary drop of partitions, only REORGANIZE and COALESCE could be
processed by this call.

For REORGANIZE m_part_info->partitions contains new partitions set and
m_part_info->temp_partitions contains PART_TO_BE_REORGED
partitions. But we cannot use temp_partitions for dropping partitions
because they go through copy_partitions() phase (see how
m_reorged_parts initialized).

For COALESCE m_part_info->partitions in the beginning of the list
contains normal partitions that will remain when alter finishes, the
end of the list contains PART_REORGED_DROPPED partitions, so Step 4
can ignore them.

Step 4 for new_file_array preparation didn't skip PART_TO_BE_DROPPED
partitions in m_file. Now we increment orig_count for them.

Step 5 now properly iterates new_file_array in respect of skipping
PART_TO_BE_DROPPED while iterating m_part_info->partitions.

For full ADD+DROP support it would be needed to rework
prep_alter_part_table() as well. Now PART_TO_BE_DROPPED state is set
in vers_create_partitions() after prep_alter_part_table().

Author: Monty
Author Date: 2023-05-23 07:02:33 UTC

Optimized version of safe_strcpy()

Note: We should replace most case of safe_strcpy() with strmake() to avoid
the not needed zerofill.

Author: =?utf-8?q?Jan_Lindstr=C3=B6m?=
Author Date: 2023-05-02 09:42:13 UTC

MDEV-30197 : Missing DBUG_RETURN or DBUG_VOID_RETURN macro in function "Wsrep_schema::restore_view()"

Here user is starting server with unsupported client charset.
We need to create wsrep_schema tables using explicit latin1
charset to avoid errors in restoring view.

Author: Sergey Petrunia
Author Date: 2023-05-22 19:51:50 UTC

MDEV-31067: selectivity_from_histogram >1.0 for a DOUBLE_PREC_HB - V3-Fix1

Variant #3, fix for trivial errors.

Author: Rex Johnston
Author Date: 2023-05-19 03:22:15 UTC

MDEV-31327 Range Histogram selectivity estimates added, not merged

When the optimizer is evaluating join order and estimating rows produced a join order, it loops through column constraints adding rather than merging selectivity estimates.

Author: Lawrin Novitsky
Author Date: 2023-05-22 13:07:05 UTC

MDEV-31064 Changes in a SP are not immediately seen in I_S.parameters

If procedure is changed in one connection, and other procedure has
already called the initial version of the procedure, the query to
INFORMATION_SCHEMA.PARAMETERS would use obsolete information from sp
cache for that connection. That happens because cache invalidating
method only increments cache version, and does not flush (all) the
cache(s), and changing of a procedure only invalidates cache, and
removes the procedure's cache entry from local thread cache only.

The fix adds the check if sp info obtained from the cache for forming of
results for the query to I_S, is not obsoleted.

The test has been added to main.information_schema. It changes the SP in
one connection, and ensures, that the change is seen in the query to the
I_S.PARAMETERS in other connection, that already has called the
procedure before the change.

Author: Oleg Smirnov
Author Date: 2023-05-19 09:53:39 UTC

MDEV-29630 WIP: remove index initialization from join_read_always_key()

Author: Brandon Nesterenko
Author Date: 2023-04-18 19:22:43 UTC

MDEV-31038: Parallel Replication Breaks if XA PREPARE Fails Updating Slave GTID State

If a replica failed to update the GTID slave state when committing
an XA PREPARE, the replica would retry the transaction and get an
out-of-order GTID error. This is because the commit phase of an XA
PREPARE is bifurcated. That is, first, the prepare is handled by the
relevant storage engines. Then second, the GTID slave state is
updated as a separate autocommit transaction. If the second phase
fails, and the transaction is retried, then the same transaction is
attempted to be committed again, resulting in a GTID out-of-order
error.

This patch fixes this error by immediately stopping the slave and
reporting the appropriate error. That is, there was logic to bypass
the error when updating the GTID slave state table if the underlying
error is allowed for retry on a parallel slave. This patch adds a
parameter to disallow the error bypass, thereby forcing the error
state to still happen.

Reviewed By
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Teemu Ollakka
Author Date: 2023-04-17 13:04:01 UTC

MDEV-29293 MariaDB stuck on starting commit state

This is a backport from 10.5.

The problem seems to be a deadlock between KILL command execution
and BF abort issued by an applier, where:
* KILL has locked victim's LOCK_thd_kill and LOCK_thd_data.
* Applier has innodb side global lock mutex and victim trx mutex.
* KILL is calling innobase_kill_query, and is blocked by innodb
  global lock mutex.
* Applier is in wsrep_innobase_kill_one_trx and is blocked by
  victim's LOCK_thd_kill.

The fix in this commit removes the TOI replication of KILL command
and makes KILL execution less intrusive operation. Aborting the
victim happens now by using awake_no_mutex() and ha_abort_transaction().
If the KILL happens when the transaction is committing, the
KILL operation is postponed to happen after the statement
has completed in order to avoid KILL to interrupt commit
processing.

Notable changes in this commit:
* wsrep client connections's error state may remain sticky after
  client connection is closed. This error message will then pop
  up for the next client session issuing first SQL statement.
  This problem raised with test galera.galera_bf_kill.
  The fix is to reset wsrep client error state, before a THD is
  reused for next connetion.
* Release THD locks in wsrep_abort_transaction when locking
  innodb mutexes. This guarantees same locking order as with applier
  BF aborting.
* BF abort from MDL was changed to do BF abort on server/wsrep-lib
  side first, and only then do the BF abort on InnoDB side. This
  removes the need to call back from InnoDB for BF aborts which originate
  from MDL and simplifies the locking.
* Removed wsrep_thd_set_wsrep_aborter() from service_wsrep.h.
  The manipulation of the wsrep_aborter can be done solely on
  server side. Moreover, it is now debug only variable and
  could be excluded from optimized builds.
* Remove LOCK_thd_kill from wsrep_thd_LOCK/UNLOCK to allow more
  fine grained locking for SR BF abort which may require locking
  of victim LOCK_thd_kill. Added explicit call for
  wsrep_thd_kill_LOCK/UNLOCK where appropriate.
* Wsrep-lib was updated to version which allows external
  locking for BF abort calls.

Changes to MTR tests:
* Disable galera_bf_abort_group_commit. This test is going to
  be removed (MDEV-30855).
* Record galera_gcache_recover_manytrx as result file was incomplete.
  Trivial change.
* Make galera_create_table_as_select more deterministic:
  Wait until CTAS execution has reached MDL wait for multi-master
  conflict case. Expected error from multi-master conflict is
  ER_QUERY_INTERRUPTED. This is because CTAS does not yet have open
  wsrep transaction when it is waiting for MDL, query gets interrupted
  instead of BF aborted. This should be addressed in separate task.
* A new test galera_kill_group_commit to verify correct behavior
  when KILL is executed while the transaction is committing.

Co-authored-by: Seppo Jaakola <seppo.jaakola@iki.fi>
Co-authored-by: Jan Lindström <jan.lindstrom@galeracluster.com>
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Teemu Ollakka
Author Date: 2023-04-19 13:51:55 UTC

MDEV-29293 MariaDB stuck on starting commit state

This commit contains a merge from 10.5-MDEV-29293-squash
into 10.6.

Although the bug MDEV-29293 was not reproducible with 10.6,
the fix contains several improvements for wsrep KILL query and
BF abort handling, and addresses the following issues:

* MDEV-30307 KILL command issued inside a transaction is
  problematic for galera replication:
  This commit will remove KILL TOI replication, so Galera side
  transaction context is not lost during KILL.
* MDEV-21075 KILL QUERY maintains nodes data consistency but
  breaks GTID sequence: This is fixed as well as KILL does not
  use TOI, and thus does not change GTID state.
* MDEV-30372 Assertion in wsrep-lib state: This was caused by
  BF abort or KILL when local transaction was in the middle
  of group commit. This commit disables THD::killed handling
  during commit, so the problem is avoided.
* MDEV-30963 Assertion failure !lock.was_chosen_as_deadlock_victim
  in trx0trx.h:1065: The assertion happened when the victim was
  BF aborted via MDL while it was committing. This commit changes
  MDL BF aborts so that transactions which are committing cannot
  be BF aborted via MDL. The RQG grammar attached in the issue
  could not reproduce the crash anymore.

Original commit message from 10.5 fix:

    MDEV-29293 MariaDB stuck on starting commit state

    The problem seems to be a deadlock between KILL command execution
    and BF abort issued by an applier, where:
    * KILL has locked victim's LOCK_thd_kill and LOCK_thd_data.
    * Applier has innodb side global lock mutex and victim trx mutex.
    * KILL is calling innobase_kill_query, and is blocked by innodb
      global lock mutex.
    * Applier is in wsrep_innobase_kill_one_trx and is blocked by
      victim's LOCK_thd_kill.

    The fix in this commit removes the TOI replication of KILL command
    and makes KILL execution less intrusive operation. Aborting the
    victim happens now by using awake_no_mutex() and ha_abort_transaction().
    If the KILL happens when the transaction is committing, the
    KILL operation is postponed to happen after the statement
    has completed in order to avoid KILL to interrupt commit
    processing.

    Notable changes in this commit:
    * wsrep client connections's error state may remain sticky after
      client connection is closed. This error message will then pop
      up for the next client session issuing first SQL statement.
      This problem raised with test galera.galera_bf_kill.
      The fix is to reset wsrep client error state, before a THD is
      reused for next connetion.
    * Release THD locks in wsrep_abort_transaction when locking
      innodb mutexes. This guarantees same locking order as with applier
      BF aborting.
    * BF abort from MDL was changed to do BF abort on server/wsrep-lib
      side first, and only then do the BF abort on InnoDB side. This
      removes the need to call back from InnoDB for BF aborts which originate
      from MDL and simplifies the locking.
    * Removed wsrep_thd_set_wsrep_aborter() from service_wsrep.h.
      The manipulation of the wsrep_aborter can be done solely on
      server side. Moreover, it is now debug only variable and
      could be excluded from optimized builds.
    * Remove LOCK_thd_kill from wsrep_thd_LOCK/UNLOCK to allow more
      fine grained locking for SR BF abort which may require locking
      of victim LOCK_thd_kill. Added explicit call for
      wsrep_thd_kill_LOCK/UNLOCK where appropriate.
    * Wsrep-lib was updated to version which allows external
      locking for BF abort calls.

    Changes to MTR tests:
    * Disable galera_bf_abort_group_commit. This test is going to
      be removed (MDEV-30855).
    * Make galera_var_retry_autocommit result more readable by echoing
      cases and expectations into result. Only one expected result for
      reap to verify that server returns expected status for query.
    * Record galera_gcache_recover_manytrx as result file was incomplete.
      Trivial change.
    * Make galera_create_table_as_select more deterministic:
      Wait until CTAS execution has reached MDL wait for multi-master
      conflict case. Expected error from multi-master conflict is
      ER_QUERY_INTERRUPTED. This is because CTAS does not yet have open
      wsrep transaction when it is waiting for MDL, query gets interrupted
      instead of BF aborted. This should be addressed in separate task.
    * A new test galera_bf_abort_registering to check that registering trx gets
      BF aborted through MDL.
    * A new test galera_kill_group_commit to verify correct behavior
      when KILL is executed while the transaction is committing.

    Co-authored-by: Seppo Jaakola <seppo.jaakola@iki.fi>
    Co-authored-by: Jan Lindström <jan.lindstrom@galeracluster.com>

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Teemu Ollakka
Author Date: 2023-04-17 13:04:01 UTC

MDEV-29293 MariaDB stuck on starting commit state

The problem seems to be a deadlock between KILL command execution
and BF abort issued by an applier, where:
* KILL has locked victim's LOCK_thd_kill and LOCK_thd_data.
* Applier has innodb side global lock mutex and victim trx mutex.
* KILL is calling innobase_kill_query, and is blocked by innodb
  global lock mutex.
* Applier is in wsrep_innobase_kill_one_trx and is blocked by
  victim's LOCK_thd_kill.

The fix in this commit removes the TOI replication of KILL command
and makes KILL execution less intrusive operation. Aborting the
victim happens now by using awake_no_mutex() and ha_abort_transaction().
If the KILL happens when the transaction is committing, the
KILL operation is postponed to happen after the statement
has completed in order to avoid KILL to interrupt commit
processing.

Notable changes in this commit:
* wsrep client connections's error state may remain sticky after
  client connection is closed. This error message will then pop
  up for the next client session issuing first SQL statement.
  This problem raised with test galera.galera_bf_kill.
  The fix is to reset wsrep client error state, before a THD is
  reused for next connetion.
* Release THD locks in wsrep_abort_transaction when locking
  innodb mutexes. This guarantees same locking order as with applier
  BF aborting.
* BF abort from MDL was changed to do BF abort on server/wsrep-lib
  side first, and only then do the BF abort on InnoDB side. This
  removes the need to call back from InnoDB for BF aborts which originate
  from MDL and simplifies the locking.
* Removed wsrep_thd_set_wsrep_aborter() from service_wsrep.h.
  The manipulation of the wsrep_aborter can be done solely on
  server side. Moreover, it is now debug only variable and
  could be excluded from optimized builds.
* Remove LOCK_thd_kill from wsrep_thd_LOCK/UNLOCK to allow more
  fine grained locking for SR BF abort which may require locking
  of victim LOCK_thd_kill. Added explicit call for
  wsrep_thd_kill_LOCK/UNLOCK where appropriate.
* Wsrep-lib was updated to version which allows external
  locking for BF abort calls.

Changes to MTR tests:
* Disable galera_bf_abort_group_commit. This test is going to
  be removed (MDEV-30855).
* Record galera_gcache_recover_manytrx as result file was incomplete.
  Trivial change.
* Make galera_create_table_as_select more deterministic:
  Wait until CTAS execution has reached MDL wait for multi-master
  conflict case. Expected error from multi-master conflict is
  ER_QUERY_INTERRUPTED. This is because CTAS does not yet have open
  wsrep transaction when it is waiting for MDL, query gets interrupted
  instead of BF aborted. This should be addressed in separate task.
* A new test galera_kill_group_commit to verify correct behavior
  when KILL is executed while the transaction is committing.

Co-authored-by: Seppo Jaakola <seppo.jaakola@iki.fi>
Co-authored-by: Jan Lindström <jan.lindstrom@galeracluster.com>
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Yuchen Pei
Author Date: 2023-05-19 10:06:31 UTC

MDEV-22534 Decorrelate IN subquery

Transform

in (select inner_col' from inner_table where inner_col = outer_col)

to

, outer_col in (select inner_col', inner_col from inner_table)

Achieved by implementing Item_in_subselect::exists2in_processor(),
accompanied with comprehensive test coverage.

Caveat:

- The fix of 2nd ps execution segfault requires
  HAVE_PSI_STATEMENT_INTERFACE because it checks
  thd->m_statement_state.m_parent_prepared_stmt to determine whether we
  are inside a ps execution. Is there a better way to determine this?
- Cannot recognise bad item mismatch in equalities that causes
  materialization to not materialize down the road

Author: Vlad Lesin
Author Date: 2023-05-12 09:11:53 UTC

MDEV-31185 rw_trx_hash_t::find() unpins pins too early

rw_trx_hash_t::find() acquires element->mutex, then unpins pins, used for
lf_hash element search. After that the "element" can be deallocated and
reused by some other thread.

If we take a look rw_trx_hash_t::insert()->lf_hash_insert()->lf_alloc_new()
calls, we will not find any element->mutex acquisition, as it was not
initialized yet before it's allocation. rw_trx_hash_t::insert() can reuse
the chunk, unpinned in rw_trx_hash_t::find().

The scenario is the following:

1. Thread 1 have just executed lf_hash_search() in
rw_trx_hash_t::find(), but have not acquired element->mutex yet.
2. Thread 2 have removed the element from hash table with
rw_trx_hash_t::erase() call.
3. Thread 1 acquired element->mutex and unpinned pin 2 pin with
lf_hash_search_unpin(pins) call.
4. Some thread purged memory of the element.
5. Thread 3 reused the memory for the element, filled element->id,
element->trx.
6. Thread 1 crashes with failed "DBUG_ASSERT(trx_id == trx->id)"
assertion.

Note that trx_t objects are also reused, see the code around trx_pools
for details.

The fix is to invoke "lf_hash_search_unpin(pins);" after element->trx is
stored in local variable in rw_trx_hash_t::find().

Reviewed by: Nikita Malyavin, Marko Mäkelä.

Author: Vlad Lesin
Author Date: 2023-05-12 09:11:53 UTC

MDEV-31185 rw_trx_hash_t::find() unpins pins too early

rw_trx_hash_t::find() acquires element->mutex, then unpins pins, used for
lf_hash element search. After that the "element" can be deallocated and
reused by some other thread.

If we take a look rw_trx_hash_t::insert()->lf_hash_insert()->lf_alloc_new()
calls, we will not find any element->mutex acquisition, as it was not
initialized yet before it's allocation. rw_trx_hash_t::insert() can reuse
the chunk, unpinned in rw_trx_hash_t::find().

The scenario is the following:

1. Thread 1 have just executed lf_hash_search() in
rw_trx_hash_t::find(), but have not acquired element->mutex yet.
2. Thread 2 have removed the element from hash table with
rw_trx_hash_t::erase() call.
3. Thread 1 acquired element->mutex and unpinned pin 2 pin with
lf_hash_search_unpin(pins) call.
4. Some thread purged memory of the element.
5. Thread 3 reused the memory for the element, filled element->id,
element->trx.
6. Thread 1 crashes with failed "DBUG_ASSERT(trx_id == trx->id)"
assertion.

Note that trx_t objects are also reused, see the code around trx_pools
for details.

The fix is to invoke "lf_hash_search_unpin(pins);" after element->trx is
stored in local variable in rw_trx_hash_t::find().

Reviewed by: Nikita Malyavin, Marko Mäkelä.

Author: Vlad Lesin
Author Date: 2023-05-12 09:11:53 UTC

MDEV-31185 rw_trx_hash_t::find() unpins pins too early

rw_trx_hash_t::find() acquires element->mutex, then unpins pins, used for
lf_hash element search. After that the "element" can be deallocated and
reused by some other thread.

If we take a look rw_trx_hash_t::insert()->lf_hash_insert()->lf_alloc_new()
calls, we will not find any element->mutex acquisition, as it was not
initialized yet before it's allocation. rw_trx_hash_t::insert() can reuse
the chunk, unpinned in rw_trx_hash_t::find().

The scenario is the following:

1. Thread 1 have just executed lf_hash_search() in
rw_trx_hash_t::find(), but have not acquired element->mutex yet.
2. Thread 2 have removed the element from hash table with
rw_trx_hash_t::erase() call.
3. Thread 1 acquired element->mutex and unpinned pin 2 pin with
lf_hash_search_unpin(pins) call.
4. Some thread purged memory of the element.
5. Thread 3 reused the memory for the element, filled element->id,
element->trx.
6. Thread 1 crashes with failed "DBUG_ASSERT(trx_id == trx->id)"
assertion.

Note that trx_t objects are also reused, see the code around trx_pools
for details.

The fix is to invoke "lf_hash_search_unpin(pins);" after element->trx is
stored in local variable in rw_trx_hash_t::find().

Reviewed by: Nikita Malyavin, Marko Mäkelä.

Author: Vlad Lesin
Author Date: 2023-02-09 10:51:57 UTC

MDEV-27701 Race on trx->lock.wait_lock between lock_rec_move() and lock_sys_t::cancel()

The initial issue was in assertion failure, which checked the equality
of lock to cancel with trx->lock.wait_lock in lock_sys_t::cancel().

If we analyze lock_sys_t::cancel() code from the perspective of
trx->lock.wait_lock racing, we won't find the error there, except the
cases when we need to reload it after the corresponding latches
acquiring.

So the fix is just to remove the assertion and reload
trx->lock.wait_lock after acquiring necessary latches.

Reviewed by: Marko Mäkelä <marko.makela@mariadb.com>

Author: tanruixiang
Author Date: 2022-06-16 07:05:35 UTC

MDEV-26182: Implement JSON_INTERSECT()

The idea is to have simple functions that the user can combine to produce
the exact result one wants, whether the user wants JSON object that has
common keys with another JSON object, or same key/value pair etc. So
making simpler function helps here.

We accomplish this by making three separate functions.
1) JSON_OBJECT_FILTER_KEYS(Obj, Arr_keys):
Put keys ( which are basically strings ) in hash, go over the object and
get key one by one. If the key is present in the hash,
add the key-value pair to result.

2) JSON_OBJECT_TO_ARRAY(Obj) : Create a string variable, Go over the json
object, and add each key value pair as an array into the result.

3) JSON_ARRAY_INTERSECT(arr1, arr2) :
Go over one of the json and add each item of the array
in hash (after normalizing each item). Go over the second array,
search the normalized item one by one in the hash. If item is found,
add it to the result.

Implementation Idea: Holyfoot ( Alexey Botchkov)
Author: tanruixiang and Rucha Deodhar

Author: Vanislavsky
Author Date: 2023-04-19 15:45:09 UTC

add local_hash.h

Author: midenok
Author Date: 2022-12-13 06:03:10 UTC

MDEV-27180 Fully atomic partitioning DDL operations

Atomic DDL for partitioning originally covers crash-safety but it does
not recover fully from failures. F.ex. if error happened during the
rename or drop of partitions the ALTER operation does not return the
table to its original state.

The patch solves the above problem similarly to MDEV-25292 (atomic
CREATE OR REPLACE): new partitions are created as temporary, old
partitions are backed up, everything is guarded by two DDL log chains:
rollback and cleanup. Rollback chain contains the actions that bring
back the original table. Cleanup chain deletes the backup files.

The generic operation of alter partition is as follows:

  1. Create new partitions as tmp partitions;
  2. Fill TMP partitions with data;
  3. Rename old and not needed partitions to backup partitions;
  4. Rename tmp partitions to original partititions;
  5. Do any required logging (mariabackup, binary);
  6. If everything is ok, drop the backup partitions;
  7. In the case any of pp.1-5 fails drop new partitions, rename back
     backup partitions.

Each rename operation is executed right after the corresponding DDL
logging. Originally they were done in different loops:
write_log_changed_partitions(), mysql_change_partitions(), etc. The
actual table operations were done in ha_partition handler:
ha_change_partitions(), ha_drop_partitions(),
ha_rename_partitions(). Now these calls are removed.

Instead the above deprecated interfaces the following classes now
handle DDL logging and table operations:

      Alter_partition_logger
      Alter_partition_add
      Alter_partition_change

Alter_partition_logger handles the basic operation, DDL logging and
table renames. Alter_partition_add does the stuff for adding the
partitions. Alter_partition_change combines Alter_partition_logger and
Alter_partition_add for complex operations such as REORGANIZE.

ha_partition::change_partitions() call was not fully removed. Stages
1-4 from that call was renamed to allocate_partitions() and that is
used by Alter_partition_add for initializing file handler arrays
(m_added_file, m_reorged_file). ha_partition::prepare_new_partition()
is renamed to create_partition() because it actually does ha_create()
and the new name better describes what it does (see previous commit
"ha_partition refactoring").

DDL_LOG_DELETE/DDL_LOG_RENAME/DDL_LOG_REPLACE are now pure file
operations. To process both .par and .frm files we now issue two
actions instead of one. That makes interface more simple and that will
be used by MDEV-16417 in future. All the table operations are done via
other actions, such as DDL_LOG_RENAME_TABLE, DDL_LOG_DROP_TABLE, etc.

DDL_LOG_RENAME_TABLE has new alter_partition option. With this option
it does simple ha_rename_table() which is needed for renaming the
partitions. The action for renaming the partitions must start from
phase DDL_RENAME_PHASE_TABLE.

The testing was refactored a bit. Many new fail/crash points was
added, the test was powered by binary and mariabackup logging, it
tests more ALTER commands with variations of partitions-only and
subpartitions, without and under LOCK TABLES.

EXCHANGE PARTITION was fixed: after failure under LOCK TABLES the
table became unlocked. Fixed by correcting place of
reopen_tables(). Tested by partition_debug.

HA_PARTITION_ONE_PHASE handling was removed from
fast_alter_partition() as native partitioning is not supported.

Author: midenok
Author Date: 2022-12-13 06:03:10 UTC

MDEV-27180 Fully atomic partitioning DDL operations

Atomic DDL for partitioning originally covers crash-safety but it does
not recover fully from failures. F.ex. if error happened during the
rename or drop of partitions the ALTER operation does not return the
table to its original state.

The patch solves the above problem similarly to MDEV-25292 (atomic
CREATE OR REPLACE): new partitions are created as temporary, old
partitions are backed up, everything is guarded by two DDL log chains:
rollback and cleanup. Rollback chain contains the actions that bring
back the original table. Cleanup chain deletes the backup files.

The generic operation of alter partition is as follows:

  1. Create new partitions as tmp partitions;
  2. Fill TMP partitions with data;
  3. Rename old and not needed partitions to backup partitions;
  4. Rename tmp partitions to original partititions;
  5. Do any required logging (mariabackup, binary);
  6. If everything is ok, drop the backup partitions;
  7. In the case any of pp.1-5 fails drop new partitions, rename back
     backup partitions.

Each rename operation is executed right after the corresponding DDL
logging. Originally they were done in different loops:
write_log_changed_partitions(), mysql_change_partitions(), etc. The
actual table operations were done in ha_partition handler:
ha_change_partitions(), ha_drop_partitions(),
ha_rename_partitions(). Now these calls are removed.

Instead the above deprecated interfaces the following classes now
handle DDL logging and table operations:

      Alter_partition_logger
      Alter_partition_add
      Alter_partition_change

Alter_partition_logger handles the basic operation, DDL logging and
table renames. Alter_partition_add does the stuff for adding the
partitions. Alter_partition_change combines Alter_partition_logger and
Alter_partition_add for complex operations such as REORGANIZE.

ha_partition::change_partitions() call was not fully removed. Stages
1-4 from that call was renamed to allocate_partitions() and that is
used by Alter_partition_add for initializing file handler arrays
(m_added_file, m_reorged_file). ha_partition::prepare_new_partition()
is renamed to create_partition() because it actually does ha_create()
and the new name better describes what it does (see previous commit
"ha_partition refactoring").

DDL_LOG_DELETE/DDL_LOG_RENAME/DDL_LOG_REPLACE are now pure file
operations. To process both .par and .frm files we now issue two
actions instead of one. That makes interface more simple and that will
be used by MDEV-16417 in future. All the table operations are done via
other actions, such as DDL_LOG_RENAME_TABLE, DDL_LOG_DROP_TABLE, etc.

DDL_LOG_RENAME_TABLE has new alter_partition option. With this option
it does simple ha_rename_table() which is needed for renaming the
partitions. The action for renaming the partitions must start from
phase DDL_RENAME_PHASE_TABLE.

The testing was refactored a bit. Many new fail/crash points was
added, the test was powered by binary and mariabackup logging, it
tests more ALTER commands with variations of partitions-only and
subpartitions, without and under LOCK TABLES.

EXCHANGE PARTITION was fixed: after failure under LOCK TABLES the
table became unlocked. Fixed by correcting place of
reopen_tables(). Tested by partition_debug.

HA_PARTITION_ONE_PHASE handling was removed from
fast_alter_partition() as native partitioning is not supported.

Author: Kristian Nielsen
Author Date: 2023-05-16 14:33:07 UTC

MDEV-25611: RESET MASTER still causes the server to hang

Testcase that reliably causes RESET MASTER to hang because it doesn't
receive a binlog checkpoint notification.

This testcase disables the InnoDB sync-the-redo-log-every-second, without
doing that I don't see a hang.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Nikita Malyavin
Author Date: 2023-05-15 18:07:50 UTC

MDEV-30984 Online ALTER table is denied with non-informative error messages

Unify online support checks under a single function, add warning.

Author: Alexander Barkov
Author Date: 2023-05-15 08:41:31 UTC

MDEV-30680 Warning: Memory not freed: 280 on mangled query, LeakSanitizer: detected memory leaks

The parser works as follows:

The rule expr_lex returns a pointer to a newly created sp_expr_lex
instance which is not linked to any MariaDB structures yet - it is
pointed only from a Bison stack variable. The sp_expr_lex instance
gets linked to other structures (such as sp_instr_jump_if_not) later,
after scanning some following grammar.

Problem before the fix:
If a parse error happened immediately after expr_lex (before it got linked),
the created sp_expr_lex value got lost causing a memory leak.

Fix:

- Using Bison's "destructor" directive to free the results of expr_lex
  on parse/oom errors.

- Moving the call for LEX::cleanup_lex_after_parse_error() from
  MYSQL_YYABORT and yyerror inside parse_sql().
  This is needed because Bison calls destructors after yyerror(),
  while it's important to delete the sp_expr_lex instance before
  LEX::cleanup_lex_after_parse_error().
  The latter frees the memory root containing the sp_expr_lex instance.

  After this change the code block are executed in the following order:

  - yyerror() -- now only raises the error to DA (no cleanup done any more)
  - %destructor { delete $$; } <expr_lex> -- destructs the sp_expr_lex instance
  - LEX::cleanup_lex_after_parse_error() -- frees the memory root containing
                                              the sp_expr_lex instance

Author: Rucha Deodhar
Author Date: 2023-04-03 08:04:51 UTC

MDEV-23187: Assorted assertion failures in json_find_path with certain
collations

Analysis:
When we have negative index, the value in array_counter[] array is going to
be -1 at some point ( because in case of negative index in json path, the
initial value for a path with negative index is -<size_of_array>, and as we
move forward in array while parsing it and finding path, this value
increments). Since SKIPPED_STEP_MARK, is maximum uint value, it gets
compared to some int value in the array and eventually equates to -1
and messes with path.
Fix:
Make SKIPPED_STEP_MARK maximum of INT32.

Author: Kristian Nielsen
Author Date: 2023-05-13 18:49:10 UTC

MDEV-30386: Test case for cleaning up extra domain_ids

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Igor Babaev
Author Date: 2023-05-12 06:34:41 UTC

MDEV-31240 Crash with condition pushable into derived and containing outer reference

This bug could affect queries containing a subquery over splittable derived
tables and having an outer references in its WHERE clause. If such subquery
contained an equality condition whose left part was a reference to a column
of the derived table and the right part referred only to outer columns
then the server crashed in the function st_join_table::choose_best_splitting()
The crashing code was added in the commit ce7ffe61d836fe9f0cfc1087f058bc40d66e5cfb
that made the code of the function sensitive to presence of the flag
OUTER_REF_TABLE_BIT in the KEYUSE_EXT::needed_in_prefix fields.

The field needed_in_prefix of the KEYUSE_EXT structure should not contain
table maps with OUTER_REF_TABLE_BIT or RAND_TABLE_BIT.

Note that this fix is quite conservative: for affected queries it just
returns the query plans that were used before the above mentioned commit.
In fact the equalities causing crashes should be pushed into derived tables
without any usage of split optimization.

Approved by Sergei Petrunia <sergey@mariadb.com>

Author: Daniele Sciascia
Author Date: 2023-03-16 08:29:10 UTC

MDEV-30473 Remove test galera.MDEV-27713

Remove test galera.MDEV-27713. This test relies on GET_LOCK() and has
stopped working since commit 844ddb1 (see MDEV-30473). This commit
disabled GET_LOCK() in combination with Galera.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Sergey Petrunia
Author Date: 2023-05-11 12:15:53 UTC

MDEV-31240, MDEV-31241: Crashes in subselects in choose_best_splitting after upgrade

In JOIN_TAB::choose_best_splitting(), take into account that
best_param_tables may contain OUTER_REF_TABLE_BIT (and no table
in join->positions[] has this bit set).

Author: Alexander Barkov
Author Date: 2022-04-04 10:50:21 UTC

MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption

The crash happened with an indexed virtual column whose
value is evaluated using a function that has a different meaning
in sql_mode='' vs sql_mode=ORACLE:

- DECODE()
- LTRIM()
- RTRIM()
- LPAD()
- RPAD()
- REPLACE()
- SUBSTR()

For example:

CREATE TABLE t1 (
  b VARCHAR(1),
  g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL,
  KEY g(g)
);

So far we had replacement XXX_ORACLE() functions for all mentioned function,
e.g. SUBSTR_ORACLE() for SUBSTR(). So it was possible to correctly re-parse
SUBSTR_ORACLE() even in sql_mode=''.

But it was not possible to re-parse the MariaDB version of SUBSTR()
after switching to sql_mode=ORACLE. It was erroneously mis-interpreted
as SUBSTR_ORACLE().

As a result, this combination worked fine:

SET sql_mode=ORACLE;
CREATE TABLE t1 ... g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL, ...;
INSERT ...
FLUSH TABLES;
SET sql_mode='';
INSERT ...

But the other way around it crashed:

SET sql_mode='';
CREATE TABLE t1 ... g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL, ...;
INSERT ...
FLUSH TABLES;
SET sql_mode=ORACLE;
INSERT ...

At CREATE time, SUBSTR was instantiated as Item_func_substr and printed
in the FRM file as substr(). At re-open time with sql_mode=ORACLE, "substr()"
was erroneously instantiated as Item_func_substr_oracle.

Fix:

The fix proposes a symmetric solution. It provides a way to re-parse reliably
all sql_mode dependent functions to their original CREATE TABLE time meaning,
no matter what the open-time sql_mode is.

We take advantage of the same idea we previously used to resolve sql_mode
dependent data types.

Now all sql_mode dependent functions are printed by SHOW using a schema
qualifier when the current sql_mode differs from the function sql_mode:

SET sql_mode='';
CREATE TABLE t1 ... SUBSTR(a,b,c) ..;
SET sql_mode=ORACLE;
SHOW CREATE TABLE t1; -> mariadb_schema.substr(a,b,c)

SET sql_mode=ORACLE;
CREATE TABLE t2 ... SUBSTR(a,b,c) ..;
SET sql_mode='';
SHOW CREATE TABLE t1; -> oracle_schema.substr(a,b,c)

Old replacement names like substr_oracle() are still understood for
backward compatibility and used in FRM files (for downgrade compatibility),
but they are not printed by SHOW any more.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-11 07:07:45 UTC

Merge branch '10.4' into 10.4.29 release

Author: Marko Mäkelä
Author Date: 2023-05-11 06:00:27 UTC

Merge 10.6 into 10.8

Author: Daniele Sciascia
Author Date: 2023-03-16 08:29:10 UTC

MDEV-30473 Remove test galera.MDEV-27713

Remove test galera.MDEV-27713. This test relies on GET_LOCK() and has
stopped working since commit 844ddb1 (see MDEV-30473). This commit
disabled GET_LOCK() in combination with Galera.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Teemu Ollakka
Author Date: 2023-02-26 10:44:49 UTC

MENT-1708 Donor in non-Primary causes assertion in wsrep-lib

Constructed a test which makes donor to go into non-Primary configuration
before `sst_sent()` is called, causing an assertion in wsrep-lib
if the bug is present.

Updated wsrep-lib to version which contains the fix.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Daniel Bartholomew
Author Date: 2023-05-10 12:48:11 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2023-05-10 12:47:29 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2023-05-10 12:46:44 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2023-05-10 12:46:01 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2023-05-10 12:45:08 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2023-05-10 12:43:49 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2023-05-10 12:11:15 UTC

bump the VERSION

Author: Marko Mäkelä
Author Date: 2023-05-10 09:43:17 UTC

MDEV-29967 innodb_read_ahead_threshold (linear read-ahead) does not work

buf_read_ahead_linear(): Correct some calculations that were broken
in commit b1ab211dee599eabd9a5b886fafa3adea29ae041 (MDEV-15053).

Thanks to Daniel Black for providing a test case and initial debugging.

Author: Sergey Petrunia
Author Date: 2023-05-09 10:09:00 UTC

MDEV-31223: UBSan error: sql_select.h:969:7: runtime error: load of value...

In Loose_scan_opt::save_to_position, initialize
POSITION::firstmatch_with_join_buf.

Author: Sergey Petrunia
Author Date: 2023-05-03 12:15:37 UTC

MDEV-31022: SIGSEGV in maria_create from create_internal_tmp_table

The code in create_internal_tmp_table() didn't take into account that
now temporary (derived) tables may have multiple indexes:

- one index due to duplicate removal
   = In this example created by conversion of big-IN(...) into subquery
   = this index might be converted into a "unique constraint" if the key
     length is too large.
- one index added by derived_with_keys optimization.

Make create_internal_tmp_table() handle multiple indexes.

Before this patch, use of a unique constraint was indicated in
TABLE_SHARE::uniques. This was ok as unique constraint was the only index
in the table. Now it's no longer the case so TABLE_SHARE::uniques is removed
and replaced with an in-memory-only flag HA_UNIQUE_HASH.

This patch is based on Monty's patch.
Co-Author: Monty <monty@mariadb.org>

Author: Yuchen Pei
Author Date: 2023-05-08 10:21:54 UTC

MDEV-22534 [do-not-push] a PoC that works for the example case

Test case:

--8<---------------cut here---------------start------------->8---
set @@optimizer_switch='materialization=on,firstmatch=off';
create table ten(a int primary key);
insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
create table t1 (a int, b int, c int);
insert into t1 select a,a,a from ten;
create table t2 select * from t1;

set optimizer_trace="enabled=on";
explain select * from t1 where a in (select a from t2 where t1.b=t2.b);
set optimizer_trace="enabled=off";
select * from information_schema.OPTIMIZER_TRACE;

set optimizer_trace="enabled=on";
explain select * from t1 where (a, b) in (select a, b from t2);
set optimizer_trace="enabled=off";
select * from information_schema.OPTIMIZER_TRACE;

drop table ten, t1, t2;
--8<---------------cut here---------------end--------------->8---

Exactly the same explain output:

--8<---------------cut here---------------start------------->8---
explain select * from t1 where a in (select a from t2 where t1.b=t2.b);
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY t1 ALL NULL NULL NULL NULL 10
1 PRIMARY <subquery2> eq_ref distinct_key distinct_key 8 func,func 1
2 MATERIALIZED t2 ALL NULL NULL NULL NULL 10

explain select * from t1 where (a, b) in (select a, b from t2);
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY t1 ALL NULL NULL NULL NULL 10
1 PRIMARY <subquery2> eq_ref distinct_key distinct_key 8 func,func 1
2 MATERIALIZED t2 ALL NULL NULL NULL NULL 10
--8<---------------cut here---------------end--------------->8---

Author: Hartmut Holzgraefe
Author Date: 2023-05-05 10:05:44 UTC

MDEV-30904 "rpm --setugids" breaks PAM authentication

Set auto_pam_tool_dir and auth_pam_tool ownership in RPM specs,
not in POSTINST hook script, so that they are properly restored
when running "rpm --setugids" instead of wrongly changing
auth_pam_tool_dir ownershipt from "mysql" to "root"

Author: Sergey Petrunia
Author Date: 2023-05-03 12:15:37 UTC

MDEV-31022: SIGSEGV in maria_create from create_internal_tmp_table

The code in create_internal_tmp_table() didn't take into account that
now temporary (derived) tables may have multiple indexes:

- one index due to duplicate removal
   = In this example created by conversion of big-IN(...) into subquery
   = this index might be converted into a "unique constraint" if the field(s)
     length is too large.
- one index added by derived_with_keys optimization.

Make it handle multiple indexes.

As a consequence of this, remove TABLE_SHARE::uniques and replace it with
per-index HA_UNIQUE_HASH flag.
- TABLE_SHARE::uniques made sense when it referred to the only index in the
  table.
- Now, we need to know which of table indexes should be a unique constraint,
  so we indicate that as an index flag.

This patch is based on Monty's patch:
Co-Author: Monty <monty@mariadb.org>

Author: Alexander Barkov
Author Date: 2023-04-28 12:13:38 UTC

MDEV-31184 Remove parser tokens DECODE_MARIADB_SYM and DECODE_ORACLE_SYM

Changing the code handling sql_mode-dependent function DECODE():

- removing parser tokens DECODE_MARIADB_SYM and DECODE_ORACLE_SYM
- removing the DECODE() related code from sql_yacc.yy/sql_yacc_ora.yy
- adding handling of DECODE() with help of a new Create_func_func_decode

Author: Sergei Golubchik
Author Date: 2023-05-03 13:37:05 UTC

MDEV-31164 default current_timestamp() not working when used INSERT ON DUPLICATE KEY in some cases

select_insert::store_values() must reset
has_value_set bitmap before every row, just like mysql_insert() does.
because ON DUPLICATE KEY UPDATE and triggers modify it

Author: Kristian Nielsen
Author Date: 2023-04-23 08:59:32 UTC

MDEV-31140: FLUSH BINARY LOGS DELETE_DOMAIN_ID=(D) can errorneously delete active domains

Fix the code in rpl_binlog_state::drop_domain(), so that _all_ entries for
the domain in the binlog state must match an entry in the initial GTID_LIST,
not just one entry match.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Sergey Petrunia
Author Date: 2023-05-03 10:49:32 UTC

MDEV-26301: Split optimization refills: Optimizer Trace coverage

Add Optimizer Trace printouts.

Author: Rucha Deodhar
Author Date: 2023-05-02 11:04:07 UTC

MDEV-30677: Incorrect result for "SELECT JSON_SCHEMA_VALID('{}', NULL)"
Analysis: null_value is not set if any one of the arguments is NULL. So it
returns 1.
Fix: when either argument is NULL, set null_value to true, so that null can
be returned

Author: Yuchen Pei
Author Date: 2023-05-03 02:06:59 UTC

MDEV-22979 MDEV-27233 defer spider init from plugin-load-add to a temporary init file

- Move spider init query execution from a background thread to
  spider_db_init() so that it is fully synchronous, adapted from part
  of the reverted commit c160a115b8b6dcd54bb3daf1a751ee9c68b7ee47
- if spider is initialised too early from init_server_components then
  defer the init to a temporary init file
- also updated other init bug tests, for MDEV-29904 and MDEV-30370

Author: sara
Author Date: 2023-05-02 23:34:32 UTC

MDEV-30892 test galera.galera_log_bin is not deterministic

galera.galera_log_bin test created the test tables and executed initial DML into node 2
Then connection is switched to node 1, where ALTER TABLE was attempted. But there is no guarantee that the table to alter was yet replicated to node 1.

The fix in this commit, creates the test tables in node 1 instead, so it is guaranteed that they are available for the later ALTER

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-02 18:13:48 UTC

New CC 3.1

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-05-02 18:13:48 UTC

New CC 3.1

Author: Elena Stepanova
Author Date: 2023-05-02 14:02:56 UTC

MacOS tests

Author: Sergei Golubchik
Author Date: 2022-11-27 16:03:29 UTC

cleanup: remove vcol_info->stored_in_db

it was redundant, duplicating vcol_type == VCOL_GENERATED_STORED.

Note that VCOL_DEFAULT is not "stored", "stored vcol" means that after
rnd_next or index_read/etc the field value is already in the record[0]
and does not need to be calculated separately

Author: Sergei Golubchik
Author Date: 2023-02-12 12:13:57 UTC

Revert "don't use ssl for windows named pipes - it doesn't work"

This reverts commit e1caa4bd5e8b4645944b85d4b603bf9fc9ef6ca4.

Will be fixed in CONC-635 instead

Author: Sergei Golubchik
Author Date: 2023-01-17 09:58:00 UTC

MDEV-26506 Over-quoted JSON when combining JSON_ARRAYAGG with JSON_OBJECT

add the test case

Author: Igor Babaev
Author Date: 2023-05-02 07:31:57 UTC

MDEV-31162 Crash for query using ROWNUM over multi-table view with ORDER BY

This bug could cause a crash of the server when processing a query with
ROWNUM() if it used in its FROM list a reference to a mergeable view
defined as SELECT over more than one table that contained ORDER BY clause.
When a mergeable view with ORDER BY clause and without LIMIT clause is used
in the FROM list of a query that does not have ORDER BY clause the ORDER BY
clause of the view is moved to the query. The code that performed this
transformation forgot to delete the moved ORDER BY list from the view.
If a query contains ROWNUM() and uses a mergeable multi-table view with
ORDER BY then according to the current code of TABLE_LIST::init_derived()
the view has to be forcibly materialized. As the query and the view shared
the same items in its ORDER BY lists they could not be properly resolved
either in the query or in the view. This led to a crash of the server.

This patch has returned back the original signature of LEX::can_not_use_merged()
to comply with 10.4 code of the condition that checks whether a megeable
view has to be forcibly materialized.

Approved by Oleksandr Byelkin <sanja@mariadb.com>