Author: Sergei Golubchik
Author Date: 2024-04-19 22:17:30 UTC

archive.archive and main.mysqlbinlog_row_compressed

fixes for zlib-ng

Author: Sergei Golubchik
Author Date: 2024-04-19 20:09:06 UTC

MDEV-33952 galera_create_table_as_select fails sporadically

disable until fixed

Author: Sergei Golubchik
Author Date: 2024-04-19 20:09:06 UTC

MDEV-33952 galera_create_table_as_select fails sporadically

disable until fixed

Author: Brandon Nesterenko
Author Date: 2024-04-19 15:28:03 UTC

Completion of MDEV-33475 merge

The merge of MDEV-33475 was incomplete, the change to
rpl_parallel.cc were missing. This patch adds them
back in

Author: Alexander Barkov
Author Date: 2024-04-12 05:25:13 UTC

MDEV-20610 Assertion failed or btr_validate_index(..) in row_upd_sec_index_entry on a time_zone change

Author: Marko Mäkelä
Author Date: 2024-04-19 13:01:26 UTC

Merge 10.5 into 10.6

Author: Julius Goryavsky
Author Date: 2024-04-19 11:55:45 UTC

HASHICORP_KEY_MANAGEMENT: MDEV-33420 post-fix

Compilation correction for Windows after MDEV-33420 in combination
with exclusion of <global.h> from the list of included headers.

Author: Elena Stepanova
Author Date: 2024-04-19 11:10:07 UTC

MDEV-33622 Server crashes when the UPDATE statement (which has duplicate key) is run after setting a low thread_stack

Added the missing test case

Author: Monty
Author Date: 2024-04-19 10:10:58 UTC

Remove high stack usage

I checked all stack potential problems found with
gcc -W -Wstack-usage=16384

Fixes:
- read_bootstrap_query() now allocates line_buffer (20000 bytes) with
  malloc() instead of using stack. This has a small performance impact
  but this is not releant for bootstrap.
- mroonga grn_select() used 65856 bytes on stack. Changed it to use
  malloc().

Not fixed:
- mroonga/vendor/groonga/lib/expr.c grn_proc_call() uses
  43712 byte on stack. However this is not easy to fix as the stack
  used is caused by a lot of code generated by defines.
- rocksdb/options/options_helper.cc uses 20288 of stack space.
  (no reason to fix except to get rid of the compiler warning)
- Cases using alloca() where the allocation size is resonable.
- An issue in libmariadb (reported to connectors).

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-04-19 09:18:21 UTC

MDEV-33087 ALTER TABLE...ALGORITHM=COPY should build indexes more efficiently

- During copy algorithm, InnoDB should use bulk insert operation
for row by row insert operation. By doing this, copy algorithm
can effectively build indexes. This optimization is disabled
for temporary table, versioning table and table which has
foreign key relation.

Introduced the variable innodb_allow_bulk_insert to avoid
the bulk insert operation inside InnoDB

ha_innobase::extra(): HA_EXTRA_END_ALTER_COPY mode tries to apply
the buffered bulk insert operation, updates the non-persistent
table stats.

row_merge_bulk_t::write_to_index(): Update stat_n_rows after
applying the bulk insert operation

row_ins_clust_index_entry_low(): In case of copy algorithm,
switch to bulk insert operation.

copy_data_error_ignore(): Handles the error while copying
the data from source to target file.

Author: Marko Mäkelä
Author Date: 2024-04-19 08:35:39 UTC

fixup! 9bb7961a964bc15de0de7a50ac15580f4080a09a

Author: Marko Mäkelä
Author Date: 2024-04-19 07:54:08 UTC

MDEV-33447: libpmem is not available in RHEL 8

Because the Red Hat Enterprise Linux 8 core repository does not include
libpmem, let us implement the necessary subset ourselves.

pmem_persist(): Implement for 64-bit x86, ARM, POWER, RISC-V, Loongarch
in a way that should be compatible with the https://github.com/pmem/pmdk/
implementation of pmem_persist().

The CMake option WITH_INNODB_PMEM can be used for enabling or disabling
this interface at compile time. By default, it is enabled on all applicable
systems that are covered by our CI system.

Note: libpmem had not been previously enabled for Loongarch in our
Debian packaging. It was enabled for RISC-V, but we will not enable it
by default on RISC-V or Loongarch because we lack CI coverage.

The generated code for x86_64 was reviewed and tested on two
Intel implementations: one that only supports clflush, and
another that supports both clflushopt and clwb.

The generated machine code was also reviewed on https://godbolt.org
using various compiler versions. Godbolt helpfully includes an option
to compile to binary code and display the encoding, which was
useful on POWER.

Reviewed by: Vladislav Vaintroub

Author: Yuchen Pei
Author Date: 2024-04-18 23:42:07 UTC

[demo] spider.spider_fixes_part: wait and retry

Author: Yuchen Pei
Author Date: 2024-04-18 23:41:25 UTC

[demo] spider.spider_fixes_part: wait and die

Author: Monty
Author Date: 2024-04-18 14:17:46 UTC

squash! 137d2c6fa298f04066a87dda160aac8dbee94e61

MDEV-33826 Assertion `tl->table == __null' failed in
THD::open_temporary_table

The bug was that admin_recreate_table() did not expect that there where
many tables in able_list,

Author: Brandon Nesterenko
Author Date: 2024-04-18 21:45:27 UTC

MDEV-33924: Gtid_log_event thread_id should be 32-bit

In accordance with MDEV-15089, and to be consistent with
Query_log_event::thread_id, the Gtid_log_event::thread_id
should also be 32-bit when writing/reading to the binary
log.

Author: VladislavVaintroub
Author Date: 2024-03-04 22:12:34 UTC

MDEV-33625 Add option --dir to mariadb-dump

New option works just like --tab, wrt output (sql file for table definition
and tab-separated for data, same options, e.g --parallel)

Compared to --tab it allows --databases and --all-databases.
When --dir is used , it creates directory structure in the output directory,
pointed to by --dir. For every database to be dumped, there will be a
directory with database name.

All options that --tab supports, are also supported by --dir, in particular
--parallel

Author: VladislavVaintroub
Author Date: 2024-03-04 22:12:34 UTC

MDEV-33625 Add option --dir to mariadb-dump

New option works just like --tab, wrt output (sql file for table definition
and tab-separated for data, same options, e.g --parallel)

Compared to --tab it allows --databases and --all-databases.
When --dir is used , it creates directory structure in the output directory,
pointed to by --dir. For every database to be dumped, there will be a
directory with database name.

All options that --tab supports, are also supported by --dir, in particular
--parallel

Author: Monty
Author Date: 2024-04-18 14:17:46 UTC

squash! 137d2c6fa298f04066a87dda160aac8dbee94e61

MDEV-33826 Assertion `tl->table == __null' failed in
THD::open_temporary_table

The bug was that admin_recreate_table() did not expect that there where
many tables in able_list,

Author: Monty
Author Date: 2024-04-18 13:43:33 UTC

squash! 8372d62a863008ea6faa1547db5c275a5863f167

MDEV-33913 TABLE_STATISTICS increments ROWS_CHANGED twice upon UPDATE and does not count DELETE

Author: Sergei Golubchik
Author Date: 2024-03-17 14:00:09 UTC

reset cached client plugin when it's no longer needed

it's a pointer into the net buffer, so it might be overwritten by the
next read or write. And the next plugin switch (in multi-auth) will
try to compare it (in send_plugin_request_packet) which is normally
harmless but fails the assert with Lex_ident::is_valid_ident()

Author: Vicențiu Ciorbaru
Author Date: 2024-04-18 10:15:04 UTC

Bug fixes - on top of Hugo's patch

Author: Sergey Petrunia
Author Date: 2024-03-13 08:53:31 UTC

MDEV-12404: Add assertions about Index Condition Pushdown use

Add assertions about limitations one has when using Index Condition
Pushdown:
- add handler::assert_icp_limitations()
- call this function from functions that may attempt violations.

Verified that assert_icp_limitations() as well as calls to it are
compiled away in release build.

Author: Yuchen Pei
Author Date: 2024-04-18 07:16:44 UTC

[merge fixup] spider: restore extern declarations of some vars

How did they get removed during the merge?? The removal did not appear
in the resolution diff at all.

Author: Yuchen Pei
Author Date: 2024-04-18 01:25:41 UTC

[demo] spider.spider_fixes_part: sleep until not running?

Author: Anel Husakovic
Author Date: 2024-04-12 06:55:27 UTC

MDEV-33420: HASHICORP_KEY_MANAGEMENT fails on Windows with libcurl installed

- When `libcurl` is installed in path out of default path, like on
Windows, `include_directories` failed to find `curl/curl.h`.
- Fix `cmake` by using modern syntax with imported target and
`find_package`
- Fix warnings treated as the errors
  - Remove `HASHICORP_HAVE_EXCEPTIONS` macro and related code
- Add package to `Server` component in Windows
- Tested with `$ ./mysql-test/mtr --suite=vault`
- Closes PR #3068
- Reviewer: <wlad@mariadb.com>
            <julius.goryavsky@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2024-04-17 20:38:40 UTC

Restore thread existence check

Author: Yuchen Pei
Author Date: 2024-04-17 07:26:41 UTC

MDEV-30727 Check spider_hton_ptr in spider udfs

We have to #undef my_error and find it from udfs when spider is not
installed.

Author: =?utf-8?q?Jan_Lindstr=C3=B6m?=
Author Date: 2024-04-12 06:15:53 UTC

MDEV-33895 : Galera test failure on galera_sr.MDEV-25718

Test was waiting INSERT-clause to make rollback but
wait_condition was too tight. State could be
Freeing items or Rollback. Fixed wait_condition
to expect one of them.

Author: VladislavVaintroub
Author Date: 2024-04-16 20:04:59 UTC

MDEV-33840 - fixup.

Be more careful, when switching on/off maintenance timer.

As long as there are active tasks and not enough threads, this will now
trigger the periodic deadlock check.

Previously, pool could block, without timer even starting.

Author: Sergei Golubchik
Author Date: 2024-04-16 17:07:28 UTC

Revert "Reapply "MDEV-33840 tpool : switch off maintenance timer when not needed.""

This reverts commit 289e76e192797144bea8ec761779945b32a0d548.

Author: Kristian Nielsen
Author Date: 2024-04-16 10:39:54 UTC

MDEV-19415: use-after-free on charsets_dir from slave connect

The slave IO thread sets MYSQL_SET_CHARSET_DIR. The code for this option
however is not thread-safe in sql-common/client.c. The value set is
temporarily written to mysys global variable `charsets-dir` and can be seen
by other threads running in parallel, which can result in use-after-free
error.

Problem was visible as random failures of test cases in suite multi_source
with Valgrind or MSAN.

Work-around by not setting this option for slave connect, it is redundant
anyway as it is just setting the default value.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Sergey Petrunia
Author Date: 2024-04-16 09:40:47 UTC

MDEV-33752: EXPLAIN UPDATE acquires locks on table rows

Make EXPLAIN commands behave like SELECTs when it comes to locking:
- In the SQL parser, make EXPLAIN DELETE and EXPLAIN UPDATE not to
  request write MDL or table locks.
- Same as above in multi_delete_set_locks_and_link_aux_tables().
- InnoDB analyzes the return value of thd_sql_command() and sets its
  locking behavior accordingly.
  EXPLAIN UPDATE|DELETE have thd_sql_command()=SQLCOM_UPDATE|DELETE.
  Introduce also thd_sql_is_explain() and if that returns true, InnoDB
  now will work as if thd_sql_command()==SQLCOM_SELECT.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-04-16 09:04:14 UTC

Merge branch '10.4' into 10.5

Author: VladislavVaintroub
Author Date: 2024-04-16 07:16:52 UTC

MDEV-16944 followup

- Better comments for do_write_line
- Add EOL to the line to be printed like "echo" does

Author: Dave Gosselin
Author Date: 2024-04-16 01:09:28 UTC

remove unused function should_collect_handler_stats

Author: Rex Johnston
Author Date: 2024-04-03 00:55:24 UTC

MDEV-28621 group by optimization incorrectly removing subquery where subject buried in a function

A group by query optimization technique, where the a group by clause in
a subquery can be removed is incorrectly removing references to subquery
items that may be defined and used elsewhere.

This is the initial implementation of a reference counter in the unit
attached to an Item_subquery.

Author: Dave Gosselin
Author Date: 2024-04-15 19:17:30 UTC

MDEV-33616 Tests failing on macOS

These tests rely on THR_KEY_mysys but it is not initialized. On
Linux, the corresponding thread variable is null, but on macOS it has a
nonzero value. In all cases, initialize the variable explicitly. We could
call MY_INIT, however some memory is leaked on test program exit in that
case. Instead, factor just enough to be able to initialize THR_KEY_mysys
from the test code (as well as from the library code).

Author: Elena Stepanova
Author Date: 2024-04-15 18:12:26 UTC

MDEV-33913 TABLE_STATISTICS increments ROWS_CHANGED twice upon UPDATE and does not count DELETE

Author: Kristian Nielsen
Author Date: 2024-04-15 16:54:30 UTC

Fix windows build failure

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Dave Gosselin
Author Date: 2024-04-15 16:15:04 UTC

MDEV-33616 Tests failing on macOS

compat/oracle.sp-inout now uses lowercase object names for test
compatibility with both case-insensitive and case-sensitive
filesystems.

Author: Dave Gosselin
Author Date: 2024-03-07 20:31:26 UTC

MTR flag to mark tests as incompatible with macOS

Introduces a new MTR include, not_mac.inc, which when included
at the top of a test, prevents that test from running on macOS.

Author: Daniel Black
Author Date: 2024-03-05 03:46:28 UTC

MDEV-33592: Use X509v3 for compatibility with libraries

According to rfc5280, x509v3 was standardised in June 1996.

RusTLS only accepts v3 certificates rejected the v1 default.
(ref:
https://github.com/rustls/webpki/issues/29#issuecomment-1453783741)

It seems reasonable that all client libraries can accept v3.

X509_VERSION_3 has a fixed value of 2, but isn't defined in
OpenSSL-1.1.1.

Thanks Austin Bonander for the suggested patch.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-04-12 12:43:21 UTC

and even more fixes

Author: Dave Gosselin
Author Date: 2024-03-18 17:03:33 UTC

MDEV-32777 DDL operations very slow under macOS

macOS is a development platform for our product but calls to
fcntl(F_FULLFSYNC) are too slow for developer needs. macOS offers
at least three different disk flush calls, making tradeoffs between
reliability and performance (see analysis notes on this MDEV).
On macOS only and as a compromise, we will back out the change to
call fcntl(F_FULLFSYNC), sacrificing a full flush in favor of more
performant writes.

Author: Monty
Author Date: 2024-04-12 15:59:59 UTC

Added Lock_time_ms and Table_catalog columns to metadata_lock_info

If compiled for debugging, LOCK_DURATION is also filled in.

Author: Monty
Author Date: 2024-04-12 15:59:59 UTC

Added Lock_time_ms and Table_catalog columns to metadata_lock_info

If compiled for debugging, LOCK_DURATION is also filled in.

Author: Andre Alves
Author Date: 2024-03-25 16:37:22 UTC

MDEV-33659 Server crashed at Create_func_aes_decrypt::create_native

Author: Julius Goryavsky
Author Date: 2021-06-20 22:54:11 UTC

MDEV-33677 : Port Performance Schema for Galera from ES

Added performance schema tables for Galera.
This is a port of the code from ES 10.6

Refs:
MENT-346 commit 86c06851
MENT-346 commit 10d0fcaf
MENT-1333 commit 2005adc3
MENT-347 commit aaca43b4
MENT-1370 commit 1f460e6b

Author: Julius Goryavsky
Author Date: 2024-04-10 04:32:06 UTC

MDEV-30732: additional changes after fix

Author: Yuchen Pei
Author Date: 2024-04-11 00:09:40 UTC

[merge fix] MDEV-33728 redundant macros MYSQL_VERSION_ID etc.

generated by

unifdef -DMYSQL_VERSION_ID=100500 -DMARIADB_BASE_VERSION -m storage/spider/spd_* storage/spider/ha_spider.* storage/spider/hs_client/*

Author: Alexander Barkov
Author Date: 2024-04-10 13:58:36 UTC

MDEV-32458 ASAN unknown-crash in Inet6::ascii_to_fbt when casting character string to inet6

The condition checked the value of the leftmost byte before checking if
at least one byte is still available in the buffer.
Changing the order in the condition: check for a byte availability before
checking the byte value.

Author: Andrei
Author Date: 2024-04-10 15:19:43 UTC

 MDEV-31779 Server crash in Rows_log_event::update_sequence upon replaying binary log

The crash at running mysqlbinlog on a SEQUENCE containing binlog file
was caused MDEV-29621 fixes that did not check which of the slave
or binlog applier executes a block introduced there.

The block is meaningful only for the parallel slave applier, so
it's safe to fix this bug with identified the actual applier and
skipping the block when it's the mysqlbinlog one.

Author: Kristian Nielsen
Author Date: 2024-04-10 14:12:20 UTC

MDEV-32020: Partial commit, start of slave-side XA replication

The slave can replicate prepared xa data by simply binlogging the data.
The slave can apply the committed xa as normal trx, but preserving XID on the GTID.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Brandon Nesterenko
Author Date: 2024-04-10 11:51:28 UTC

reb

Author: Yuchen Pei
Author Date: 2024-03-01 01:59:21 UTC

MDEV-31609 Send initial values of system variables in first OK packet

Values of all session tracking system variables will be sent in the
first ok packet upon connection after successful authentication.

Also updated mtr to print session track info on connection (h/t Sergei
Golubchik) so that we can write mtr tests for this change.

Author: Brandon Nesterenko
Author Date: 2024-04-08 19:04:59 UTC

MDEV-30260: Slave crashed:reload_acl_and_cache during shutdown

The signal handler thread can use various different runtime
resources when processing a SIGHUP (e.g. master-info information)
due to calling into reload_acl_and_cache(). Currently, the shutdown
process waits for the termination of the signal thread after
performing cleanup. However, this could cause resources actively
used by the signal handler to be freed while reload_acl_and_cache()
is processing.

The specific resource that caused MDEV-30260 is a race condition for
the hostname_cache, such that mysqld would delete it in
clean_up()::hostname_cache_free(), before the signal handler would
use it in reload_acl_and_cache()::hostname_cache_refresh().

Another similar resource is the active_mi/master_info_index. There
was a race between its deletion by the main thread in end_slave(),
and their usage by the Signal Handler as a part of
Master_info_index::flush_all_relay_logs.read(active_mi) in
reload_acl_and_cache().

This patch fixes these race conditions by relocating where server
shutdown waits for the signal handler to die until after
server-level threads have been killed (i.e., as a last step of
close_connections()). With respect to the hostname_cache, active_mi
and master_info_cache, this ensures that they cannot be destroyed
while the signal handler is still active, and potentially using
them.

Additionally:

 1) This requires that Events memory is still in place for SIGHUP
handling's mysql_print_status(). So event deinitialization is moved
into clean_up(), but the event scheduler still needs to be stopped
in close_connections() at the same spot.

 2) The function kill_server_thread is no longer used, so it is
deleted

 3) The timeout to wait for the death of the signal thread was not
consistent with the comment. The comment mentioned up to 10 seconds,
whereas it was actually 0.01s. The code has been fixed to wait up to
10 seconds.

 4) A warning has been added if the signal handler thread fails to
exit in time.

 5) Added pthread_join() to end of wait_for_signal_thread_to_end()
if it hadn't ended in 10s with a warning. Note this also removes
the pthread_detached attribute from the signal_thread to allow
for the pthread_join().

Reviewed By:
===========
Vladislav Vaintroub <wlad@mariadb.com>
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2024-04-08 19:04:59 UTC

MDEV-30260: Slave crashed:reload_acl_and_cache during shutdown

The signal handler thread can use various different runtime
resources when processing a SIGHUP (e.g. master-info information)
due to calling into reload_acl_and_cache(). Currently, the shutdown
process waits for the termination of the signal thread after
performing cleanup. However, this could cause resources actively
used by the signal handler to be freed while reload_acl_and_cache()
is processing.

The specific resource that caused MDEV-30260 is a race condition for
the hostname_cache, such that mysqld would delete it in
clean_up()::hostname_cache_free(), before the signal handler would
use it in reload_acl_and_cache()::hostname_cache_refresh().

Another similar resource is the active_mi/master_info_index. There
was a race between its deletion by the main thread in end_slave(),
and their usage by the Signal Handler as a part of
Master_info_index::flush_all_relay_logs.read(active_mi) in
reload_acl_and_cache().

This patch fixes these race conditions by relocating where server
shutdown waits for the signal handler to die until after
server-level threads have been killed (i.e., as a last step of
close_connections()). With respect to the hostname_cache, active_mi
and master_info_cache, this ensures that they cannot be destroyed
while the signal handler is still active, and potentially using
them.

Additionally:

 1) This requires that Events memory is still in place for SIGHUP
handling's mysql_print_status(). So event deinitialization is moved
into clean_up(), but the event scheduler still needs to be stopped
in close_connections() at the same spot.

 2) The function kill_server_thread is no longer used, so it is
deleted

 3) The timeout to wait for the death of the signal thread was not
consistent with the comment. The comment mentioned up to 10 seconds,
whereas it was actually 0.01s. The code has been fixed to wait up to
10 seconds.

 4) A warning has been added if the signal handler thread fails to
exit in time.

 5) Added pthread_join() to end of wait_for_signal_thread_to_end()
if it hadn't ended in 10s with a warning. Note this also removes
the pthread_detached attribute from the signal_thread to allow
for the pthread_join().

Reviewed By:
===========
Vladislav Vaintroub <wlad@mariadb.com>
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2024-04-09 17:35:22 UTC

MDEV-33672: 10.11 Fix for Two Phase Alter Flags

Extends 89c907bd4f713b77e248f7c8e6247b5f3be18fb4 to account for
binlog_two_phase_alter flags in a Gtid log event. I.e., if the
FL_COMMIT_ALTER_E1 or FL_ROLLBACK_ALTER_E2 flags are set in the
event flags, yet the length of the event is too short to hold
the value, then set the event as invalid

Author: Dave Gosselin
Author Date: 2024-01-26 16:13:48 UTC

MDEV-32401: expression cache heap-use-after-free

Queries selecting a TEXT column by alias, then joining that with a subquery
referencing the same alias may invoke a subquery expression cache. Typically,
these caches expect to rely on heap memory for their backing temporary table,
but TEXT (and BLOB) types require an ARIA backing; in this case, a crash may
result. Such a query may look like:

SELECT t1.d FROM (SELECT c AS d FROM t0) AS t1 JOIN t0 ON (
  SELECT ORD((SELECT BIT_OR(d))) FROM t0 GROUP BY d
);

This patch checks if there are blob types present before attempting to create
the SQL expression cache and if so, it does not, leaving the associated item
cache transparent.

The Expression_cache_tmptable implements the Expression_cache using a temporary
table. During Expression_cache_tmptable::init, we expect the created temporary
table to have the heap handlerton. However, for the crash in this ticket, it
is created using the ARIA handlerton instead. This happens because the
Create_tmp_table::choose_engine selects the ARIA engine because the temporary
table's associated TABLE_SHARE indicates that there are BLOB fields present.
This is true because TEXT fields are handled in the same way as BLOB fields in
our system; in our grammar, both TEXT and BLOB types rely on the
type_handler_blob handler. After the temporary table is created and the engine
associated with it, we then invoke move_field on each field associated with
the TABLE_SHARE. However, once this is complete and we return to
Expression_cache_tmptable::init, we check to see that the heap hton is used as
expected and it is not: so we delete the temporary table. Unfortunately, this
leaves the pointers inside the Field objects dangling, and we crash later as
indicated in the stack traces on this ticket. Non-BLOB types avoid this
problem because they end up with a temporary table backed by the heap
handlerton.

Author: Dave Gosselin
Author Date: 2024-04-03 17:16:25 UTC

MDEV-32397, MDEV-32403 Crashes during join processing.

Queries having the following form may cause a crash
  SELECT t1.a FROM ( SELECT a AS a1 FROM t1 ) dt
  JOIN t1 ON a1 LIKE EXISTS ( SELECT a + RAND () FROM t1 UNION SELECT a FROM t1);
because the table t1 has some JOIN cleanup operations performed prematurely
during the subselect.

In this particular case, the presence of RAND() makes the subquery
uncacheable, necessitating the need to execute the subquery multiple
times during join record evaluation. Each time the subquery runs, it
creates its own JOIN structure which has references to the table t1.
When the subquery completes, JOIN::cleanup and functions called by it
result in ha_end_keyread() being called on table t1. However, we are
not done with table t1 because the upper level `select t1.a from...`
query requires table t1 to be open. To solve this, we make the executor
aware of when we're in subqueries like this and delay JOIN cleanup
until the end of the query.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-04-09 10:12:33 UTC

Merge branch '11.1' into 11.2

Author: Yuchen Pei
Author Date: 2024-04-09 06:43:18 UTC

MDEV-29363 [demo] avoid equality propagation of items with subqueries

We update one place to avoid equality propagation of items with
subqueries. It may not be thorough, but is sufficient for the
counterexample in this demo: simply run tests on
main.multiple_equality and notice how it fails in the MDEV-32539 case.

Author: Rucha Deodhar
Author Date: 2024-04-08 13:42:28 UTC

MDEV-30646: View created via JSON_ARRAYAGG returns incorrect json object

Analysis:
For the JSON typehandler ::type_handler_for_tmp_table() returns
plain VARCHAR. Hence so the VARCHAR field is created in the temporary table.
So 'is_json_type()' check in the append_json_value() fails and we end up
quoting the string additionally.
Fix:
Force treat it as json, because append_json_value() will of course always
append json.

Author: Monty
Author Date: 2024-04-05 12:51:44 UTC

squash! 3a7a3e9b3be3504e1bfc32c931e14ed47a317d9b

- Uptime_since_flush_status is now always provided, independent if
  ENABLED_PROFILING is enabled when compiling MariaDB.
- @@global.Uptime_since_flush_status is reset on FLUSH GLOBAL STATUS
  and @@session.Uptime_since_flush_status is reset on FLUSH SESSION STATUS.
- When connected, @@session.Uptime_since_flush_status is set to 0.

Author: Monty
Author Date: 2024-04-06 09:07:49 UTC

squash! 9831521fc852f72317024778c723e28d121635e6

MDEV-33680 Server hangs or assertion fails upon SELECT with limited max_tmp_space_usage

The bug was that Aggregator_distinct::add() did not properly handle
write errors. (Old bug exposed by the new code).

Author: Alexander Barkov
Author Date: 2024-04-05 06:18:28 UTC

Fixup 2: MDEV-11339 Implement native UUID4 function

Adding a template Item_func_uuid_vx to share the code
between the implementations of UUID() and UUIDv4().

Author: Yuchen Pei
Author Date: 2024-03-26 03:37:17 UTC

MDEV-33734 Improve the sequence increment inequality testing

We add an extra condition that makes the inequality testing in
SEQUENCE::increment_value() mathematically watertight, and we cast to
and from unsigned in potential underflow and overflow addition and
subtractions to avoid undefined behaviour.

Let's start by distinguishing between c++ expressions and mathematical
expressions. by c++ expression I mean an expression with the outcome
determined by the compiler/runtime. by mathematical expression I mean
an expression whose value is mathematically determined. So a c++
expression -9223372036854775806 - 1000 at worst can evaluate to any
value due to underflow. A mathematical expression -9223372036854775806
- 1000 evaluates to -9223372036854776806.

The problem boils down to how to write a c++ expression equivalent to
an mathematical expression x + y < z where x and z can take any values
of long long int, and y < 0 is also a long long int. Ideally we want
to avoid underflow, but I'm not sure how this can be done.

The correct c++ form should be (x + y < z || x < z - y || x < z).
Let M=9223372036854775808 i.e. LONGLONG_MAX + 1. We have

-M < x < M - 1
-M < y < 0
-M < z < M - 1

Let's consider the case where x + y < z is true as a mathematical
expression.

If the first disjunct underflows, i.e. the mathematical expression x
+ y < -M. If the arbitrary value resulting from the underflow causes
the c++ expression to hold too, then we are done. Otherwise we move
onto the next expression x < z - y. If there's no overflow in z
- y then we are done. If there's overflow i.e. z - y > M - 1,
and the c++ expression evals to false, then we are onto x < z.
There's no over or underflow here, and it will eval to true. To see
this, note that

x + y < -M means x < -M - y < -M - (-M) = 0
z - y > M - 1 means z > y + M - 1 > - M + M - 1 = -1
so x < z.

Now let's consider the case where x + y < z is false as a mathematical
expression.

The first disjunct will not underflow in this case, so we move to (x <
z - y). This will not overflow. To see this, note that

x + y >= z means z - y <= x < M - 1

So it evals to false too. And the third disjunct x < z also evals to
false because x >= z - y > z.

I suspect that in either case the expression x < z does not determine
the final value of the disjunction in the vast majority cases, which
is why we leave it as the final one in case of the rare cases of both
an underflow and an overflow happening.

Here's an example of both underflow and overflow happening and the
added inequality x < z saves the day:

x = - M / 2
y = - M / 2 - 1
z = M / 2

x + y evals to M - 1 which is > z
z - y evals to - M + 1 which is < x

We can do the same to test x + y > z where the increment y is positive:

(x > z - y || x + y > z || x > z)

And the same analysis applies to unsigned cases.

Author: Yuchen Pei
Author Date: 2024-03-18 02:11:49 UTC

MDEV-33679 Spider group by handler: skip on multiple equalities

The spider group by handler is created in
JOIN::make_aggr_tables_info(), by which time calls to
substitute_for_best_equal_field() should have already removed all the
multiple equalities (i.e. Item_equal, with MULT_EQUAL_FUNC func_type).
Therefore, if there is still such items, it is deemed as an optimizer
bug and should be skipped.

Author: Yuchen Pei
Author Date: 2024-03-20 03:03:52 UTC

MDEV-33728 spider: remove use of MYSQL_VERSION_ID and MARIADB_BASE_VERSION

change created by:

unifdef -DMYSQL_VERSION_ID=100400 -DMARIADB_BASE_VERSION -m storage/spider/spd_* storage/spider/ha_spider.* storage/spider/hs_client/*

basically MDEV-27637, MDEV-27641, MDEV-27655

Author: Yuchen Pei
Author Date: 2024-03-19 23:36:25 UTC

MDEV-28992 Spider group by handler: Push down TIMESTAMPDIFF function

Also removed ITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC.

Similar to pr#2225, with the testcase adapted from that patch:

--8<---------------cut here---------------start------------->8---
From 884f7c6df16236748ca975339e0b1c267e195309 Mon Sep 17 00:00:00 2001
From: "Norio Akagi (norakagi)" <norakagi@amazon.com>
Date: Wed, 3 Aug 2022 23:30:34 -0700
Subject: [PATCH] [MDEV-28992] Push down TIMESTAMP_DIFF in spider

This changes so that TIMESTAMP_DIFF function in a query is pushed down and works natively in Spider.
Instead of directly accessing item's member, now we can rely on a public accessor method to make it work.
Unit tests are added under spider.pushdown_timestamp_diff.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
--8<---------------cut here---------------end--------------->8---

Author: Rex Johnston
Author Date: 2024-04-08 02:46:57 UTC

MDEV-31793 LooseScan join on derived table crashes

Test to see if not adding the offending key on the end has any consequences.

Author: Kristian Nielsen
Author Date: 2024-04-05 16:06:40 UTC

MDEV-33668: More efficient XA dependency tracking in SQL driver thread

Avoid linear scan of all recently queued XIDs in the SQL driver thread,
which might be expensive in XA-heavy workloads and large number of parallel
replication worker threads.

Instead keep a hash in the rpl_parallel_entry of where recently queued XIDs
were scheduled. This allows direct lookup of any potential scheduling
dependency.

Keep a list in each scheduling bucket of recently queued XIDs, and purge the
list (based on generations) when queueing next XA.

Also implement a more fine-grained dependency check based on sub_id
comparison. This can sometimes avoid a scheduling dependency that would
otherwise look necessary based solely on the generation check.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Rex Johnston
Author Date: 2024-04-03 00:55:24 UTC

MDEV-28621 group by optimization incorrectly removing subquery where subject buried in a function

A group by query optimization technique, where the a group by clause in
a subquery can be removed is incorrectly removing references to subquery
items that may be defined and used elsewhere.

This is the initial implementation of a reference counter in the unit
attached to an Item_subquery.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-04-05 10:39:56 UTC

MDEV-20094 InnoDB blob allocation allocates extra extents

- InnoDB reserves the free extents unnecessarily during blob
page allocation even though btr_page_alloc() can handle
reserving the extent when the existing ran out of pages to be used.

Author: Julius Goryavsky
Author Date: 2024-04-05 11:10:32 UTC

galera: wsrep-lib submodule update

Author: Daniele Sciascia
Author Date: 2024-01-10 13:34:12 UTC

MDEV-33216 stack-use-after-return in Wsrep_schema_impl::open_table()

Fix a case of stack-use-after-return reported by ASAN in
Wsrep_schema_impl::open_table(). This function has a stack allocated
TABLE_LIST object and return TABLE_LIST::table to the caller.
Changed the function to take a TABLE_LIST pointer as argument.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Daniele Sciascia
Author Date: 2024-01-10 13:34:12 UTC

MDEV-33216 stack-use-after-return in Wsrep_schema_impl::open_table()

Fix a case of stack-use-after-return reported by ASAN in
Wsrep_schema_impl::open_table(). This function has a stack allocated
TABLE_LIST object and return TABLE_LIST::table to the caller.
Changed the function to take a TABLE_LIST pointer as argument.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-04-05 09:45:22 UTC

MDEV-14231 MATCH() AGAINST( IN BOOLEAN MODE), results mismatch

st_mysql_ftparser_boolean_info(): Add variable position to
store the correct offset for the default fts parser.

Added plugin_debug.test, multiple_index.test from mysql-5.7

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-04-05 09:45:22 UTC

MDEV-14321 MATCH() AGAINST( IN BOOLEAN MODE), results mismatch

st_mysql_ftparser_boolean_info(): Add variable position to
store the correct offset for the default fts parser.

Added plugin_debug.test, multiple_index.test from mysql-5.7

Author: Alexander Barkov
Author Date: 2024-04-04 13:13:09 UTC

MDEV-33827 UUID() returns a NULL-able result

It was wrong to derive Item_func_uuid from Item_func_sys_guid,
because the former is a function returning the UUID data type,
while the latter is a string function returning VARCHAR.

As a result of the wrong hierarchy, Item_func_uuid erroneously derived
Item_str_func::fix_fields(), which contains this code:

  /*
    In Item_str_func::check_well_formed_result() we may set null_value
    flag on the same condition as in test() below.
  */
  if (thd->is_strict_mode())
    set_maybe_null();

This code is not relevant to UUID() at all.
A simple fix would be to set_maybe_null(false) in
Item_func_uuid::fix_length_and_dec(). However,
it'd fix only exactly this single consequence of the wrong
class hierarchy, and similar bugs could appear again in
the future. Moreover, we're going to add functions UUIDv4()
and UUIDv7() soon (in 11.6). So it's better to fix the class hierarchy
in the right way before adding these new functions.

Fix:

- Adding a new abstract class Item_fbt_func in the template
  in sql_type_fixedbin.h
- Deriving Item_typecast_fbt from Item_fbt_func
- Deriving Item_func_uuid from Item_fbt_func
- Adding a new helper class UUIDv1. It derives from UUID, and additionally
  initializes the value to "UUID version 1" right in the constructor.
  Note, the new coming soon SQL functions UUIDv4() and UUIDv7()
  will also have corresponding classes UUIDv4 and UUIDv7.

So now UUID() is a pure "returning UUID" function,
like CAST(expr AS UUID) used to be, without any unintentional
artifacts of functions returning VARCHAR/TEXT.

Cleanup:
- Removing the member Item_func_sys_guid::with_dashes,
  as it's not needed any more:
  * Item_func_sys_guid now does not have any descendants any more
  * Item_func_sys_guid::val_str() itself always displays without dashes

Author: Monty
Author Date: 2024-04-04 13:51:04 UTC

fixup! ddab8e0840f1b11c44bb05accae1291b1191aa1a

Author: sjaakola
Author Date: 2024-04-04 14:12:09 UTC

MDEV-33749 hyphen in table name can cause galera certification failures

Fix in this commit handles foreign key value appending into write set
so that db and table names are converted from the filepath format
to tablename format. This is compatible with key values appended from
elsewhere in the code base

There is a mtr test galera.galera_table_with_hyphen for regression testing

Reviewer: monty@mariadb.com

Author: Vicențiu Ciorbaru
Author Date: 2024-02-23 09:46:46 UTC

Vec insert and search working on a multi-layer

Author: Marko Mäkelä
Author Date: 2024-04-04 06:13:19 UTC

Merge 10.11 into 11.0

Author: Brandon Nesterenko
Author Date: 2024-04-01 15:46:50 UTC

MDEV-33799: mysql_manager_submit Segfault at Startup Still Possible During Recovery

MDEV-26473 fixed a segmentation fault at startup between the handle
manager thread and the binlog background thread, such that the
binlog background thread could be started and submit a job to the
handle manager, before it had initialized. Where MDEV-26473 made it
so the handle manager would initialize before the main thread
started the normal binary logs, it did not account for the recovery
case. That is, there is still a possibility of a segmentation fault
when a server is recovering using the binary logs such that it can
open the binary logs, start the binlog background thread, and submit
a job to the handle manager before it is initialized.

This patch fixes this by moving the initialization of the mysql
handler manager to happen prior to recovery.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: midenok
Author Date: 2024-04-03 13:39:36 UTC

Columnstore empty submodule fix 2

Original problem was error when configuring without initialized
submodule Columnstore:

  The source directory

    /home/midenok/src/mariadb/10.6c/src/storage/columnstore/columnstore

  does not contain a CMakeLists.txt file.

The original fix disabled Columnstore build when PLUGIN_COLUMNSTORE
was not defined, but this seems to be wrong and any plugin should be
built if it is not explicitly disabled. This is expected by buildbots.

Thanks to Vladislav Vaintroub <vvaintroub@gmail.com> for the fix

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-04-03 13:11:41 UTC

MDEV-33533: Crash at execution of DELETE when trying to use rowid filter

Make innodb ignore an rowid filter in rnd_pos()

Workaround about above bug removed from UPDATE.

Author: Marko Mäkelä
Author Date: 2024-04-02 06:54:56 UTC

MDEV-33802 Weird read view after ROLLBACK of another transaction

Even after commit b8a671988954870b7db22e20d1a1409fd40f8e3d there
is an anomaly where a locking read could return inconsistent results.
If a locking read would have to wait for a record lock, then by the
definition of a read view, the modifications made by the current lock
holder cannot be visible in the read view. This is because the read
view must exclude any transactions that had not been committed at the
time when the read view was created.

lock_rec_convert_impl_to_expl_for_trx(), lock_rec_convert_impl_to_expl():
Return an unsafe-to-dereference pointer to a transaction that holds or
held the lock, or nullptr if the lock was available.

lock_clust_rec_modify_check_and_lock(),
lock_sec_rec_read_check_and_lock(),
lock_clust_rec_read_check_and_lock():
Return DB_RECORD_CHANGED if innodb_strict_isolation=ON and the
lock was being held by another transaction.

The test case, which is based on a bug report by Zhuang Liu,
covers the function lock_sec_rec_read_check_and_lock().

Author: Alexey Botchkov
Author Date: 2024-04-02 10:04:15 UTC

test if the find_select_handler is actually working

    (expecially for the partition engine)
       Seems it implemented for the federatedex egngine only.

Author: Alexander Barkov
Author Date: 2024-03-29 09:28:40 UTC

MDEV-33701 upgrades from 11.4 to 11.5 don't work

A user variable and a literal with different collation
produce an illegal mix of collation error.
Rewriting the script to avoid such arguments.

Author: midenok
Author Date: 2024-04-01 23:51:18 UTC

MDEV-33125 CHECK TABLE does not recognize corruption
after EXCHANGE WITHOUT VALIDATION on system-time partitioning

MDEV-21011 disabled this check if there is no INTERVAL on history
partitions as the rows may be in any history partition. Now we
reenable the check and allow this condition.

Author: Oleg Smirnov
Author Date: 2024-04-01 07:03:46 UTC

MDEV-33010 Condition with CHARSET()/COERCIBILITY() erroneously pushed into derived table

Based on the current logic, objects of classes Item_func_charset and
Item_func_coercibility (responsible for CHARSET() and COERCIBILITY()
functions) are always considered const.
However, SQL syntax allows their use in a non-constant manner, such as
CHARSET(t1.a), COERCIBILITY(t1.a), and similar.

This "const-forcing" was introduced with
commit aa1002a35cefcc1851cbfb6b6b60463bda6f9aa3.

When those objects are erroneously deemed const, it could lead to
unwanted consequences like pushing down conditions to materialized
derived tables illegitimately.

This commits removes the "const-forcing" for Item_func_charset and
Item_func_coercibility, making them determine their constness in
the general way

Author: Alexander Barkov
Author Date: 2024-03-28 10:01:43 UTC

Cherry-picking from 10.5: MDEV-33788 HEX(COLUMN_CREATE(.. AS CHAR ...)) fails with --view-protocol

Item_func_dyncol_create::print_arguments() printed only CHARSET clause
without COLLATE.

Therefore,

HEX(column_create(1,'1212' AS CHAR CHARACTER SET utf8mb3 COLLATE utf8mb3_bin))

inside a VIEW changed to just:

HEX(column_create(1,'1212' AS CHAR CHARACTER SET utf8mb3))

which changed the collation ID seen in the HEX output.

Note, the collation ID inside column_create() is not really much important.
(It's only important what the character set is).
And for COLLATE, the more important thing is what's later written
in the AS clause of COLUMN_GET:

SELECT
   COLUMN_GET(
    column_create(1,'1212' AS CHAR CHARACTER SET utf8mb3 COLLATE utf8mb3_bin)
    column_nr AS type -- this type is more important
   );

Still, let's add the COLLATE clause into the COLUMN_CREATE() print output,
although it's not important for now for anything else than just the HEX output.
At least to make VIEW work in a more predictable way with HEX(COLUMN_CREATE()).

Also, in the future we can start using somehow the collation ID written inside
COLUMN_CREATE(), for example by making the `AS type` clause optional in
COLUMN_GET():
  COLUMN_GET(dyncol_blob, column_nr [AS type]);
instead of:
  COLUMN_GET(dyncol_blob, column_nr AS type);

SQL Server compatibility layer may need this for
the SQL_Variant data type support.

Author: Nikita Malyavin
Author Date: 2024-03-30 17:34:26 UTC

MDEV-15990 REPLACE on a precise-versioned table returns ER_DUP_ENTRY

Author: Elena Stepanova
Author Date: 2024-03-30 15:17:22 UTC

A build of MDEV-33655 + MDEV-33449 + MDEV-32188 for better coverage

And for getting packages stored

Author: Elena Stepanova
Author Date: 2024-03-30 12:58:50 UTC

Test commit

Author: Vlad Lesin
Author Date: 2024-03-29 12:02:44 UTC

MDEV-33796 innodb.deadlock_in_subqueries_join fails

It looks like the following code is not strict enough for assuring that a
statement is locked:

let $wait_condition=
  SELECT count(*) = 1 FROM information_schema.processlist
  WHERE (state = 'Sending data' OR state = "Updating")
  AND (info LIKE 'delete from t2 where%' OR
       info LIKE 'UPDATE t2 SET pkey=pkey+10 WHERE%');
--source include/wait_condition.inc

Use debug syncpoints instead.

Author: Marko Mäkelä
Author Date: 2024-03-28 10:15:36 UTC

Merge 11.0 into 11.1

Author: midenok
Author Date: 2024-03-27 19:55:38 UTC

MDEV-33780 Server crashes on renaming of table if table is locked

Under LOCK TABLES there is no ticket for old_table.