Author: Yuchen Pei
Author Date: 2024-03-13 05:11:28 UTC

MDEV-29363 avoid adding items with subqueries to item_equal

The optimizer constructs multiple equalities, and swap them in during
equality propagation. In doing so, it may create multiple pointers to
the same item. When the item contain subqueries, bad things can
happen. For example, in pushdown of HAVING to WHERE, if the HAVING
conditions contain multiple equalities with a const subquery, the
subquery will not be excluded from pushdown into WHERE. And if there
are multiple pointers to the same subquery, it may unfix the subquery
and destroy and rebuild its join, which could cause further
use-after-free if there are external pointers to items in the
join (e.g. tmp table fields as in the main case in MDEV-29363).

This patch also fixes cases in MDEV-32424, MDEV-32608, MDEV-32539.

Author: Yuchen Pei
Author Date: 2024-03-13 05:11:28 UTC

MDEV-29363 avoid adding items with subqueries to item_equal

The optimizer constructs multiple equalities, and swap them in during
equality propagation. In doing so, it may create multiple pointers to
the same item. When the item contain subqueries, bad things can
happen. For example, in pushdown of HAVING to WHERE, if the HAVING
conditions contain multiple equalities with a const subquery, the
subquery will not be excluded from pushdown into WHERE. And if there
are multiple pointers to the same subquery, it may unfix the subquery
and destroy and rebuild its join, which could cause further
use-after-free if there are external pointers to items in the
join (e.g. tmp table fields as in the main case in MDEV-29363).

This patch also fixes cases in MDEV-32424, MDEV-32608, MDEV-32539.

Author: Rex Johnston
Author Date: 2024-03-12 20:21:12 UTC

Merge branch '11.5' into 11.5-MDEV-31466

Author: Alexander Barkov
Author Date: 2024-03-07 08:00:36 UTC

MDEV-33621 Unify duplicate code in my_wildcmp_uca_impl() and my_wildcmp_unicode_impl()

This is a refactoring patch, it does not change the behaviour.
The MTR tests are being added only to cover the LIKE predicate better.
(these tests should have been added earlier under terms of MDEV 9711).
This patch does not need its own specific MTR tests.

Moving the duplicate code into a new shared file ctype-wildcmp.inl
and including it from multiple places, to define the following functions:

- my_wildcmp_uca_impl(), in ctype-uca.c

  For utf8mb3, utf8mb4, ucs2, utf16, utf32, using cs->cset->mb_wc().
  For UCA based collations.

- my_wildcmp_mb2_or_mb4_general_ci_impl(), in ctype-ucs2.c:

  For ucs2, utf16, utf32, using cs->cset->mb_wc().
  For general_ci-style collations:
      - xxx_general_ci
      - xxx_general_mysql500_ci
      - xxx_general_nopad_ci

- my_wildcmp_mb2_or_mb4_bin_impl(), in ctype-ucs2.c:

  For ucs2, utf16, utf32, using cs->cset->mb_wc().
  For _bin collations:
      - xxx_bin
      - xxx_nopad_bin

- my_wildcmp_utf8mb3_general_ci_impl(), in ctype-utf8.c

  Optimized for utf8mb3, using my_mb_wc_utf8mb3_quick().

  For general_ci-style collations:
      - utf8mb3_general_ci
      - utf8mb3_general_mysql500_ci
      - utf8mb3_general_nopad_ci

- my_wildcmp_utf8mb4_general_ci_impl(), in ctype-utf8.c

  Optimized for utf8mb4, using my_mb_wc_utf8mb4_quick().

  For general_ci-style collations:
      - utf8mb4_general_ci
      - utf8mb4_general_nopad_ci

Author: Yuchen Pei
Author Date: 2024-03-12 02:09:11 UTC

MDEV-33608 Skip spider/bugfix.quick_mode_N for valgrind builds

ASAN builds pass so it's ok

Author: Sergey Petrunia
Author Date: 2024-03-11 19:07:56 UTC

tmp

Author: Sergey Petrunia
Author Date: 2024-03-11 12:53:25 UTC

MDEV-33648: range optimizer works incorrectly for date_col=expression yielding invalid_date

range optimizer constructs index lookup value by calling
value_item->save_in_field(). When that function returns an error, it
calls Field::stored_field_make_mm_leaf_truncated().

That function has logic that "truncated value means there won't
be any matches for equality lookup". This is not true for DATE columns:
invalid date is equal to '0000-00-00', which can be present in the table.

Fix stored_field_make_mm_leaf_truncated() to not return SEL_ARG_IMPOSSIBLE
if the field is a date field.

Author: Sergei Golubchik
Author Date: 2024-02-29 17:16:06 UTC

MDEV-33554 Upgrade from 11.2 to 11.3 changes root's privileges

let ALL PRIVILEGES to always mean ALL PRIVILEGES over all
upgrades, no matter what new privileges were added in later versions.

Author: Kristian Nielsen
Author Date: 2024-03-08 21:43:26 UTC

MDEV-24622: Replication does not support bulk insert into empty table

Remove work-around that disables bulk insert optimization in replication

The root cause of the original problem is now fixed (MDEV-33475). Though the
bulk insert optimization will still be disabled in replication, as it is
only enabled in special circumstances meant for loading a mysqldump.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Kristian Nielsen
Author Date: 2024-03-08 15:26:07 UTC

MDEV-32976: Un-deprecate MASTER_USE_GTID=Current_Pos

Remove incorrect deprecation.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Monty
Author Date: 2024-03-08 13:36:41 UTC

MDEV-33620 Improve times and states in show processlist for replication

- Slave_IO thread time is now reset between reading events. Before
  this commit Slave_IO always showed "Waiting for master to send
  event" and the time was from SLAVE START. Now it shows time since
  reading last event.

Author: Rucha Deodhar
Author Date: 2024-02-27 14:13:51 UTC

MDEV-32854: Make JSON_DEPTH_LIMIT configurable

Create server variable to store limit for json depth with default value of
32, minimum 0 and maximum 100. Replace use of the previous direct or
indirect use JSON_DEPTH_LIMIT examples in array.

Author: Yuchen Pei
Author Date: 2024-03-08 04:40:41 UTC

MENT-1591 [demo] For debugging use.

Author: Alexander Barkov
Author Date: 2024-03-07 13:08:41 UTC

Extra tests for MDEV-30716 Wrong casefolding in xxx_unicode_520_ci for U+0700..U+07FF

New tests display additional information about characters from the BMP range:

- A summary with a COUNT(*) for all distinct combinations of properties
  telling how the "=" and the "LIKE" predicates compare characters to their
  LOWER() and UPPER() variants.

- A detailed list of trciky characters
  for which the "=" and the "LIKE" predicates compare
  LOWER(c)/UPPER(c) variants as not equal to just "c".

Tricky characters include:
 - Turkish letters: ı - small dotless letter i
 - Croatian letters: precombined contractions for Dž, Dz, Lj, Nj
 - Units of measurement: Ω,K,Å (Ohm, Kelvin, Angstrom)
   These ones look very similar to Greek letter Omega,
   Latin letter Kra, Swedish/Finnish letter A with a ring above.

Author: midenok
Author Date: 2024-03-07 12:24:33 UTC

Overwrite options from mysql_release.cmake

Author: VladislavVaintroub
Author Date: 2024-03-04 22:12:34 UTC

MDEV-33188 Add option --dir to mariadb-dump

New option works just like --tab, w.r.t output (sql file for table definition
and tab-separated for data).

Compared to --tab it allows --databases and --all-databases.
When --dir is used , it creates directory structure in the output directory,
pointed to by --dir. For every database to be dumped, there will be a directory
with database name.

All options that --tab supports, are also supported by --dir, in particular
--parallel

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-03-05 16:02:30 UTC

MDEV-32346 Assertion failure sym_node->table != NULL in pars_retrieve_table_def on UPDATE

- During update operation, InnoDB should avoid the initializing
the FTS_DOC_ID of foreign table if the foreign table is discarded

Author: Nikita Malyavin
Author Date: 2024-03-05 01:48:14 UTC

fix race in the MDEV-32614 test

Sometimes 'continue' signal could be missed.

Author: Brandon Nesterenko
Author Date: 2024-02-29 18:51:09 UTC

MDEV-33465: an option to enable semisync recovery

The current semi-sync binlog fail-over recovery process uses
rpl_semi_sync_slave_enabled==TRUE as its condition to truncate a
primary server’s binlog, as it is anticipating the server to re-join
a replication topology as a replica. However, for servers configured
with both rpl_semi_sync_master_enabled=1 and
rpl_semi_sync_slave_enabled=1, if a primary is just re-started (i.e.
retaining its role as master), it can truncate its binlog to drop
transactions which its replica(s) has already received and executed.
If this happens, when the replica reconnects, its gtid_slave_pos can
be ahead of the recovered primary’s gtid_binlog_pos, resulting in an
error state where the replica’s state is ahead of the primary’s.

This patch changes the condition for semi-sync recovery to truncate
the binlog to instead use the configuration variable
--init-rpl-role, when set to SLAVE. This allows for both
rpl_semi_sync_master_enabled and rpl_semi_sync_slave_enabled to be
set for a primary that is restarted, and no transactions will be
lost, so long as --init-rpl-role is not set to SLAVE.

Reviewed By:
============
<TODO>

Author: Dave Gosselin
Author Date: 2024-03-04 13:53:30 UTC

update tests where the index condition is used now instead of where clause

Author: Kristian Nielsen
Author Date: 2024-02-29 18:42:00 UTC

MDEV-25923: Aria parallel repair MY_THREAD_SPECIFIC mismatch in realloc

maria_repair_parallel() clears the MY_THREAD_SPECIFIC flag for allocations
since it uses different threads. But it still did one _ma_alloc_buffer()
call as thread-specific which would later assert if another thread needed
to extend the buffer with realloc.

This patch, due to Monty, removes the MY_THREAD_SPECIFIC flag for
allocations that need to realloc in different threads, and preserves
it for those that are allocated/freed in the user's thread.

Also fixes MDEV-33562.

Reviewed-by: Monty <monty@mariadb.org>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Yuchen Pei
Author Date: 2024-02-26 04:00:54 UTC

MDEV-33539 spider: remove some unused code in self reference checks

Author: Yuchen Pei
Author Date: 2024-03-03 23:25:34 UTC

MDEV-33538 make auxiliary spider plugins init depend on actual spider

The two I_S plugins SPIDER_ALLOC_MEM and SPIDER_WRAPPER_PROTOCOL
only makes sense if the main SPIDER plugin is installed. Further,
SPIDER_ALLOC_MEM requires a mutex that requires SPIDER init to fill
the table.

We also update the spider init query to override
--transaction_read_only=on so that it does not affect the spider init.

Also fixed error handling in spider_db_init() so that failure in
spider table init does not result in memory leak

Author: Dave Gosselin
Author Date: 2024-01-31 16:59:21 UTC

MDEV-12404 Index condition pushdown on partitioned tables

WIP, support index condition pushdown within partitioned tables.

Author: Dave Gosselin
Author Date: 2024-02-02 19:29:08 UTC

MDEV-32424 Crash in subselect engine

According to MDEV-29363, recompute the with_subquery() attribute
in the pushdown code to avoid writing to freed memory.

Author: Dave Gosselin
Author Date: 2023-11-30 16:18:17 UTC

MDEV-28509 Dereferenced null pointer of type 'struct JOIN_TAB' in add_key_field

This patch fixes a crash when calculating join statistics during query
optimization for queries with dangling WINDOW references. Put another way,
the system may crash when a query defines a WINDOW but doesn't then refer to
it.

Item::marker is overloaded for different uses, the most typical refer to it as
a bit field. However, the setup_group function uses it to mark that a field
was found when traversing a GROUP BY. Originally, this marking set the
Item::marker field to 1 to indicate that it was found. Later on in setup_group
(and only when SQL mode ONLY_FULL_GROUP_BY is enabled), we would skip any such
marked fields when checking that fields only referenced those found in the
GROUP BY; otherwise, it would be silly to find fields of the GROUP BY within
the GROUP BY field itself. Setting Item::marker to 1 seemed mostly harmless
at that point in time. But later, in git sha 4d143a6ff6, we introduced
several changes: (1) the value of marker in setup_group was changed from 1 to
UNDEF_POS, (2) Item::marker was changed from uint8 to int8 (which has since
been changed to an int), and (3) UNDEF_POS which is defined to be -1 was also
added.

Queries that define WINDOWs internally will setup groups and orders as part
of query processing via the setup_group function. Consequently because of
the behavior described earlier above, such queries may have items with markers
as UNDEF_POS (-1, or 0xffffffff) which is the same as marking all of the flag
bits as set. This is disastrous for those users of Item::marker which refer
to it as a bit field as some of those bits are mutually exclusive in meaning,
and in many places we don't mask the bits we're interested in, we just
compare the value of the field as a whole to some flag value with direct
comparison. In particular, the method
Item_direct_view_ref::grouping_field_transformer_for_where would
incorrectly see that the ref's marker was set for substitution when it was
actually -1, all bits set, taking the wrong execution path leading to the
crash.

Upon return from the setup_group function, we set the value of the marker
flag to zero if we set it to -1. This preserves the marker behavior for
'full group by' (if configured) while not otherwise allowing the marker
flag state to leak outside of this function.

Author: Sergei Golubchik
Author Date: 2024-02-29 17:16:06 UTC

MDEV-33554 Upgrade from 11.2 to 11.3 changes root's privileges

let ALL PRIVILEGES to always mean ALL PRIVILEGES over all
upgrades, no matter what new privileges were added in later versions.

Author: Sergei Golubchik
Author Date: 2024-02-29 17:16:06 UTC

MDEV-33554 Upgrade from 11.2 to 11.3 changes root's privileges

let ALL PRIVILEGES to always mean ALL PRIVILEGES over all
upgrades, no matter what new privileges were added in later versions.

Author: midenok
Author Date: 2024-02-29 04:38:48 UTC

incremental_encrypted fix

Author: Brandon Nesterenko
Author Date: 2023-07-10 15:53:19 UTC

MDEV-7850: Extend GTID Binlog Events with Thread Id

This patch augments Gtid_log_event with the user thread-id.
In particular that compensates for the loss of this info in
Rows_log_events.

Gtid_log_event::thread_id gets visible in mysqlbinlog output like

  #231025 16:21:45 server id 1 end_log_pos 537 CRC32 0x1cf1d963 GTID 0-1-2 ddl thread_id=10

as 64 bit unsigned integer.

While the size of Gtid event has grown by 8-9 bytes
replication from OLD <-> NEW is not affected by it. This patch
also slightly changes the logic to convert Gtid events to Query
events for older replicas which don't support Gtid. Instead of
hard-coding the padding of the sys var section of the generated
Query event, the length to pad is dynamically calculated based
on the length of the Gtid event.

This work was started by the late Sujatha Sivakumar.
Brandon Nesterenko took it over, reviewed initial patches and extended
the work.

Reviewed-by:
=============
Andrei Elkin <andrei.elkin@mariadb.com>
Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Igor Babaev
Author Date: 2024-02-28 22:56:45 UTC

MDEV-30073 Wrong result on 2nd execution of PS for query with NOT EXISTS

This is a preliminary patch to check possible failures in buildbot.
Variant 6

Author: Alexander Barkov
Author Date: 2023-06-23 09:24:02 UTC

MDEV-31531 Remove my_casedn_str() and my_caseup_str()

Under terms of MDEV 27490 we'll add support for non-BMP identifiers
and upgrade casefolding information to Unicode version 14.0.0.
In Unicode-14.0.0 conversion to lower and upper cases can increase octet length
of the string, so conversion won't be possible in-place any more.

This patch removes virtual functions performing in-place casefolding:
  - my_charset_handler_st::casedn_str()
  - my_charset_handler_st::caseup_str()
and fixes the code to use the non-inplace functions instead:
  - my_charset_handler_st::casedn()
  - my_charset_handler_st::caseup()

Author: Oleg Smirnov
Author Date: 2024-02-28 08:35:05 UTC

Move %union from sql_yacc.yy to sql_lex.h

Author: Georg Richter
Author Date: 2024-02-28 06:17:23 UTC

Update to C/C 3.4

Author: midenok
Author Date: 2024-02-27 14:53:07 UTC

Repeat without restart enabled in test

But it fails after ~100 repetitions because SHOW ENGINE INNODB STATUS
inserts " into output:

mysqltest: At line 132: query 'let $i= `select instr(\"\$s\", "LATEST
DETECTED DEADLOCK")`' failed: ER_PARSE_ERROR (1064): You have an error
in your SQL syntax; check the manual that corresponds to your MariaDB
server version for the right syntax to use near ';;
 2: len 7; hex 7f000000320110; asc 2 ;;
 3: len 4; hex 80000001; asc ...' at line 1

Author: midenok
Author Date: 2024-02-27 12:13:45 UTC

mysqltest intelligible error message about empty variable name

Author: Sergei Golubchik
Author Date: 2024-02-25 18:00:48 UTC

MDEV-33429 compilation of MariaDB 10.11.7 fails on arm32, sizeof(MYSQL) is wrong

Author: Monty
Author Date: 2023-12-02 15:58:26 UTC

Some changes to prepare for updated maria-backup

- Updated prototype for is_binary_frm_header().
- Added extra argument to ma_control_file_open().
- Added ma_control_file_open_or_create() for usage by tests.
  (to make test a bit simpler).

Author: Igor Babaev
Author Date: 2024-02-23 06:58:52 UTC

MDEV-31276 Wrong warnings on 2-nd execution of PS for query with GROUP_CONCAT

If a query with GROUP_CONCAT is executed then the server reports a warning
every time when the length of the result of this function exceeds the set
value of the system variable group_concat_max_len. This bug led to the set
of warnings from the second execution of the prepared statement that did
not coincide with the one from the first execution if the executed query
was a grouping query over a join of tables using GROUP_CONCAT function and
join cache was not allowed to be employed.
The descrepancy of the sets of warnings was due to lack of cleanup for
Item_func_group_concat::row_count after execution of the query.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Kristian Nielsen
Author Date: 2024-02-25 16:41:50 UTC

Binlog in Engine: Very first sketch, able to create and write an InnoDB tablespace

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Monty
Author Date: 2024-02-26 11:01:38 UTC

Moved test for online alter in connect to separate test

This was needed as mtr --embedded connect.alter failed as online is
ignored for embedder server.
This fixes a wrong commit 30c965f8664cd0aad44847c0b99b618d385c2ffe

Author: Julius Goryavsky
Author Date: 2024-02-19 17:17:36 UTC

galera: correction after wsrep-lib update

Correction to ensure compatibility with the updated wsrep-lib library.

Author: Marko Mäkelä
Author Date: 2024-02-23 08:09:14 UTC

fixup! 4d04025ef98ed82b41aecb4a17c66d96a4a65de6

Improve tests

Author: Marko Mäkelä
Author Date: 2024-02-22 11:53:17 UTC

WIP fix MDEV-26643

Author: VladislavVaintroub
Author Date: 2024-01-24 18:28:51 UTC

MDEV-32841 Add tests for new counters.

Only add innodb_async_reads/writes_total_count here.
For all other counters, writing reproducible tests is rather tricky.

Author: Nikita Malyavin
Author Date: 2024-02-21 16:11:09 UTC

MDEV-23729 INFORMATION_SCHEMA for users locked due to max_password_errors

Author: Georg Richter
Author Date: 2024-02-21 15:59:50 UTC

test fixes:

Since TLSv1.0 is no longer supported in C/C 3.4, TLS1.0 tests which
were expected to fail are removed.

Author: Marko Mäkelä
Author Date: 2024-02-21 10:57:48 UTC

Merge

Unresolved conflicts: storage/innobase/os/os0file.cc
FIXME: Unchecked return value of log_write_and_flush() in
mtr_t::commit_file()

Author: Yuchen Pei
Author Date: 2024-02-21 01:19:26 UTC

MDEV-31789 Deprecate spider_casual_read

The corresponding table param was deprecated as part of MDEV-28861

Author: Yuchen Pei
Author Date: 2024-01-04 01:12:50 UTC

MDEV-28152 Features for sequences

- Add `as <int_type>` to sequence creation options
  - int_type can be signed or unsigned integer types, including
    tinyint, smallint, mediumint, int and bigint
  - Limitation: when alter sequence as <new_int_type>, cannot have any
    other alter options in the same statement
  - Limitation: increment remains signed longlong, and the hidden
    constraint (cache_size x abs(increment) < longlong_max) stays for
    unsigned types. This means for bigint unsigned, neither
    abs(increment) nor (cache_size x abs(increment)) can be between
    longlong_max and ulonglong_max
- Truncating maxvalue and minvalue from user input to the nearest max
  or min value of the type, plus or minus 1. When the truncation
  happens, a warning is emitted
- Information schema table for sequences

Author: Alexander Barkov
Author Date: 2023-09-25 17:48:01 UTC

MDEV-12252 ROW data type for stored function return values

Adding support for the ROW data type in the stored function RETURNS clause:

- explicit ROW(..members...) for both sql_mode=DEFAULT and sql_mode=ORACLE

  CREATE FUNCTION f1() RETURNS ROW(a INT, b VARCHAR(32)) ...

- anchored "ROW TYPE OF [db1.]table1" declarations for sql_mode=DEFAULT

  CREATE FUNCTION f1() RETURNS ROW TYPE OF test.t1 ...

- anchored "[db1.]table1%ROWTYPE" declarations for sql_mode=ORACLE

  CREATE FUNCTION f1() RETURN test.t1%ROWTYPE ...

Adding support for anchored scalar data types in RETURNS clause:

- "TYPE OF [db1.]table1.column1" for sql_mode=DEFAULT

  CREATE FUNCTION f1() RETURNS TYPE OF test.t1.column1;

- "[db1.]table1.column1" for sql_mode=ORACLE

  CREATE FUNCTION f1() RETURN test.t1.column1%TYPE;

Details:

- Adding a new sql_mode_t parameter to
    sp_head::create()
    sp_head::sp_head()
    sp_package::create()
    sp_package::sp_package()
  to guarantee early initialization of sp_head::m_sql_mode.
  Before this change, this member was not initialized at all during
  CREATE FUNCTION/PROCEDURE/PACKAGE statements, and was not used.
  Now it needs to be initialized to write properly the
  mysql.proc.returns column, according to the create time sql_mode.

- Moving the code from the loop body in sp_rcontext::init_var_items()
  into a separate member Spvar_definition::make_item_field_row(),
  to reuse this code for variable/parameter definitions
  and for the RETURNS clause.

- Changing the data type of sp_head::m_return_field_def
  from Column_definition to Spvar_definition.
  So now it supports not only SQL column field types,
  but also explicit ROW and anchored ROW data types.

- Modifying sql_yacc.yy to support the new grammar.
  Adding new helper methods:
    * sf_return_fill_definition_row()
    * sf_return_fill_definition_rowtype_of()
    * sf_return_fill_definition_type_of()

- Fixing tests in:
  * Virtual_tmp_table::setup_field_pointers() in sql_select.cc
  * Send_field::normalize() in field.h
  to prevent calling Type_handler_row::field_type(),
  which is implemented a DBUG_ASSERT(0).
  Before this patch the affected methods were called only
  for scalar data types. Now ROW is also possible.

- Adding a new virtual method Field::cols()

- Overriding methods:
   Item_func_sp::cols()
   Item_func_sp::element_index()
   Item_func_sp::check_cols()
   Item_func_sp::bring_value()
  to support the ROW data type.

- Extending the rule sp_return_type to support
  * explicit ROW and anchored ROW data types
  * anchored scalar data types

- Overriding Field_row::sql_type() to print
  the data type of an explicit ROW.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-02-20 08:32:26 UTC

Merge branch '11.3' into mariadb-11.3.2

Author: VladislavVaintroub
Author Date: 2024-02-20 01:26:50 UTC

x

Author: Sergey Petrunia
Author Date: 2024-02-19 22:38:37 UTC

Trivial fix in my_bitmap.c

Author: Alexander Barkov
Author Date: 2023-10-07 15:35:18 UTC

MDEV-32380 Array data type for stored routnes

Author: Alexey Botchkov
Author Date: 2024-02-19 10:04:39 UTC

MDEV-20250

preliminary solution.

Author: Yuchen Pei
Author Date: 2024-02-19 04:12:16 UTC

MDEV-33434 spider direct sql: Check length before memcpy

similar to MDEV-30981

Author: Daniel Bartholomew
Author Date: 2024-02-17 22:18:13 UTC

Merge branch 'bb-11.4-bumpversion' of github.com:MariaDB/server into bb-11.4-bumpversion

Author: Daniel Bartholomew
Author Date: 2024-02-16 21:33:35 UTC

Merge branch 'bb-11.3-bumpversion' of github.com:MariaDB/server into bb-11.3-bumpversion

Author: VladislavVaintroub
Author Date: 2024-02-16 12:54:47 UTC

speed up SSL connection time by approx factor of 2, for WolfSSL.

Optimize common RSA math operations (WOLFSSL_HAVE_SP_RSA),
Use intel assembly optimizations for it.

"slow" ssl tests, such as main.ssl-big and main.ssl_connect now execute
in half the time, with this patch.

Author: Marko Mäkelä
Author Date: 2024-02-16 15:58:35 UTC

MDEV-32175 References to buf_page_t::frame may cost some performance

This is work in progress, for performance testing.

Based on the outcome, we might want to revert changes that cause
additional calls to page_offset() or page_align().

Author: Oleg Smirnov
Author Date: 2024-02-16 07:48:26 UTC

MDEV-33436 Transform OUTER JOIN to INNER JOIN in subqueries used with IN operator

This commit adds logging of the transformations made to joins
to the optimizer trace

Author: Andrei
Author Date: 2024-02-15 18:48:58 UTC

Another increment to Xlle support.

- xid:s of Xlle are treated as if they are XAP:s at the start of
  the (recovery) initial binlog file
- xid_cache_insert() needs an explicit "was binlogged" argument
  when xid is recovered (into prepared state)
- Xid_list_log_event::Xid_list_log_event() made to compute
  the size of the list inside the ctor's body; its
  List_log_event base is inited with zero.
  That's 'cos thd_arg may be also null and then a THD instance
  has to be dynamically created, to be passed to methods dealing
  with the list, also at time of `mysqld_server_started` is still false.

Author: Sergei Golubchik
Author Date: 2024-02-14 15:19:01 UTC

MDEV-31857 fix galera.MW-284

followup for abcd23add202

Author: midenok
Author Date: 2023-10-11 15:40:40 UTC

MDEV-30973 Run-time configuration and fixes

Logging slave retries is only needed for short period of time during
measurement. Requiring server restart makes this feature inconvenient
(or impossible in some cases).

This patch provides run-time configuration which requires concurrency
protection between writing output file and switching logging off.

ConfigFactory.pm provides correctly suffixed logname for master and
slave.

FLUSH ERROR LOGS flushes slave retries log as well.

Configuration options (Global):

  log_slave_retries
  log_slave_retries_file
  log_slave_retries_max

log_slave_retries enables or disables logging of slave retries.

log_slave_retries_file specifies path to log file.
Defaults to mysqld-retries.err in the default error log location.

log_slave_retries_max specifies maximum retry attempt at which the
repeated message is printed (see below).

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-02-13 08:24:32 UTC

fix view protocol in MDEV-29179

Author: Yuchen Pei
Author Date: 2024-02-12 10:08:22 UTC

MDEV-33441 Do not deinit plugin variables when retry requested

After MDEV-31400, plugins are allowed to ask for retries when failing
initialisation. However, such failures also cause plugin system
variables to be deleted (plugin_variables_deinit()) before retrying
and are not re-added during retry.

We fix this by checking that if the plugin has requested a retry the
variables are not deleted. Because plugin_deinitialize() also calls
plugin_variables_deinit(), if the retry fails, the variables will
still be deleted.

Alternatives considered:

- remove the plugin_variables_deinit() from plugin_initialize() error
handling altogether. We decide to take a more conservative approach
here.

- re-add the system variables during retry. It is more complicated
than simply iterating over plugin->system_vars and call
my_hash_insert(). For example we will need to assign values to
the test_load field and extract more code from test_plugin_options(),
if that is possible.

Author: Yuchen Pei
Author Date: 2024-02-12 13:11:21 UTC

MDEV-33434 spider direct sql: Check length before memcpy

similar to MDEV-30981

Author: Yuchen Pei
Author Date: 2024-02-12 10:08:22 UTC

MDEV-33441 Do not deinit plugin variables when retry requested

After MDEV-31400, plugins are allowed to ask for retries when failing
initialisation. However, such failures also cause plugin system
variables to be deleted (plugin_variables_deinit()) before retrying
and are not re-added during retry.

We fix this by checking that if the plugin has requested a retry the
variables are not deleted. Because plugin_deinitialize() also calls
plugin_variables_deinit(), if the retry fails, the variables will
still be deleted.

Alternatives considered:

- remove the plugin_variables_deinit() from plugin_initialize() error
handling altogether. We decide to take a more conservative approach
here.

- re-add the system variables during retry. It is more complicated
than simply iterating over plugin->system_vars and call
my_hash_insert(). For example we will need to assign values to
the test_load field and extract more code from test_plugin_options(),
if that is possible.

Author: Sergei Golubchik
Author Date: 2024-01-17 14:32:45 UTC

initial support for vector indexes

MDEV-33407 Parser support for vector indexes

The syntax is

  create table t1 (... vector index (v) ...);

limitation:
* v is a binary string and NOT NULL
* only one vector index per table
* temporary tables are not supported

MDEV-33404 Engine-independent indexes: subtable method

added support for so-called "high level indexes", they are not visible
to the storage engine, implemented on the sql level. For every such
an index in a table, say, t1, the server implicitly creates a second
table named, like, t1#i#05 (where "05" is the index number in t1).
This table has a fixed structure, no frm, not accessible directly,
doesn't go into the table cache, needs no MDLs.

MDEV-33406 basic optimizer support for k-NN searches

for a query like SELECT ... ORDER BY func() optimizer will use
item_func->part_of_sortkey() to decide what keys can be used
to resolve ORDER BY.

Author: VladislavVaintroub
Author Date: 2024-02-09 01:18:32 UTC

MDEV-33430 - Fix self-signed certificate errors on Windows

Adjust test after fixing the C/C.

On Windows, use --host=127.0.0.2 to fake "insecure" transport
with TCP connection for test purposes. 127.0.0.2 is loopback address,
that can be used instead of usual 127.0.0.1

Unfortunately, this technique does not work on all *nixes the same,
notably neither on BSDs nor Solaris. Thus default --host=localhost
remains "insecure" transport,when TCP is used. but it is not that critical,
the "self-signed" is not nearly as annoying on *nixes as it is on Windows.

Author: Brandon Nesterenko
Author Date: 2024-02-08 16:55:02 UTC

MDEV-14357: rpl.rpl_domain_id_filter_io_crash failed in buildbot with wrong result

A race condition with the SQL thread, where depending on if it was
killed before or after it had executed the fake/generated IGN_GTIDS
Gtid_list_log_event, may or may not update gtid_slave_pos with the
position of the ignored events. Then, the slave would be restarted
while resetting IGNORE_DOMAIN_IDS to be empty, which would result in
the slave requesting different starting locations, depending on
whether or not gtid_slave_pos was updated. And, because previously
ignored events could now be requested and executed (no longer
ignored), their presence would fail the test.

This patch fixes this in two ways. First, to use GTID positions for
synchronization rather than binlog file positions. Then second, to
synchronize the SQL thread’s gtid_slave_pos with the ignored events
before killing the SQL thread.

To consistently reproduce the test failure, the following patch can
be applied:

diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
index f51f5b7deec..de62233acff 100644
--- a/sql/log_event_server.cc
+++ b/sql/log_event_server.cc
@@ -3686,6 +3686,12 @@ Gtid_list_log_event::do_apply_event(rpl_group_info *rgi)
     void *hton= NULL;
     uint32 i;

+ sleep(1);
+ if (rli->sql_driver_thd->killed || rli->abort_slave)
+ {
+ return 0;
+ }
+

Author: Alexander Barkov
Author Date: 2024-02-08 11:58:26 UTC

MDEV-33428 Error messages ER_PACKAGE_ROUTINE_* are not good enough

Changing the format in error messages:
- ER_PACKAGE_ROUTINE_IN_SPEC_NOT_DEFINED_IN_BODY
- ER_PACKAGE_ROUTINE_FORWARD_DECLARATION_NOT_DEFINED

from
  "Subroutine 'db.pkg.f1' ..."

to a more clear:
  "FUNCTION `db.pkg.f1` ..."
  "PROCEDURE `db.pkg.p1` ..."

Author: Yuchen Pei
Author Date: 2024-02-08 07:32:34 UTC

MDEV-29363 avoid adding items with subqueries to item_equal

The optimizer constructs multiple equalities, and swap them in during
equality propagation. In doing so, it may create multiple pointers to
the same item. When the item contain subqueries, bad things can
happen. For example, in pushdown of HAVING to WHERE, if the HAVING
conditions contain multiple equalities with a const subquery, the
subquery will not be excluded from pushdown into WHERE. And if there
are multiple pointers to the same subquery, it may unfix the subquery
and destroy and rebuild its join, which could cause further
use-after-free if there are external pointers to items in the
join (e.g. tmp table fields as in the main case in MDEV-29363).

This patch also fixes cases in MDEV-32424, MDEV-32608, MDEV-32539.

Author: Dmitry Shulga
Author Date: 2024-02-08 05:17:02 UTC

MDEV-15703: Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT

This patch fixes the issue with passing the DEFAULT or IGNORE values to
positional parameters for some kind of SQL statements to be executed
as prepared statements.

The main idea of the patch is to associate an actual value being passed
by the USING clause with the positional parameter represented by
the Item_param class. Such association must be performed on execution of
UPDATE statement in PS/SP mode. Other corner cases that results in
server crash is on handling CREATE TABLE when positional parameter
placed after the DEFAULT clause or CALL statement and passing either
the value DEFAULT or IGNORE as an actual value for the positional parameter.
This case is fixed by checking whether an error is set in diagnostics
area at the function pack_vcols() on return from the function pack_expression()

Author: Daniel Black
Author Date: 2024-02-08 01:49:20 UTC

Merge branch '11.4' into bb-11.4-MDEV-18478

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-02-07 07:50:01 UTC

Merge branch '11.2' into mariadb-11.2.3

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-02-07 07:47:37 UTC

Merge branch '11.1' into mariadb-11.1.4

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-02-07 07:21:29 UTC

Merge branch '11.0' into mariadb-11.0.5

Author: Alexander Barkov
Author Date: 2024-02-07 05:05:28 UTC

MDEV-33392 Server crashes when using RANDOM_BYTES function and GROUP BY clause on a column with a negative value

Item_func_random_bytes did not set its NULL-ability flag.

Author: Yuchen Pei
Author Date: 2024-02-06 13:30:25 UTC

MDEV-32640 allow sys_var_charptr to update session value inplace

If true, it allocates the session value once, and updates it with
strmake(), instead of freeing existing char* and creating new ones.

Make default_master_connection such a variable, so that the session
updates syncs automatically with statement ones. This fixes a
regression introduced by MDEV-15935.

Author: Daniel Bartholomew
Author Date: 2024-02-06 13:29:54 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-02-06 13:28:35 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-02-06 13:27:16 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-02-06 13:25:30 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-02-06 13:23:21 UTC

bump the VERSION

Author: Daniel Bartholomew
Author Date: 2024-02-06 13:21:25 UTC

bump the VERSION

Author: Yuchen Pei
Author Date: 2024-02-06 11:26:01 UTC

MDEV-32640 [wip] add an in_place_update field to sys_var_charptr

Author: Oleg Smirnov
Author Date: 2024-02-05 10:43:09 UTC

MDEV-32034: Second SELECT from view created from INFORMATION_SCHEMA.OPTIMIZER_TRACE gives NULL

Function list_has_optimizer_trace_table() is responsible for checking
whether the query includes I_S.OPTIMIZER_TRACE. However, the function
is called at an early stage, before references are resolved,
resulting in the omission of cases where a SELECT is made from a view
constructed from I_S.OPTIMIZER_TRACE.

To address this, an supplementary call to
list_has_optimizer_trace_table() is added at the end of tracing,
when all the information is available. That allows to identify cases
in which the SELECT is made from a view or a stored procedure
containing I_S.OPTIMIZER_TRACE

Author: Brandon Nesterenko
Author Date: 2024-02-05 12:56:53 UTC

Revert "MDEV-7850: Extend GTID Binlog Events with Thread Id"

This reverts commit c37b2087b4abe576f1b0391c8d379dba6299dcb5.

In c37b20887, when re-binlogging a GTID event on a replica,
it will overwrite the thread_id from the primary to be the
value of the slave applier (SQL thread or parallel worker).

This should be the value of the original thread_id on the
master connection though, to both help track temporary
tables, and be consistent with Query_log_event.

Reverting the commit to re-target 11.5, so we can re-test
with the corrected thread_id.

Author: Alexander Barkov
Author Date: 2024-02-05 10:41:44 UTC

MDEV-33386 Wrong error message on `GRANT .. ON PACKAGE no_such_package ..`

When displaying the ER_SP_DOES_NOT_EXIST error, use
Sp_handler::type_lex_cstring() to the the underlying
object type:
- PROCEDURE
- FUNCTION
- PACKAGE
- PACKAGE BODY
instead of hard-coded "FUNCTION or PROCEDURE".

Author: Yuchen Pei
Author Date: 2024-02-05 09:56:51 UTC

[fixup] Spider: the MDEV-29718 fix requires use of spider_same_server_link

...for spider testcases where the remote data node is specified by a
server on the same server

Author: Sergei Golubchik
Author Date: 2023-10-15 14:03:14 UTC

MDEV-32473 --disable-ssl doesn't disable it

Author: Nikita Malyavin
Author Date: 2024-02-03 05:40:06 UTC

MDEV-33348 ALTER TABLE lock waiting stages are indistinguishable

Several points of synchronization during ALTER TABLE COPY looked identical
in the progress report query. Besides, if its the late lock upgrade stage,
the data would be:
STAGE 0
MAX_STAGE 0
PROGRESS 0.000

which looks irrelevant.

This patch moves thd_progress_deinit call after the last lock upgrade.

Also, for online alter, if there is nothing to replicate, the
progress and max_progress values would be 0, which discard the result data
on the side of sql_show, see processlist_callback in sql_show.cc.
So now the minimal max_progress will be 1. To avoid 0% progress in the
report, minimax progress value is also set to 1, so we will see 100% if
there's nothing to replicate.

Author: Igor Babaev
Author Date: 2024-02-03 04:33:40 UTC

MDEV-31361 Wrong result on 2nd execution of PS for query with derived table

This bug led to wrong result sets returned by the second execution of
prepared statements from selects using mergeable derived tables pushed
into external engine. Such derived tables are always materialized. The
decision that they have to be materialized is taken late in the function
mysql_derived_optimized(). For regular derived tables this decision is
usually taken at the prepare phase. However in some cases for some derived
tables this decision is made in mysql_derived_optimized() too. It can be
seen in the code of mysql_derived_fill() that for such a derived table it's
critical to change its translation table to tune it to the fields of the
temporary table used for materialization of the derived table and this
must be done after each refill of the derived table. The same actions are
needed for derived tables pushed into external engines.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Kristian Nielsen
Author Date: 2024-02-01 22:29:55 UTC

MDEV-31949: XA review, test cases for binlog recovery

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Rex Johnston
Author Date: 2023-08-18 03:51:54 UTC

MDEV-31466 Add optional correlation column list for derived tables

extend Mariadb to support column name assignment in derived sytax
(some expression) [as|=] ident [(comma separated column name list)]

Author: Sergey Petrunia
Author Date: 2024-01-26 13:54:35 UTC

MDEV-33314: Crash in calculate_cond_selectivity_for_table() with many columns

Variant#3: moved the logic out of create_key_parts_for_pseudo_indexes

Range Analyzer (get_mm_tree functions) can only process up to MAX_KEY=64
indexes. The problem was that calculate_cond_selectivity_for_table used
it to estimate selectivities for columns, and since a table can
have > MAX_KEY columns, would invoke Range Analyzer with more than MAX_KEY
"pseudo-indexes".

Fixed by making calculate_cond_selectivity_for_table() to run Range
Analyzer with at most MAX_KEY pseudo-indexes. If there are more
columns to process, Range Analyzer will be invoked multiple times.

Also made this change:
- param.real_keynr[0]= 0;
+ MEM_UNDEFINED(&param.real_keynr, sizeof(param.real_keynr));

Range Analyzer should have no use on real_keynr when it is run with
pseudo-indexes.

Author: Alexander Barkov
Author Date: 2024-02-01 11:20:37 UTC

Add a replication MTR test cloning the slave with mariadb-backup

Part2: adding a test for Galera->MariaDB replication

Author: Igor Babaev
Author Date: 2024-02-01 07:50:41 UTC

MDEV-31305 Crash caused by query with aggregation over materialized derived

This bug was fixed by the patch for bug MDEV-30706.
Only a test case is added in this commit.