Author: Dave Gosselin
Author Date: 2024-04-03 17:16:25 UTC

MDEV-32397, MDEV-32403 Crashes during join processing.

Queries having the following form may cause a crash
  SELECT t1.a FROM ( SELECT a AS a1 FROM t1 ) dt
  JOIN t1 ON a1 LIKE EXISTS ( SELECT a + RAND () FROM t1 UNION SELECT a FROM t1);
because the table t1 has some JOIN cleanup operations performed prematurely
during the subselect.

In this particular case, the presence of RAND() makes the subquery
uncacheable, necessitating the need to execute the subquery multiple
times during join record evaluation. Each time the subquery runs, it
creates its own JOIN structure which has references to the table t1.
When the subquery completes, JOIN::cleanup and functions called by it
result in ha_end_keyread() being called on table t1. However, we are
not done with table t1 because the upper level `select t1.a from...`
query requires table t1 to be open. To solve this, we make the executor
aware of when we're in subqueries like this and delay JOIN cleanup
until the end of the query.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-04-09 10:12:33 UTC

Merge branch '11.1' into 11.2

Author: Yuchen Pei
Author Date: 2024-04-09 06:43:18 UTC

MDEV-29363 [demo] avoid equality propagation of items with subqueries

We update one place to avoid equality propagation of items with
subqueries. It may not be thorough, but is sufficient for the
counterexample in this demo: simply run tests on
main.multiple_equality and notice how it fails in the MDEV-32539 case.

Author: Rucha Deodhar
Author Date: 2024-04-08 13:42:28 UTC

MDEV-30646: View created via JSON_ARRAYAGG returns incorrect json object

Analysis:
For the JSON typehandler ::type_handler_for_tmp_table() returns
plain VARCHAR. Hence so the VARCHAR field is created in the temporary table.
So 'is_json_type()' check in the append_json_value() fails and we end up
quoting the string additionally.
Fix:
Force treat it as json, because append_json_value() will of course always
append json.

Author: Monty
Author Date: 2024-04-06 09:07:49 UTC

squash! 9831521fc852f72317024778c723e28d121635e6

MDEV-33680 Server hangs or assertion fails upon SELECT with limited max_tmp_space_usage

The bug was that Aggregator_distinct::add() did not properly handle
write errors. (Old bug exposed by the new code).

Author: Alexander Barkov
Author Date: 2024-04-05 06:18:28 UTC

Fixup 2: MDEV-11339 Implement native UUID4 function

Adding a template Item_func_uuid_vx to share the code
between the implementations of UUID() and UUIDv4().

Author: Yuchen Pei
Author Date: 2024-03-26 03:37:17 UTC

MDEV-33734 Improve the sequence increment inequality testing

We add an extra condition that makes the inequality testing in
SEQUENCE::increment_value() mathematically watertight, and we cast to
and from unsigned in potential underflow and overflow addition and
subtractions to avoid undefined behaviour.

Let's start by distinguishing between c++ expressions and mathematical
expressions. by c++ expression I mean an expression with the outcome
determined by the compiler/runtime. by mathematical expression I mean
an expression whose value is mathematically determined. So a c++
expression -9223372036854775806 - 1000 at worst can evaluate to any
value due to underflow. A mathematical expression -9223372036854775806
- 1000 evaluates to -9223372036854776806.

The problem boils down to how to write a c++ expression equivalent to
an mathematical expression x + y < z where x and z can take any values
of long long int, and y < 0 is also a long long int. Ideally we want
to avoid underflow, but I'm not sure how this can be done.

The correct c++ form should be (x + y < z || x < z - y || x < z).
Let M=9223372036854775808 i.e. LONGLONG_MAX + 1. We have

-M < x < M - 1
-M < y < 0
-M < z < M - 1

Let's consider the case where x + y < z is true as a mathematical
expression.

If the first disjunct underflows, i.e. the mathematical expression x
+ y < -M. If the arbitrary value resulting from the underflow causes
the c++ expression to hold too, then we are done. Otherwise we move
onto the next expression x < z - y. If there's no overflow in z
- y then we are done. If there's overflow i.e. z - y > M - 1,
and the c++ expression evals to false, then we are onto x < z.
There's no over or underflow here, and it will eval to true. To see
this, note that

x + y < -M means x < -M - y < -M - (-M) = 0
z - y > M - 1 means z > y + M - 1 > - M + M - 1 = -1
so x < z.

Now let's consider the case where x + y < z is false as a mathematical
expression.

The first disjunct will not underflow in this case, so we move to (x <
z - y). This will not overflow. To see this, note that

x + y >= z means z - y <= x < M - 1

So it evals to false too. And the third disjunct x < z also evals to
false because x >= z - y > z.

I suspect that in either case the expression x < z does not determine
the final value of the disjunction in the vast majority cases, which
is why we leave it as the final one in case of the rare cases of both
an underflow and an overflow happening.

Here's an example of both underflow and overflow happening and the
added inequality x < z saves the day:

x = - M / 2
y = - M / 2 - 1
z = M / 2

x + y evals to M - 1 which is > z
z - y evals to - M + 1 which is < x

We can do the same to test x + y > z where the increment y is positive:

(x > z - y || x + y > z || x > z)

And the same analysis applies to unsigned cases.

Author: Yuchen Pei
Author Date: 2024-03-18 02:11:49 UTC

MDEV-33679 Spider group by handler: skip on multiple equalities

The spider group by handler is created in
JOIN::make_aggr_tables_info(), by which time calls to
substitute_for_best_equal_field() should have already removed all the
multiple equalities (i.e. Item_equal, with MULT_EQUAL_FUNC func_type).
Therefore, if there is still such items, it is deemed as an optimizer
bug and should be skipped.

Author: Yuchen Pei
Author Date: 2024-03-20 03:03:52 UTC

MDEV-33728 spider: remove use of MYSQL_VERSION_ID and MARIADB_BASE_VERSION

change created by:

unifdef -DMYSQL_VERSION_ID=100400 -DMARIADB_BASE_VERSION -m storage/spider/spd_* storage/spider/ha_spider.* storage/spider/hs_client/*

basically MDEV-27637, MDEV-27641, MDEV-27655

Author: Yuchen Pei
Author Date: 2024-03-19 23:36:25 UTC

MDEV-28992 Spider group by handler: Push down TIMESTAMPDIFF function

Also removed ITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC.

Similar to pr#2225, with the testcase adapted from that patch:

--8<---------------cut here---------------start------------->8---
From 884f7c6df16236748ca975339e0b1c267e195309 Mon Sep 17 00:00:00 2001
From: "Norio Akagi (norakagi)" <norakagi@amazon.com>
Date: Wed, 3 Aug 2022 23:30:34 -0700
Subject: [PATCH] [MDEV-28992] Push down TIMESTAMP_DIFF in spider

This changes so that TIMESTAMP_DIFF function in a query is pushed down and works natively in Spider.
Instead of directly accessing item's member, now we can rely on a public accessor method to make it work.
Unit tests are added under spider.pushdown_timestamp_diff.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
--8<---------------cut here---------------end--------------->8---

Author: Rex Johnston
Author Date: 2024-04-08 02:46:57 UTC

MDEV-31793 LooseScan join on derived table crashes

Test to see if not adding the offending key on the end has any consequences.

Author: Kristian Nielsen
Author Date: 2024-04-05 16:06:40 UTC

MDEV-33668: More efficient XA dependency tracking in SQL driver thread

Avoid linear scan of all recently queued XIDs in the SQL driver thread,
which might be expensive in XA-heavy workloads and large number of parallel
replication worker threads.

Instead keep a hash in the rpl_parallel_entry of where recently queued XIDs
were scheduled. This allows direct lookup of any potential scheduling
dependency.

Keep a list in each scheduling bucket of recently queued XIDs, and purge the
list (based on generations) when queueing next XA.

Also implement a more fine-grained dependency check based on sub_id
comparison. This can sometimes avoid a scheduling dependency that would
otherwise look necessary based solely on the generation check.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Rex Johnston
Author Date: 2024-04-03 00:55:24 UTC

MDEV-28621 group by optimization incorrectly removing subquery where subject buried in a function

A group by query optimization technique, where the a group by clause in
a subquery can be removed is incorrectly removing references to subquery
items that may be defined and used elsewhere.

This is the initial implementation of a reference counter in the unit
attached to an Item_subquery.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-04-05 10:39:56 UTC

MDEV-20094 InnoDB blob allocation allocates extra extents

- InnoDB reserves the free extents unnecessarily during blob
page allocation even though btr_page_alloc() can handle
reserving the extent when the existing ran out of pages to be used.

Author: Julius Goryavsky
Author Date: 2024-04-05 11:10:32 UTC

galera: wsrep-lib submodule update

Author: Daniele Sciascia
Author Date: 2024-01-10 13:34:12 UTC

MDEV-33216 stack-use-after-return in Wsrep_schema_impl::open_table()

Fix a case of stack-use-after-return reported by ASAN in
Wsrep_schema_impl::open_table(). This function has a stack allocated
TABLE_LIST object and return TABLE_LIST::table to the caller.
Changed the function to take a TABLE_LIST pointer as argument.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Daniele Sciascia
Author Date: 2024-01-10 13:34:12 UTC

MDEV-33216 stack-use-after-return in Wsrep_schema_impl::open_table()

Fix a case of stack-use-after-return reported by ASAN in
Wsrep_schema_impl::open_table(). This function has a stack allocated
TABLE_LIST object and return TABLE_LIST::table to the caller.
Changed the function to take a TABLE_LIST pointer as argument.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-04-05 09:45:22 UTC

MDEV-14231 MATCH() AGAINST( IN BOOLEAN MODE), results mismatch

st_mysql_ftparser_boolean_info(): Add variable position to
store the correct offset for the default fts parser.

Added plugin_debug.test, multiple_index.test from mysql-5.7

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2024-04-05 09:45:22 UTC

MDEV-14321 MATCH() AGAINST( IN BOOLEAN MODE), results mismatch

st_mysql_ftparser_boolean_info(): Add variable position to
store the correct offset for the default fts parser.

Added plugin_debug.test, multiple_index.test from mysql-5.7

Author: Alexander Barkov
Author Date: 2024-04-04 13:13:09 UTC

MDEV-33827 UUID() returns a NULL-able result

It was wrong to derive Item_func_uuid from Item_func_sys_guid,
because the former is a function returning the UUID data type,
while the latter is a string function returning VARCHAR.

As a result of the wrong hierarchy, Item_func_uuid erroneously derived
Item_str_func::fix_fields(), which contains this code:

  /*
    In Item_str_func::check_well_formed_result() we may set null_value
    flag on the same condition as in test() below.
  */
  if (thd->is_strict_mode())
    set_maybe_null();

This code is not relevant to UUID() at all.
A simple fix would be to set_maybe_null(false) in
Item_func_uuid::fix_length_and_dec(). However,
it'd fix only exactly this single consequence of the wrong
class hierarchy, and similar bugs could appear again in
the future. Moreover, we're going to add functions UUIDv4()
and UUIDv7() soon (in 11.6). So it's better to fix the class hierarchy
in the right way before adding these new functions.

Fix:

- Adding a new abstract class Item_fbt_func in the template
  in sql_type_fixedbin.h
- Deriving Item_typecast_fbt from Item_fbt_func
- Deriving Item_func_uuid from Item_fbt_func
- Adding a new helper class UUIDv1. It derives from UUID, and additionally
  initializes the value to "UUID version 1" right in the constructor.
  Note, the new coming soon SQL functions UUIDv4() and UUIDv7()
  will also have corresponding classes UUIDv4 and UUIDv7.

So now UUID() is a pure "returning UUID" function,
like CAST(expr AS UUID) used to be, without any unintentional
artifacts of functions returning VARCHAR/TEXT.

Cleanup:
- Removing the member Item_func_sys_guid::with_dashes,
  as it's not needed any more:
  * Item_func_sys_guid now does not have any descendants any more
  * Item_func_sys_guid::val_str() itself always displays without dashes

Author: sjaakola
Author Date: 2024-04-04 14:12:09 UTC

MDEV-33749 hyphen in table name can cause galera certification failures

Fix in this commit handles foreign key value appending into write set
so that db and table names are converted from the filepath format
to tablename format. This is compatible with key values appended from
elsewhere in the code base

There is a mtr test galera.galera_table_with_hyphen for regression testing

Reviewer: monty@mariadb.com

Author: Vicențiu Ciorbaru
Author Date: 2024-02-23 09:46:46 UTC

Vec insert and search working on a multi-layer

Author: Marko Mäkelä
Author Date: 2024-04-04 06:13:19 UTC

Merge 10.11 into 11.0

Author: Brandon Nesterenko
Author Date: 2024-04-01 15:46:50 UTC

MDEV-33799: mysql_manager_submit Segfault at Startup Still Possible During Recovery

MDEV-26473 fixed a segmentation fault at startup between the handle
manager thread and the binlog background thread, such that the
binlog background thread could be started and submit a job to the
handle manager, before it had initialized. Where MDEV-26473 made it
so the handle manager would initialize before the main thread
started the normal binary logs, it did not account for the recovery
case. That is, there is still a possibility of a segmentation fault
when a server is recovering using the binary logs such that it can
open the binary logs, start the binlog background thread, and submit
a job to the handle manager before it is initialized.

This patch fixes this by moving the initialization of the mysql
handler manager to happen prior to recovery.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: midenok
Author Date: 2024-04-03 13:39:36 UTC

Columnstore empty submodule fix 2

Original problem was error when configuring without initialized
submodule Columnstore:

  The source directory

    /home/midenok/src/mariadb/10.6c/src/storage/columnstore/columnstore

  does not contain a CMakeLists.txt file.

The original fix disabled Columnstore build when PLUGIN_COLUMNSTORE
was not defined, but this seems to be wrong and any plugin should be
built if it is not explicitly disabled. This is expected by buildbots.

Thanks to Vladislav Vaintroub <vvaintroub@gmail.com> for the fix

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-04-03 13:11:41 UTC

MDEV-33533: Crash at execution of DELETE when trying to use rowid filter

Make innodb ignore an rowid filter in rnd_pos()

Workaround about above bug removed from UPDATE.

Author: Marko Mäkelä
Author Date: 2024-04-02 06:54:56 UTC

MDEV-33802 Weird read view after ROLLBACK of another transaction

Even after commit b8a671988954870b7db22e20d1a1409fd40f8e3d there
is an anomaly where a locking read could return inconsistent results.
If a locking read would have to wait for a record lock, then by the
definition of a read view, the modifications made by the current lock
holder cannot be visible in the read view. This is because the read
view must exclude any transactions that had not been committed at the
time when the read view was created.

lock_rec_convert_impl_to_expl_for_trx(), lock_rec_convert_impl_to_expl():
Return an unsafe-to-dereference pointer to a transaction that holds or
held the lock, or nullptr if the lock was available.

lock_clust_rec_modify_check_and_lock(),
lock_sec_rec_read_check_and_lock(),
lock_clust_rec_read_check_and_lock():
Return DB_RECORD_CHANGED if innodb_strict_isolation=ON and the
lock was being held by another transaction.

The test case, which is based on a bug report by Zhuang Liu,
covers the function lock_sec_rec_read_check_and_lock().

Author: Alexey Botchkov
Author Date: 2024-04-02 10:04:15 UTC

test if the find_select_handler is actually working

    (expecially for the partition engine)
       Seems it implemented for the federatedex egngine only.

Author: Alexander Barkov
Author Date: 2024-03-29 09:28:40 UTC

MDEV-33701 upgrades from 11.4 to 11.5 don't work

A user variable and a literal with different collation
produce an illegal mix of collation error.
Rewriting the script to avoid such arguments.

Author: midenok
Author Date: 2024-04-01 23:51:18 UTC

MDEV-33125 CHECK TABLE does not recognize corruption
after EXCHANGE WITHOUT VALIDATION on system-time partitioning

MDEV-21011 disabled this check if there is no INTERVAL on history
partitions as the rows may be in any history partition. Now we
reenable the check and allow this condition.

Author: Oleg Smirnov
Author Date: 2024-04-01 07:03:46 UTC

MDEV-33010 Condition with CHARSET()/COERCIBILITY() erroneously pushed into derived table

Based on the current logic, objects of classes Item_func_charset and
Item_func_coercibility (responsible for CHARSET() and COERCIBILITY()
functions) are always considered const.
However, SQL syntax allows their use in a non-constant manner, such as
CHARSET(t1.a), COERCIBILITY(t1.a), and similar.

This "const-forcing" was introduced with
commit aa1002a35cefcc1851cbfb6b6b60463bda6f9aa3.

When those objects are erroneously deemed const, it could lead to
unwanted consequences like pushing down conditions to materialized
derived tables illegitimately.

This commits removes the "const-forcing" for Item_func_charset and
Item_func_coercibility, making them determine their constness in
the general way

Author: Alexander Barkov
Author Date: 2024-03-28 10:01:43 UTC

Cherry-picking from 10.5: MDEV-33788 HEX(COLUMN_CREATE(.. AS CHAR ...)) fails with --view-protocol

Item_func_dyncol_create::print_arguments() printed only CHARSET clause
without COLLATE.

Therefore,

HEX(column_create(1,'1212' AS CHAR CHARACTER SET utf8mb3 COLLATE utf8mb3_bin))

inside a VIEW changed to just:

HEX(column_create(1,'1212' AS CHAR CHARACTER SET utf8mb3))

which changed the collation ID seen in the HEX output.

Note, the collation ID inside column_create() is not really much important.
(It's only important what the character set is).
And for COLLATE, the more important thing is what's later written
in the AS clause of COLUMN_GET:

SELECT
   COLUMN_GET(
    column_create(1,'1212' AS CHAR CHARACTER SET utf8mb3 COLLATE utf8mb3_bin)
    column_nr AS type -- this type is more important
   );

Still, let's add the COLLATE clause into the COLUMN_CREATE() print output,
although it's not important for now for anything else than just the HEX output.
At least to make VIEW work in a more predictable way with HEX(COLUMN_CREATE()).

Also, in the future we can start using somehow the collation ID written inside
COLUMN_CREATE(), for example by making the `AS type` clause optional in
COLUMN_GET():
  COLUMN_GET(dyncol_blob, column_nr [AS type]);
instead of:
  COLUMN_GET(dyncol_blob, column_nr AS type);

SQL Server compatibility layer may need this for
the SQL_Variant data type support.

Author: Elena Stepanova
Author Date: 2024-03-30 15:17:22 UTC

A build of MDEV-33655 + MDEV-33449 + MDEV-32188 for better coverage

And for getting packages stored

Author: Elena Stepanova
Author Date: 2024-03-30 12:58:50 UTC

Test commit

Author: Vlad Lesin
Author Date: 2024-03-29 12:02:44 UTC

MDEV-33796 innodb.deadlock_in_subqueries_join fails

It looks like the following code is not strict enough for assuring that a
statement is locked:

let $wait_condition=
  SELECT count(*) = 1 FROM information_schema.processlist
  WHERE (state = 'Sending data' OR state = "Updating")
  AND (info LIKE 'delete from t2 where%' OR
       info LIKE 'UPDATE t2 SET pkey=pkey+10 WHERE%');
--source include/wait_condition.inc

Use debug syncpoints instead.

Author: Marko Mäkelä
Author Date: 2024-03-28 10:15:36 UTC

Merge 11.0 into 11.1

Author: midenok
Author Date: 2024-03-27 19:55:38 UTC

MDEV-33780 Server crashes on renaming of table if table is locked

Under LOCK TABLES there is no ticket for old_table.

Author: Yuchen Pei
Author Date: 2024-03-27 02:41:44 UTC

MDEV-33731 Only iterate over m_locked_partitions in update_next_auto_inc_val()

Author: Yuchen Pei
Author Date: 2024-03-27 00:15:25 UTC

MDEV-33777 Spider: Correct checks for show index column numbers

It was updated for 10.6+ in MDEV-7317. Because a lower version spider
node may connect to a higher version data node, we need to change this
for 10.4 and 10.5 as well.

Author: Yuchen Pei
Author Date: 2024-03-27 00:15:25 UTC

MDEV-33777 Spider: Correct checks for show index column numbers

It was updated for 10.6+ in MDEV-7317. Because a lower version spider
node may connect to a higher version data node, we need to change this
for 10.4 and 10.5 as well.

Author: midenok
Author Date: 2024-03-13 22:39:14 UTC

MDEV-27293 Allow converting a versioned table from implicit
           to explicit row_start/row_end columns

In case of adding both system fields of same type (length, unsigned
flag) as old implicit system fields do the rename of implicit system
fields to the ones specified in ALTER, remove SYSTEM_INVISIBLE flag in
that case. Correct PERIOD clause must be specified in ALTER as well.

Author: Alexey Botchkov
Author Date: 2024-03-07 12:02:35 UTC

MDEV-33393 audit plugin do not report user did the action.

Specific test for the auditing plugin v1 instead of the rpl_auditing
test.

Author: Sergey Petrunia
Author Date: 2024-03-25 10:53:55 UTC

MDEV-28621: group by optimization incorrectly removing subquery

Try a different approach at removal: remove only "local" subqueries
for which we can't find any other references in the select list.

Author: Denis Protivensky
Author Date: 2024-02-06 08:55:04 UTC

MDEV-33136: Properly BF-abort user transactions with explicit locks

User transactions may acquire explicit MDL locks from InnoDB level
when persistent statistics is re-read for a table.
If such a transaction would be subject to BF-abort, it was improperly
detected as a system transaction and wouldn't get aborted.

The fix: Check if a transaction holding explicit MDL locks is a user
transaction in the MDL conflict handling code.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Daniele Sciascia
Author Date: 2024-03-21 13:03:19 UTC

MDEV-33509 Failed to apply write set with flags=(rollback|pa_unsafe)

Fix function `remove_fragment()` in wsrep_schema so that no error is
raised if the fragment to be removed is not found in the
wsrep_streaming_log table. This is necessary to handle the case where
streaming transaction in idle state is BF aborted. This may result in
the case where the rollbacker thread successfully removes the
transaction's fragments, followed by the applier's attempt to remove
the same fragments. Causing the node to leave the cluster after
reporting a "Failed to apply write set" error.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Daniele Sciascia
Author Date: 2024-03-21 13:03:19 UTC

MDEV-33509 Failed to apply write set with flags=(rollback|pa_unsafe)

Fix function `remove_fragment()` in wsrep_schema so that no error is
raised if the fragment to be removed is not found in the
wsrep_streaming_log table. This is necessary to handle the case where
streaming transaction in idle state is BF aborted. This may result in
the case where the rollbacker thread successfully removes the
transaction's fragments, followed by the applier's attempt to remove
the same fragments. Causing the node to leave the cluster after
reporting a "Failed to apply write set" error.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Yuchen Pei
Author Date: 2024-03-22 04:29:41 UTC

MDEV-33739 Check field type of the first field in check_sequence_fields()

This avoids non-integral types breaking the call of
sequence_structure().

Author: Yuchen Pei
Author Date: 2024-03-26 03:52:16 UTC

remove "for ubsan" see what happens

Author: Rex Johnston
Author Date: 2024-03-21 23:57:16 UTC

MDEV-28621 what if we delay actual subquery removal to the end of the optimization phase?

not sure this is the best way to do it, but at least is has the virtue
of being mildly cunning by leaving the actual subquery removal to end of
the optimization phase and untaging something for removal if we walk
into it earlier in the optimization.

Author: Daniele Sciascia
Author Date: 2023-05-11 08:19:26 UTC

Re-enable MTR test galera_sr.GCF-1060

Test seems to pass reliably now.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Daniele Sciascia
Author Date: 2023-05-11 08:19:26 UTC

Re-enable MTR test galera_sr.GCF-1060

Test seems to pass reliably now.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Nikita Malyavin
Author Date: 2023-01-18 11:49:35 UTC

REfactor REPLACE!!!!

Author: Denis Protivensky
Author Date: 2024-03-25 13:40:55 UTC

MDEV-32633: Fix Galera cluster <-> native replication interaction

It's possible to establish Galera multi-cluster setups connected
through the native replication when every Galera cluster is configured
to have a separate domain ID.
For this setup to work, we need to replace domain ID values in generated
GTID events when they are written at transaction commit to the values
configured by Wsrep replication.

At the same time, it's possible that the GTID event already contains
a correct domain ID if it comes through the native replication from
another Galera cluster.
In this case, when such an event is applied either through a native
replication slave thread or through Wsrep applier, we write GTID event
on transaction start and avoid writing it during transaction commit.

The code contained multiple problems that were fixed:
- applying GTID events didn't work because it's applied without a
running server transaction and Wsrep transaction was not started
- GTID event generation on transaction start didn't contain proper
"standalone" and "is_transactional" flags that the original applied
GTID event contained
- condition determining that GTID event is written on transaction start
to avoid writing it on commit relied on the fact that the GTID event
is the first found in transaction/statement caches, which wasn't the
case and resulted in duplicate GTID events written
- instead of relying on the caches to find a GTID event, a simple check
is introduced that follows the exact rules for checking if event is
written at transaction start as described above
- the test case is improved to check that exact GTID events are
applied after two Galera clusters have synced.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Denis Protivensky
Author Date: 2024-03-05 08:51:04 UTC

MDEV-32633: Fix Galera cluster <-> native replication interaction

It's possible to establish Galera multi-cluster setups connected
through the native replication when every Galera cluster is configured
to have a separate domain ID.
For this setup to work, we need to replace domain ID values in generated
GTID events when they are written at transaction commit to the values
configured by Wsrep replication.

At the same time, it's possible that the GTID event already contains
a correct domain ID if it comes through the native replication from
another Galera cluster.
In this case, when such an event is applied either through a native
replication slave thread or through Wsrep applier, we write GTID event
on transaction start and avoid writing it during transaction commit.

The code contained multiple problems that were fixed:
- applying GTID events didn't work because it's applied without a
running server transaction and Wsrep transaction was not started
- GTID event generation on transaction start didn't contain proper
"standalone" and "is_transactional" flags that the original applied
GTID event contained
- condition determining that GTID event is written on transaction start
to avoid writing it on commit relied on the fact that the GTID event
is the first found in transaction/statement caches, which wasn't the
case and resulted in duplicate GTID events written
- instead of relying on the caches to find a GTID event, a simple check
is introduced that follows the exact rules for checking if event is
written at transaction start as described above
- the test case is improved to check that exact GTID events are
applied after two Galera clusters have synced.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Yuchen Pei
Author Date: 2024-03-21 06:17:53 UTC

MDEV-33220 Fix -wmaybe-uninitialized warnings for g++-13

Author: Yuchen Pei
Author Date: 2024-02-21 01:19:26 UTC

MDEV-31789 Deprecate spider_casual_read

The corresponding table param was deprecated as part of MDEV-28861

Author: VladislavVaintroub
Author Date: 2024-03-22 17:09:21 UTC

MDEV-33748 get rid of pthread_key_t and pthread_get/setspecific APIs.

Replaced with C-style _Thread_local(standard since C11), C++11 thread_local
or compiler-dependent equivalents.

thread_local is easier to use, does not require specfic order
of initialization/termination, unlike pthread thread specific variables.

It is also slightly faster to access these variables. This comes handy in
threadpool code, where a bunch of TLS variables are reset on every query.

Author: Brandon Nesterenko
Author Date: 2024-02-27 19:11:06 UTC

MDEV-33551: Semi-sync Wait Point AFTER_COMMIT Slow on Workloads with Heavy Concurrency

When using semi-sync replication with
rpl_semi_sync_master_wait_point=AFTER_COMMIT, the performance of the
primary can significantly reduce compared to AFTER_SYNC's
performance for workloads with many concurrent users executing
transactions. This is because all connections on the primary share
the same cond_wait variable/mutex pair, so any time an ACK is
received from a replica, all waiting connections are awoken to check
if the ACK was for itself, which is done in mutual exclusion.

This patch changes this such that the waiting THD will use its own
local condition variable, and the ACK receiver thread only signals
connections which have been ACKed for wakeup. That is, the
THD::LOCK_wakeup_ready condition variable is re-used for this
purpose, and the Active_tranx queue nodes are extended to hold the
waiting thread, so it can be signalled once ACKed.

Additionally:

 1) Removed part of MDEV-11853 additions, which allowed suspended
connection threads awaiting their semi-sync ACKs to live until their
ACKs had been received. This part, however, wasn't needed. That is,
all that was needed was for the Ack_thread to survive. So now the
connection threads are killed during phase 1. Thereby
THD::is_awaiting_semisync_ack, and all its related code was removed.

 2) COND_binlog_send is repurposed to signal on the condition when
Active_tranx is emptied during clear_active_tranx_nodes.

 3) At master shutdown (when waiting for slaves), instead of the
main loop individually waiting for each ACK, await_slave_reply()
(renamed await_all_slave_replies()) just waits once for the
repurposed COND_binlog_send to signal it is empty.

 4) Test rpl_semi_sync_shutdown_await_ack is updates as following:
   4.1) Added test case (adapted from Kristian Nielsen) to ensure
that if a thread awaiting its ACK is killed while SHUTDOWN WAIT FOR
ALL SLAVES is issued, the primary will still wait for the ACK from
the killed thread.
   4.2) As connections which by-passed phase 1 of thread killing no
longer are delayed for kill until phase 2, we can no longer query
yes/no tx after receiving an ACK/timeout. The check for these
variables is removed.
   4.3) Comment descriptions are updated which mention that the
connection is alive; and adjusted to be the Ack_thread.

Reviewed By:
============
Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Marko Mäkelä
Author Date: 2024-03-21 15:20:54 UTC

PMEM: Invoke mtr_t::finish_write via a function pointer

Author: Nikita Malyavin
Author Date: 2024-03-20 16:47:39 UTC

perf measures

Author: Marko Mäkelä
Author Date: 2024-03-20 13:36:44 UTC

MDEV-33515 attempt 1 (halving throughput on Haswell)

Let us make log_sys.lsn_lock a pure spin lock, using the x86 instruction
xchg for acquisition. On a Haswell microarchitecture dual
Intel Xeon E5-2630 v4 at 256 virtual users, this is nearly halving the
throughput. But, the Linux kernel function native_queued_spin_lock_slowpath
surely gets replaced with lsn_delay().

Author: Andrei
Author Date: 2024-03-19 20:31:26 UTC

Review suggestion.

As maybe_active_xid can expand in runtime and its extra allocated elements
may not be needed in a while, lets check for that and shrink
the size to a double of the initial one.

Author: Rucha Deodhar
Author Date: 2024-03-19 19:12:28 UTC

MDEV-8578: Wrong error code/message with enforce_storage_engine and
NO_ENGINE_SUBSTITUTION

Analysis:
When the error is hit, wrong error code is passed in my_error
Fix:
Pass a better error code.

Author: Nikita Malyavin
Author Date: 2024-03-19 12:40:54 UTC

Fix integer types

Author: Rex Johnston
Author Date: 2024-03-19 01:31:58 UTC

MDEV-32297 pushed condition into context with empty table list crashes.

When a condition is pushed into a subquery that has no table references,
under some circumstances, the context for the purposes of name resolution
may be left in an unexpected state.

Author: Rex Johnston
Author Date: 2024-03-18 20:50:19 UTC

MDEV-32583 UUID() should be treated as stochastic for the purposes of forcing query materialization

RAND() and UUID() are treated differently with respect to subquery materialization
both should be marked as uncacheable, forcing materialization.
Altered Create_func_uuid::create_builder() to mark it as uncacheable.

Author: Rucha Deodhar
Author Date: 2024-03-18 20:00:13 UTC

MDEV-19487: JSON_TYPE doesnt detect the type of String Values
(returns NULL) and for Date/DateTime returns "INTEGER"

Analysis:
When the first character of json is scanned it is number. Based on that
integer is returned.
Fix:
Scan rest of the json before returning the final result to ensure json is
valid in the first place in order to have a valid type.

Author: Dave Gosselin
Author Date: 2024-03-13 13:25:50 UTC

MDEV-33652 compat/oracle.sp-inout fails on macOS

On systems where the database is running from a case-insensitive
filesystem (like APFS which is case-insensitive by default), we
discard the SP name case information. Later, if we print or
otherwise display this name to the user, it’s shown as all
lowercase, in contradiction to our documentation.

When specifying the name of a SP, we remember that name both in its
original case and in a lowercase form. We show the original case
to the user whenever it may appear in an error message or show output.
We retain the all-lowercase value for internal processing as we do today,
so internal SP lookup procedures are unaffected.

Author: Sergei Golubchik
Author Date: 2024-03-15 17:42:06 UTC

mysqltest: support MARIADB_OPT_RESTRICTED_AUTH

C/C 3.4 disables mysql_old_password by default, so

add an option for the `connect` command to support specifying
allowed authentication plugins (MARIADB_OPT_RESTRICTED_AUTH).

use it to enable mysql_old_password when needed for testing

Author: VladislavVaintroub
Author Date: 2024-03-18 12:30:03 UTC

MDEV-33519 Remove WITH_SSL=<custom_location_of_openssl> option

This commit removes the WITH_SSL=<custom_location_of_openssl> option,
leaving only -DWITH_SSL=bundled/system.

The rationale behind this removal is as follows:

1. The WITH_SSL=<custom_location_of_openssl> option is obscure
and not widely used.

2. There is no added value in this option compared to using
OPENSSL_ROOT_DIR. In fact, the availability of "helpful" MySQL options
might discourage users from exploring proper CMake options independently.

3. Users may incorrectly assume full MySQL compatibility even with this
option, including undocumented behaviors such as MySQL's preference for static libraries
with WITH_SSL=<custom_location_of_openssl>.

This change simplifies the configuration options and encourages users to
adopt more standardized and documented practices.

Author: Dmitry Shulga
Author Date: 2024-03-18 10:43:16 UTC

MDEV-33525: Recreate/reuse temporary table

Calling a stored function that uses a cursor inside its body
could produce the error ER_NO_SUCH_TABLE on the second execution
in case the cursor uses multi-table query and one of the tables
is a temporary table just created before querying the cursor and
dropped just after the query has been executed.

The reason for issue is that re-parsing of failed a SP instruction
caused be create/drop of the temporary table used LEX object
left from previous parsing of a SP instruction's query instead
re-initialize the lex object before parsing.

To fix the issue, add initialization of lex for cursor's
statement before re-parsing the query of a failed SP instruction.

Author: Sergey Petrunia
Author Date: 2024-03-18 10:12:23 UTC

Leak-proof error cleanup in Item_in_subselect::setup_mat_engine()

The only pointer to allocated "engine" is in mat_engine local variable.
Free the mat_engine if we've failed to initialize it.

Author: Sergei Golubchik
Author Date: 2024-03-15 21:12:30 UTC

MDEV-32218 PASSWORD_EXPIRATION_TIME column

* show it as a datetime, not number of seconds
* show all users
* show manually expired users as 0000-00-00 00:00:00
* show default expiration interval correctly
* numerous test fixes, add more tests
* fix compilation of embedded

Author: Rex Johnston
Author Date: 2023-08-18 03:51:54 UTC

MDEV-31466 Add optional correlation column list for derived tables

extend Mariadb to support column name assignment in derived sytax
(some expression) [as|=] ident [(comma separated column name list)]

Author: Yuchen Pei
Author Date: 2024-02-20 00:20:21 UTC

MDEV-33494 fix spider init with no_zero_date global sql mode

Like the fix for MDEV-32753 and MDEV-33242, spider init queries
creates new connections that use the global sql_mode of the existing
connection.

Author: Yuchen Pei
Author Date: 2024-03-18 02:11:49 UTC

MDEV-33679 Spider group by handler: skip on multiple equalities

The spider group by handler is created in
JOIN::make_aggr_tables_info(), by which time calls to
substitute_for_best_equal_field() should have already removed all the
multiple equalities (i.e. Item_equal, with MULT_EQUAL_FUNC func_type).
Therefore, if there is still such items, it is deemed as an optimizer
bug and should be skipped.

Author: Monty
Author Date: 2024-01-04 18:44:38 UTC

Optimize checking if a table is a statistics table

Author: Sergei Golubchik
Author Date: 2024-03-12 13:08:23 UTC

cleanup: remove convert_dash_to_underscore

it was a no-op, plugin variables don't have dashes

Author: Monty
Author Date: 2023-12-31 10:41:25 UTC

MDEV-33144 Implement the Percona variable slow_query_log_always_write_time

This task is inspired by the Percona implementation of
slow_query_log_always_write_time.

This task implements the variable log_slow_always_query_time (name
matching other MariaDB variables using the slow query log). The
default value for the variable is 0, which makes MariaDB compatible
with older installations.

For queries with execution time longer than log_slow_always_query_time
the variables log_slow_rate_limit and log_slow_min_examined_row_limit
will not ignored and the query will be written to the slow query log
if there is no other limitations (like log_slow_filter etc).

Other things:
- long_query_time internal variable renamed to log_slow_query_time.
- More descriptive information for "log_slow_query_time".

Author: Sergei Golubchik
Author Date: 2024-03-16 22:47:59 UTC

fixup MDEV-9101 Limit size of created disk temporary files and tables

Author: Sergey Petrunia
Author Date: 2024-03-15 18:05:05 UTC

MDEV-28621: What if we didn't eliminate subqueries?

We would just remove the items.

So far I've hit only one issue - testcase for MDEV-28617.
The issue is that if you don't have an Item pointing to the
subquery, fix_fields() won't be called at all.

However, the tables are still in the table list. And code like

INSERT INTO t1 SELECT FROM ... t1 ...

will search for the second reference to t1, and if it is in
a merged-derived will try to switch it into materialized.
However this will fail as the derived table's SELECT_LEX_UNIT is
not fixed.

Author: Sergey Petrunia
Author Date: 2024-03-15 18:05:05 UTC

MDEV-28621: What if we didn't eliminate subqueries?

We would just remove the items.

So far I've hit only one issue - testcase for MDEV-28617.
The issue is that if you don't have an Item pointing to the
subquery, fix_fields() won't be called at all.

However, the tables are still in the table list. And code like

INSERT INTO t1 SELECT FROM ... t1 ...

will search for the second reference to t1, and if it is in
a merged-derived will try to switch it into materialized.
However this will fail as the derived table's SELECT_LEX_UNIT is
not fixed.

Author: Oleg Smirnov
Author Date: 2024-03-10 10:03:04 UTC

WIP: Parser for optimizer hints

Author: Rucha Deodhar
Author Date: 2023-11-10 10:31:15 UTC

MDEV-32726: Fix failing test fir freebsd for json

Json test about max statement time fails with freebsd because on some
architectures the test might execute faster and the statement may not fail.

To simulate failure regardless of architecture, introduce a wait of seconds
longer than the max_statement_time.

Author: Rucha Deodhar
Author Date: 2024-03-15 11:17:21 UTC

MDEV-17943: Unfortunate error code/message upon SET DEFAULT ROLE FOR

Analysis: Wrong error message.
Fix: Change the error message.

Author: Yuchen Pei
Author Date: 2024-03-15 06:13:21 UTC

MDEV-33220 Fix -wmaybe-uninitialized warnings

Author: Yuchen Pei
Author Date: 2024-03-15 03:46:26 UTC

MDEV-33679 [poc] Construct queries for Item_equal in spider group by handler

Author: Sergei Golubchik
Author Date: 2024-03-14 18:52:08 UTC

Merge branch '10.5' into 10.6

Author: Andrei
Author Date: 2024-03-14 16:49:52 UTC

Test fixes: rpl_parallel_xa_same_xid needs greater slave_transaction_retries

The reason is that in the duplicate xid case
  XAC_k(xid) -> XAP_k+i(xid)
dependency is handled with XAP's retry when the xid still in XAC's
possession.
Todo: implement a sleep interval in between of two consequent retry cycles.

Author: Nikita Malyavin
Author Date: 2024-03-13 20:13:22 UTC

adopt library

Author: Marko Mäkelä
Author Date: 2024-03-14 13:45:26 UTC

squash! 5704eeb9df09b12c9da3736a7e470c93aa7f0b87

This reverts the incorrect "bool combined" part of
22e7eca3501310f800a9dbc221359472a7beca39 in buf_buddy_shrink().

buf_buddy_free_low(): Do not care about buffer pool shrinking.
This will be handled by buf_buddy_shrink() and
buf_buddy_condense_free().

buf_pool_t::contains_zip(): Added a parameter for the
number of least significant pointer bits to disregard,
so that we can find any pointers to within a block
that is supposed to be free.

buf_buddy_alloc_zip(): Assert !buf_pool.contains_zip()
when we are allocating from the binary buddy system.
Previously we were asserting this on multiple recursion levels.

buf_buddy_block_free(), buf_buddy_free_low():
Assert !buf_pool.contains_zip().

buf_buddy_alloc_from(): Remove the redundant parameter j.

buf_pool_t::resize(): Always zero out b->page.zip.data.
Failure to do so would cause crashes or corruption in
the test innodb.innodb_buffer_pool_resize due to
duplicated allocation in the buddy system.

buf_pool_t::LRU_shrink(): Check if buffer pool shrinking needs
to process a buffer page.

buf_flush_LRU_list_batch(): Add the parameter shrinking.
If we are shrinking, invoke buf_pool_t::LRU_shrink() to see
if we must keep going.

buf_do_LRU_batch(): Skip buf_free_from_unzip_LRU_list_batch()
if we are shrinking the buffer pool. In that case, we want
to minimize the page relocations and just finish as quickly
as possible.

Author: Sergey Petrunia
Author Date: 2024-03-14 11:22:05 UTC

MDEV-18478: part 3: late code cleanups, no functional changes.

Author: Yuchen Pei
Author Date: 2024-03-14 06:34:51 UTC

MDEV-32609 [poc] Implement Item_window_func::change_context_processor()

Author: Julius Goryavsky
Author Date: 2024-03-13 12:52:36 UTC

galera: wsrep-lib submodule update

Author: =?utf-8?q?Jan_Lindstr=C3=B6m?=
Author Date: 2023-11-01 09:07:16 UTC

MDEV-25089 : Assertion `error.len > 0' failed in wsrep_status_t galera::ReplicatorSMM::handle_apply_error(galera::TrxHandleSlave&, const wsrep_buf_t&, const string&)

Problem is that Galera starts TOI (total order isolation) i.e.
it sends query to all nodes. Later it is discovered that
used engine or other feature is not supported by Galera.
Because TOI is executed parallelly in all nodes appliers
could execute given TOI and ignore the error and
start inconsistency voting causing node to leave from
cluster or we might have a crash as reported.

For example SEQUENCE engine does not support GEOMETRY data
type causing either inconsistency between nodes (because
some errors are ignored on applier) or crash.

Fixed my adding new function wsrep_check_support to check
can Galera support provided CREATE TABLE/SEQUENCE before TOI is
started and if not clear error message is provided to
the user.

Currently, not supported cases:

* CREATE TABLE ... AS SELECT when streaming replication is used
* CREATE TABLE ... WITH SYSTEM VERSIONING AS SELECT
* CREATE TABLE ... ENGINE=SEQUENCE
* CREATE SEQUENCE ... ENGINE!=InnoDB
* ALTER TABLE t ... ENGINE!=InnoDB where table t is SEQUENCE

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Kristian Nielsen
Author Date: 2024-03-08 21:18:44 UTC

MDEV-33475: --gtid-ignore-duplicate can double-apply event in case of parallel replication retry

When rolling back and retrying a transaction in parallel replication, don't
release the domain ownership (for --gtid-ignore-duplicates) as part of the
rollback. Otherwise another master connection could grab the ownership and
double-apply the transaction in parallel with the retry.

Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Oleksandr "Sanja" Byelkin
Author Date: 2024-03-13 11:38:40 UTC

CC 3.4

Author: Monty
Author Date: 2024-02-20 13:03:31 UTC

Fixed typo in mariadb-install-db

Author: Sergey Petrunia
Author Date: 2024-03-13 08:53:31 UTC

MDEV-12404: Add assertions about Index Condition Pushdown use

Add assertions about limitations one has when using Index Condition
Pushdown:
- add handler::assert_icp_limitations()
- call this function from functions that may attempt violations.

Verified that assert_icp_limitations() as well as calls to it are
compiled away in release build.

Author: Yuchen Pei
Author Date: 2024-03-13 06:47:00 UTC

MDEV-29363 avoid adding items with subqueries to item_equal

The optimizer constructs multiple equalities, and swap them in during
equality propagation. In doing so, it may create multiple pointers to
the same item. When the item contain subqueries, bad things can
happen. For example, in pushdown of HAVING to WHERE, if the HAVING
conditions contain multiple equalities with a const subquery, the
subquery will not be excluded from pushdown into WHERE. And if there
are multiple pointers to the same subquery, it may unfix the subquery
and destroy and rebuild its join, which could cause further
use-after-free if there are external pointers to items in the
join (e.g. tmp table fields as in the main case in MDEV-29363).

This patch also fixes cases in MDEV-32424, MDEV-32608, MDEV-32539.