lp:maria

Author: Yuchen Pei
Author Date: 2023-10-19 00:01:29 UTC

[fixup] Spider fixup of the merge

Fixing the following
- Some commented out code were added in the merge
- Some changes from the following was reverted

b5d317197c6 MDEV-29676 refactored and documented spider_get_share() and friends

Author: Brandon Nesterenko
Author Date: 2023-10-18 14:32:28 UTC

MDEV-31949: rpl_xa_prepare_gtid_fail deterministic paths

happy_xac is where the XA COMMIT completes before noticing the
error signalled by the prior XAP

sad_xac is where the XA COMMIT notices the error signalled by
the prior XAC and rolls back, leaving a dangling XAP.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-10-18 13:27:51 UTC

MDEV-32308 Server crash on cleanup of non-fully-constructed-due-to-an-error CTE

Check and do not try to access With_element::rec_result on cleanup
if it is not assigned.

Author: Brandon Nesterenko
Author Date: 2023-10-17 21:48:03 UTC

MDEV-31949: Fix rpl_xa_prepare_gtid_fail

rpl.rpl_xa_prepare_gtid_fail would sporadically fail, as an XA COMMIT,
running concurrently with a failed prepare, could sometimes complete
succesfully, or be rolled back due to failure of prior commit. The
test expected the completion case, but if it failed, the xa transaction
would be left in a prepared state with a lock on its table. This then
created a lock time-out during test cleanup, as it tries to drop that
table.

The fix is to extend the test to check if the transaction is still
prepared, and silently commit it if so, thus releasing the locks.
The non-determinism is fine (i.e. DEBUG_SYNC isn't needed to force
one path), as the verification performed by the test has already
completed. This is just for cleanup.

Author: Brandon Nesterenko
Author Date: 2023-10-17 21:48:03 UTC

MDEV-31949: Fix rpl_xa_prepare_gtid_fail

rpl.rpl_xa_prepare_gtid_fail would sporadically fail, as an XA COMMIT,
running concurrently with a failed prepare, could sometimes complete
succesfully, or be rolled back due to failure of prior commit. The
test expected the completion case, but if it failed, the xa transaction
would be left in a prepared state with a lock on its table. This then
created a lock time-out during test cleanup, as it tries to drop that
table.

The fix is to extend the test to check if the transaction is still
prepared, and silently commit it if so, thus releasing the locks.
The non-determinism is fine (i.e. DEBUG_SYNC isn't needed to force
one path), as the verification performed by the test has already
completed. This is just for cleanup.

Author: Yuchen Pei
Author Date: 2023-10-18 07:18:24 UTC

MDEV-32486 Fix duplicate spider_bulk_malloc ID

The id 38 is already used elsewhere, we use 44 instead, which hasn't
been used

Author: Yuchen Pei
Author Date: 2023-10-18 07:18:24 UTC

MDEV-32486 Fix duplicate spider_bulk_malloc ID

The id 38 is already used elsewhere, we use 44 instead, which hasn't
been used

Author: Yuchen Pei
Author Date: 2023-10-17 04:02:12 UTC

MDEV-32485 Fix Spider upgrade failure caused by duplication in mysql.func

Author: Igor Babaev
Author Date: 2023-10-18 03:52:34 UTC

MDEV-32351 Significant slowdown for query with many outer joins

This is a preliminary patch. To be used just for testing.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-09-04 11:04:06 UTC

MDEV-29167 new db-level SHOW CREATE ROUTINE privilege

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-16 17:43:18 UTC

ibuf_delete_recs() doesn't remove all entries of page id

ibuf_delete_recs() should use the counter field for searching
the tuple when we're reopening the cursor.

Author: VladislavVaintroub
Author Date: 2023-10-17 09:58:20 UTC

Innodb cleanup - log_t::append_prepare

- make the function non-static
- pass lsn to it, and do log_write_up_to to wait for this lsn, rather than
  always wait for the current one.

Author: Yuchen Pei
Author Date: 2023-10-17 07:14:05 UTC

Merge branch '10.10' into 10.11

Author: Yuchen Pei
Author Date: 2023-10-17 04:02:12 UTC

MDEV-32485 Fix Spider upgrade failure caused by duplication in mysql.func

Author: Yuchen Pei
Author Date: 2023-10-11 05:40:30 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Christian Hesse
Author Date: 2023-08-09 00:18:36 UTC

MDEV-19210: do not run pre and post scripts as root

Now that we do not pollute systemd's environment but write private
environment files running these as root is not longer required. So
let's drop `PermissionsStartOnly=true`.

Debian adds extra `ExecStartPre=` and `ExecStartPost=`, though.
Use special executable prefix for full privileges there. (See
systemd.service(5) for details.)

Author: Tuukka Pasanen
Author Date: 2023-10-10 06:45:48 UTC

MDEV-30953: Add package for Galera server for RPM

Add new package mariadb-server-galera to RPM build
recude mariadb-server package dependencies.
Galera cluster scripts and libraries are currently
in package mariadb-server. Move Galera cluster
scripts and libraries to new package mariadb-server-galera
as most of the people does not need them and
it allows to slim down mariadb-server package

Author: Elena Stepanova
Author Date: 2023-10-14 16:58:51 UTC

The list is not relevant anymore, bugs are fixed

Author: Sergey Petrunia
Author Date: 2023-10-13 09:17:25 UTC

MDEV-32320: Server crashes at TABLE::add_tmp_key

The code inside Item_subselect::fix_fields() could fail to check
that left expression had an Item_row, like this:

  (('x', 1.0) ,1) IN (SELECT 'x', 1.23 FROM ... UNION ...)

In order to hit the failure, the first SELECT of the subquery had
to be a degenerate no-tables select. In this case, execution will
not enter into Item_in_subselect::create_row_in_to_exists_cond()
and will not check if left_expr is composed of scalars.

But the subquery is a UNION so as a whole it is not degenerate.
We try to create an expression cache for the subquery.
We create a temp.table from left_expr columns. No field is created
for the Item_row. Then, we crash when trying to add an index over a
non-existent field.

Fixed by moving the left_expr cardinality check to a point in
check_and_do_in_subquery_rewrites() which gets executed for all
cases.
It's better to make the check early so we don't have to care about
subquery rewrite code hitting Item_row in left_expr.

Author: Yuchen Pei
Author Date: 2023-10-13 06:16:57 UTC

MDEV-29963 MDEV-31357 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, or simply TRUNCATEd, when executing
LOCK TABLES again, the lock lists would be queried again in
store_lock(), which could cause access to freed space associated with
the dropped table.

Author: Yuchen Pei
Author Date: 2023-10-13 06:16:57 UTC

MDEV-29963 MDEV-31357 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, or simply TRUNCATEd, when executing
LOCK TABLES again, the lock lists would be queried again in
store_lock(), which could cause access to freed space associated with
the dropped table.

Author: Yuchen Pei
Author Date: 2023-09-14 05:19:50 UTC

MDEV-29502 Fix some issues with spider direct aggregate

The direct aggregate mechanism sems to be only intended to work when
otherwise a full table scan query will be executed from the spider
node and the aggregation done at the spider node too. Typically this
happens in sub_select(). In the test spider.direct_aggregate_part
direct aggregate allows to send COUNT statements directly to the data
nodes and adds up the results at the spider node, instead of iterating
over the rows one by one at the spider node.

By contrast, the group by handler (GBH) typically sends aggregated
queries directly to data nodes, in which case DA does not improve the
situation here.

That is why we should fix it by disabling DA when GBH is used.

There are other reasons supporting this change. First, the creation of
GBH results in a call to change_to_use_tmp_fields() (as opposed to
setup_copy_fields()) which causes the spider DA function
spider_db_fetch_for_item_sum_funcs() to work on wrong items. Second,
the spider DA function only calls direct_add() on the items, and the
follow-up add() needs to be called by the sql layer code. In
do_select(), after executing the query with the GBH, it seems that the
required add() would not necessarily be called.

Disabling DA when GBH is used does fix the bug. There are a few
other things included in this commit to improve the situation with
spider DA:

1. Add a session variable that allows user to disable DA completely,
this will help as a temporary measure if/when further bugs with DA
emerge.

2. Move the increment of direct_aggregate_count to the spider DA
function. Currently this is done in rather bizarre and random
locations.

3. Fix the spider_db_mbase_row creation so that the last of its row
field (sentinel) is NULL. The code is already doing a null check, but
somehow the sentinel field is on an invalid address, causing the
segfaults. With a correct implementation of the row creation, we can
avoid such segfaults.

Author: Yuchen Pei
Author Date: 2023-07-05 07:55:30 UTC

MDEV-31117 Fix spider connection info parsing

Spider connection string is a comma-separated parameter definitions,
where each definition is of the form "<param_title> <param_value>",
where <param_value> is quote delimited on both ends, with backslashes
acting as an escaping prefix.

Despite the simple syntax, the existing spider connection string
parser was poorly-written, complex, hard to reason and error-prone,
causing issues like the one described in MDEV-31117. For example it
treated param title the same way as param value when assigning, and
have nonsensical fields like delim_title_len and delim_title.

Thus as part of the bugfix, we clean up the spider comment connection
string parsing, including:

- Factoring out some code from the parsing function
- Simplify the struct `st_spider_param_string_parse`
- And any necessary changes caused by the above changes

Author: Rucha Deodhar
Author Date: 2023-10-12 11:41:07 UTC

MDEV-31684: More tests

Author: Sergey Petrunia
Author Date: 2023-10-12 11:19:08 UTC

MDEV-32324: Server crashes inside filesort at my_decimal::to_binary

A subquery in form "(SELECT not_null_value LIMIT 1 OFFSET 1)" will
produce no rows which will translate into scalar SQL NULL value.

The code Item_singlerow_subselect::fix_length_and_dec() failed to
take LIMIT clause into account and set item_subselect->maybe_null=0.

This caused a crash in filesort() code when it got a NULL value for a
not nullable item.

Author: Yuchen Pei
Author Date: 2023-10-12 03:39:14 UTC

MDEV-29963 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, when executing LOCK TABLES again,
the lock lists would be queried again in store_lock(), which could
cause access to freed stuff.

This also fixes MDEV-31357

Author: Yuchen Pei
Author Date: 2023-10-12 00:52:12 UTC

MDEV-27902 MDEV-29962 Spider check trx and get conn before locking

In 10.4 the testcase for MDEV-27902 displays symptoms of MDEV-29962,
which is why we do the check trx and get conn before locking,
different from other MDEV-27902 fixes for higher versions.

Author: Sergey Petrunia
Author Date: 2023-10-11 16:02:25 UTC

MDEV-32301: Server crashes at Arg_comparator::compare_row

In Item_bool_rowready_func2::build_clone(): if we're setting
  clone->cmp.comparators=0
also set
  const_item_cache=0
as the Item is currently in a state where one cannot compute it.

Author: Dmitry Shulga
Author Date: 2023-10-11 07:47:05 UTC

MDEV-32123: require_secure_transport doesn't allow TCP connections

In case the option require_secure_transport is on the user can't
establish a secure ssl connection over TCP protocol. Inability to set up
a ssl session over TCP was caused by the fact that a type of client's
connection was checked before ssl handshake performed (ssl handshake
happens at the function acl_authenticate()). At that moment vio type has
the value VIO_TYPE_TCPIP for client connection that uses TCP transport.
In result, checking for allowable vio type for fails despite the fact
that SSL session being established. To fix the issue move checking of
vio type for allowable values inside the function
  parse_client_handshake_packet()
right after client's capabilities discovered that SSL is not requested
by the client.

Author: Lena Startseva
Author Date: 2023-08-30 05:22:07 UTC

MDEV-31933: Make working view-protocol + ps-protocol (running two protocols together)

Now tests from suite main can be run with view-protocol + ps-protocol (together)

Author: Yuchen Pei
Author Date: 2023-10-11 05:40:30 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Yuchen Pei
Author Date: 2023-10-11 05:29:51 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Yuchen Pei
Author Date: 2023-10-11 05:29:51 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Yuchen Pei
Author Date: 2023-10-11 05:29:51 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Yuchen Pei
Author Date: 2023-10-11 04:04:14 UTC

[experiment] is have_innodb.inc the problem?

Author: Yuchen Pei
Author Date: 2023-09-25 05:46:59 UTC

MDEV-32238 Adding a switch to disable the spider group by handler

The system variable spider_disable_group_by_handler, if on, will
disable the spider group by handler (gbh), and such disablement serves
as workaround to bugs caused by gbh, labelled with spider-gbh on jira,
including MDEV-26247, MDEV-28998, MDEV-29163, MDEV-30392, MDEV-31645.
Tests for these tickets are added accordingly with the workaround in
place.

Author: Yuchen Pei
Author Date: 2023-09-25 05:46:59 UTC

MDEV-32238 Adding a switch to disable the spider group by handler

The system variable spider_disable_group_by_handler, if on, will
disable the spider group by handler (gbh), and such disablement serves
as workaround to bugs caused by gbh, labelled with spider-gbh on jira,
including MDEV-26247, MDEV-28998, MDEV-29163, MDEV-30392, MDEV-31645.
Tests for these tickets are added accordingly with the workaround in
place.

Author: Yuchen Pei
Author Date: 2023-09-25 05:46:59 UTC

MDEV-32238 Adding a switch to disable the spider group by handler

The system variable spider_disable_group_by_handler, if on, will
disable the spider group by handler (gbh), and such disablement serves
as workaround to bugs caused by gbh, labelled with spider-gbh on jira,
including MDEV-26247, MDEV-28998, MDEV-29163, MDEV-30392, MDEV-31645.
Tests for these tickets are added accordingly with the workaround in
place.

Author: Yuchen Pei
Author Date: 2023-10-11 01:06:44 UTC

MDEV-29963 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, when executing LOCK TABLES again,
the lock lists would be queried again in store_lock(), which could
cause access to freed stuff.

Author: Brandon Nesterenko
Author Date: 2023-10-09 17:52:47 UTC

MDEV-31757: Compressed binary log with two-phase alter events cannot be replayed

WIP

Extended mariadb-binlog to be able to handle Query_compressed event,
yet on event replay, there is replica specific logic that causes the
server to segfault because of an uninitialized data structure.

Put in a quick prototype fix to first check if the data structure
exists before accessing its inner members, but I'm not sold on it.

Author: =?utf-8?q?Jan_Lindstr=C3=B6m?=
Author Date: 2023-04-12 08:26:39 UTC

MDEV-31062 : Reduce number of wsrep calls to server code from InnoDB

Thread executing wsrep transaction can't change during transaction
execution. Similarly, thread executing high priority brute force
(BF) transaction does not change during transaction execution.
Therefore, in both cases there is no need to call server
code after transaction has initialized.

InnoDB already stores information is this wsrep transaction to
trx_t::wsrep and this is checked using trx->is_wsrep() function.
Because, brute force transaction is always a wsrep transaction
we can extend trx_t::wsrep variable so that value

0 == not wsrep transaction (and not BF)
1 == normal wsrep transaction
2 == high priority BF wsrep transaction
4 == high priority BF transaction is performing unique secondary index scan

These values can be set by calling server code on innobase_trx_init().
After that we can use trx_t::is_wsrep() and new function
introduced in this patch trx_t::is_wsrep_BF(). Unique
secondary index scan is determined later but it implies BF,
so this patch removes unnecessary variable trx_t::wsrep_UK_scan.
Patch introduces new functions trx_t::is_wsrep_UK_scan(),
wsrep_begin_UK_scan() and wsrep_end_UK_scan() to handle
unique secondary index scan.

This change reduces number of calls to server code from InnoDB and
reduces code bloat on performance critical stages like acquiring
record locks. Furthermore, it simplifies code on several locations.

This patch also removes unnecessary trx_t::wsrep_event that
was always set to NULL and not used.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2023-10-06 21:08:20 UTC

MDEV-27247: Change SQL_BEFORE_GTIDS to go before the GTID in each domain

Author: midenok
Author Date: 2023-10-06 12:37:01 UTC

MDEV-25547 Auto-create: Undetected deadlock lasts longer than the configured timeout

While the principle of fixing the stated problem is simple: remove
forcing result=false after ER_LOCK_WAIT_TIMEOUT, there is much serious
problem. While the waiting threads wait for the creating thread to
complete they run the loop continuously, acquiring and releasing
MDL_SHARED_WRITE. If the creating thread is blocked for a long time
waiting for MDL_EXCLUSIVE and the lock wait timeout is usually very
long as MDEV-25547 states, this long time the waiting threads will hog
up the CPU time.

The patch introduces vers_create_signal based on condition variable
which holds down the waiting threads until creating thread is busy
adding the partition. The problem is, there is no ready common storage
for vers_create_signal as the share may be freed once we released
MDL_SHARED_WRITE.

To solve this we allocate vers_auto_create_signal from heap, store it
into TABLE_SHARE temporarily, only for passing it to the waiting
threads. Then we re-lay it into Open_table_context to be able to use
at time when TABLE_SHARE is already can be freed.

All the waiting threads after they release MDL_SHARED_WRITE are
blocked until creating thread signals they can continue. But there is
another problem: how to free vers_auto_create_signal. If done by
waiting threads this leads to race condition as waiting happens in
free from locks zone. Like if the waiting thread decides he is the
last one and frees, this moment another waiting thread wants to
initiate another wait.

So the freeing of vers_auto_create_signal is only possible from the
creating thread as it is under MDL_EXCLUSIVE and this guarantees all
the waiting threads already in free from locks zone and all of them
stated they want to wait. So the creating thread can know when the
last waiting ended the wait. This is implemented by waiters count and
by feedback signal.

The waiting threads tell they are going_wait() under MDL_SHARED_WRITE,
creating thread didn't acquire MDL_EXCLUSIVE yet. This increments
waiters counter. When creating thread acquires MDL_EXCLUSIVE all the
waiting threads are on its way to wait() and the waiters counter will
not increase anymore. After creating thread is done its main logic it
broadcasts to the waiters so they start to exit waiting and decrement
waiters counter. When the waiters counter is zero it is the time for
the creating thread to free vers_auto_create_signal.

The logic of waiting is based on mysql_cond_timedwait(), so we can not
delay over lock_wait_timeout limit. And since there were more delays
(acquiring MDL f.ex.) we use only the remainder of lock_wait_timeout
duration since query start time.

Author: Yuchen Pei
Author Date: 2023-10-06 04:44:47 UTC

MDEV-28683 Spider: create conn on demand when direct delete

Same issue as MDEV-31996

Author: Yuchen Pei
Author Date: 2023-10-06 03:54:40 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: midenok
Author Date: 2023-10-05 12:24:49 UTC

MDEV-31014 Database privileges are insufficient for CONVERT TABLE TO PARTITION

check_access() updates ACL of first TABLE_LIST (t1), but not second
(tp). After it is done we copy t1 ACL to tp.

Author: Yuchen Pei
Author Date: 2023-10-05 07:29:45 UTC

MDEV-32046 [DEMO] Do not ignore error in spider_get_share()

Author: Yuchen Pei
Author Date: 2023-07-20 02:10:26 UTC

MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs

Fix spider init bugs (MDEV-22979, MDEV-27233, MDEV-28218) while
preventing regression on old ones (MDEV-30370, MDEV-29904)

Two things are changed:

First, Spider initialisation is made fully synchronous, i.e. it no
longer happens in a background thread. Adapted from the original fix
by nayuta for MDEV-27233. This change itself would cause failure when
spider is initialised early, by plugin-load-add, due to dependency on
Aria and udf function creation, which are fixed in the second and
third parts below. Requires SQL Service, thus porting earlier versions
requires MDEV-27595

Second, if spider is initialised before udf_init(), create udf by
inserting into `mysql.func`, otherwise do it by `CREATE FUNCTION` as
usual. This change may be generalised in MDEV-31401.

Also factor out some clean-up queries from deinit_spider.inc for use
of spider init tests.

A minor caveat is that early spider initialisation will fail if the
server is bootstrapped for the first time, due to missing `mysql`
database which needs to be created by the bootstrap script.

Author: Yuchen Pei
Author Date: 2023-07-20 02:10:26 UTC

MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs

Fix spider init bugs (MDEV-22979, MDEV-27233, MDEV-28218) while
preventing regression on old ones (MDEV-30370, MDEV-29904)

Two things are changed:

First, Spider initialisation is made fully synchronous, i.e. it no
longer happens in a background thread. Adapted from the original fix
by nayuta for MDEV-27233. This change itself would cause failure when
spider is initialised early, by plugin-load-add, due to dependency on
Aria and udf function creation, which are fixed in the second and
third parts below. Requires SQL Service, thus porting earlier versions
requires MDEV-27595

Second, if spider is initialised before udf_init(), create udf by
inserting into `mysql.func`, otherwise do it by `CREATE FUNCTION` as
usual. This change may be generalised in MDEV-31401.

Also factor out some clean-up queries from deinit_spider.inc for use
of spider init tests.

A minor caveat is that early spider initialisation will fail if the
server is bootstrapped for the first time, due to missing `mysql`
database which needs to be created by the bootstrap script.

Author: Yuchen Pei
Author Date: 2023-10-05 02:18:10 UTC

MDEV-26247 Re-implement spider gbh query rewrite of tables

Spider GBH's query rewrite of table joins is overly complex and
error-prone. We replace it with something closer to what
dbug_print() (more specifically, print_join()) does, but catered to
spider.

More specifically, we replace the body of
spider_db_mbase_util::append_from_and_tables() with a call to
spider_db_mbase_util::append_join(), and remove downstream append_X
functions.

We make it handle const tables by rewriting them as (select 1). This
fixes the main issue in MDEV-26247.

We also ban semijoin from spider gbh, which fixes MDEV-31645 and
MDEV-30392, as semi-join is an "internal" join, and "semi join" does
not parse, and it is different from "join" in that it deduplicates the
right hand side

Not all queries passed to a group by handler are valid (MDEV-32273),
for example, a join on expr may refer outer fields not in the current
context. We detect this during the handler creation when walking the
join. See also gbh_outer_fields_in_join.test.

It also skips eliminated tables, which fixes MDEV-26193.

Author: Gagan Goel
Author Date: 2023-10-04 18:04:04 UTC

Update ColumnStore submodule to version 23.10.0-1.

Author: Gagan Goel
Author Date: 2023-10-04 17:51:35 UTC

Update ColumnStore submodule to version 23.10.0-1.

Author: Brandon Nesterenko
Author Date: 2023-10-04 17:08:44 UTC

MDEV-32184: Timeout test fix

Another WIP commit

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-04 14:45:32 UTC

MDEV-32337 Assertion `pos < table->n_def' failed in dict_table_get_nth_col

While checking for altered column in foreign key constraints,
InnoDB fails to ignore virtual columns. This issue caused
by commit 5f09b53bdb4e973e7c7ec2c53a24c98321223f98(MDEV-31086).

Author: Vlad Lesin
Author Date: 2023-10-03 17:07:39 UTC

MDEV-30658 lock_row_lock_current_waits counter in information_schema.innodb_metrics may become negative

MONITOR_OVLD_ROW_LOCK_CURRENT_WAIT monitor should has
MONITOR_DISPLAY_CURRENT flag set in its definition, as it shows the
current state and does not accumulate anything.

Author: Vlad Lesin
Author Date: 2023-10-03 17:07:39 UTC

MDEV-30658 lock_row_lock_current_waits counter in information_schema.innodb_metrics may become negative

MONITOR_OVLD_ROW_LOCK_CURRENT_WAIT monitor should has
MONITOR_DISPLAY_CURRENT flag set in its definition, as it shows the
current state and does not accumulate anything.

Author: Alexey Botchkov
Author Date: 2023-10-04 12:17:55 UTC

comment added.

Author: VladislavVaintroub
Author Date: 2023-09-27 10:54:28 UTC

MDEV-32252 addendum - refactor CPackWixConfig.cmake

Refactor CPackWixConfig.cmake so that it complains about unknown CPack
components.

This would prevent missing components,or discrepancy between MSI and ZIP
content, whenever new components are added or renamed.

Also, remove some old stuff, and use macros add_component_group() and
add_component() to reduce the verbosity.

Author: Sergey Petrunia
Author Date: 2023-10-03 12:40:42 UTC

MDEV-32335: main.partition_innodb2 fails in BB with different #rows

use --replace_column for rows, as the numbers are not important.

Author: Yuchen Pei
Author Date: 2023-10-03 06:07:05 UTC

MDEV-26247 [demo] optimized_away seems to imply const_table

No assert failure with when running test over the main suite

Author: Brandon Nesterenko
Author Date: 2022-07-18 16:12:57 UTC

MDEV-19415 Regression

This patch reproduces MDEV-19415.

Pushing this old commit so it isn't lost on my laptop.

Author: Oleg Smirnov
Author Date: 2023-10-02 05:12:15 UTC

MDEV-28506 Possible item replacement during fix_fields() is not taken into account when pushing conditions from HAVING to WHERE

1. During the 5th stage of pushdown_from_having_into_where() there is
a sequence of cleanup/fix_fields calls for items involved. fix_fields()
may sometimes replace the initial item, which is done, for example, for
Item_func_not. This commits adds the correct handling for such cases.

2. Sometimes fix_fields() calls are made with the 2nd parameter
(Item **ref) set to NULL. This may lead to access violation when the item
is of class Item_func_not or Item_func_sp. This commits add a proper check
before dereferencing **ref.

Author: Sergei Golubchik
Author Date: 2023-09-20 16:28:56 UTC

MDEV-32104 add removed command line options back as noops

Author: Alexander Barkov
Author Date: 2023-09-25 17:48:01 UTC

MDEV-12252 ROW data type for stored function return values

Adding support for the ROW data type in the stored function RETURNS clause:

- explicit ROW(..members...) for both sql_mode=DEFAULT and sql_mode=ORACLE

  CREATE FUNCTION f1() RETURNS ROW(a INT, b VARCHAR(32)) ...

- anchored "ROW TYPE OF [db1.]table1" declarations for sql_mode=DEFAULT

  CREATE FUNCTION f1() RETURNS ROW TYPE OF test.t1 ...

- anchored "[db1.]table1%ROWTYPE" declarations for sql_mode=ORACLE

  CREATE FUNCTION f1() RETURN test.t1%ROWTYPE ...

Adding support for anchored scalar data types in RETURNS clause:

- "TYPE OF [db1.]table1.column1" for sql_mode=DEFAULT

  CREATE FUNCTION f1() RETURNS TYPE OF test.t1.column1;

- "[db1.]table1.column1" for sql_mode=ORACLE

  CREATE FUNCTION f1() RETURN test.t1.column1%TYPE;

Details:

- Adding a new sql_mode_t parameter to
    sp_head::create()
    sp_head::sp_head()
    sp_package::create()
    sp_package::sp_package()
  to guarantee early initialization of sp_head::m_sql_mode.
  Before this change, this member was not initialized at all during
  CREATE FUNCTION/PROCEDURE/PACKAGE statements, and was not used.
  Now it needs to be initialized to write properly the
  mysql.proc.returns column, according to the create time sql_mode.

- Moving the code from the loop body in sp_rcontext::init_var_items()
  into a separate member Spvar_definition::make_item_field_row(),
  to reuse this code for variable/parameter definitions
  and for the RETURNS clause.

- Changing the data type of sp_head::m_return_field_def
  from Column_definition to Spvar_definition.
  So now it supports not only SQL column field types,
  but also explicit ROW and anchored ROW data types.

- Cleanup: adding the const qualifier to Spvar_definition::m_column_type_ref.
  Fixing the following methods accordingly:
  * Qualified_column_ident::resolve_type_ref()
  * LEX::sp_variable_declarations_column_type_finalize()

- Modifying sql_yacc.yy to support the new grammar.
  Adding new helper methods:
    * sf_return_fill_definition_row()
    * sf_return_fill_definition_rowtype_of()
    * sf_return_fill_definition_type_of()

- Fixing tests in:
  * Virtual_tmp_table::setup_field_pointers() in sql_select.cc
  * Send_field::normalize() in field.h
  to prevent calling Type_handler_row::field_type(),
  which is implemented a DBUG_ASSERT(0).
  Before this patch the affected methods were called only
  for scalar data types. Now ROW is also possible.

- Adding a new virtual method Field::cols()

- Overriding methods:
   Item_func_sp::cols()
   Item_func_sp::element_index()
   Item_func_sp::check_cols()
   Item_func_sp::bring_value()
  to support the ROW data type.

- Extending sp_returns_type() to support
  * explicit ROW and anchored ROW data types
  * anchored scalar data types

- Overriding Field_row::sql_type() to print
  the data type of an explicit ROW.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-09-28 13:01:07 UTC

MDEV-32017 Auto-increment no longer works for explicit FTS_DOC_ID

- InnoDB increments the next document id whenever InnoDB does
commit the fulltext sync operation. InnoDB should avoid the
sync commit operation when there is nothing in fulltext cache.

Author: Yuchen Pei
Author Date: 2023-09-28 06:16:06 UTC

MDEV-28998 remove a known reference to a SPIDER_CONN when it is freed

Author: Brandon Nesterenko
Author Date: 2023-09-27 20:40:34 UTC

MDEV-32265: seconds_behind_master is inaccurate for Delayed replication

This patch fixes this by separating the negation of
sql_thread_caught_up out of the timestamp-dependent block, so it is
called any time an idle parallel slave queues an event to a worker.

Reviewed By:
============
<TODO>

Author: Vlad Lesin
Author Date: 2023-09-12 19:17:13 UTC

MDEV-25163 Rowid filter does not process storage engine error correctly.

The fix is to return 3-state value from Range_rowid_filter::build()
call:
1. The filter was built successfully;
2. The filter was not built, but the error was not fatal, i.e. there is
no need to rollback transaction. For example, if the size of container to
storevrow ids is not enough;
3. The filter was not built because of fatal error, for example,
deadlock or lock wait timeout from storage engine. In this case we
should stop query plan execution and roll back transaction.

Reviewed by: Sergey Petrunya

Author: Vlad Lesin
Author Date: 2023-09-12 19:17:13 UTC

MDEV-25163 Rowid filter does not process storage engine error correctly.

The fix is to return 3-state value from Range_rowid_filter::build()
call:
1. The filter was built successfully;
2. The filter was not built, but the error was not fatal, i.e. there is
no need to rollback transaction. For example, if the size of container to
storevrow ids is not enough;
3. The filter was not built because of fatal error, for example,
deadlock or lock wait timeout from storage engine. In this case we
should stop query plan execution and roll back transaction.

Reviewed by: Sergey Petrunya

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-09-27 08:28:44 UTC

MDEV-29180: Description of log_warnings incorrectly mentions "general log"

Author: Yuchen Pei
Author Date: 2023-09-27 03:08:34 UTC

MDEV-32254 Make NULL an illegal value for redirect_url

Author: Yuchen Pei
Author Date: 2023-09-27 03:08:34 UTC

MDEV-32254 Make NULL an illegal value for redirect_url

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-09-26 17:59:50 UTC

MDEV-29289 uncleaned LIMIT ROWS EXAMINED

Clean thd->lex->limit_rows_examined_cnt at the end of dispatch_command to avoid side effects

Author: Daniel Black
Author Date: 2023-07-25 00:58:45 UTC

MDEV-30520 SELinux/Apparmor incorrectly labeled mariadbd

From 10.4 -> 10.5 renamed a few MariaDB executables.

The default selinux profile expects mariadbd to be installed in
/usr/libexec/mariadbd however our packages install in in /usr/sbin.

Change non-existant named executables in the apparmor and selinux
profiles to use the new names like mariadb-backup, mariadb, mbstream
etc and use the right paths.

Author: =?utf-8?q?Jan_Lindstr=C3=B6m?=
Author Date: 2023-08-15 07:12:34 UTC

MDEV-24912: post-fix for test regression

Problem is that mysql.galera_slave_pos table is replicated,
thus it should be InnoDB to allow rolling back in case
of replay.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Yuchen Pei
Author Date: 2023-09-26 06:34:32 UTC

MDEV-26247 [wip] Make spider_create_gbh go through append_join too

After this, we can get rid of append_from_and_tables().

There are two tricky issues remaining.

1. By the time the query reaches the spider gbh, the optimizer may
have modified the query so that it is no longer valid. See the test
tmppp.test

2. Somehow this change also fails spider/bg.ha and spider.ha because
failing links are no longer detected with the link status updated to
NG, causing spider to use them even though they are unavailable.

Author: Monty
Author Date: 2023-09-20 12:46:55 UTC

MDEV-32203 Raise notes when an index cannot be used on data type mismatch

Raise notes if indexes cannot be used:
- in case of data type or collation mismatch (diferent error messages)
- in case if a table field was replaced to something else
  (e.g. Item_func_conv_charset) during a condition rewrite

Added option to write warnings and notes to the slow query log for
slow queries.

New variables added/changed:

- note_verbosity, with is a set of the following options:
  normal - All old notes
  unusable_keys - Print warnings about keys that cannot be used
                     for select, delete or update.
  explain - Print unusable_keys warnings for EXPLAIN querys

  The default is 'normal,explain'.

- log_slow_verbosity has a new option 'warnings'. If this is set
  then warnings and notes generated are printed in the slow query log
  (up to log_slow_max_warnings times per statement).

- log_slow_max_warnings - Max number of warnings written to
                            slow query log.

Other things:
- One can now use =ALL for any 'set' variable to set all options at once.
  For example using "note_verbosity=ALL" in a config file or
  "SET @@note_verbosity=ALL' in SQL.
- mysqldump will in the future use @@note_verbosity=""' instead of
  @sql_notes=0 to disable notes.
- Added "enum class Data_type_compatibility" and changing the return type
  of all Field::can_optimize*() methods from "bool" to this new data type.

Reviewer & Co-author: Alexander Barkov <bar@mariadb.com>
- The code that prints out the notes comes mainly from Alexander

Author: =?utf-8?q?Jan_Lindstr=C3=B6m?=
Author Date: 2023-04-12 08:26:39 UTC

MDEV-31062 : Reduce number of wsrep calls to server code from InnoDB

Thread executing wsrep transaction can't change during transaction
execution. Similarly, thread executing high priority brute force
(BF) transaction does not change during transaction execution.
Therefore, in both cases there is no need to call server
code after transaction has initialized.

InnoDB already stores information is this wsrep transaction to
trx_t::wsrep and this is checked using trx->is_wsrep() function.
Because, brute force transaction is always a wsrep transaction
we can extend trx_t::wsrep variable so that value

0 == not wsrep transaction (and not BF)
1 == normal wsrep transaction
2 == high priority BF wsrep transaction
4 == high priority BF transaction is performing unique secondary index scan

These values can be set by calling server code on innobase_trx_init().
After that we can use trx_t::is_wsrep() and new function
introduced in this patch trx_t::is_wsrep_BF(). Unique
secondary index scan is determined later but it implies BF,
so this patch removes unnecessary variable trx_t::wsrep_UK_scan.
Patch introduces new functions trx_t::is_wsrep_UK_scan(),
wsrep_begin_UK_scan() and wsrep_end_UK_scan() to handle
unique secondary index scan.

This change reduces number of calls to server code from InnoDB and
reduces code bloat on performance critical stages like acquiring
record locks. Furthermore, it simplifies code on several locations.

This patch also removes unnecessary trx_t::wsrep_event that
was always set to NULL and not used.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-09-20 14:15:24 UTC

MDEV-29989 binlog_do_db option breaks versioning table

Problem:
=========
During commit, server calls prepare_commit_versioned to
determine the transaction modified system-versioned data.
Due to binlog_do_db option, we disable the binlog for the
statement. But prepare_commit_versioned() is being
called only when binlog is enabled for the statement.

Fix:
===
prepare_commit_versioned() should happen irrespective
of binlog state. So if the server has any read-write operation
then we should call prepare_commit_versioned().

Author: Yuchen Pei
Author Date: 2023-09-25 06:55:59 UTC

MDEV-32234 Fix missing space in Spider variable descriptions

Author: Yuchen Pei
Author Date: 2023-09-25 01:19:11 UTC

MDEV-31936 Simplify deinit_spider.inc

Spider is part of the server, and there's no need to check the
version.

All spider plugins are uninstalled in clean_up_spider.inc

DROP SERVER IF EXISTS makes things easier

Author: Sergei Golubchik
Author Date: 2023-09-17 08:09:47 UTC

MDEV-3953 Add columns for ROWS_EXAMINED, ROWS_SENT, and ROWS_READ to I_S and processlist

Added ROWS_SENT to information_schema.processlist

This is to have the same information as Percona server (SENT_ROWS)

I also extended the display size for the columns in processlist to ensure
that most results will fit in the display size.

Author: VladislavVaintroub
Author Date: 2023-09-24 09:20:38 UTC

MDEV-25870 followup - some Windows ARM64 improvements

- optimize atomic store64/load64 implementation.
- allow CRC32 optimization. Do not allow pmull yet, as this fails like in
  https://stackoverflow.com/questions/54048837/how-to-perform-polynomial-multiplication-using-arm64

Author: Yuchen Pei
Author Date: 2023-09-22 03:11:12 UTC

MDEV-26247 [demo] address the first pass during gbh creation

Spider GBH does two passes of spider_db_print_item_type() for all
items in the query. The first pass during the GBH creation has
str==NULL, to construct a chain of fields pointing to table alias and
the correct associated ha_spider. The second pass during the
init_scan() walks the query again to translate to the query for remote
tables. See also MDEV-29447.

After re-implementing rewriting the FROM part of the query, we also
have to update the corresponding first pass part that processes FROM.

For the second pass to work, the two passes have to walk the queries
so that the items are in the exact same order. But the simple update
of the first pass in this patch seems to break that order: the test
spider.direct_right_left_join_nullable fails an assertion of the
order.

There are two ways to address this:

1. Just do one pass of walking the item. The complexity should be the
   same, up to a constant
2. Figure out why the order is wrong now, and fix it.

Of the two ideas, 1 is better, because it can significantly simplify
the code by getting rid of the first pass and the different behaviours
of the spider item printing functions depending on the nullness of
str, and it will also make it much easier to debug with ordering
issues, because we don't need to go back and forth between the two
passes, and it will make the code easier to maintain.

Author: Yuchen Pei
Author Date: 2023-09-21 10:58:14 UTC

MDEV-26247 [demo] gbh::init_scan() passes first table holder spider

With the assertion that one of the following is true:

- !str: we are creating a gbh, rather than doing a init_scan
- !use_fields: probably means the spider_db_open_item_field is called
  from outside of gbh, which also implies fields is nonnull
- fields->table_holder->spider == spider: the spider passed to
  spider_db_open_item_field() is the same as the first table_holder
  of fields

Basically this means the spider passed to spider_db_open_item_field is
irrelevant to the item_field during init_scan(), and we should find
another way of recovering the table the item_field belongs to.
Currently it does so by relying on the field chain from fields, which
was constructed during creation of the gbh (first pass). We need to
find another way, because this way is hacky and unstable when making
changes to query rewrite code and it would be better to skip the first
pass.

Author: Nikita Malyavin
Author Date: 2023-09-21 10:25:21 UTC

fix rdb_i_s.cc build

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-09-20 14:02:07 UTC

MDEV-29989 binlog_do_db option breaks importing sql dumps

Problem:
=========
During commit, InnoDB does apply the buffered bulk insert
operation in innodb_prepare_commit_versioned(). Due to
binlog_do_db option, we disable the binlog for the
statement. But innodb_prepare_commit_versioned() is being
called only when binlog is enabled for the statement.

Fix:
===
innodb_prepare_commit_versioned() should happen irrespective
of binlog state. So if the server has any read-write operation
then we should call prepare_commit_versioned(). This eventually
fixes bulk insert operation.

versioning.trx_id : Added the option --binlog_do_db

Author: Yuchen Pei
Author Date: 2023-09-21 02:59:54 UTC

MDEV-26247 [demo] disabling the spider gbh fixes the issue

Before 10.10, the spider group by handler could be disabled by setting
spider_use_handler to nonzero, but this was removed in MDEV-27256.
That variable and its corresponding table param did a few other things
as well. We could introduce a system variable that disables group by
handler, something like spider_disable_group_by_handler, which can be
a workaround for bugs caused by spider gbh while we try to fix them.

Author: VladislavVaintroub
Author Date: 2023-09-19 16:30:57 UTC

MDEV-32189 Use icu for timezones on windows

Use ICU to work with timezones, to retrieve current timezone name,
abbreviation, and offset from GMT.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-09-20 06:29:13 UTC

MDEV-31826 InnoDB may fail to recover after being killed
  in fil_delete_tablespace()

- InnoDB is first setting the fil_space_t::STOPPING flag and
only then writing the FILE_DELETE record. Before writing the redo
log record, InnoDB server has been killed. So there are few pages
which were never written to the file because of STOPPING flag

- InnoDB should write FILE_DELETE redo record before checking for
pending operations.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-09-18 12:49:48 UTC

MDEV-30802 Assertion `index->is_btree() || index->is_ibuf()'
   failed in btr_search_guess_on_hash

Problem:
=======
 - There is a race condition between purge and rollback of alter
operation. Alter rollback marks the index as corrupted.
At the same time, purge is working on the same index and leads
to assert failure. This is caused by
commit 7c0b9c6020101f7655c494327babb274d3af6dc3 (MDEV-15250).

Solution:
=======
 - After MDEV-15250, InnoDB logs the operation only at the
end of transaction commit and applies the log in commit phase
and also via dml thread. So there is no need for purge to
work on uncommitted index.

Added the test case to make the assert fail. But we need
to add the following in row_purge_remove_sec_if_poss_leaf()

  if (!index->is_committed()) sleep(5);

Author: Rex Johnston
Author Date: 2023-08-18 03:51:54 UTC

MDEV-31466 Add optional correlation column list for derived tables

extend Mariadb to support column name assignment in derived sytax
(some expression) [as|=] ident [comma separated column name list].

Author: Yuchen Pei
Author Date: 2023-09-13 00:03:28 UTC

MDEV-32157 MDEV-28856 Spider: Tests, documentation, small fixes and cleanups

Removed some redundant hint related string literals from
spd_db_conn.cc

Clean up SPIDER_PARAM_*_[CHAR]LEN[S]

Adding tests covering monitoring_kind=2. What it does is that it reads
from mysql.spider_link_mon_servers with matching db_name, table_name,
link_id, and does not do anything about that...

How monitoring_* can be useful: in the deprecated spider high
availability feature, when one remote fails, spider will try another
remote, which apparently makes use of these table parameters.

A test covering the query_cache_sync table param. Some further tests
on some spider table params.

Wrapper should be case insensitive.

Code documentation on spider priority binary tree.

Add an assertion that static_key_cardinality is always -1. All tests
pass still

Author: Yuchen Pei
Author Date: 2023-09-13 00:03:28 UTC

MDEV-32157 MDEV-28856 Spider: Tests, documentation, small fixes and cleanups

Removed some redundant hint related string literals from
spd_db_conn.cc

Clean up SPIDER_PARAM_*_[CHAR]LEN[S]

Adding tests covering monitoring_kind=2. What it does is that it reads
from mysql.spider_link_mon_servers with matching db_name, table_name,
link_id, and does not do anything about that...

How monitoring_* can be useful: in the deprecated spider high
availability feature, when one remote fails, spider will try another
remote, which apparently makes use of these table parameters.

A test covering the query_cache_sync table param. Some further tests
on some spider table params.

Wrapper should be case insensitive.

Code documentation on spider priority binary tree.

Add an assertion that static_key_cardinality is always -1. All tests
pass still

Author: Yuchen Pei
Author Date: 2023-09-13 00:03:28 UTC

MDEV-32157 MDEV-28856 Spider: Tests, documentation, small fixes and cleanups

Removed some redundant hint related string literals from
spd_db_conn.cc

Clean up SPIDER_PARAM_*_[CHAR]LEN[S]

Adding tests covering monitoring_kind=2. What it does is that it reads
from mysql.spider_link_mon_servers with matching db_name, table_name,
link_id, and does not do anything about that...

How monitoring_* can be useful: in the deprecated spider high
availability feature, when one remote fails, spider will try another
remote, which apparently makes use of these table parameters.

A test covering the query_cache_sync table param. Some further tests
on some spider table params.

Wrapper should be case insensitive.

Code documentation on spider priority binary tree.

Add an assertion that static_key_cardinality is always -1. All tests
pass still

Author: Yuchen Pei
Author Date: 2023-09-13 00:03:28 UTC

MDEV-32157 MDEV-28856 Spider: Tests, documentation, small fixes and cleanups

Removed some redundant hint related string literals from
spd_db_conn.cc

Clean up SPIDER_PARAM_*_[CHAR]LEN[S]

Adding tests covering monitoring_kind=2. What it does is that it reads
from mysql.spider_link_mon_servers with matching db_name, table_name,
link_id, and does not do anything about that...

How monitoring_* can be useful: in the deprecated spider high
availability feature, when one remote fails, spider will try another
remote, which apparently makes use of these table parameters.

A test covering the query_cache_sync table param. Some further tests
on some spider table params.

Wrapper should be case insensitive.

Code documentation on spider priority binary tree.

Add an assertion that static_key_cardinality is always -1. All tests
pass still

Author: Yuchen Pei
Author Date: 2023-09-15 03:26:22 UTC

MDEV-32157 MDEV-28856 Spider: Tests, documentation, small fixes and cleanups

Removed some redundant hint related string literals from
spd_db_conn.cc

Clean up SPIDER_PARAM_*_[CHAR]LEN[S]

Adding tests covering monitoring_kind=2. What it does is that it reads
from mysql.spider_link_mon_servers with matching db_name, table_name,
link_id, and does not do anything about that...

How monitoring_* can be useful: in the deprecated spider high
availability feature, when one remote fails, spider will try another
remote, which apparently makes use of these table parameters.

A test covering the query_cache_sync table param. Some further tests
on some spider table params.

Wrapper should be case insensitive.

Code documentation on spider priority binary tree.

Add an assertion that static_key_cardinality is always -1. All tests
pass still

Author: Yuchen Pei
Author Date: 2023-09-15 03:26:22 UTC

MDEV-32157 MDEV-28856 Spider: Tests, documentation, small fixes and cleanups

Removed some redundant hint related string literals from
spd_db_conn.cc

Clean up SPIDER_PARAM_*_[CHAR]LEN[S]

Adding tests covering monitoring_kind=2. What it does is that it reads
from mysql.spider_link_mon_servers with matching db_name, table_name,
link_id, and does not do anything about that...

How monitoring_* can be useful: in the deprecated spider high
availability feature, when one remote fails, spider will try another
remote, which apparently makes use of these table parameters.

A test covering the query_cache_sync table param. Some further tests
on some spider table params.

Wrapper should be case insensitive.

Code documentation on spider priority binary tree.

Add an assertion that static_key_cardinality is always -1. All tests
pass still

Author: Yuchen Pei
Author Date: 2023-09-14 04:52:43 UTC

MDEV-31673 MDEV-29502 Remove spider_db_handler::need_lock_before_set_sql_for_exec

This function trivially returns false

Author: Yuchen Pei
Author Date: 2023-09-12 01:57:07 UTC

MDEV-22534 Fix self-referencing Item_direct_view_ref

The loop is created in substitute_for_best_equal_field().

It calls Item_ref::transform (for the Item_direct_view_ref object)
which calls:

  Item *new_item= (*ref)->transform(thd, transformer, arg); // TRANSFORM-CALL

where transformer is Item::replace_equal_field().

Consider *vr1->ref==field, where vr1 is an Item_direct_view_ref, and
field is an Item_field that erroneously participates in a multiple
equality. If field->replace_equal_field() returns vr2, another
Item_direct_view_ref, with vr2->ref == vr1->ref. vr1->transform()
will, after the call on field->transform() that returns vr2 for
replacement, update its ref to point to vr2, so now *vr1->ref==vr2,
and since vr2->ref==vr1->ref, this gives us *vr2->ref == vr2, a self
reference.

This can be generalised to nested Item_direct_view_refs too, if
field->transform() returns an Item_direct_view_ref that has the same
ref field as the ref field of any Item_direct_view_ref in the chain.

On to the fix. If we maintain the rule that

  Objects inside Item_direct_view_ref do not participate in multiple
  equalities.

then the TRANSFORM-CALL will have new_item == *ref. That is, the
transform will be a no-op and no loop will be created.

The participation of multiple equalities is set during call to
Item::propagate_equal_fields(). This commit fixes
Item_ref::propagate_equal_fields() so that the rule is not violated
when a (non-Item_direct_view_ref) Item_ref points to an
Item_direct_view_ref:

Item_ref->Item_ref->...->Item_direct_view_ref->Item_direct_view_ref->...->Item_field

More specifically, we now delegate to the dereferenced
item (i.e. *ref) in such calls, rather than directly to the underlying
Item_field. By doing this, the call to propagate_equal_fields() on
toplevel Item_direct_view_ref will make it participate in the multiple
equality, instead of any of its downstream items.