lp:maria

Author: Yuchen Pei
Author Date: 2023-06-13 09:46:03 UTC

MDEV-31400 Simple plugin dependency resolution

We introduce simple plugin dependency. A plugin init function may
return HA_ERR_RETRY_INIT. If this happens during server startup when
the server is trying to initialise all plugins, the failed plugins
will be retried, until no more plugins succeed in initialisation or
want to be retried.

This will fix spider init bugs which is caused in part by its
dependency on Aria for initialisation.

The reason we need a new return code, instead of treating every
failure as a request for retry, is that it may be impossible to clean
up after a failed plugin initialisation. Take InnoDB for example, it
has a global variable `buf_page_cleaner_is_active`, which may not
satisfy an assertion during a second initialisation try, probably
because InnoDB does not expect the initialisation to be called
twice.

Author: Yuchen Pei
Author Date: 2023-06-13 09:46:03 UTC

MDEV-31400 Simple plugin dependency resolution

We introduce simple plugin dependency. A plugin init function may
return HA_ERR_RETRY_INIT. If this happens during server startup when
the server is trying to initialise all plugins, the failed plugins
will be retried, until no more plugins succeed in initialisation or
want to be retried.

This will fix spider init bugs which is caused in part by its
dependency on Aria for initialisation.

The reason we need a new return code, instead of treating every
failure as a request for retry, is that it may be impossible to clean
up after a failed plugin initialisation. Take InnoDB for example, it
has a global variable `buf_page_cleaner_is_active`, which may not
satisfy an assertion during a second initialisation try, probably
because InnoDB does not expect the initialisation to be called
twice.

Author: Yuchen Pei
Author Date: 2023-06-13 09:46:03 UTC

MDEV-31400 Simple plugin dependency resolution

We introduce simple plugin dependency. A plugin init function may
return HA_ERR_RETRY_INIT. If this happens during server startup when
the server is trying to initialise all plugins, the failed plugins
will be retried, until no more plugins succeed in initialisation or
want to be retried.

This will fix spider init bugs which is caused in part by its
dependency on Aria for initialisation.

The reason we need a new return code, instead of treating every
failure as a request for retry, is that it may be impossible to clean
up after a failed plugin initialisation. Take InnoDB for example, it
has a global variable `buf_page_cleaner_is_active`, which may not
satisfy an assertion during a second initialisation try, probably
because InnoDB does not expect the initialisation to be called
twice.

Author: Yuchen Pei
Author Date: 2023-06-13 09:46:03 UTC

MDEV-31400 Simple plugin dependency resolution

We introduce simple plugin dependency. A plugin init function may
return HA_ERR_RETRY_INIT. If this happens during server startup when
the server is trying to initialise all plugins, the failed plugins
will be retried, until no more plugins succeed in initialisation or
want to be retried.

This will fix spider init bugs which is caused in part by its
dependency on Aria for initialisation.

The reason we need a new return code, instead of treating every
failure as a request for retry, is that it may be impossible to clean
up after a failed plugin initialisation. Take InnoDB for example, it
has a global variable `buf_page_cleaner_is_active`, which may not
satisfy an assertion during a second initialisation try, probably
because InnoDB does not expect the initialisation to be called
twice.

Author: Yuchen Pei
Author Date: 2023-07-25 03:10:52 UTC

MDEV-31400 Simple plugin dependency resolution

We introduce simple plugin dependency. A plugin init function may
return HA_ERR_RETRY_INIT. If this happens during server startup when
the server is trying to initialise all plugins, the failed plugins
will be retried, until no more plugins succeed in initialisation or
want to be retried.

This will fix spider init bugs which is caused in part by its
dependency on Aria for initialisation.

The reason we need a new return code, instead of treating every
failure as a request for retry, is that it may be impossible to clean
up after a failed plugin initialisation. Take InnoDB for example, it
has a global variable `buf_page_cleaner_is_active`, which may not
satisfy an assertion during a second initialisation try, probably
because InnoDB does not expect the initialisation to be called
twice.

Author: Yuchen Pei
Author Date: 2023-07-25 03:10:52 UTC

MDEV-31400 Simple plugin dependency resolution

We introduce simple plugin dependency. A plugin init function may
return HA_ERR_RETRY_INIT. If this happens during server startup when
the server is trying to initialise all plugins, the failed plugins
will be retried, until no more plugins succeed in initialisation or
want to be retried.

This will fix spider init bugs which is caused in part by its
dependency on Aria for initialisation.

The reason we need a new return code, instead of treating every
failure as a request for retry, is that it may be impossible to clean
up after a failed plugin initialisation. Take InnoDB for example, it
has a global variable `buf_page_cleaner_is_active`, which may not
satisfy an assertion during a second initialisation try, probably
because InnoDB does not expect the initialisation to be called
twice.

Author: Yuchen Pei
Author Date: 2023-07-20 02:10:26 UTC

MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs

Fix spider init bugs (MDEV-22979, MDEV-27233, MDEV-28218) while
preventing regression on old ones (MDEV-30370, MDEV-29904)

Two things are changed:

First, Spider initialisation is made fully synchronous, i.e. it no
longer happens in a background thread. Adapted from the original fix
by nayuta for MDEV-27233. This change itself would cause failure when
spider is initialised early, by plugin-load-add, due to dependency on
Aria and udf function creation, which are fixed in the second and
third parts below. Requires SQL Service, thus porting earlier versions
requires MDEV-27595

Second, if spider is initialised before udf_init(), create udf by
inserting into `mysql.func`, otherwise do it by `CREATE FUNCTION` as
usual. This change may be generalised in MDEV-31401.

Also factor out some clean-up queries from deinit_spider.inc for use
of spider init tests.

A minor caveat is that early spider initialisation will fail if the
server is bootstrapped for the first time, due to missing `mysql`
database which needs to be created by the bootstrap script.

Author: Yuchen Pei
Author Date: 2023-06-13 09:46:03 UTC

MDEV-31400 Simple plugin dependency resolution

We introduce simple plugin dependency. A plugin init function may
return HA_ERR_RETRY_INIT. If this happens during server startup when
the server is trying to initialise all plugins, the failed plugins
will be retried, until no more plugins succeed in initialisation or
want to be retried.

This will fix spider init bugs which is caused in part by its
dependency on Aria for initialisation.

The reason we need a new return code, instead of treating every
failure as a request for retry, is that it may be impossible to clean
up after a failed plugin initialisation. Take InnoDB for example, it
has a global variable `buf_page_cleaner_is_active`, which may not
satisfy an assertion during a second initialisation try, probably
because InnoDB does not expect the initialisation to be called
twice.

Author: Yuchen Pei
Author Date: 2023-07-25 00:26:07 UTC

MDBF-535 Test the buildbot change by breaking a spider test

Author: Lena Startseva
Author Date: 2023-07-24 14:11:05 UTC

MDEV-31003: Second execution for ps-protocol

Update tests for version 11.2

Author: Lena Startseva
Author Date: 2023-07-24 11:44:18 UTC

MDEV-31003: Second execution for ps-protocol

Update tests for version 11.1

Author: Lena Startseva
Author Date: 2023-07-24 10:35:41 UTC

MDEV-31003: Second execution for ps-protocol

Update tests for version 11.0

Author: Lena Startseva
Author Date: 2023-07-24 04:40:46 UTC

MDEV-31003: Second execution for ps-protocol

Update tests for version 10.10

Author: Lena Startseva
Author Date: 2023-07-23 07:56:51 UTC

MDEV-31003: Second execution for ps-protocol

Update tests for version 10.9

Author: Brandon Nesterenko
Author Date: 2023-07-17 21:01:16 UTC

MDEV-30619: Test case for MDEV-29639 SBM spike fix

Added test case to cover SBM spike of relay log read and LMT
update that was fixed by MDEV-29639

Note that pause_sql_thread_on_next_event usage was removed
with the MDEV-30619 fixes. Rather than remove it, we adapt it
to the needs of this test case.

Additionally updated rpl_seconds_behind_master_spike.test to
use the negate_clock_diff_with_master debug eval.

Author: Brandon Nesterenko
Author Date: 2023-07-17 21:01:16 UTC

MDEV-30619: Test case for MDEV-29639 SBM spike fix

Added test case to cover SBM spike of relay log read and LMT
update that was fixed by MDEV-29639

Note that pause_sql_thread_on_next_event usage was removed
with the MDEV-30619 fixes. Rather than remove it, we adapt it
to the needs of this test case.

Additionally updated rpl_seconds_behind_master_spike.test to
use the negate_clock_diff_with_master debug eval.

Author: Lena Startseva
Author Date: 2023-07-21 11:17:18 UTC

MDEV-31003: Second execution for ps-protocol

Update tests for version 10.6

Author: Julius Goryavsky
Author Date: 2023-07-21 11:09:56 UTC

debug mtr

Author: Sergei Golubchik
Author Date: 2023-06-17 17:00:56 UTC

MDEV-26182 fixes for --ps

* invoke parent's cleanup()
* don't reinit memroot, if already inited (causes memory leak)

also move free_root() from destructor to cleanup() to not accumulate
allocations from prepare and multiple executes

Author: Andrei
Author Date: 2023-07-20 18:20:37 UTC

MDEV-31755 Replica's DML event deadlocks wit online alter table

The deadlock was caused by too strong MDL acquired by the start ALTER.

While MDL is shared by the start ALTER wait for its 2nd part
to allow concurrent DML:s to grab the lock.
The fixes uses wait_for_master reentrancy.

Author: Lena Startseva
Author Date: 2023-07-20 07:24:11 UTC

MDEV-31003: Second execution for ps-protocol

Update tests for version 10.5

Author: Sergey Petrunia
Author Date: 2023-07-06 07:41:46 UTC

MDEV-31577: Make ANALYZE FORMAT=JSON print innodb stats

ANALYZE FORMAT=JSON output now includes table.r_engine_stats which
has the engine statistics. Only non-zero members are printed.

Internally: EXPLAIN data structure Explain_table_acccess now has
handler* handler_for_stats pointer.
It is used to read statistics from handler_for_stats->handler_stats.

Explain data structures exist after the tables are closed. We avoid
walking invalid pointers using this:
- SQL layer calls Explain_query::notify_tables_are_closed() before
  closing tables.
- After that call, printing of JSON output is disabled. Non-JSON output
  can be printed but we don't access handler_for_stats when doing that.

Author: Rex Johnston
Author Date: 2023-07-19 04:59:33 UTC

MDEV-30710 Incorrect operator when comparing large unsigned integers.

When constructing a SEL_TREE, an unsigned integer greater than
its signed equivalent caused an incorrect comparison operator to
be chosen.

Approved by Alexander Barkov <bar@mariadb.com>

Author: Vicențiu Ciorbaru
Author Date: 2022-08-29 07:21:21 UTC

bugfix: userstat uses acl code, it must be recompiled for embedded

Author: Rex Johnston
Author Date: 2023-07-19 04:59:33 UTC

MDEV-30710 Incorrect operator when comparing large unsigned integers.

When constructing a SEL_TREE, an unsigned integer greater than
its signed equivalent caused an incorrect comparison operator to
be chosen.

Author: Nikita Malyavin
Author Date: 2023-07-17 10:52:30 UTC

add unit-test

usae:
buld taret open_address_hash-t (i.e. cmake your-build-dir --build open_address_hash-t)
run ctest open_address_hash (i.e. ctest -R open_address_hash, from your-build-dir)

Author: Yuchen Pei
Author Date: 2023-07-17 07:50:28 UTC

MDEV-26247 [exp] Removal of const table handling in spider GBH

This is a silly experiment, where we remove all constant table
handling in the spider group by handler, introduced in commit
52ca07c2a0977f74ccfb56363e4158f0bd0ad3a0.

It shows the same output of the spider GBH query rewrite output as
dbug_print for the following two queries:

select tbl_b.d AS d from tbl_a join tbl_b where tbl_b.c = 1;
SELECT d FROM tbl_a LEFT JOIN tbl_b ON a = c WHERE a IN (1);

Author: Brandon Nesterenko
Author Date: 2023-07-12 20:42:03 UTC

MDEV-7850: main and innodb MTR test suite fixes

Author: Sergey Petrunia
Author Date: 2023-07-06 07:41:46 UTC

MDEV-31577: Make ANALYZE FORMAT=JSON print innodb stats

(Backport to 10.6)
ANALYZE FORMAT=JSON output now includes table.r_engine_stats which
has the engine statistics. Only non-zero members are printed.

Internally: EXPLAIN data structure Explain_table_acccess now has
handler* handler_for_stats pointer.
It is used to read statistics from handler_for_stats->handler_stats.

Explain data structures exist after the tables are closed. We avoid
walking invalid pointers using this:
- SQL layer calls Explain_query::notify_tables_are_closed() before
  closing tables.
- After that call, printing of JSON output is disabled. Non-JSON output
  can be printed but we don't access handler_for_stats when doing that.

Author: Yuchen Pei
Author Date: 2023-07-13 07:04:22 UTC

MDEV-29502 Fix some issues with spider direct aggregate

The direct aggregate mechanism sems to be only intended to work when
otherwise a full table scan query will be executed from the spider
node and the aggregation done at the spider node too. Typically this
happens in sub_select(). In the test spider.direct_aggregate_part
direct aggregate allows to send COUNT statements directly to the data
nodes and adds up the results at the spider node, instead of iterating
over the rows one by one at the spider node.

By contrast, the group by handler (GBH) typically sends aggregated
queries directly to data nodes, in which case DA does not improve the
situation here.

That is why we should fix it by disabling DA when GBH is used.

There are other reasons supporting this change. First, the creation of
GBH results in a call to change_to_use_tmp_fields() (as opposed to
setup_copy_fields()) which causes the spider DA function
spider_db_fetch_for_item_sum_funcs() to work on wrong items. Second,
the spider DA function only calls direct_add() on the items, and the
follow-up add() needs to be called by the sql layer code. In
do_select(), after executing the query with the GBH, it seems that the
required add() would not necessarily be called.

Disabling DA when GBH is used does fix the bug. There are a few
other things included in this commit to improve the situation with
spider DA:

1. Add a session variable that allows user to disable DA completely,
this will help as a temporary measure if/when further bugs with DA
emerge.

2. Move the increment of direct_aggregate_count to the spider DA
function. Currently this is done in rather bizarre and random
locations.

3. Fix the spider_db_mbase_row creation so that the last of its row
field (sentinel) is NULL. The code is already doing a null check, but
somehow the sentinel field is on an invalid address, causing the
segfaults. With a correct implementation of the row creation, we can
avoid such segfaults.

Author: Yuchen Pei
Author Date: 2023-07-13 07:04:22 UTC

MDEV-29502 Fix some issues with spider direct aggregate

The direct aggregate mechanism sems to be only intended to work when
otherwise a full table scan query will be executed from the spider
node and the aggregation done at the spider node too. Typically this
happens in sub_select(). In the test spider.direct_aggregate_part
direct aggregate allows to send COUNT statements directly to the data
nodes and adds up the results at the spider node, instead of iterating
over the rows one by one at the spider node.

By contrast, the group by handler (GBH) typically sends aggregated
queries directly to data nodes, in which case DA does not improve the
situation here.

That is why we should fix it by disabling DA when GBH is used.

There are other reasons supporting this change. First, the creation of
GBH results in a call to change_to_use_tmp_fields() (as opposed to
setup_copy_fields()) which causes the spider DA function
spider_db_fetch_for_item_sum_funcs() to work on wrong items. Second,
the spider DA function only calls direct_add() on the items, and the
follow-up add() needs to be called by the sql layer code. In
do_select(), after executing the query with the GBH, it seems that the
required add() would not necessarily be called.

Disabling DA when GBH is used does fix the bug. There are a few
other things included in this commit to improve the situation with
spider DA:

1. Add a session variable that allows user to disable DA completely,
this will help as a temporary measure if/when further bugs with DA
emerge.

2. Move the increment of direct_aggregate_count to the spider DA
function. Currently this is done in rather bizarre and random
locations.

3. Fix the spider_db_mbase_row creation so that the last of its row
field (sentinel) is NULL. The code is already doing a null check, but
somehow the sentinel field is on an invalid address, causing the
segfaults. With a correct implementation of the row creation, we can
avoid such segfaults.

Author: Yuchen Pei
Author Date: 2023-07-11 07:53:59 UTC

MDEV-29502 Fix some issues with spider direct aggregate

The direct aggregate mechanism sems to be only intended to work when
otherwise a full table scan query will be executed from the spider
node and the aggregation done at the spider node too. Typically this
happens in sub_select(). In the test spider.direct_aggregate_part
direct aggregate allows to send COUNT statements directly to the data
nodes and adds up the results at the spider node, instead of iterating
over the rows one by one at the spider node.

By contrast, the group by handler (GBH) typically sends aggregated
queries directly to data nodes, in which case DA does not improve the
situation here.

That is why we should fix it by disabling DA when GBH is used.

There are other reasons supporting this change. First, the creation of
GBH results in a call to change_to_use_tmp_fields() (as opposed to
setup_copy_fields()) which causes the spider DA function
spider_db_fetch_for_item_sum_funcs() to work on wrong items. Second,
the spider DA function only calls direct_add() on the items, and the
follow-up add() needs to be called by the sql layer code. In
do_select(), after executing the query with the GBH, it seems that the
required add() would not necessarily be called.

Disabling DA when GBH is used does fix the bug. There are a few
other things included in this commit to improve the situation with
spider DA:

1. Add a session variable that allows user to disable DA completely,
this will help as a temporary measure if/when further bugs with DA
emerge.

2. Move the increment of direct_aggregate_count to the spider DA
function. Currently this is done in rather bizarre and random
locations.

3. Fix the spider_db_mbase_row creation so that the last of its row
field (sentinel) is NULL. The code is already doing a null check, but
somehow the sentinel field is on an invalid address, causing the
segfaults. With a correct implementation of the row creation, we can
avoid such segfaults.

Author: Yuchen Pei
Author Date: 2023-07-07 01:07:05 UTC

MDEV-31524 Fixing spider table param / variable overriding

The existing (incorrect) overriding mechanism is:

Non-minus-one var value overrides table param overrides default value.

Before MDEV-27169, unspecified var value is -1. So if the user sets
both the var to be a value other than -1 and the table param, the var
value will prevail, which is incorrect.

After MDEV-27169, unspecified var value is default value. So if the
user does not set the var but sets the table param, the default value
will prevail, which is even more incorrect.

This patch fixes it so that table param, if specified, always
overrides var value, and the latter if not specified or set to -1,
falls back to the default value

We achieve this by replacing all such overriding in spd_param.cc with
macros that override in the correct way, and removing all the
"overriding -1" lines involving table params in
spider_set_connect_info_default() except for those table params not
defined as sysvar/thdvar in spd_params.cc

We also introduced macros for non-overriding sysvar and thdvar, so
that the code is cleaner and less error-prone

In server versions where MDEV-27169 has not been applied, we also
backport the patch, that is, replacing -1 default values with real
default values

In server versions where MDEV-28006 has not been applied, we do the
same for udf params

Author: Yuchen Pei
Author Date: 2023-07-07 01:07:05 UTC

MDEV-31524 Fixing spider table param / variable overriding

The existing (incorrect) overriding mechanism is:

Non-minus-one var value overrides table param overrides default value.

Before MDEV-27169, unspecified var value is -1. So if the user sets
both the var to be a value other than -1 and the table param, the var
value will prevail, which is incorrect.

After MDEV-27169, unspecified var value is default value. So if the
user does not set the var but sets the table param, the default value
will prevail, which is even more incorrect.

This patch fixes it so that table param, if specified, always
overrides var value, and the latter if not specified or set to -1,
falls back to the default value

We achieve this by replacing all such overriding in spd_param.cc with
macros that override in the correct way, and removing all the
"overriding -1" lines involving table params in
spider_set_connect_info_default() except for those table params not
defined as sysvar/thdvar in spd_params.cc

We also introduced macros for non-overriding sysvar and thdvar, so
that the code is cleaner and less error-prone

In server versions where MDEV-27169 has not been applied, we also
backport the patch, that is, replacing -1 default values with real
default values

In server versions where MDEV-28006 has not been applied, we do the
same for udf params

Author: Yuchen Pei
Author Date: 2023-07-07 01:07:05 UTC

MDEV-31524 Fixing spider table param / variable overriding

The existing (incorrect) overriding mechanism is:

Non-minus-one var value overrides table param overrides default value.

Before MDEV-27169, unspecified var value is -1. So if the user sets
both the var to be a value other than -1 and the table param, the var
value will prevail, which is incorrect.

After MDEV-27169, unspecified var value is default value. So if the
user does not set the var but sets the table param, the default value
will prevail, which is even more incorrect.

This patch fixes it so that table param, if specified, always
overrides var value, and the latter if not specified or set to -1,
falls back to the default value

We achieve this by replacing all such overriding in spd_param.cc with
macros that override in the correct way, and removing all the
"overriding -1" lines involving table params in
spider_set_connect_info_default() except for those table params not
defined as sysvar/thdvar in spd_params.cc

We also introduced macros for non-overriding sysvar and thdvar, so
that the code is cleaner and less error-prone

In server versions where MDEV-27169 has not been applied, we also
backport the patch, that is, replacing -1 default values with real
default values

In server versions where MDEV-28006 has not been applied, we do the
same for udf params

Author: Yuchen Pei
Author Date: 2023-07-07 01:07:05 UTC

MDEV-31524 Fixing spider table param / variable overriding

The existing (incorrect) overriding mechanism is:

Non-minus-one var value overrides table param overrides default value.

Before MDEV-27169, unspecified var value is -1. So if the user sets
both the var to be a value other than -1 and the table param, the var
value will prevail, which is incorrect.

After MDEV-27169, unspecified var value is default value. So if the
user does not set the var but sets the table param, the default value
will prevail, which is even more incorrect.

This patch fixes it so that table param, if specified, always
overrides var value, and the latter if not specified or set to -1,
falls back to the default value

We achieve this by replacing all such overriding in spd_param.cc with
macros that override in the correct way, and removing all the
"overriding -1" lines involving table params in
spider_set_connect_info_default() except for those table params not
defined as sysvar/thdvar in spd_params.cc

We also introduced macros for non-overriding sysvar and thdvar, so
that the code is cleaner and less error-prone

In server versions where MDEV-27169 has not been applied, we also
backport the patch, that is, replacing -1 default values with real
default values

In server versions where MDEV-28006 has not been applied, we do the
same for udf params

Author: Yuchen Pei
Author Date: 2023-07-07 01:07:05 UTC

MDEV-31524 Fixing spider table param / variable overriding

The existing (incorrect) overriding mechanism is:

Non-minus-one var value overrides table param overrides default value.

Before MDEV-27169, unspecified var value is -1. So if the user sets
both the var to be a value other than -1 and the table param, the var
value will prevail, which is incorrect.

After MDEV-27169, unspecified var value is default value. So if the
user does not set the var but sets the table param, the default value
will prevail, which is even more incorrect.

This patch fixes it so that table param, if specified, always
overrides var value, and the latter if not specified or set to -1,
falls back to the default value

We achieve this by replacing all such overriding in spd_param.cc with
macros that override in the correct way, and removing all the
"overriding -1" lines involving table params in
spider_set_connect_info_default() except for those table params not
defined as sysvar/thdvar in spd_params.cc

We also introduced macros for non-overriding sysvar and thdvar, so
that the code is cleaner and less error-prone

In server versions where MDEV-27169 has not been applied, we also
backport the patch, that is, replacing -1 default values with real
default values

In server versions where MDEV-28006 has not been applied, we do the
same for udf params

Author: Yuchen Pei
Author Date: 2023-07-07 01:07:05 UTC

MDEV-31524 Fixing spider table param / variable overriding

The existing (incorrect) overriding mechanism is:

Non-minus-one var value overrides table param overrides default value.

Before MDEV-27169, unspecified var value is -1. So if the user sets
both the var to be a value other than -1 and the table param, the var
value will prevail, which is incorrect.

After MDEV-27169, unspecified var value is default value. So if the
user does not set the var but sets the table param, the default value
will prevail, which is even more incorrect.

This patch fixes it so that table param, if specified, always
overrides var value, and the latter if not specified or set to -1,
falls back to the default value

We achieve this by replacing all such overriding in spd_param.cc with
macros that override in the correct way, and removing all the
"overriding -1" lines involving table params in
spider_set_connect_info_default() except for those table params not
defined as sysvar/thdvar in spd_params.cc

We also introduced macros for non-overriding sysvar and thdvar, so
that the code is cleaner and less error-prone

In server versions where MDEV-27169 has not been applied, we also
backport the patch, that is, replacing -1 default values with real
default values

In server versions where MDEV-28006 has not been applied, we do the
same for udf params

Author: Rex Johnston
Author Date: 2023-07-12 22:38:16 UTC

MDEV-31668 Build issues with Ubuntu 20.04.6 LTS, libaio, causing slowdown

A combination of MDEV-31558: Add InnoDB engine information to the slow query log
and MDEV-31577: Make ANALYZE FORMAT=JSON print innodb stats
backported to 10.3 for testing purposes.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-07-11 09:53:11 UTC

MDEV-29283 Assertion `0' failed -or- Assertion `item->maybe_null()' failed - both in virtual void Type_handler_string_result::make_sort_key_part - on UPDATE ... ORDER BY ... LIMIT 0

LIMIT & OFFSET also can influence one raw subselect NULL

Author: Kristian Nielsen
Author Date: 2023-06-19 22:07:17 UTC

MDEV-31671: implement fast STOP SLAVE with rollback ongoing transactions

This patch implements a STOP SLAVE FORCE option that tries to stop the slave
faster by rolling back active (transactional) event groups:

 - Abort any replicating row event on the next row operation.
 - In parallel replication, abort event group on the next event (as well as
   all following event groups).

Makes rpl_parallel_entry::abort_slave an std::atomic<>, as it is read
without mutex protection in some cases. And renames it as the more
appropriate name `slave_stopping`, to avoid merges silently leaving direct
accesses to the variable (these would default to std:: memory_order_seq_cst,
which is usually not desirable).

Based on original work by Brandon Nesterenko.

Reviewed-by: Andrei Elkin <andrei.elkin@mariadb.com>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Rucha Deodhar
Author Date: 2023-07-11 11:40:34 UTC

clang test crash work in progress

Author: Andrei
Author Date: 2023-07-10 15:53:19 UTC

MDEV-7850 Thread-id in Gtid

Adopting original Sujatha's patch and Sachin's followup (7850.patch).

Todo:

1. IIRC some attention to Gtid::peek is required, the must be some problems at running

rpl.rpl_mariadb_slave_capability 'innodb,row' w12 [ fail ]
        Test ended at 2023-03-27 18:32:31

CURRENT_TEST: rpl.rpl_mariadb_slave_capability

Server [mysqld.1 - pid: 22976, winpid: 22976, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
2023-03-27 18:32:27 68 [Note] Deleted Master_info file '/dev/shm/var/12/mysqld.1/data/master.info'.
2023-03-27 18:32:27 68 [Note] Deleted Master_info file '/dev/shm/var/12/mysqld.1/data/relay-log.info'.
2023-03-27 18:32:27 70 [Note] Start binlog_dump to slave_server(2), pos(master-bin.000001, 4), using_gtid(0), gtid('')
2023-03-27 18:32:27 73 [Note] Start binlog_dump to slave_server(2), pos(master-bin.000001, 329), using_gtid(0), gtid('')
mysqld: /home/andrei/MDB.WTs/CS/10.4/A/sql/log_event.cc:5131: static int Query_log_event::begin_event(String*, ulong, enum_binlog_checksum_alg): Assertion `data_len == 19 + 19 || data_len == 19 + 19 + 2' failed.
230327 18:32:28 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary

2. check out whether the new
  static const uchar FL_EXTRA_THREAD_ID= 2;
indeed occupies the greatest bit in 11.2 (bits have been added in 10.5,6,8).

Author: Marko Mäkelä
Author Date: 2023-06-27 06:12:38 UTC

MDEV-31487: Recovery or backup failure after innodb_undo_log_truncate=ON

recv_sys_t::parse(): For undo tablespace truncation mini-transactions,
remember the start_lsn instead of the end LSN. This is what we expect
after commit 461402a56432fa5cd0f6d53118ccce23081fca18 (MDEV-30479).

Author: midenok
Author Date: 2023-07-07 11:25:55 UTC

MDEV-25644 UPDATE not working properly on transaction precise system versioned table

First UPDATE under START TRANSACTION does nothing (nstate= nstate),
but anyway generates history. Since update vector is empty we get into
(!uvect->n_fields) branch which only adds history row, but does not do
update. After that we get current row with wrong (old) row_start value
and because of that second UPDATE tries to insert history row again
because it sees trx->id != row_start which is the guard to avoid
inserting multiple trx_id-based history rows under same transaction
(because we have same trx_id and we get duplicate error and this bug
demostrates that). But this try anyway fails because PK is based on
row_end which is constant under same transaction, so PK didn't change.

The fix moves vers_make_update() to an earlier stage of
calc_row_difference(). Therefore it prepares update vector before
(!uvect->n_fields) check and never gets into that branch, hence no
need to handle versioning inside that condition anymore.

Now trx->id and row_start are equal after first UPDATE and we don't
try to insert second history row.

== Cleanups and improvements ==

ha_innobase::update_row():

vers_set_fields and vers_ins_row are cleaned up into direct condition
check. SQLCOM_ALTER_TABLE check now is not used as this is dead code,
assertion is done instead.

upd_node->is_delete is set in calc_row_difference() just to keep
versioning code as much in one place as possible. vers_make_delete()
is still located in row_update_for_mysql() as this is required for
ha_innodbase::delete_row() as well.

row_ins_duplicate_error_in_clust():

Restrict DB_FOREIGN_DUPLICATE_KEY to the better conditions.
VERSIONED_DELETE is used specifically to help lower stack to
understand what caused current insert. Related to MDEV-29813.

Author: Yuchen Pei
Author Date: 2023-07-07 01:07:05 UTC

MDEV-31524 Fixing spider table param / variable overriding

The existing (incorrect) overriding mechanism is:

Non-minus-one var value overrides table param overrides default value.

Before MDEV-27169, unspecified var value is -1. So if the user sets
both the var to be a value other than -1 and the table param, the var
value will prevail, which is incorrect.

After MDEV-27169, unspecified var value is default value. So if the
user does not set the var but sets the table param, the default value
will prevail, which is even more incorrect.

This patch fixes it so that table param, if specified, always
overrides var value, and the latter if not specified or set to -1,
falls back to the default value

We achieve this by replacing all such overriding in spd_param.cc with
macros that override in the correct way, and removing all the
"overriding -1" lines involving table params in
spider_set_connect_info_default() except for those table params not
defined as sysvar/thdvar in spd_params.cc

We also introduced macros for non-overriding sysvar and thdvar, so
that the code is cleaner and less error-prone

In server versions where MDEV-27169 has not been applied, we also
backport the patch, that is, replacing -1 default values with real
default values

In server versions where MDEV-28006 has not been applied, we do the
same for udf params

Author: Sergei Golubchik
Author Date: 2023-07-06 20:17:35 UTC

MDEV-29959 fix for aarch64

on aarch64 `char` by default is unsigned for performance reasons.
let's adjust checks to work for both signed and unsigned `char`

Author: Vlad Lesin
Author Date: 2023-07-06 08:49:28 UTC

MDEV-29311 Server Status Innodb_row_lock_time% is reported in seconds

Before MDEV-24671, the wait time was derived from my_interval_timer() /
1000 (nanoseconds converted to microseconds, and not microseconds to
milliseconds like I must have assumed). The lock_sys.wait_time and
lock_sys.wait_time_max are already in milliseconds; we should not divide
them by 1000.

In MDEV-24738 the millisecond counts lock_sys.wait_time and
lock_sys.wait_time_max were changed to a 32-bit type. That would
overflow in 49.7 days. Keep using a 64-bit type for those millisecond
counters.

Author: Vlad Lesin
Author Date: 2023-07-03 13:04:15 UTC

MDEV-10962 Deadlock with 3 concurrent DELETEs by unique key

PROBLEM:
A deadlock was possible when a transaction tried to "upgrade" an already
held Record Lock to Next Key Lock.

SOLUTION:
This patch is based on observations that:
(1) a Next Key Lock is equivalent to Record Lock combined with Gap Lock
(2) a GAP Lock never has to wait for any other lock
In case we request a Next Key Lock, we check if we already own a Record
Lock of equal or stronger mode, and if so, then we change the requested
lock type to GAP Lock, which we either already have, or can be granted
immediately, as GAP locks don't conflict with any other lock types.
(We don't consider Insert Intention Locks a Gap Lock in above statements).

The reason of why we don't upgrage Record Lock to Next Key Lock is the
following.

Imagine a transaction which does something like this:

for each row {
    request lock in LOCK_X|LOCK_REC_NOT_GAP mode
    request lock in LOCK_S mode
}

If we upgraded lock from Record Lock to Next Key lock, there would be
created only two lock_t structs for each page, one for
LOCK_X|LOCK_REC_NOT_GAP mode and one for LOCK_S mode, and then used
their bitmaps to mark all records from the same page.

The situation would look like this:

request lock in LOCK_X|LOCK_REC_NOT_GAP mode on row 1:
// -> creates new lock_t for LOCK_X|LOCK_REC_NOT_GAP mode and sets bit for
// 1
request lock in LOCK_S mode on row 1:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 1,
// so it upgrades it to X
request lock in LOCK_X|LOCK_REC_NOT_GAP mode on row 2:
// -> creates a new lock_t for LOCK_X|LOCK_REC_NOT_GAP mode (because we
// don't have any after we've upgraded!) and sets bit for 2
request lock in LOCK_S mode on row 2:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 2,
// so it upgrades it to X
    ...etc...etc..

Each iteration of the loop creates a new lock_t struct, and in the end we
have a lot (one for each record!) of LOCK_X locks, each with single bit
set in the bitmap. Soon we run out of space for lock_t structs.

If we create LOCK_GAP instead of lock upgrading, the above scenario works
like the following:

// -> creates new lock_t for LOCK_X|LOCK_REC_NOT_GAP mode and sets bit for
// 1
request lock in LOCK_S mode on row 1:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 1,
// so it creates LOCK_S|LOCK_GAP only and sets bit for 1
request lock in LOCK_X|LOCK_REC_NOT_GAP mode on row 2:
// -> reuses the lock_t for LOCK_X|LOCK_REC_NOT_GAP by setting bit for 2
request lock in LOCK_S mode on row 2:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 2,
// so it reuses LOCK_S|LOCK_GAP setting bit for 2

In the end we have just two locks per page, one for each mode:
LOCK_X|LOCK_REC_NOT_GAP and LOCK_S|LOCK_GAP.
Another benefit of this solution is that it avoids not-entirely
const-correct, (and otherwise looking risky) "upgrading".

The fix was ported from
mysql/mysql-server@bfba840dfa7794b988c59c94658920dbe556075d
mysql/mysql-server@75cefdb1f73b8f8ac8e22b10dfb5073adbdfdfb0

Reviewed by: Marko Mäkelä

Author: Yuchen Pei
Author Date: 2023-07-05 07:53:12 UTC

MDEV-31117 clean up spider connection info parsing

Spider connection string is a comma-separated parameter definitions,
where each definition is of the form "<param_title> <param_value>",
where <param_value> is quote delimited on both ends, with backslashes
acting as an escaping prefix.

The code however treated param title the same way as param value when
assigning, and have nonsensical fields like delim_title_len and
delim_title. We remove these.

We also clean up the spider comment connection string parsing,
including:

- Factoring out some code from the parsing function
- Rewriting the struct `st_spider_param_string_parse`, including
  replacing its messy methods with cleaner ones
- And any necessary changes caused by the above changes

Author: Rucha Deodhar
Author Date: 2023-07-05 11:05:01 UTC

MDEV-23187 misses resetting collation connection

MDEV-23187 misses resetting collation connection causing test failures for
10.5+ bugs.

Author: Yuchen Pei
Author Date: 2023-07-05 07:55:30 UTC

MDEV-31117 clean up spider connection info parsing

Spider connection string is a comma-separated parameter definitions,
where each definition is of the form "<param_title> <param_value>",
where <param_value> is quote delimited on both ends, with backslashes
acting as an escaping prefix.

The code however treated param title the same way as param value when
assigning, and have nonsensical fields like delim_title_len and
delim_title. We remove these.

We also clean up the spider comment connection string parsing,
including:

- Factoring out some code from the parsing function
- Rewriting the struct `st_spider_param_string_parse`, including
  replacing its messy methods with cleaner ones
- And any necessary changes caused by the above changes

Author: Yuchen Pei
Author Date: 2023-07-05 07:53:12 UTC

MDEV-31117 clean up spider connection info parsing

Spider connection string is a comma-separated parameter definitions,
where each definition is of the form "<param_title> <param_value>",
where <param_value> is quote delimited on both ends, with backslashes
acting as an escaping prefix.

The code however treated param title the same way as param value when
assigning, and have nonsensical fields like delim_title_len and
delim_title. We remove these.

We also clean up the spider comment connection string parsing,
including:

- Factoring out some code from the parsing function
- Rewriting the struct `st_spider_param_string_parse`, including
  replacing its messy methods with cleaner ones
- And any necessary changes caused by the above changes

Author: Yuchen Pei
Author Date: 2023-06-23 07:10:01 UTC

MDEV-31524 Fixing spider table param / variable overriding

The existing (incorrect) overriding mechanism is:

Non-minus-one var value overrides table param overrides default value.

Before MDEV-27169, unspecified var value is -1. So if the user sets
both the var to be a value other than -1 and the table param, the var
value will prevail, which is incorrect.

After MDEV-27169, unspecified var value is default value. So if the
user does not set the var but sets the table param, the default value
will prevail, which is even more incorrect.

This patch fixes it so that table param, if specified, always
overrides var value, and the latter if not specified or set to -1,
falls back to the default value

We achieve this by replacing all such overriding in spd_param.cc with
macros that override in the correct way, and removing all the
"overriding -1" lines involving table params in
spider_set_connect_info_default() except for those table params not
defined as sysvar/thdvar in spd_params.cc

We also introduced macros for non-overriding sysvar and thdvar, so
that the code is cleaner and less error-prone

In server versions where MDEV-27169 has not been applied, we also
backport the patch, that is, replacing -1 default values with real
default values

In server versions where MDEV-28006 has not been applied, we do the
same for udf params

Author: Vlad Lesin
Author Date: 2023-07-03 13:04:15 UTC

MDEV-10962 Deadlock with 3 concurrent DELETEs by unique key

PROBLEM:
A deadlock was possible when a transaction tried to "upgrade" an already
held Record Lock to Next Key Lock.

SOLUTION:
This patch is based on observations that:
(1) a Next Key Lock is equivalent to Record Lock combined with Gap Lock
(2) a GAP Lock never has to wait for any other lock
In case we request a Next Key Lock, we check if we already own a Record
Lock of equal or stronger mode, and if so, then we change the requested
lock type to GAP Lock, which we either already have, or can be granted
immediately, as GAP locks don't conflict with any other lock types.
(We don't consider Insert Intention Locks a Gap Lock in above statements).

The reason of why we don't upgrage Record Lock to Next Key Lock is the
following.

Imagine a transaction which does something like this:

for each row {
    request lock in LOCK_X|LOCK_REC_NOT_GAP mode
    request lock in LOCK_S mode
}

If we upgraded lock from Record Lock to Next Key lock, there would be
created only two lock_t structs for each page, one for
LOCK_X|LOCK_REC_NOT_GAP mode and one for LOCK_S mode, and then used
their bitmaps to mark all records from the same page.

The situation would look like this:

request lock in LOCK_X|LOCK_REC_NOT_GAP mode on row 1:
// -> creates new lock_t for LOCK_X|LOCK_REC_NOT_GAP mode and sets bit for
// 1
request lock in LOCK_S mode on row 1:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 1,
// so it upgrades it to X
request lock in LOCK_X|LOCK_REC_NOT_GAP mode on row 2:
// -> creates a new lock_t for LOCK_X|LOCK_REC_NOT_GAP mode (because we
// don't have any after we've upgraded!) and sets bit for 2
request lock in LOCK_S mode on row 2:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 2,
// so it upgrades it to X
    ...etc...etc..

Each iteration of the loop creates a new lock_t struct, and in the end we
have a lot (one for each record!) of LOCK_X locks, each with single bit
set in the bitmap. Soon we run out of space for lock_t structs.

If we create LOCK_GAP instead of lock upgrading, the above scenario works
like the following:

// -> creates new lock_t for LOCK_X|LOCK_REC_NOT_GAP mode and sets bit for
// 1
request lock in LOCK_S mode on row 1:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 1,
// so it creates LOCK_S|LOCK_GAP only and sets bit for 1
request lock in LOCK_X|LOCK_REC_NOT_GAP mode on row 2:
// -> reuses the lock_t for LOCK_X|LOCK_REC_NOT_GAP by setting bit for 2
request lock in LOCK_S mode on row 2:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 2,
// so it reuses LOCK_S|LOCK_GAP setting bit for 2

In the end we have just two locks per page, one for each mode:
LOCK_X|LOCK_REC_NOT_GAP and LOCK_S|LOCK_GAP.
Another benefit of this solution is that it avoids not-entirely
const-correct, (and otherwise looking risky) "upgrading".

The fix was ported from
mysql/mysql-server@bfba840dfa7794b988c59c94658920dbe556075d
mysql/mysql-server@75cefdb1f73b8f8ac8e22b10dfb5073adbdfdfb0

Reviewed by: Marko Mäkelä

Author: Vlad Lesin
Author Date: 2023-07-03 13:04:15 UTC

MDEV-10962 Deadlock with 3 concurrent DELETEs by unique key

PROBLEM:
A deadlock was possible when a transaction tried to "upgrade" an already
held Record Lock to Next Key Lock.

SOLUTION:
This patch is based on observations that:
(1) a Next Key Lock is equivalent to Record Lock combined with Gap Lock
(2) a GAP Lock never has to wait for any other lock
In case we request a Next Key Lock, we check if we already own a Record
Lock of equal or stronger mode, and if so, then we change the requested
lock type to GAP Lock, which we either already have, or can be granted
immediately, as GAP locks don't conflict with any other lock types.
(We don't consider Insert Intention Locks a Gap Lock in above statements).

The reason of why we don't upgrage Record Lock to Next Key Lock is the
following.

Imagine a transaction which does something like this:

for each row {
    request lock in LOCK_X|LOCK_REC_NOT_GAP mode
    request lock in LOCK_S mode
}

If we upgraded lock from Record Lock to Next Key lock, there would be
created only two lock_t structs for each page, one for
LOCK_X|LOCK_REC_NOT_GAP mode and one for LOCK_S mode, and then used
their bitmaps to mark all records from the same page.

The situation would look like this:

request lock in LOCK_X|LOCK_REC_NOT_GAP mode on row 1:
// -> creates new lock_t for LOCK_X|LOCK_REC_NOT_GAP mode and sets bit for
// 1
request lock in LOCK_S mode on row 1:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 1,
// so it upgrades it to X
request lock in LOCK_X|LOCK_REC_NOT_GAP mode on row 2:
// -> creates a new lock_t for LOCK_X|LOCK_REC_NOT_GAP mode (because we
// don't have any after we've upgraded!) and sets bit for 2
request lock in LOCK_S mode on row 2:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 2,
// so it upgrades it to X
    ...etc...etc..

Each iteration of the loop creates a new lock_t struct, and in the end we
have a lot (one for each record!) of LOCK_X locks, each with single bit
set in the bitmap. Soon we run out of space for lock_t structs.

If we create LOCK_GAP instead of lock upgrading, the above scenario works
like the following:

// -> creates new lock_t for LOCK_X|LOCK_REC_NOT_GAP mode and sets bit for
// 1
request lock in LOCK_S mode on row 1:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 1,
// so it creates LOCK_S|LOCK_GAP only and sets bit for 1
request lock in LOCK_X|LOCK_REC_NOT_GAP mode on row 2:
// -> reuses the lock_t for LOCK_X|LOCK_REC_NOT_GAP by setting bit for 2
request lock in LOCK_S mode on row 2:
// -> notices that we already have LOCK_X|LOCK_REC_NOT_GAP on the row 2,
// so it reuses LOCK_S|LOCK_GAP setting bit for 2

In the end we have just two locks per page, one for each mode:
LOCK_X|LOCK_REC_NOT_GAP and LOCK_S|LOCK_GAP.
Another benefit of this solution is that it avoids not-entirely
const-correct, (and otherwise looking risky) "upgrading".

The fix was ported from
mysql/mysql-server@bfba840dfa7794b988c59c94658920dbe556075d
mysql/mysql-server@75cefdb1f73b8f8ac8e22b10dfb5073adbdfdfb0

Reviewed by: Marko Mäkelä

Author: midenok
Author Date: 2023-07-03 19:51:03 UTC

Cleanups

Author: Yuchen Pei
Author Date: 2023-07-03 02:04:46 UTC

MDEV-26137 force flush?

Author: Yuchen Pei
Author Date: 2023-06-23 07:10:01 UTC

MDEV-31524 Fixing spider table param / variable overriding

The existing (incorrect) overriding mechanism is:

Non-minus-one var value overrides table param overrides default value.

Before MDEV-27169, unspecified var value is -1. So if the user sets
both the var to be a value other than -1 and the table param, the var
value will prevail, which is incorrect.

After MDEV-27169, unspecified var value is default value. So if the
user does not set the var but sets the table param, the default value
will prevail, which is even more incorrect.

This patch fixes it so that table param, if specified, always
overrides var value, and the latter if not specified or set to -1,
falls back to the default value

We achieve this by replacing all such overriding in spd_param.cc with
macros that override in the correct way, and removing all the
"overriding -1" lines involving table params in
spider_set_connect_info_default() except for those table params not
defined as sysvar/thdvar in spd_params.cc

We also introduced macros for non-overriding sysvar and thdvar, so
that the code is cleaner and less error-prone

In server versions where MDEV-27169 has not been applied, we also
backport the patch, that is, replacing -1 default values with real
default values

In server versions where MDEV-28006 has not been applied, we do the
same for udf params

Author: Rex Johnston
Author Date: 2023-06-27 04:51:10 UTC

MDEV-28622 Item_subselect eliminated flag set but Item still evaluated/used.

Subquery elimination by optimizer not taken into account elsewhere.

Author: Sergey Petrunia
Author Date: 2023-06-29 08:48:04 UTC

MDEV-31577: Make ANALYZE FORMAT=JSON print innodb stats

First implementation.

ANALYZE FORMAT=JSON_EXT now has table.r_engine_stats which has the
engine statistics. Only non-zero members are printed.

Internally: EXPLAIN data structure Explain_table_acccess now has a
pointer to the handler object. It will read the stats when producing
JSON output.
(TODO: are we ready to commit to the "EXPLAIN data structure requires
that tables used in the query still exist when output is printed") ?

Author: Yuchen Pei
Author Date: 2023-06-29 08:18:57 UTC

MDEV-31524 Further fixes of the params.

ha_part.test hangs at spider_copy_tables...

Author: Marko Mäkelä
Author Date: 2023-06-29 05:45:48 UTC

fixup! 056c83cc8f9551aa861dd9a3327584c8790562cd

Author: Yuchen Pei
Author Date: 2023-06-28 04:25:53 UTC

MDEV-31421 Fix spider test cleanup

This fixes mdev_26541.test, and the new clean_up_spider.inc will be
useful for other tests where part of deinit_spider does not apply,
e.g. those testing spider initialisation only.

Author: Yuchen Pei
Author Date: 2023-06-13 09:56:51 UTC

MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs

Fix spider init bugs (MDEV-22979, MDEV-27233, MDEV-28218) while
preventing regression on old ones (MDEV-30370, MDEV-29904)

Two things are changed:

First, Spider initialisation is made fully synchronous, i.e. it no
longer happens in a background thread. Adapted from the original fix
by nayuta for MDEV-27233. This change itself would cause failure when
spider is initialised early, by plugin-load-add, due to dependency on
Aria and udf function creation, which are fixed in the second and
third parts below. Requires SQL Service, thus porting earlier versions
requires MDEV-27595

Second, if spider is initialised before udf_init(), create udf by
inserting into `mysql.func`, otherwise do it by `CREATE FUNCTION` as
usual. This change may be generalised in MDEV-31401.

Also factor out some clean-up queries from deinit_spider.inc for use
of spider init tests.

A minor caveat is that early spider initialisation will fail if the
server is bootstrapped for the first time, due to missing `mysql`
database which needs to be created by the bootstrap script.

Author: Yuchen Pei
Author Date: 2023-06-28 04:25:53 UTC

MDEV-31421 Fix spider test cleanup

This fixes mdev_26541.test, and the new clean_up_spider.inc will be
useful for other tests where part of deinit_spider does not apply,
e.g. those testing spider initialisation only.

Author: Yuchen Pei
Author Date: 2023-06-13 09:56:51 UTC

MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs

Fix spider init bugs (MDEV-22979, MDEV-27233, MDEV-28218) while
preventing regression on old ones (MDEV-30370, MDEV-29904)

Two things are changed:

First, Spider initialisation is made fully synchronous, i.e. it no
longer happens in a background thread. Adapted from the original fix
by nayuta for MDEV-27233. This change itself would cause failure when
spider is initialised early, by plugin-load-add, due to dependency on
Aria and udf function creation, which are fixed in the second and
third parts below. Requires SQL Service, thus porting earlier versions
requires MDEV-27595

Second, if spider is initialised before udf_init(), create udf by
inserting into `mysql.func`, otherwise do it by `CREATE FUNCTION` as
usual. This change may be generalised in MDEV-31401.

Also factor out some clean-up queries from deinit_spider.inc for use
of spider init tests.

A minor caveat is that early spider initialisation will fail if the
server is bootstrapped for the first time, due to missing `mysql`
database which needs to be created by the bootstrap script.

Author: Yuchen Pei
Author Date: 2023-06-13 09:56:51 UTC

MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs

Fix spider init bugs (MDEV-22979, MDEV-27233, MDEV-28218) while
preventing regression on old ones (MDEV-30370, MDEV-29904)

Two things are changed:

First, Spider initialisation is made fully synchronous, i.e. it no
longer happens in a background thread. Adapted from the original fix
by nayuta for MDEV-27233. This change itself would cause failure when
spider is initialised early, by plugin-load-add, due to dependency on
Aria and udf function creation, which are fixed in the second and
third parts below. Requires SQL Service, thus porting earlier versions
requires MDEV-27595

Second, if spider is initialised before udf_init(), create udf by
inserting into `mysql.func`, otherwise do it by `CREATE FUNCTION` as
usual. This change may be generalised in MDEV-31401.

Also factor out some clean-up queries from deinit_spider.inc for use
of spider init tests.

A minor caveat is that early spider initialisation will fail if the
server is bootstrapped for the first time, due to missing `mysql`
database which needs to be created by the bootstrap script.

Author: Yuchen Pei
Author Date: 2023-06-28 04:25:53 UTC

MDEV-31421 Fix spider test cleanup

This fixes mdev_26541.test, and the new clean_up_spider.inc will be
useful for other tests where part of deinit_spider does not apply,
e.g. those testing spider initialisation only.

Author: Yuchen Pei
Author Date: 2023-06-13 09:56:51 UTC

MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs

Fix spider init bugs (MDEV-22979, MDEV-27233, MDEV-28218) while
preventing regression on old ones (MDEV-30370, MDEV-29904)

Two things are changed:

First, Spider initialisation is made fully synchronous, i.e. it no
longer happens in a background thread. Adapted from the original fix
by nayuta for MDEV-27233. This change itself would cause failure when
spider is initialised early, by plugin-load-add, due to dependency on
Aria and udf function creation, which are fixed in the second and
third parts below. Requires SQL Service, thus porting earlier versions
requires MDEV-27595

Second, if spider is initialised before udf_init(), create udf by
inserting into `mysql.func`, otherwise do it by `CREATE FUNCTION` as
usual. This change may be generalised in MDEV-31401.

Also factor out some clean-up queries from deinit_spider.inc for use
of spider init tests.

A minor caveat is that early spider initialisation will fail if the
server is bootstrapped for the first time, due to missing `mysql`
database which needs to be created by the bootstrap script.

Author: Yuchen Pei
Author Date: 2023-06-28 04:25:53 UTC

MDEV-31421 Fix spider test cleanup

This fixes mdev_26541.test, and the new clean_up_spider.inc will be
useful for other tests where part of deinit_spider does not apply,
e.g. those testing spider initialisation only.

Author: Yuchen Pei
Author Date: 2023-06-28 04:25:53 UTC

MDEV-31421 Fix spider test cleanup

This fixes mdev_26541.test, and the new clean_up_spider.inc will be
useful for other tests where part of deinit_spider does not apply,
e.g. those testing spider initialisation only.

Author: Yuchen Pei
Author Date: 2023-06-28 04:25:53 UTC

MDEV-31421 Fix spider test cleanup

This fixes mdev_26541.test, and the new clean_up_spider.inc will be
useful for other tests where part of deinit_spider does not apply,
e.g. those testing spider initialisation only.

Author: Yuchen Pei
Author Date: 2023-06-28 04:25:53 UTC

MDEV-31421 Fix spider test cleanup

This fixes mdev_26541.test, and the new clean_up_spider.inc will be
useful for other tests where part of deinit_spider does not apply,
e.g. those testing spider initialisation only.

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func

spider_db_mbase_util::open_item_func() is a monster function.
It is difficult to maintain while it is expected that we need to
modify it when a new SQL function or a new func_type is added.

We split the function into two distinct functions: one handles the
case of str != NULL and the other handles the case of str == NULL.

This refactoring was done in a conservative way because we do not
have comprehensive tests on the function.

It also fixes MDEV-29447 and MDEV-31338 where field items that are
arguments of a func item may be used before created / initialised.

Note this commit is adapted from a patch by Nayuta for MDEV-26285.

Author: Yuchen Pei
Author Date: 2023-06-28 02:22:56 UTC

MDEV-30542 Fixing spider/bugfix.self_reference_multi

The server needs to have a unique name

Author: Yuchen Pei
Author Date: 2023-06-08 07:35:14 UTC

MDEV-26137 Improve import tablespace workflow.

Allow ALTER TABLE ... IMPORT TABLESPACE without creating the table
followed by discarding the tablespace.

That is, assuming we want to import table t1 to t2, instead of

CREATE TABLE t2 LIKE t1;
ALTER TABLE t2 DISCARD TABLESPACE;
FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We can simply do

FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.frm $MYSQLD_DATADIR/test/t2.frm
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We achieve this by creating a "stub" table in the second scenario
while opening the table, where t2 does not exist but needs to import
from t1. The "stub" table is similar to a table that is created but
then instructed to discard its tablespace.

We include tests with various row formats, encryption, with indexes
and auto-increment.

Signed-off-by: Yuchen Pei <ycp@mariadb.com>

Author: Yuchen Pei
Author Date: 2023-06-13 09:56:51 UTC

MDEV-22979 MDEV-27233 MDEV-28218 Fixing spider init bugs

Fix spider init bugs (MDEV-22979, MDEV-27233, MDEV-28218) while
preventing regression on old ones (MDEV-30370, MDEV-29904)

Two things are changed:

First, Spider initialisation is made fully synchronous, i.e. it no
longer happens in a background thread. Adapted from the original fix
by nayuta for MDEV-27233. This change itself would cause failure when
spider is initialised early, by plugin-load-add, due to dependency on
Aria and udf function creation, which are fixed in the second and
third parts below. Requires SQL Service, thus porting earlier versions
requires MDEV-27595

Second, if spider is initialised before udf_init(), create udf by
inserting into `mysql.func`, otherwise do it by `CREATE FUNCTION` as
usual. This change may be generalised in MDEV-31401.

Also factor out some clean-up queries from deinit_spider.inc for use
of spider init tests.

A minor caveat is that early spider initialisation will fail if the
server is bootstrapped for the first time, due to missing `mysql`
database which needs to be created by the bootstrap script.

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func

spider_db_mbase_util::open_item_func() is a monster function.
It is difficult to maintain while it is expected that we need to
modify it when a new SQL function or a new func_type is added.

We split the function into two distinct functions: one handles the
case of str != NULL and the other handles the case of str == NULL.

This refactoring was done in a conservative way because we do not
have comprehensive tests on the function.

It also fixes MDEV-29447 and MDEV-31338 where field items that are
arguments of a func item may be used before created / initialised.

Note this commit is adapted from a patch by Nayuta for MDEV-26285.

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func

spider_db_mbase_util::open_item_func() is a monster function.
It is difficult to maintain while it is expected that we need to
modify it when a new SQL function or a new func_type is added.

We split the function into two distinct functions: one handles the
case of str != NULL and the other handles the case of str == NULL.

This refactoring was done in a conservative way because we do not
have comprehensive tests on the function.

It also fixes MDEV-29447 and MDEV-31338 where field items that are
arguments of a func item may be used before created / initialised.

Note this commit is adapted from a patch by Nayuta for MDEV-26285.

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func

spider_db_mbase_util::open_item_func() is a monster function.
It is difficult to maintain while it is expected that we need to
modify it when a new SQL function or a new func_type is added.

We split the function into two distinct functions: one handles the
case of str != NULL and the other handles the case of str == NULL.

This refactoring was done in a conservative way because we do not
have comprehensive tests on the function.

It also fixes MDEV-29447 and MDEV-31338 where field items that are
arguments of a func item may be used before created / initialised.

Note this commit is adapted from a patch by Nayuta for MDEV-26285.

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func

spider_db_mbase_util::open_item_func() is a monster function.
It is difficult to maintain while it is expected that we need to
modify it when a new SQL function or a new func_type is added.

We split the function into two distinct functions: one handles the
case of str != NULL and the other handles the case of str == NULL.

This refactoring was done in a conservative way because we do not
have comprehensive tests on the function.

It also fixes MDEV-29447 and MDEV-31338 where field items that are
arguments of a func item may be used before created / initialised.

Note this commit is adapted from a patch by Nayuta for MDEV-26285.

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func

spider_db_mbase_util::open_item_func() is a monster function.
It is difficult to maintain while it is expected that we need to
modify it when a new SQL function or a new func_type is added.

We split the function into two distinct functions: one handles the
case of str != NULL and the other handles the case of str == NULL.

This refactoring was done in a conservative way because we do not
have comprehensive tests on the function.

It also fixes MDEV-29447 and MDEV-31338 where field items that are
arguments of a func item may be used before created / initialised.

Note this commit is adapted from a patch by Nayuta for MDEV-26285.

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func

spider_db_mbase_util::open_item_func() is a monster function.
It is difficult to maintain while it is expected that we need to
modify it when a new SQL function or a new func_type is added.

We split the function into two distinct functions: one handles the
case of str != NULL and the other handles the case of str == NULL.

This refactoring was done in a conservative way because we do not
have comprehensive tests on the function.

It also fixes MDEV-29447 and MDEV-31338 where field items that are
arguments of a func item may be used before created / initialised.

Note this commit is adapted from a patch by Nayuta for MDEV-26285.

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func

spider_db_mbase_util::open_item_func() is a monster function.
It is difficult to maintain while it is expected that we need to
modify it when a new SQL function or a new func_type is added.

We split the function into two distinct functions: one handles the
case of str != NULL and the other handles the case of str == NULL.

This refactoring was done in a conservative way because we do not
have comprehensive tests on the function.

It also fixes MDEV-29447 and MDEV-31338 where field items that are
arguments of a func item may be used before created / initialised.

Note this commit is adapted from a patch by Nayuta for MDEV-26285.

Author: midenok
Author Date: 2023-06-22 12:58:43 UTC

Test: definite binlog positions

Author: Nikita Malyavin
Author Date: 2023-06-20 12:33:19 UTC

MDEV-27293 preview

Author: Rucha Deodhar
Author Date: 2023-06-20 11:25:38 UTC

MDEV-31477: Inconsistent handling while fetching values in json

Analysis:
JSON_KEY_VALUE() uses Json_path_extractor::extract() to reach to path and
extract values. One error it only returns true, but no error.
Fix:
report error when there is error in scanning json document.

Author: Rucha Deodhar
Author Date: 2023-06-20 10:51:57 UTC

MDEV-31477: Inconsistent handling while fetching values in json

Analysis:
When we get value in json_value(), if any error occurs while scanning the
json, it is not reported. Only true is returned.
Fix:
Report error along with returning true (error)

Author: Alexey Botchkov
Author Date: 2023-06-20 10:20:25 UTC

MDEV-22168 Supporting multiple engines with table partitioning

Author: Rucha Deodhar
Author Date: 2023-06-16 11:17:50 UTC

MDEV-31477: The JSON_KEY_VALUE function should print an error message
if json_doc is passed as a variable

Analysis:
JSON_KEY_VALUE() uses Json_path_extractor::extract() to reach to path and
extract values. One error it only returns true, but no error.
Fix:
report error when there is error in scanning json document.

Author: Sergei Golubchik
Author Date: 2023-06-19 14:20:21 UTC

MDEV-31474 KDF() function

KDF(key_str, salt [, {info | iterations} [, kdf_name [, width ]]])

kdf_name is hkdf (default) or pbkdf2_hmac.
width (in bits) can be any number divisible by 8,
by default it's taken from @@block_encryption_mode

Author: Yuchen Pei
Author Date: 2023-06-08 07:35:14 UTC

MDEV-26137 Improve import tablespace workflow.

Allow ALTER TABLE ... IMPORT TABLESPACE without creating the table
followed by discarding the tablespace.

That is, assuming we want to import table t1 to t2, instead of

CREATE TABLE t2 LIKE t1;
ALTER TABLE t2 DISCARD TABLESPACE;
FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We can simply do

FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.frm $MYSQLD_DATADIR/test/t2.frm
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We achieve this by creating a "stub" table in the second scenario
while opening the table, where t2 does not exist but needs to import
from t1. The "stub" table is similar to a table that is created but
then instructed to discard its tablespace.

We include tests with various row formats, encryption, with indexes
and auto-increment.

Signed-off-by: Yuchen Pei <ycp@mariadb.com>

Author: Vlad Lesin
Author Date: 2023-06-16 09:52:47 UTC

Two system variables are added to manage gcore dump location and debug it.

gcore_dump_dir - the directory, to which gcore dump files are generated,
                 by default they will be generated into data directory.

generate_gcore_dump - boolean variable for debug purpose to test gcore
                      dump generating functionality, generates gcore dump
                      on its value update.

Author: Vlad Lesin
Author Date: 2023-06-16 09:52:47 UTC

Two system variables are added to manage gcore dump location and debug it.

gcore_dump_dir - the directory, to which gcore dump files are generated,
                 by default they will be generated into data directory.

generate_gcore_dump - boolean variable for debug purpose to test gcore
                      dump generating functionality, generates gcore dump
                      on its value update.

Author: Sergey Petrunia
Author Date: 2023-06-19 08:27:51 UTC

Make the new tests --view-protocol friendly. Still don't pass.

Author: Rucha Deodhar
Author Date: 2023-05-25 10:15:43 UTC

MDEV-30145: JSON_TABLE: allow to retrieve the key when iterating on JSON
objects

Idea behind implementation:
We get the json object specified by the json path. Then, transform it into
key-value pairs by going over the json. Get each key-value pair
one-by-one and return the result.

Author: Alexander Barkov
Author Date: 2023-05-24 05:18:14 UTC

Basic support for timestamps with time zone.

Author: Daniel Black
Author Date: 2023-06-15 05:18:40 UTC

MDEV-27038 Custom configuration file procedure does not work with Docker Desktop for Windows 10+

Docker when mounting a configuration file into a Windows exposes the
file with permission 0777. These world writable files are ignored by
by MariaDB.

Add the access check such that filesystem RO or immutable file is
counted as sufficient protection on the file.

Test:
$ mkdir /tmp/src
$ vi /tmp/src/my.cnf
$ chmod 666 /tmp/src/my.cnf
$ mkdir /tmp/dst
$ sudo mount --bind /tmp/src /tmp/dst -o ro
$ ls -la /tmp/dst
total 4
drwxr-xr-x. 2 dan dan 60 Jun 15 15:12 .
drwxrwxrwt. 25 root root 660 Jun 15 15:13 ..
-rw-rw-rw-. 1 dan dan 10 Jun 15 15:12 my.cnf
$ mount | grep dst
tmpfs on /tmp/dst type tmpfs (ro,seclabel,nr_inodes=1048576,inode64)

strace client/mariadb --defaults-file=/tmp/dst/my.cnf

newfstatat(AT_FDCWD, "/tmp/dst/my.cnf", {st_mode=S_IFREG|0666, st_size=10, ...}, 0) = 0
access("/tmp/dst/my.cnf", W_OK) = -1 EROFS (Read-only file system)
openat(AT_FDCWD, "/tmp/dst/my.cnf", O_RDONLY|O_CLOEXEC) = 3

The one failing test, but this isn't a regression, just not a total fix:

$ chmod u-w /tmp/src/my.cnf
$ ls -la /tmp/src/my.cnf
-r--rw-rw-. 1 dan dan 18 Jun 16 10:22 /tmp/src/my.cnf
$ strace -fe trace=access client/mariadb --defaults-file=/tmp/dst/my.cnf
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
access("/etc/system-fips", F_OK) = -1 ENOENT (No such file or directory)
access("/tmp/dst/my.cnf", W_OK) = -1 EACCES (Permission denied)
Warning: World-writable config file '/tmp/dst/my.cnf' is ignored

Windows test (Docker Desktop ~4.21) which was the important one to fix:

dan@LAPTOP-5B5P7RCK:~$ docker run --rm -v /mnt/c/Users/danie/Desktop/conf:/etc/mysql/conf.d/:ro -e MARIADB_ROOT_PASSWORD=bob quay.io/m
ariadb-foundation/mariadb-devel:10.4-MDEV-27038-ro-mounts-pkgtest ls -la /etc/mysql/conf.d
total 4
drwxrwxrwx 1 root root 512 Jun 15 13:57 .
drwxr-xr-x 4 root root 4096 Jun 15 07:32 ..
-rwxrwxrwx 1 root root 43 Jun 15 13:56 myapp.cnf

root@a59b38b45af1:/# strace -fe trace=access mariadb
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
access("/etc/mysql/conf.d/myapp.cnf", W_OK) = -1 EROFS (Read-only file system)

Author: Yuchen Pei
Author Date: 2023-05-19 10:06:31 UTC

MDEV-22534 Decorrelate IN subquery

Transform

in (select inner_col' from inner_table where inner_col = outer_col)

to

, outer_col in (select inner_col', inner_col from inner_table)

Achieved by implementing Item_in_subselect::exists2in_processor(),
accompanied with comprehensive test coverage. Factored out common code
between the two transformations.

Caveat:

- Cannot recognise bad item mismatch in equalities that causes
  materialization to not materialize down the road