lp:maria

Author: midenok
Author Date: 2023-11-15 16:28:07 UTC

MDEV-22164 without validation for exchange partition/convert in

1. WITHOUT/WITH VALIDATION may be added to EXCHANGE PARTITION or CONVERT TABLE:

  alter table tp exchange partition p1 with table t with validation;
  alter table tp exchange partition p1 with table t; -- same as with validation
  alter table tp exchange partition p1 with table t without validation;

2. Optional THAN keyword for RANGE partitioning. Normally you type:

  create table tp (a int primary key) partition by range (a) (
    partition p0 values less than (100),
    partition p1 values less than maxvalue);

  Now you may type (PARTITION keyword is also optional):

  create table tp (a int primary key) partition by range (a) (
    p0 values less (100),
    p1 values less maxvalue);

Author: Lena Startseva
Author Date: 2023-11-14 12:03:16 UTC

MDEV-32177: Exclude double sorting of results for ps2-protocol

Author: Sergei Golubchik
Author Date: 2023-11-04 09:11:51 UTC

don't disable lto in DEB builds

Author: Yuchen Pei
Author Date: 2023-11-15 07:19:53 UTC

MDEV-27254 Delete spider_crd_type and spider_crd_weight

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-11-14 17:29:53 UTC

Merge branch 'nn-11.2-merge' into bb-11.3-release

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-11-14 17:22:29 UTC

Merge branch '11.2' into bb-11.2-release

Author: midenok
Author Date: 2023-11-14 10:39:48 UTC

MDEV-29872 MSAN/Valgrind uninitialised value errors in TABLE::vers_switch_partition

Delayed_insert has its own THD (initialized at mysql_insert()) and
hence its own LEX. Delayed_insert initalizes a very few parameters for
LEX and 'duplicates' is not in this list. Now we copy this missing
parameter from parser LEX (as well as sql_command).

Author: Kristian Nielsen
Author Date: 2023-11-14 14:00:41 UTC

MDEV-32673 Handle InnoDB binlog position recovery for RESET MASTER or binlog name change

Another attempt, to do most of what Marko's solution does without
requiring a new handlerton call: Compare each binlog position against
the previous one, and reset all old entries if the position moves
backwards.

Does not work though, it asserts in multiple places about pages not
marked as modified (IIUC) during the reset of commit processing. Also,
even if the mark-as-modified could be added, I'n not sure what the
consequences will be of trying to modify all rseg headers in the final
mtr during commit-to-memory. How would the locking work, if multiple
threads would try to do this simulaneosly? (though that might not be
possible due to LOCK_commit_ordered). Or could there be other locking
problems?

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Anel Husakovic
Author Date: 2023-10-05 14:06:27 UTC

MDEV-32168: 10.10

Author: Anel Husakovic
Author Date: 2023-10-05 13:57:03 UTC

MDEV-32168: 10.9 reviewer

Author: Anel Husakovic
Author Date: 2023-11-13 10:00:22 UTC

MDEV-32168: 10.6 reviewer 1

Author: Anel Husakovic
Author Date: 2023-10-05 13:26:01 UTC

MDEV-32168: 10.5 reviewr

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-11-14 07:35:10 UTC

MDEV-32242 innodb.doublewrite test case always gets skipped

- Rewrote the doublewrite test case and removes the usage of
innodb_buf_flush_list variable, added search pattern to make
sure that InnoDB uses doublewrite buffer. Replaced all
kill_mysqld.inc with shutdown_mysqld.inc and zero shutdown timeout

Author: Yuchen Pei
Author Date: 2023-11-14 01:48:52 UTC

MDEV-27576 Use reverse index for max/min optimization

We use a bool to indicate that the key part used is a descending
index, which will flip the functions and flags used in
get_index_max_value() and get_index_min_value(), that allows correct
optimization for max/min for descending index.

Author: Dmitry Shulga
Author Date: 2023-11-13 12:18:51 UTC

MDEV-32788: Building of server fails on MacOS on compilation of trx0purge.cc

The method srw_lock_impl of the template class srw_lock_impl
is conditionally defined under #ifndef SUX_LOCK_GENERIC but
called unconditionally that could result in compilation failure

Author: Rucha Deodhar
Author Date: 2023-11-10 10:31:15 UTC

MDEV-32726: Fix failing test fir freebsd for json

Json test about max statement time fails with freebsd because on some
architectures the test might execute faster and the statement may not fail.

To simulate failure regardless of architecture, introduce a wait of seconds
longer than the max_statement_time.

Author: midenok
Author Date: 2023-11-09 13:26:11 UTC

MDEV-29932 Invalid expr in cleanup_session_expr() upon INSERT DELAYED

There are two TABLE objects in each thread: first one is created in
delayed thread by Delayed_insert::open_and_lock_table(), second one is
created in connection thread by Delayed_insert::get_local_table(). It
is copied from the delayed thread table.

When the second table is copied copy-assignment operator copies
vcol_refix_list which is already filled with an item from delayed
thread. Then get_local_table() adds its own item. Thus both tables
contains the same list with two items which is wrong. Then connection
thread finishes and its item freed. Then delayed thread tries to
access it in vcol_cleanup_expr().

The fix just clears vcol_refix_list in the copied table.

Another problem is that copied table contains the same mem_root, any
allocations on it will be invalid if the original table is freed (and
that is indeterministic as it is done in another thread). Since copied
table is allocated in connection THD and lives not longer than
thd->mem_root we may assign its mem_root from thd->mem_root.

Third, it doesn't make sense to do open_and_lock_tables() on NULL
pointer.

Author: Yuchen Pei
Author Date: 2023-11-10 05:40:45 UTC

[demo] mysql.spider_table_sts is empty, but it returns something when read

Author: Andrei
Author Date: 2023-11-09 12:36:46 UTC

MDEV-32365: post-fixes to rpl_semi_sync_slave_reply_fail

Author: Yuchen Pei
Author Date: 2023-11-09 06:42:49 UTC

MDEV-28739 [demo] Commenting out spider sts and crd table updates fixes the problem

see https://jira.mariadb.org/browse/MDEV-28739?focusedCommentId=273704&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-273704

Author: Yuchen Pei
Author Date: 2023-11-09 00:43:27 UTC

MDEV-27576 Use reverse index for max/min optimization

We use a bool to indicate that the key part used is a descending
index, which will flip the functions and flags used in
get_index_max_value() and get_index_min_value(), that allows correct
optimization for max/min for descending index.

Author: Brandon Nesterenko
Author Date: 2023-11-08 23:57:58 UTC

MDEV-32551: ‘Read semi-sync reply magic number error’ warnings on master

A test to show how to induce the ‘Read semi-sync reply magic number
error’ message on a primary. In short, if semi-sync is turned on
during the hand-shake process between a primary and replica, but
later a user negates the rpl_semi_sync_slave_enabled variable while
the replica’s IO thread is running; if the io thread exits, the
replica can skip a necessary call to kill_connection() in
repl_semisync_slave.slave_stop() due to its reliance on a global
variable. Then, the replica will send a COM_QUIT packet to the
primary on an active semi-sync connection, causing the magic number
error.

The test in this patch exits the IO thread by forcing an error;
though note a call to STOP SLAVE could also do this, but it ends up
needing more synchronization. That is, the STOP SLAVE command also
tries to kill the vio of the replica, which makes a race with the IO
thread to try and send the COM_QUIT before this happens (which would
need more debug_sync to get around). See THD::awake_no_mutex for
details as to the killing of the replica’s vio.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-11-08 16:59:16 UTC

Merge branch 'nn-11.0-merge' into bb-11.1-release

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-11-08 15:49:48 UTC

Merge branch 'nn-10.11-merge' into bb-11.0-release

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-11-08 15:25:40 UTC

Merge branch 'nn-10.10-merge' into nn-10.11-merge

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-11-08 15:22:42 UTC

Merge branch '10.6' into 10.10

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-11-08 15:22:42 UTC

Merge branch '10.6' into 10.10

Author: Marko Mäkelä
Author Date: 2023-11-08 15:22:15 UTC

MDEV-32673 Handle InnoDB binlog position recovery for RESET MASTER or binlog name change

ha_reset_binlog_pos(), trx_rseg_reset_binlog_pos(): Reset the
binlog file name and position to the specified one.
Store it in the first rollback segment (in the system tablespace),
and invalidate all other rollback segments as well as the
legacy information in the TRX_SYS page (from before
commit 947efe17ed8188ca4feef6deb0c2831a246b5c8f).

MYSQL_BIN_LOG::reset_logs(), init_server_components():
Invoke ha_reset_binlog_pos().

The test case was written by and the code co-developed with
Kristian Nielsen.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-11-08 14:59:59 UTC

Merge branch 'nn-10.6-merge' into nn-10.10-merge

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-11-08 14:39:10 UTC

Merge branch 'nn-10.6-start' into nn-10.6-merge

Author: VladislavVaintroub
Author Date: 2023-09-29 10:37:10 UTC

MDEV-32189 Use icu for timezones on windows

Use ICU to work with timezones, to retrieve current timezone name,
abbreviation, and offset from GMT. However in case TZ environment variable
is used to set timezone, and ICU does not have corresponding one,
C runtime functions will be used.

Moved some of timezone handling to mysys.
Added unit tests.

Author: Yuchen Pei
Author Date: 2023-11-08 02:10:38 UTC

MDEV-32559 ensure spider alter table init queries are executed after ddl recovery

If spider was initialised in init_server_components(), i.e. with
--plugin-load-add, then all alter table init queries are to be
executed in the signal_ddl_recovery_done() handlerton callback. Since
this is part of the spider plugin init, we update the callback
signature to return an int, and deinit the plugin when it fails the
callback.

Author: Sergey Petrunia
Author Date: 2023-11-07 09:20:02 UTC

MDEV-32682: Assertion `range->rows >= s->found_records' failed in best_access_path

Fix the issue introduced in ec2574fd8f, fix for MDEV-31983:

get_quick_record_count() must set quick_count=0 when it got
IMPOSSIBLE_RANGE from test_quick_select.

Failure to do so will cause an assertion in 11.0, when the number of
quick select rows (0) is checked to be lower than the number of
found_records (which is capped up to 1).

Author: Julius Goryavsky
Author Date: 2023-11-06 13:15:23 UTC

galera: wsrep-lib submodule update

Author: Sergei Golubchik
Author Date: 2023-11-04 09:11:51 UTC

enable lto

Author: Yuchen Pei
Author Date: 2023-11-03 07:00:51 UTC

MDEV-27575 Remove thd from spider_db_done

It is unused, and causing segfaults

Author: Yuchen Pei
Author Date: 2023-11-03 00:58:58 UTC

MDEV-32559 [demo] Add a bad spider init query to show failure in ddl_recovery_done causes deinit of the plugin

See the test spider/bugfix.plugin_load_add_all

Author: Rucha Deodhar
Author Date: 2023-09-18 19:24:19 UTC

MDEV-31599: Assertion `0' failed in Item_param::can_return_value
from Item::val_json, UBSAN: member access within null pointer of
type 'struct String' in sql/item_jsonfunc.cc

Analysis:
The first argument of json_schema_valid() needs to be a constant.
Fix:
Parse the schema if the item is constant otherwise set it to return null.

Author: Rucha Deodhar
Author Date: 2023-07-04 09:09:57 UTC

MDEV-31411: JSON_ARRAY_INTERSECT/JSON_OBJECT_FILTER_KEYS should fetch
data from a table similar to other JSON functions

Analysis:
Since we are fetching values for every row ( because we are running SELECT
for all rows of a table ), correct value can be only obtained at the time of
calling val_int() because it is called to get value for each row.
Fix:
Set up hash for each row instead of doing it during fixing fields.

Author: Marko Mäkelä
Author Date: 2023-11-02 09:23:23 UTC

MDEV-32374 log_sys.lsn_lock is a performance hog

The log_sys.lsn_lock that was introduced in
commit a635c40648519fd6c3729c9657872a16a0a20821
had better be located in the same cache line with log_sys.latch
so that log_t::append_prepare() needs to modify only two first
cache lines where log_sys is stored.

log_t::lsn_lock: On Linux, change the type from pthread_mutex_t to
something that may be as small as 32 bits, to pack more data members
in the same cache line. On Microsoft Windows, CRITICAL_SECTION works
better.

log_t::check_flush_or_checkpoint_: Renamed to need_checkpoint.
There is no need to pause all writer threads in log_free_check() when
we only need to write log_sys.buf to ib_logfile0. That will be done in
mtr_t::commit().

log_t::append_prepare_wait(): Make the member function non-static
to simplify the call interface, and add a parameter for the LSN.

log_t::append_prepare(): Invoke append_prepare_wait() at most once.
Only set_check_for_checkpoint() if a log checkpoint needs to
be written. If the log buffer needs to be written, we will take care
of it ourselves later in our caller. This will reduce interference
with log_free_check() in other threads.

mtr_t::commit(): Call log_write_up_to() if needed.

log_t::get_write_target(): Return a log_write_up_to() target
to mtr_t::commit().

buf_flush_ahead(): If we are in furious flushing, call
log_sys.set_check_for_checkpoint() so that all writers will wait
in log_free_check() until the checkpoint is done. Otherwise,
the test innodb.insert_into_empty could occasionally report
an error "Crash recovery is broken".

log_check_margins(): Replaced by log_free_check().

log_flush_margin(): Removed. This is part of mtr_t::commit()
and other operations that write log.

log_t::create(), log_t::attach(): Guarantee that buf_free < max_buf_free
will always hold on PMEM, to satisfy an assumption of
log_t::get_write_target().

Author: Yuchen Pei
Author Date: 2023-11-02 02:30:36 UTC

MDEV-22534 [demo] Use Item_ref instead of Item_direct_ref

This passes the infinite ref cycle test.

This also depends on the simplification of the MDEV-31269 workaround
in the parent commit that increases coverage / false positives in
detecting potential 2nd ps segfaults, otherwise MDEV-3906 would
resurface, i.e. the corresponding case in main.subselect.exists2in
fails.

Author: Yuchen Pei
Author Date: 2023-11-02 02:20:31 UTC

MDEV-22534 [demo] MDEV-30073 fixes MDEV-31269 and MDEV-3906

MDEV-31269: run the test mdev_31269
MDEV-3906: replacing Item_direct_ref with Item_ref in
exists2in_processor when AND IS_NOT_NULLS is fine

Author: Kristian Nielsen
Author Date: 2023-10-31 12:54:24 UTC

MDEV-22351: InnoDB may report incorrect binlog position information after RESET MASTER

From MariaDB 10.3.5, the saved binlog position is stored in multiple
entries in each rseg header, and during recovery the most recent of
them is selected as the postion to recover. The comparison between
different entries is by comparing the binlog file name and offset.
This fails if the binlog file name changes to something that compares
smaller, which can happen from RESET MASTER (which goes back to the
suffix .000001) or from renaming the basename of the binlog. As a
result, the wrong position can be recovered after RESET MASTER or
binlog rename.

This commit writes an 8-byte incrementing version number with each
position entry. The version is allocated during
innobase_commit_ordered(), which ensures that a higher version number
will correspond to a more recent commit. The recovered position is
then the entry with the highest version number.

If starting on a database written by earlier server version, the code
falls back to the old methods of comparing position entries.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: Dmitry Shulga
Author Date: 2023-10-31 14:32:09 UTC

MDEV-32517: Memory leak on running query in PS mode with condition that has 'ANY/SOME (subquery)' clause and any kind of the following comparisons '>', '>=', '<', '<=' '

Instantiation of Item_direct_ref_to_item for storing a refence
on Item_func_conv_charset should take place on first execution of
a query in PS mode, else it would lead to a memory leak.

Added extra check for
  current_select->first_cond_optimization
before entering into the block where a new instance of the class
Item_direct_ref_to_item is created on PS memroy_root.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-10-31 09:32:57 UTC

Make it compile with clang

Author: Yuchen Pei
Author Date: 2023-10-06 03:54:40 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Sergey Petrunia
Author Date: 2023-10-30 12:24:37 UTC

Merge bb-10.11-release -> 11.0

Author: Marko Mäkelä
Author Date: 2023-10-30 14:08:01 UTC

Merge 10.6 into 10.10

Author: Luis Lizardo
Author Date: 2023-10-30 13:24:39 UTC

Merge branch '11.3' into bb-11.3-mdev-32568

Author: Yuchen Pei
Author Date: 2023-10-27 08:04:02 UTC

MDEV-28856 Add remaining Spider table options

MDEV-27106 added REMOTE_TABLE, REMOTE_DATABASE, REMOTE_SERVER spider
table options. In this commit, we add all remaining options for table
params that are not marked to be deprecated.

All these options are parsed as strings from sql statements and have
string values at the sql level, so that we can determine whether it is
specified by checking its nullness.

The string values are further parsed by Spider into their actual types
in the SPIDER_SHARE, including string list, bounded nonnegative int,
bounded nonnegative int list, nonnegative longlong, boolean, and key
hints. Except for string lists, all other types are validated during
this parsing process.

Most of the options are backward compatible, i.e. they accept any
values that is accepted by there corresponding param parser. The only
exception is the index hint IDX which corresponds to the idxNNN param
name. For example,

'idx000 "f PRIMARY", idx001 "u k1"'

translates to

IDX="f PRIMARY u k1".

We include a test with all options specified, and tests involving
spider table options of all actual types.

Any table options, if present, will cause comments to be ignored with
a warning. The warning can be disabled by setting a new spider
global/session system variable spider_suppress_comment_ignored_warning
to 1.

Another global/session variable introduced is spider_ignore_comments,
which if set to 1, will cause COMMENT and CONNECTION strings to be
ignored unconditionally, whether or not table options are specified.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-10-26 20:33:00 UTC

columnstore 6.4.7-2

Author: Sergey Petrunia
Author Date: 2023-10-26 17:44:10 UTC

MDEV-32351: Significant slowdown with outer joins: Test coverage

Make ANALYZE FORMAT=JSON print block-nl-join.r_unpack_ops when
analyze_print_r_unpack_ops debug flag is set.

Then, add a testcase.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-10-26 13:59:20 UTC

columnstore-5.6.8-2

Author: Yuchen Pei
Author Date: 2023-10-24 05:07:07 UTC

MDEV-32525 Validate --redirect_url supplied as server flag

Like sql_mode, we factor out of ON_CHECK function for export, to be
used in get_options() during server startup, for validation of
--redirect_url value.

Author: Daniel Black
Author Date: 2023-10-25 07:25:29 UTC

MDEV-24670 avoid OOM by linux kernel co-operative memory management

Author: Kristian Nielsen
Author Date: 2023-10-25 17:09:21 UTC

MDEV-26632: GTID master switch when slave position is filtered on new master

If an intermediate slave S1 has replication filters enabled, its
@@gtid_slave_pos may contain a GTID that is filtered and doesn't propagate
to lower-level slaves with S1 as master.

If then later S1 is demoted to a slave, it may attempt to connect to the
filtered position. This is normally disallowed in --gtid-strict-mode. But if
--gtid-ignore-duplicates is enabled, we should allow it, as in this case we
can trust the GTID sequence numbers between different server ids. So we can
know that the next GTID is the right one for the filtered slave GTID
position.

This allows advanced users to use replication filtering in topologies like
this and still run with --gtid-strict-mode enabled.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Author: VladislavVaintroub
Author Date: 2023-10-24 15:32:28 UTC

MDEV-32537 Name threads to improve debugging experience and diagnostics.

Use SetThreadDescription/pthread_setname_np to give threads a name.

Author: Rex Johnston
Author Date: 2023-09-14 21:44:49 UTC

MDEV-31983 jointable materialization subquery optimization ignoring

...errors, then failing ASSERT.

In optimizing a subselect, transforming it to a join, then attempting to
materialize that join, errors in evaluation of join conditions, flagged in
the thread are ignored for suffiently long to cause problems.

When executing an update SQL command, thd->abort_on_warning can cause
thd->error() to be set during evaluation in get_mm_tree() of attached
conditions. This need to be caught ASAP in test_quick_select(), the
caller of get_mm_tree() and returned as an error condition.

We change the return type of SQL_SELECT::test_quick_select() to allow
unambiguous detection of an error by the caller.

Sergei Petrunia adds:
Add error handling:
- JOIN::make_range_rowid_filters() should return error immediately if
  it fails to build a filter.
- test_if_skip_sort_order() now will return an error if it gets an error
  from test_quick_select() call.
   = All callers check for this condition, too.

Also, added error handling logic: now, a fatal error in
any SQL_SELECT::test_quick_select() call will be caught
by any caller which will abort the query execution.
- "Range checked for each record" code now checks for test_quick_select
  errors, too.

Author: Oleg Smirnov
Author Date: 2023-10-02 05:12:15 UTC

MDEV-28506 Possible item replacement during fix_fields() is not taken into account when pushing conditions from HAVING to WHERE

1. During the 5th stage of pushdown_from_having_into_where() there is
a sequence of cleanup/fix_fields calls for items involved. fix_fields()
may sometimes replace the initial item, which is done, for example, for
Item_func_not. This commits adds the correct handling for such cases.

2. Sometimes fix_fields() calls are made with the 2nd parameter
(Item **ref) set to NULL. This may lead to access violation when the item
is of class Item_func_not or Item_func_sp. This commits add a proper check
before dereferencing **ref.

Author: Monty
Author Date: 2023-10-23 17:24:24 UTC

MDEV-32532 Assertion failure in ddl_log_increment_phase_no_lock upon ..

The reason for the bug was that spider plugin does some ALTER TABLE
commands before recovery is done, which was not expected.

Fixed by resetting the ddl_log lists in create_ddl_log()

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-10-24 07:21:31 UTC

Merge branch '10.5' into bb-10.5-release

Author: Marko Mäkelä
Author Date: 2023-10-24 06:47:46 UTC

MDEV-31953 madvise(..., MADV_FREE) is causing a performance regression

buf_page_t::set_os_unused(): Remove the system call that had been added in
commit 16c9718758cb3bbff76672405d4ce1bce6da6c6f and revised in
commit c1fd082e9c7369f4511eb5a52e58cb15489caa74 for Microsoft Windows.

buf_pool_t::garbage_collect(): A new function to collect any garbage
from the InnoDB buffer pool that can be removed without writing any
log or data files. This will also invoke madvise() or
DiscardVirtualMemory() for all of buf_pool.free.

buf_resize_callback(): Simply invoke buf_pool.resize().

buf_resize_start(): If there is no actual change to
innodb_buffer_pool_size, invoke buf_pool_t::garbage_collect()
instead of starting buf_resize_callback() in another thread.

Author: Sergey Petrunia
Author Date: 2023-10-23 16:02:04 UTC

MDEV-31983, more fixes.

Fix formatting

Add error handling:
- JOIN::make_range_rowid_filters() should return error immediately if
  it fails to build a filter.
- test_if_skip_sort_order() now will return an error if it gets an error
  from test_quick_select() call.
   = All callers check for this condition, too.

- "Range checked for each record" code now checks for test_quick_select
  errors, too.

Author: Vlad Lesin
Author Date: 2023-10-23 13:33:24 UTC

MDEV-32554 mon_lock_wait_current_count test causes innodb.monitor failure

This commit will be applied during 10.5->10.6 merging as a part of
conflicts resolution.

-------------------------
SET GLOBAL innodb_monitor_disable=all;
-------------------------

was set at the the beggining of mon_lock_wait_current_count test,
because lock_row_lock_time_avg is filled under lock_sys.wait_mutex lock,
which is held by connection 2:

--------------------------
--connect (con2,localhost,root,,)
SET DEBUG_SYNC="lock_wait_before_suspend SIGNAL blocked WAIT_FOR cont";
BEGIN;
--send SELECT * FROM t FOR UPDATE
-------------------------

That is why the followin SELECT is blocked:

-----------------------
SELECT name, count FROM information_schema.innodb_metrics
  WHERE name ='lock_row_lock_current_waits';
-----------------------

And setting innodb_monitor_disable=all causes innodb monitor enabling
status changing. And the status is not restored to default value even
after
-----------------------
SET GLOBAL innodb_monitor_reset_all=default;
SET GLOBAL innodb_monitor_enable=default;
-----------------------
execution. See MDEV-32553 for details.

The fix is to disable not all the monitors at the beggining of the test,
but only the monitor, which causes hanging, i.e. lock_row_lock_time_avg.

MDEV-32553 it the root case, the current fix is just workaround.
10.5 is not affected, as there is difference in innodb.monitor.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-23 08:16:29 UTC

MDEV-29092 FOREIGN_KEY_CHECKS does not prevent non-copy alter from creating invalid FK structure

Problem:
========
- InnoDB should have two keys on the same column for the self
referencing foreign key relation. Inplace alter algorithm allows
to drop the index if there is only one key for the self referencing
foreign key and it leads to invalid foreign key table structure.

Solution:
=========
When foreign_key_check is enabled, InnoDB shouldn't allow
to drop index if there is no suitable key for self referencing
foreign key relation.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-22 14:30:01 UTC

MDEV-32523 ASAN errors or assertion failure in row_merge_buf_add

innobase_build_col_map_add(): InnoDB wrongly copies the old field
record into default value even when new field length has been changed.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-22 14:30:01 UTC

MDEV-32523 ASAN errors or assertion failure in row_merge_buf_add

innobase_build_col_map_add(): InnoDB wrongly copies the old field
record into default value even when new field length has been changed.

Author: Julius Goryavsky
Author Date: 2023-10-21 16:15:58 UTC

galera: enabling tests for fixed problems

Author: Yuchen Pei
Author Date: 2023-10-20 07:05:54 UTC

MDEV-30014 Spider should not second guess server when locking / unlocking

This fixes MDEV-30014, MDEV-29456, MDEV-29667, and MDEV-30049.

The server may ask storage engines to unlock when the original sql
command is not UNLOCK. This patch makes sure that spider honours these
requests, so that the server has the correct idea which tables are
locked and which are not.

MDEV-29456, MDEV-29667, MDEV-30049: a later LOCK statement would, as
the first step, unlock locked tables and clear the OPTION_TABLE_LOCK
bit in thd->variables.option_bits, as well as locked_tables_list,
indicating no tables are locked. If Spider does not unlock because the
sql command is not UNLOCK, and if after this the LOCK statement fails
to lock any tables, these indications that no tables are locked
remains, so a later UNLOCK TABLES; statement would not try to unlock
any table. Causing later statements requiring mdl locks to hang on
waiting until lock_wait_timeout (default 1h) has passed.

MDEV-30014: when a LOCK statement tries to lock more than one tables,
say t2 and t3 as in mdev_30014.test, and t2 succeeds but t3 fails, the
sql layer would try to undo by unlocking t2, and again, if Spider does
not honour this request, the sql layer would assume t2 has been
unlocked, but later actions on t2 or t2's remote table could hang on
waiting for the mdl.

Author: Yuchen Pei
Author Date: 2023-10-20 07:03:54 UTC

MDEV-30014 Spider should not second guess server when locking / unlocking

This fixes MDEV-30014, MDEV-29456, MDEV-29667, and MDEV-30049.

The server may ask storage engines to unlock when the original sql
command is not UNLOCK. This patch makes sure that spider honours these
requests, so that the server has the correct idea which tables are
locked and which are not.

MDEV-29456, MDEV-29667, MDEV-30049: a later LOCK statement would, as
the first step, unlock locked tables and clear the OPTION_TABLE_LOCK
bit in thd->variables.option_bits, as well as locked_tables_list,
indicating no tables are locked. If Spider does not unlock because the
sql command is not UNLOCK, and if after this the LOCK statement fails
to lock any tables, these indications that no tables are locked
remains, so a later UNLOCK TABLES; statement would not try to unlock
any table. Causing later statements requiring mdl locks to hang on
waiting until lock_wait_timeout (default 1h) has passed.

MDEV-30014: when a LOCK statement tries to lock more than one tables,
say t2 and t3 as in mdev_30014.test, and t2 succeeds but t3 fails, the
sql layer would try to undo by unlocking t2, and again, if Spider does
not honour this request, the sql layer would assume t2 has been
unlocked, but later actions on t2 or t2's remote table could hang on
waiting for the mdl.

Author: Elena Stepanova
Author Date: 2023-10-19 16:01:13 UTC

Test commit: run feature-related suites on Windows

Author: Yuchen Pei
Author Date: 2023-10-19 04:42:43 UTC

MDEV-32515 [experiment] Add --bootstrap to the $MYSQLD_CMD invocation in mdev_30370

After all, spider/bugfix.mdev_22979 passes, and the only difference
that may matter is the --bootstrap flag

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-19 00:42:55 UTC

MDEV-32453 Bulk insert fails to apply when trigger does insert operation

- InnoDB fails to apply the bulk operation when trigger initiated
another statement on the operated table which is same as
associated table. InnoDB should apply all bulk operation
before ending the bulk insert operation on the operated table.

Author: Yuchen Pei
Author Date: 2023-10-19 00:01:29 UTC

[fixup] Spider fixup of the merge

Fixing the following
- Some commented out code were added in the merge
- Some changes from the following was reverted

b5d317197c6 MDEV-29676 refactored and documented spider_get_share() and friends

Author: Brandon Nesterenko
Author Date: 2023-10-18 14:32:28 UTC

MDEV-31949: rpl_xa_prepare_gtid_fail deterministic paths

happy_xac is where the XA COMMIT completes before noticing the
error signalled by the prior XAP

sad_xac is where the XA COMMIT notices the error signalled by
the prior XAC and rolls back, leaving a dangling XAP.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-10-18 13:27:51 UTC

MDEV-32308 Server crash on cleanup of non-fully-constructed-due-to-an-error CTE

Check and do not try to access With_element::rec_result on cleanup
if it is not assigned.

Author: Brandon Nesterenko
Author Date: 2023-10-17 21:48:03 UTC

MDEV-31949: Fix rpl_xa_prepare_gtid_fail

rpl.rpl_xa_prepare_gtid_fail would sporadically fail, as an XA COMMIT,
running concurrently with a failed prepare, could sometimes complete
succesfully, or be rolled back due to failure of prior commit. The
test expected the completion case, but if it failed, the xa transaction
would be left in a prepared state with a lock on its table. This then
created a lock time-out during test cleanup, as it tries to drop that
table.

The fix is to extend the test to check if the transaction is still
prepared, and silently commit it if so, thus releasing the locks.
The non-determinism is fine (i.e. DEBUG_SYNC isn't needed to force
one path), as the verification performed by the test has already
completed. This is just for cleanup.

Author: Brandon Nesterenko
Author Date: 2023-10-17 21:48:03 UTC

MDEV-31949: Fix rpl_xa_prepare_gtid_fail

rpl.rpl_xa_prepare_gtid_fail would sporadically fail, as an XA COMMIT,
running concurrently with a failed prepare, could sometimes complete
succesfully, or be rolled back due to failure of prior commit. The
test expected the completion case, but if it failed, the xa transaction
would be left in a prepared state with a lock on its table. This then
created a lock time-out during test cleanup, as it tries to drop that
table.

The fix is to extend the test to check if the transaction is still
prepared, and silently commit it if so, thus releasing the locks.
The non-determinism is fine (i.e. DEBUG_SYNC isn't needed to force
one path), as the verification performed by the test has already
completed. This is just for cleanup.

Author: Yuchen Pei
Author Date: 2023-10-18 07:18:24 UTC

MDEV-32486 Fix duplicate spider_bulk_malloc ID

The id 38 is already used elsewhere, we use 44 instead, which hasn't
been used

Author: Yuchen Pei
Author Date: 2023-10-18 07:18:24 UTC

MDEV-32486 Fix duplicate spider_bulk_malloc ID

The id 38 is already used elsewhere, we use 44 instead, which hasn't
been used

Author: Yuchen Pei
Author Date: 2023-10-17 04:02:12 UTC

MDEV-32485 Fix Spider upgrade failure caused by duplication in mysql.func

Author: Igor Babaev
Author Date: 2023-10-18 03:52:34 UTC

MDEV-32351 Significant slowdown for query with many outer joins

This is a preliminary patch. To be used just for testing.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-09-04 11:04:06 UTC

MDEV-29167 new db-level SHOW CREATE ROUTINE privilege

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-10-16 17:43:18 UTC

ibuf_delete_recs() doesn't remove all entries of page id

ibuf_delete_recs() should use the counter field for searching
the tuple when we're reopening the cursor.

Author: VladislavVaintroub
Author Date: 2023-10-17 09:58:20 UTC

Innodb cleanup - log_t::append_prepare

- make the function non-static
- pass lsn to it, and do log_write_up_to to wait for this lsn, rather than
  always wait for the current one.

Author: Yuchen Pei
Author Date: 2023-10-17 07:14:05 UTC

Merge branch '10.10' into 10.11

Author: Yuchen Pei
Author Date: 2023-10-17 04:02:12 UTC

MDEV-32485 Fix Spider upgrade failure caused by duplication in mysql.func

Author: Yuchen Pei
Author Date: 2023-10-11 05:40:30 UTC

MDEV-31996 Create connection on demand in spider_db_delete_all_rows

When spider_db_delete_all_rows() is called, the supplied spider->conns
may have already been freed. The existing mechanism has spider_trx own
the connections in trx_conn_hash and it may free a conn during the
cleanup after a query. When running a delete query and if the table is
in the table cache, ha_spider::open() would not be called which would
recreate the conn. So we recreate the conn when necessary during
delete by calling spider_check_trx_and_get_conn().

We also reduce code duplication as delete_all_rows() and truncate()
has almost identical code, and there's no need to assign
wide_handler->sql_command in these functions because it has already
been correctly assigned.

Author: Christian Hesse
Author Date: 2023-08-09 00:18:36 UTC

MDEV-19210: do not run pre and post scripts as root

Now that we do not pollute systemd's environment but write private
environment files running these as root is not longer required. So
let's drop `PermissionsStartOnly=true`.

Debian adds extra `ExecStartPre=` and `ExecStartPost=`, though.
Use special executable prefix for full privileges there. (See
systemd.service(5) for details.)

Author: Tuukka Pasanen
Author Date: 2023-10-10 06:45:48 UTC

MDEV-30953: Add package for Galera server for RPM

Add new package mariadb-server-galera to RPM build
recude mariadb-server package dependencies.
Galera cluster scripts and libraries are currently
in package mariadb-server. Move Galera cluster
scripts and libraries to new package mariadb-server-galera
as most of the people does not need them and
it allows to slim down mariadb-server package

Author: Elena Stepanova
Author Date: 2023-10-14 16:58:51 UTC

The list is not relevant anymore, bugs are fixed

Author: Sergey Petrunia
Author Date: 2023-10-13 09:17:25 UTC

MDEV-32320: Server crashes at TABLE::add_tmp_key

The code inside Item_subselect::fix_fields() could fail to check
that left expression had an Item_row, like this:

  (('x', 1.0) ,1) IN (SELECT 'x', 1.23 FROM ... UNION ...)

In order to hit the failure, the first SELECT of the subquery had
to be a degenerate no-tables select. In this case, execution will
not enter into Item_in_subselect::create_row_in_to_exists_cond()
and will not check if left_expr is composed of scalars.

But the subquery is a UNION so as a whole it is not degenerate.
We try to create an expression cache for the subquery.
We create a temp.table from left_expr columns. No field is created
for the Item_row. Then, we crash when trying to add an index over a
non-existent field.

Fixed by moving the left_expr cardinality check to a point in
check_and_do_in_subquery_rewrites() which gets executed for all
cases.
It's better to make the check early so we don't have to care about
subquery rewrite code hitting Item_row in left_expr.

Author: Yuchen Pei
Author Date: 2023-10-13 06:16:57 UTC

MDEV-29963 MDEV-31357 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, or simply TRUNCATEd, when executing
LOCK TABLES again, the lock lists would be queried again in
store_lock(), which could cause access to freed space associated with
the dropped table.

Author: Yuchen Pei
Author Date: 2023-10-13 06:16:57 UTC

MDEV-29963 MDEV-31357 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, or simply TRUNCATEd, when executing
LOCK TABLES again, the lock lists would be queried again in
store_lock(), which could cause access to freed space associated with
the dropped table.

Author: Yuchen Pei
Author Date: 2023-09-14 05:19:50 UTC

MDEV-29502 Fix some issues with spider direct aggregate

The direct aggregate mechanism sems to be only intended to work when
otherwise a full table scan query will be executed from the spider
node and the aggregation done at the spider node too. Typically this
happens in sub_select(). In the test spider.direct_aggregate_part
direct aggregate allows to send COUNT statements directly to the data
nodes and adds up the results at the spider node, instead of iterating
over the rows one by one at the spider node.

By contrast, the group by handler (GBH) typically sends aggregated
queries directly to data nodes, in which case DA does not improve the
situation here.

That is why we should fix it by disabling DA when GBH is used.

There are other reasons supporting this change. First, the creation of
GBH results in a call to change_to_use_tmp_fields() (as opposed to
setup_copy_fields()) which causes the spider DA function
spider_db_fetch_for_item_sum_funcs() to work on wrong items. Second,
the spider DA function only calls direct_add() on the items, and the
follow-up add() needs to be called by the sql layer code. In
do_select(), after executing the query with the GBH, it seems that the
required add() would not necessarily be called.

Disabling DA when GBH is used does fix the bug. There are a few
other things included in this commit to improve the situation with
spider DA:

1. Add a session variable that allows user to disable DA completely,
this will help as a temporary measure if/when further bugs with DA
emerge.

2. Move the increment of direct_aggregate_count to the spider DA
function. Currently this is done in rather bizarre and random
locations.

3. Fix the spider_db_mbase_row creation so that the last of its row
field (sentinel) is NULL. The code is already doing a null check, but
somehow the sentinel field is on an invalid address, causing the
segfaults. With a correct implementation of the row creation, we can
avoid such segfaults.

Author: Yuchen Pei
Author Date: 2023-07-05 07:55:30 UTC

MDEV-31117 Fix spider connection info parsing

Spider connection string is a comma-separated parameter definitions,
where each definition is of the form "<param_title> <param_value>",
where <param_value> is quote delimited on both ends, with backslashes
acting as an escaping prefix.

Despite the simple syntax, the existing spider connection string
parser was poorly-written, complex, hard to reason and error-prone,
causing issues like the one described in MDEV-31117. For example it
treated param title the same way as param value when assigning, and
have nonsensical fields like delim_title_len and delim_title.

Thus as part of the bugfix, we clean up the spider comment connection
string parsing, including:

- Factoring out some code from the parsing function
- Simplify the struct `st_spider_param_string_parse`
- And any necessary changes caused by the above changes

Author: Rucha Deodhar
Author Date: 2023-10-12 11:41:07 UTC

MDEV-31684: More tests

Author: Sergey Petrunia
Author Date: 2023-10-12 11:19:08 UTC

MDEV-32324: Server crashes inside filesort at my_decimal::to_binary

A subquery in form "(SELECT not_null_value LIMIT 1 OFFSET 1)" will
produce no rows which will translate into scalar SQL NULL value.

The code Item_singlerow_subselect::fix_length_and_dec() failed to
take LIMIT clause into account and set item_subselect->maybe_null=0.

This caused a crash in filesort() code when it got a NULL value for a
not nullable item.

Author: Yuchen Pei
Author Date: 2023-10-12 03:39:14 UTC

MDEV-29963 Spider should clear its lock lists when locking fails

Spider populates its lock lists (a hash) in store_lock(), and normally
clears them in the actual lock_tables(). However, if lock_tables()
fails, there's no reset_lock() method for storage engine handlers,
which can cause bad things to happen. For example, if one of the table
involved is dropped and recreated, when executing LOCK TABLES again,
the lock lists would be queried again in store_lock(), which could
cause access to freed stuff.

This also fixes MDEV-31357

Author: Yuchen Pei
Author Date: 2023-10-12 00:52:12 UTC

MDEV-27902 MDEV-29962 Spider check trx and get conn before locking

In 10.4 the testcase for MDEV-27902 displays symptoms of MDEV-29962,
which is why we do the check trx and get conn before locking,
different from other MDEV-27902 fixes for higher versions.