Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-06-14 11:35:15 UTC

more test fixes

Author: Igor Babaev
Author Date: 2022-06-13 22:31:53 UTC

MDEV-27159 Re-design the upper level of handling DML commands

This is the second commit for the task. This patch allows to execute only
single-table and multi-table DELETE statements using the method
Sql_cmd_dml::execute(). The code that handles INSERT statements has not
been touched.

This patch still does not have the final changes to handle UPDATE/DELETE
statements.
All tests from the main suite passed. With --ps-protocol one test from
opt_trace_security returns not the same result. This will be fixed soon.

Author: Oleg Smirnov
Author Date: 2022-06-13 15:28:21 UTC

MDEV-28817 Derived table elimination issue for multiple occurences

If the same field occurs more than once in the SELECT list
(using aliases) we need to store all those occurences
to be able to eliminate the derived table. For example,
  SELECT t1.* from t1 LEFT JOIN
    (SELECT a AS a1, MAX(c), b AS b1, b AS b2, a AS a2
      FROM t2 GROUP BY a, b) D
    ON D.a1=t1.a and D.b2=t1.b
Derived table D is joined using aliases a1 and b2 so the eliminating
algorithm must know that fields (a1,b2) are the same as (a,b)
which are on the GROUP BY list

Author: Oleg Smirnov
Author Date: 2022-02-17 15:53:37 UTC

MDEV-26278 Add functionality to eliminate derived tables from the query

Elimination of unnecessary tables from SQL queries is already present
in MariaDB. But it only works for regular tables and not for derived ones.

Imagine we have a view:
  CREATE VIEW v1 AS SELECT a, b, max(c) AS maxc FROM t1 GROUP BY a, b

Due to "GROUP BY a, b" the values of combinations {a, b} are unique,
and this fact can be treated as like derived table "v1" has a unique key
on fields {a, b}.

Suppose we have a SQL query:
  SELECT t2.* FROM t2 LEFT JOIN v1 ON t2.a=v1.a and t2.b=v1.b

1. Since {v1.a, v1.b} is unique and both these fields are bound to t2,
   "v1" is functionally dependent on t2.
   This means every record of "t2" will be either joined with
   a single record of "v1" or NULL-complemented.
2. No fields of "v1" are present on the SELECT list

These two facts allow the server to completely exclude (eliminate)
the derived table "v1" from the query.

Author: Nayuta Yanagisawa
Author Date: 2022-05-27 07:11:19 UTC

MDEV-27902 SIGSEGV's in spider_check_and_set_trx_isolation, spider_db_open_handler, and ha_spider::external_lock and __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed in psi_mutex_lock and inline_mysql_mutex_lock, and SIGABRT in safe_mutex_lock

Author: Jan Lindström
Author Date: 2022-06-11 09:31:58 UTC

Restore thread creation as we now do my_thread_init()/end()

Author: Dmitry Shulga
Author Date: 2022-06-09 08:27:29 UTC

MDEV-5816: Stored programs: validation of stored program statements

This is the prerequisite patch introducing the class sp_lex_instr
that encapsulates access to an instance of sp_lex_keeper. Every SP instruction
that does need access to a LEX object on its processing should inherit this class
and implement two abstract methods:
  is_invalid(),
  invalidate().

These method will be used in subsequent patches to implement recomplilation of
SP instructions on failure.

Currently, the following instructions are derived from the class sp_lex_instr:
  sp_instr_stmt,
  sp_instr_set,
  sp_instr_set_trigger_field,
  sp_instr_jump_if_not,
  sp_instr_freturn,
  sp_instr_cpush,
  sp_instr_cursor_copy_struct,
  sp_instr_set_case_expr

Additionally, this patch converts the class sp_instr_opt_meta
to the base abstract class (that is, not inhereted from
the class sp_instr).

Every jump SP instruction now must be inhereted directly from
the class sp_instr_opt_meta and additionally from either the class
sp_lex_instr or sp_instr depending on whether this SP instruction
does need access to LEX object or not.

Moreover, the class sp_cursor is no more owner of sp_lex_keeper.
Instead, the virtual method get_lex_keeper() has been added to the
class sp_cursor() that returns nullptr and this method is overriden
in the derived class sp_instr_cpush to provide a pointer to a real
instance of the class sp_lex_keeper.

Author: Nikita Malyavin
Author Date: 2022-06-09 14:13:47 UTC

fix long tpool execution

Author: Julius Goryavsky
Author Date: 2022-06-09 13:59:03 UTC

MDEV-28690 addendum: fixed regression in tests for qpress

Author: Daniel Black
Author Date: 2022-06-09 08:27:59 UTC

MDEV-28782 mariadb-tzinfo-to-sql to work in bootstrap mode

Work around MDEV-28718 for now, but also optimize the interation
of information_schema.SYSTEM_VARIABLES.

Add test case to show that tzinfo data into bootstrap is
desired functionality.

Bug report thanks to Dan Lenski of AWS.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-05-11 15:11:49 UTC

MDEV-5215 prerequisite: remove test and test_* database hacks in the test suite

Author: Roman Nozdrin
Author Date: 2022-06-08 15:05:31 UTC

MCOL-5114 Removing ctor specialization b/c it violates c++20 syntax needed in ColumnStorew

Author: Sergei Golubchik
Author Date: 2022-06-06 18:34:49 UTC

fix the test for FreeBSD

Author: Sergey Petrunia
Author Date: 2022-06-06 19:21:22 UTC

MDEV-28749: restore_prev_nj_state() doesn't update cur_sj_inner_tables correctly

Errors:
1. restore_prev_sj_state() assumed that
  "we assume remaining_tables doesnt contain @tab"
  which wasn't true.
2. Another bug in this function: it did remove bits from
   join->cur_sj_inner_tables but never added them.
3. greedy_search() adds tables into the join prefix but
   neglects to update the semi-join optimization state.
   It does update nested outer join state, see this call:

  check_interleaving_with_nj(best_table)

   but there's no matching call to update the semi-join state.
   (This wasn't visible because most of the state is in the POSITION
    structure which is updated. But there is also state in JOIN, too)

The patch:
- Fixes all of the above
- Adds JOIN::dbug_verify_sj_inner_tables() which is used to verify the
  state is correct at every step.
- Renamed advance_sj_state() to optimize_semi_joins().
  = Introduced update_sj_state() which ideally should be called
    "advance_sj_state" but I didn't reuse the name to avoid confusion.

Author: Igor Babaev
Author Date: 2022-06-04 03:05:31 UTC

MDEV-7487 Semi-join optimization for single-table UPDATE/DELETEs

This is a preliminary patch.

Author: Oleg Smirnov
Author Date: 2022-05-24 13:41:59 UTC

MDEV-27366 SIGSEGV in handler_index_cond_check with rowid_filter

Cause:
After the optimizer has generated a query plan using "ref access"
plus "rowid filter", the execution reaches "join_cache_level" check.
As the result the server may decide to employ full table scan and
hash join but does not discard the rowid filter. This leads to an
inconsistency in the table access.

Solution:
Discard "rowid filter" when full table scan | hash join are employed

Author: Oleg Smirnov
Author Date: 2022-05-31 09:47:13 UTC

MDEV-28598 Assertion 'got_name == named_item_expected()' failed

This assertion triggered on an attempt to write a piece of corrupt
JSON to the optimizer trace:
  "transformation": {
    "select_id": 2,
    "from": "IN (SELECT)",
    "to": "semijoin",
    {
       "join_optimization": {
       "select_id": 3,
       "steps": []
    }
  }

This happened because some parts of the query may be evaluated right
during the optimization stage. To handle such cases Json_writer will now
implicitly add named members with automatically assigned names
"implicitly_added_member_#1", "implicitly_added_member_#2", etc
to preserve JSON document correctness. Also the tail of the JSON document
will be printed to stderr before abort.
Those two improvements only apply for Debug builds, not for Release ones
as the Release builds skip JSON correctness checks

Author: Marko Mäkelä
Author Date: 2022-05-31 12:07:02 UTC

fixup! 424ad322ea096d6552c49e0bc268c730e897e915

Author: Marko Mäkelä
Author Date: 2022-05-31 11:43:46 UTC

Merge 10.5 into 10.6

Author: tmokmss
Author Date: 2022-05-30 08:57:26 UTC

use unlikely instead of likely

Author: Nikita Malyavin
Author Date: 2022-05-29 16:07:29 UTC

seems working kinda eh

Author: Nayuta Yanagisawa
Author Date: 2022-05-26 23:55:42 UTC

MDEV-28522 Delete constant SPIDER_SQL_TYPE_*_HS

The HandlerSocket support of Spider has been deleted by MDEV-26858.
Thus, the constants, SPIDER_SQL_TYPE_*_HS, are no longer necessary.

Author: Daniel Black
Author Date: 2022-05-19 23:31:18 UTC

MDEV-28597: double free in maria_open()

disk_cache is freed error handling functions.

Because we've freed this earlier in maria_open,
disk_cache is assigning to 0 (like the 10.3+ commit
 c127a1ba1450) to prevents the double free happing
in the error path that can be triggered later in
this function.

Author: Marko Mäkelä
Author Date: 2022-05-25 09:55:44 UTC

fixup! 036d091d2ef314bf5faab6c01e15c4030adee51e

Author: Marko Mäkelä
Author Date: 2022-05-14 08:45:50 UTC

Performance test: Disable some counter updates

Author: Jan Lindström
Author Date: 2022-05-24 04:17:16 UTC

Update galera disabled.def file

Author: Jan Lindström
Author Date: 2022-05-24 04:37:08 UTC

Update galera_sr disabled.def file

Author: Nikita Malyavin
Author Date: 2022-05-22 19:00:50 UTC

WIP resolve sql_type and field linkage

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-05-11 15:11:49 UTC

MDEV-5215 prerequisite: remove test and test_* database hacks in the test suite

Author: Julius Goryavsky
Author Date: 2022-05-19 02:53:32 UTC

MDEV-28423 continuation: Galera progress reporting + rolling updates

A new feature to support progress reporting (including progress
reporting in the JSON format) that was added in 10.9 can crash the
rolling update scenario if the old version donor node does not pass
the "total" tag in the MAGIC_FILE to the joiner node. The "grep"
utility that is used to extract this tag in this case will return
an empty string, which will then be substituted into the "if"
statement, resulting in a syntactically invalid construct.

This commit corrects this issue.

Also, when using the previous fixes (from MDEV-28423 and from
MDEV-28593), authentication errors are possible in case of mixing
versions at the opposite sides - if the donor is newer than joiner,
due to incorrect reading of the new MAGIC_FILE format by older node
versions (from the joiners side). This problem is also fixed here,
taking advantage of the fact that older versions are still able
to read GUID coordinates even if multiple strings are passed to
them at the end.

In addition, the new progress reporting exchanges the length of
the transmitted data between the donor and the joiner before the
start of the transfer, but does not use this information on the
joiner side, which can lead to incorrect calculation of progress
by the joiner (not synchronous with calculations from the donor
side) if from the joiner side reporting works in compatibility
mode with older versions. This commit moves the construction
of the command line for progress reporting to a point where
we already know the size and can use that knowledge.

Also, to facilitate SST script support in the future, the
script code now has been made unified between all versions
from 10.3 to 10.9.

Author: Julius Goryavsky
Author Date: 2022-05-19 01:45:19 UTC

MDEV-28423 continuation: Galera progress reporting + rolling updates

A new feature to support progress reporting (including progress
reporting in the JSON format) that was added in 10.9 can crash the
rolling update scenario if the old version donor node does not pass
the "total" tag in the MAGIC_FILE to the joiner node. The "grep"
utility that is used to extract this tag in this case will return
an empty string, which will then be substituted into the "if"
statement, resulting in a syntactically invalid construct.

This commit corrects this issue.

Also, when using the previous fixes (from MDEV-28423 and from
MDEV-28593), authentication errors are possible in case of mixing
versions at the opposite sides - if the donor is newer than joiner,
due to incorrect reading of the new MAGIC_FILE format by older node
versions (from the joiners side). This problem is also fixed here,
taking advantage of the fact that older versions are still able
to read GUID coordinates even if multiple strings are passed to
them at the end.

In addition, the new progress reporting exchanges the length of
the transmitted data between the donor and the joiner before the
start of the transfer, but does not use this information on the
joiner side, which can lead to incorrect calculation of progress
by the joiner (not synchronous with calculations from the donor
side) if from the joiner side reporting works in compatibility
mode with older versions. This commit moves the construction
of the command line for progress reporting to a point where
we already know the size and can use that knowledge.

Also, to facilitate SST script support in the future, the
script code now has been made unified between all versions
from 10.3 to 10.9.

Author: Oleg Smirnov
Author Date: 2022-05-18 14:36:32 UTC

MDEV-24837 HAVING clause yields wrong results

Set items references to point to the temporary table fields
when processing aggregate operations

Author: Kiyoshi Takeda
Author Date: 2022-05-18 12:49:24 UTC

MDEV-14642 Assertion 'table->s->db_create_options == part_table->s->db_create_options' failed in compare_table_with_partition

When trying to execute ALTER TABLE EXCHANGE PARTITION with different definitions, assertion

    table->s->db_create_options == part_table->s->db_create_options

failed in compare_table_with_partition().
However, this execution should not be allowed since executing 'exchange partition' requires the identical structure of the two tables.

To fix the problem, I deleted the assertion code and added code that returns an error that indicates tables have different definitions.

Author: Sergei Golubchik
Author Date: 2022-05-18 11:11:16 UTC

MDEV-28606 Server crashes in st_select_lex::add_table_to_list instead of error 1066: Not unique table/alias

10.2-only fix, 10.3+ uses LEX_STRING's and checks the length first

Author: Vlad Lesin
Author Date: 2022-05-18 11:09:40 UTC

MDEV-28607 add option to mtr to start mariabackup under rr

The new --rr-backup mtr option is added.

Author: Oleg Smirnov
Author Date: 2022-05-16 11:32:39 UTC

MDEV-27366 SIGSEGV in handler_index_cond_check on SELECT in connection with rowid_filter setting

Cause: when optimizer picks a full table scan it does not reset previously
chosen rowid filter.
Solution: reset rowid filter for full table scan in best_access_path()

Author: Julius Goryavsky
Author Date: 2022-05-17 21:49:13 UTC

MDEV-28583: Galera: binlogs disappear after rsync IST

This commit sends a flag indicating the presence of the "--bypass"
option from the donor node to the joiner nodes during rsync IST,
because without such a flag it is impossible to distinguish IST
from the SST on the joiner nodes (in IST/SST scripts, because the
"--bypass" option is still not passed to scripts from server code).
Specifically, this fixes an issue with binary logs disappearing
after IST (via rsync). There are also changes to diagnostic messages
here that will make it easier to diagnose script-related problems
in the future when debugging and when checking the logs. This commit
also adds more robust signal handlers - to handle exceptions during
script execution. These handlers won't mask some crashes and it
also unifies exit codes between different scripts. These changes
have already been helpful to debugging "bypass" flag handling.

Author: Julius Goryavsky
Author Date: 2022-05-17 21:43:32 UTC

MDEV-28583: Galera: binlogs disappear after rsync IST

This commit sends a flag indicating the presence of the "--bypass"
option from the donor node to the joiner nodes during rsync IST,
because without such a flag it is impossible to distinguish IST
from the SST on the joiner nodes (in IST/SST scripts, because the
"--bypass" option is still not passed to scripts from server code).
Specifically, this fixes an issue with binary logs disappearing
after IST (via rsync). There are also changes to diagnostic messages
here that will make it easier to diagnose script-related problems
in the future when debugging and when checking the logs. This commit
also adds more robust signal handlers - to handle exceptions during
script execution. These handlers won't mask some crashes and it
also unifies exit codes between different scripts. These changes
have already been helpful to debugging "bypass" flag handling.

Author: Julius Goryavsky
Author Date: 2022-05-16 12:17:15 UTC

rnew

Author: midenok
Author Date: 2022-05-16 13:11:20 UTC

MDEV-27328 MSAN fix

Author: Sergey Petrunia
Author Date: 2022-05-14 17:19:05 UTC

Code cleanup in/around check_interleaving_with_nj()

- In best_extension_by_limited_search(), do not check for
  "(remaining_tables & real_table_bit)", it is guaranteed to be true.
  Make it an assert.
- In (!idx || check_interleaving_with_nj())", remove the !idx part.
  This check made sense only in the original version of this function.
- "micro optimization" in check_interleaving_with_nj().

Author: Nayuta Yanagisawa
Author Date: 2022-05-14 06:03:12 UTC

MDEV-28564 Remove option value '-1' from spider_bgs_first_read, spider_bgs_mode, and spider_bgs_second_read

Remove the option value '-1', which has been deprecated by MDEV-27169,
of spider_bgs_first_read, spider_bgs_mode, and spider_bgs_second_read.

Author: Sergey Petrunia
Author Date: 2022-05-07 16:10:59 UTC

Optimizer trace: Make ref_optimizer_key_uses[*] show the index name

Author: Daniel Black
Author Date: 2022-05-11 03:28:42 UTC

MDEV-28534: clang-12 compile warnings (innodb)

Solves compile error:

/buildbot/amd64-ubuntu-2004-msan/build/storage/innobase/include/buf0types.h:149:23: error: 'constexpr' non-static member function will not be implicitly 'const' in C++14; add 'const' to avoid a change in behavior [-Werror,-Wconstexpr-not-const]
  constexpr ulonglong raw() { return m_id; }
                      ^
                            const

Author: Brandon Nesterenko
Author Date: 2022-05-10 20:25:35 UTC

MDEV-28530: Revoking privileges from a non-existing user on a master breaks replication on the slave in the presence of replication filters

Problem:
========
Replication can break while applying a query log event if its
respective command errors on the primary, but is ignored by the
replication filter within Grant_tables on the replica. The bug
reported by MDEV-28530 shows this with REVOKE ALL PRIVILEGES using a
non-existent user. The primary will binlog the REVOKE command with
an error code, and the replica will think the command executed with
success because the replication filter will ignore the command while
accessing the Grant_tables classes. When the replica performs an
error check, it sees the difference between the error codes, and
replication breaks.

Solution:
========
If the replication filter check done by Grant_tables logic ignores
the tables, reset thd->slave_expected_error to 0 so that
Query_log_event::do_apply_event() can be made aware that the
underlying query was ignored when it compares errors.

Note that this bug also effects DROP USER if not all users exist
in the provided list, and the patch fixes and tests this case.

Reviewed By:
============
<TODO>

Author: Roman Nozdrin
Author Date: 2022-05-10 14:47:04 UTC

The patch fixes openssl3-related compilation failure found using fresh U22 image

Author: Julius Goryavsky
Author Date: 2022-05-10 06:18:25 UTC

MDEV-28015: Galera: GTID value is missing when mariabackup SST is used

Author: Julius Goryavsky
Author Date: 2022-05-10 06:22:21 UTC

MDEV-28500: Hashicorp: Debian packaging is broken

This commit fixes a documentation installation
issue (for debian packaging) and generally brings
the installation control files up to date (as for
the rest of the components).

Author: Daniel Black
Author Date: 2022-05-10 02:43:28 UTC

deb: limit columnstore to tested platforms

"We only support Debian 10, but not Debian 11 and from today
not Debian 9, and only Ubuntu 18.04 and 20.04, but not 22.04"

Author: Norio Akagi
Author Date: 2022-05-07 21:48:30 UTC

MDEV-27892 Improve an error message for foreign server exists

Improve and add more languages for the ERROR 1476: Foreign Server
already exists.

Author: Monty
Author Date: 2022-05-09 08:36:44 UTC

Simple optimization: Remove JOIN::set_group_rpa as it is not needed

Author: Elena Stepanova
Author Date: 2022-05-07 14:27:39 UTC

Hashicorp debian build/installation test #2

Author: Daniel Black
Author Date: 2022-05-06 02:00:01 UTC

Deb: dh_missing --fail-missing - columnstore

Per man dh_missing, not-installed will exand wildcards
since debhelper 11.1. Since Stretch is on 10.2.5, this won't happen.

As columnstore is still only x86_64 we can use that in the file.

Author: Brandon Nesterenko
Author Date: 2022-04-27 01:51:42 UTC

MDEV-28294: set default role bypasses Replicate_Wild_Ignore_Table: mysql.%

Problem:
========
When replicating SET DEFAULT ROLE, the pre-update check (i.e. that
in set_var_default_role::check()) tries to validate the existence of
the given rules/user even when the targeted tables are ignored. When
previously issued CREATE USER/ROLE commands are ignored by the
replica because of the replication filtering rules, this results in
an error because the targeted data does not exist.

Solution:
========
Before checking that the given roles/user exist of a SET DEFAULT
ROLE command, first ensure that the mysql.user and
mysql.roles_mapping tables are not excluded by replication filters.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
Sergei Golubchik <serg@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2022-04-27 01:51:42 UTC

MDEV-28294: set default role bypasses Replicate_Wild_Ignore_Table: mysql.%

Problem:
========
When replicating SET DEFAULT ROLE, the pre-update check (i.e. that
in set_var_default_role::check()) tries to validate the existence of
the given rules/user even when the targeted tables are ignored. When
previously issued CREATE USER/ROLE commands are ignored by the
replica because of the replication filtering rules, this results in
an error because the targeted data does not exist.

Solution:
========
Before checking that the given rules/user exist of a SET DEFAULT
ROLE command, first ensure that the mysql.user and
mysql.roles_mapping tables are not excluded by replication filters.

Reviewed By
===========
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2022-04-27 01:51:42 UTC

MDEV-28294: set default role bypasses Replicate_Wild_Ignore_Table: mysql.%

Problem:
========
When replicating SET DEFAULT ROLE, the pre-update check (i.e. that
in set_var_default_role::check()) tries to validate the existence of
the given rules/user even when the targeted tables are ignored. When
previously issued CREATE USER/ROLE commands are ignored by the
replica because of the replication filtering rules, this results in
an error because the targeted data does not exist.

Solution:
========
Before checking that the given rules/user exist of a SET DEFAULT
ROLE command, first ensure that the mysql.user and
mysql.roles_mapping tables are not excluded by replication filters.

Reviewed By
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Jan Lindström
Author Date: 2022-04-29 06:49:14 UTC

MDEV-24688 : galera.galera_ist_progress MTR failed: assert_grep.inc failed

Grep is not safe as it will confuse if there is more than one
matching line or if progress report is slightly different as
progress reporting is afected by how fast IST is send/received.

Fix is to use sed and we are interested only that there is at
least one progress report from IST.

Author: Marko Mäkelä
Author Date: 2022-04-28 06:01:37 UTC

Merge

Author: Nikita Malyavin
Author Date: 2022-04-27 17:29:03 UTC

fix ha_innobase plugin initialization race with purge table open

Now that purge opens a table very early, at undo node parsing stage,
it slips before plugin initialization ends.
This causes ASAN global-buffer-overflow and segfault.

The problem is that `resolve_sysvars` function that normalizes table
options (used in parse_engine_table_options during table open) is called
after purge starts (srv_start() call in innodb_init()).

To solve this, additional resolve_sysvars call is addid before srv_start
in innodb_init.

Author: Nikita Malyavin
Author Date: 2022-04-26 01:38:14 UTC

add deterministic tests

Author: Marko Mäkelä
Author Date: 2022-04-27 10:16:07 UTC

MDEV-28422 Page split breaks a gap lock

btr_insert_into_right_sibling(): Inherit any gap lock from the
left sibling to the right sibling before inserting the record
to the right sibling and updating the node pointer(s).

lock_update_node_pointer(): Update locks in case a node pointer
will move.

Based on mysql/mysql-server@c7d93c274fdc5c56e36458fa4000fa3a483ffffd

Author: Julius Goryavsky
Author Date: 2022-04-26 13:18:22 UTC

MDEV-28291: Post-review changes

Author: Sergey Petrunia
Author Date: 2022-04-26 12:18:15 UTC

Merge branch '10.2' of github.com:MariaDB/server into 10.2

Author: Sergey Petrunia
Author Date: 2022-04-26 12:18:15 UTC

Merge branch '10.2' of github.com:MariaDB/server into 10.2

Author: Julius Goryavsky
Author Date: 2022-04-25 22:47:24 UTC

MDEV-26171: Post-merge fix

Author: Igor Babaev
Author Date: 2022-04-22 14:53:16 UTC

MDEV-27212 Crash in Item_equal::sort on second execution of stored procedure

This bug could cause a crash of the server at the second call of a stored
procedure when it executed a query containing a mergeable derived table /
view whose specification used another mergeable derived_table or view and a
subquery with outer reference in the select list of the specification.
Such queries could cause the same problem when they were executed for the
second time in a prepared mode.
The problem appeared due to a typo mistake in the legacy code of the function
create_view_field() that prevented building Item_direct_view_ref wrapper
for the mentioned outer reference at the second execution of the query and
setting the depended_from field for the outer reference.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Jan Lindström
Author Date: 2022-04-25 07:18:50 UTC

MDEV-28377 : galera.galera_as_slave_nonprim bind: Address already in use

Reported error is caused by misconfiguration of node_2. Additionally,
node_4 that should have been non-galera node contained also
incorrect configuration parameters. Removed some sleeps from test
case and replaced them with wait_condition.

Author: Julius Goryavsky
Author Date: 2022-04-22 23:57:35 UTC

MDEV-26171: Post-merge fixes

Author: Julius Goryavsky
Author Date: 2022-04-20 16:35:35 UTC

MDEV-28277: post-review fix

Author: Jan Lindström
Author Date: 2022-04-19 09:02:34 UTC

MDEV-28314 : The Galera cluster primary node goes into hang mode when innodb_encryption_threads is enabled

When we enable writes after Galera SST srv_n_fil_crypt_threads needs
to be set temporally to 0 (as was done when writes were disabled)
to make sure that encryption threads will be really started based
on old value.

Fix provided by Marko Mäkelä and added a test case.

Author: Brandon Nesterenko
Author Date: 2022-04-04 13:35:48 UTC

MDEV-25768: ALTER gtid_slave_pos table storage engine should not be allowed when slave is running.

Problem:
========
The mysql.gtid_slave_pos table can have its engine modified while
the SQL thread is alive.

Solution:
========
Use MDL locks to ensure that the table can only have its
engine modified when the slave SQL thread is not running.

Reviewed By:
============
TODO

Author: Julius Goryavsky
Author Date: 2020-07-13 17:12:00 UTC

MDEV-19281: Plugin implementation for the Hashicorp Vault KMS

- Authentication is done using the Hashicorp Vault's token
  authentication method;
- If additional client authentication is required, then the
  path to the CA authentication bundle file may be passed
  as a plugin parameter;
- The creation of the keys and their management is carried
  out using the Hashicorp Vault KMS and their tools;
- Key values stored as hexadecimal strings;
- Key values caching is supported.
- Implemented a time-invalidated cache for key values and
  for key version numbers received from the Hashicorp Valult
  server;
- The plugin uses libcurl (https) as an interface to
  the HashiCorp Vault server;
- JSON parsing is performed through the JSON service
  (through the include/mysql/service_json.h);
- HashiCorp Vault 1.2.4 was used for development and testing.

Author: Oleg Smirnov
Author Date: 2022-04-17 15:30:31 UTC

MDEV-25994 Crash with union of my_decimal type in ORDER BY clause

During execution of Type_handler_decimal_result::make_sort_key_part()
the call to item->val_decimal_result() may fail and return NULL pointer.
Such case was not analyzed which led to the crash.
This commit adds necessary checks to the case described above and also
adds an indicator of successful execution for make_sort_key() functions
(return value = FALSE for success and TRUE failure)

Author: Nayuta Yanagisawa
Author Date: 2022-04-05 09:20:41 UTC

MDEV-28010 Deprecate spider_crd_type and spider_crd_weight

Deprecate the variables spider_crd_type and spider_crd_weight.
The value should be defined by the engine developers.

Author: Vlad Lesin
Author Date: 2022-04-14 11:27:23 UTC

MDEV-17843 Assertion `page_rec_is_leaf(rec)' failed in lock_rec_queue_validate upon SHOW ENGINE INNODB STATUS

lock_validate() accumulates page ids under locked LockMutexGuard, then
releases the latch, and invokes lock_rec_block_validate() for each page.
Some other thread has ability to add/remove locks and change pages
between releasing the latch in lock_validate() and acquiring it in
lock_rec_validate_page().

lock_rec_validate_page() can invoke lock_rec_queue_validate() for
non-locked supremum, what can cause ut_ad(page_rec_is_leaf(rec)) failure
in lock_rec_queue_validate().

The fix is to invoke lock_rec_queue_validate() only for locked records
in lock_rec_validate_page().

The error message in lock_rec_block_validate() is not necessary as
BUF_GET_POSSIBLY_FREED mode is used to get block from buffer pool, and
this is not error if a block was evicted.

The test case would require new debug sync point. I think it's not
necessary as the fixed code is debug-only.

Author: Marko Mäkelä
Author Date: 2022-04-14 11:30:26 UTC

Merge

Author: Marko Mäkelä
Author Date: 2022-04-14 09:59:59 UTC

MDEV-21423: Replace trx_sys.rw_trx_hash with a locked hash table

The embedded page_hash_latch in buf_pool.page_hash have served us well.
Let us attempt the same approach for trx_sys.rw_trx_hash,
instead of using the lock-free hash table that is unable to shrink.

Compared to the lock-free hash table, any iteration of the entire
hash table, as in rw_trx_hash.for_each() or rw_trx_hash.for_each_until()
may potentially modify a large number of cache lines.
Outside debug checks, trx_sys.rw_trx_hash is being iterated in
ReadView::open(), trx_sys_t::clone_oldest_view(), and
trx_sys_t::get_min_trx_id() (invoked on secondary index lock checks).

To reduce the impact of locking on processors
that support transactional memory, we introduce the predicate
rw_trx_hash_t::empty() so that unnecessary acquisition and releasing
of hash array latches can be avoided when an entire slice of the
hash array is empty.

FIXME: This is reducing throughput in many cases. How to improve this
further?

TODO: Find a reasonable value for the constexpr size_t n_cells.

Author: Oleg Smirnov
Author Date: 2022-04-13 12:56:34 UTC

MDEV-27699 ANALYZE FORMAT=JSON fields are incorrect for UNION ALL queries

UNION ALL queries are a subject of optimization introduced in MDEV-334
when creation of a temporary table is skipped.
While there is a check for this optimization in Explain_union::print_explain()
there was no such in Explain_union::print_explain_json(). This resulted in
printing irrelevant data like:
  "union_result": {
    "table_name": "<union2,3>",
    "access_type": "ALL",
    "r_loops": 0,
    "r_rows": null
in case when creation of the temporary table was actually optimized out.
This commits adds a check whether the temporary table was actually created
during the UNION ALL processing and eliminates printing of the irrelevant data.

Author: Nikita Malyavin
Author Date: 2022-04-13 19:08:06 UTC

MDEV-26281 ASAN use-after-poison when complex conversion involved in blob

The problem existed for long. Item_func_in::create_array allocates `array`
on mem_root but releases it with C++ `delete` in Item_func_in::cleanup().

Allocating on a mem root is incorrect in a case of Item, since it controls
its memory by an independent mechanism, and its lifetime differs from
mem_roots (thd, stmt, table).

It was also incorrect to allocate in_vector::base on mem_root by the same
reasons (also done in in_row).

So far, thd argument has been completely removed from constructors of the
hierarchy, and RAII memory management is favored.

Author: Nikita Malyavin
Author Date: 2021-10-28 23:04:07 UTC

MDEV-26508 heap-use-after-free in Item_default_value::walk on 2nd SP run

The bug is a regression of c47e4aab62c6.

In Item_default_value::enchant_default_with_arg_processor `arg` argument is
set with f_item from values list.

Then on the 2nd execution this item turns out to be freed.

This is because of check_fields() call where value items are subctituted
the temporary ones with thd->change_item_tree().

Solution: we should keep Item_default_value::arg clean after SP run.
Thus, register the change of the item tree. After SP run NULL will be set
back to Item_default_value::arg.

Author: Daniel Black
Author Date: 2022-04-13 05:05:19 UTC

MDEV-27791: rocksdb_log_dir test postfix

We can only remove a subdirectory in mtr on an installed instance

Example failure previously:

CURRENT_TEST: rocksdb.rocksdb_log_dir
mysqltest: At line 15: Path '/usr/local/mariadb-10.9.0-linux-systemd-x86_64/mysql-test/var/tmp/1' is not a subdirectory of MYSQLTEST_VARDIR '/usr/local/mariadb-10.9.0-linux-systemd-x86_64/mysql-test/var/1'or MYSQL_TMP_DIR '/usr/local/mariadb-10.9.0-linux-systemd-x86_64/mysql-test/var/tmp/1'

Author: Daniel Black
Author Date: 2022-04-13 05:05:19 UTC

MDEV-27791: rocksdb_log_dir test postfix

We can only remove a subdirectory in mtr on an installed instance

Example failure previously:

CURRENT_TEST: rocksdb.rocksdb_log_dir
mysqltest: At line 15: Path '/usr/local/mariadb-10.9.0-linux-systemd-x86_64/mysql-test/var/tmp/1' is not a subdirectory of MYSQLTEST_VARDIR '/usr/local/mariadb-10.9.0-linux-systemd-x86_64/mysql-test/var/1'or MYSQL_TMP_DIR '/usr/local/mariadb-10.9.0-linux-systemd-x86_64/mysql-test/var/tmp/1'

Author: Julius Goryavsky
Author Date: 2022-04-11 23:01:26 UTC

MDEV-26171: wsrep_sst_receive_address does not parse IPv6 address correctly

This commit fixes problems with parsing ipv6 addresses given via
the wsrep_sst_receive_address and wsrep_node_address options.

Also, this commit removes extra lines in the configuration files
in the mtr test suites for Galera related to these parameters.

Author: Dmitry Shulga
Author Date: 2022-04-11 06:37:55 UTC

MDEV-16128: Server crash in Item_func::print_op on 2nd execution of PS

For some queries that involve tables with different but convertible
character sets for columns taking part in the query, repeatable
execution of such queries in PS mode or as part of a stored routine
would result in server abnormal termination.

For example,
CREATE TABLE t1 (a2 varchar(10));
CREATE TABLE t2 (u1 varchar(10) CHARACTER SET utf8);
CREATE TABLE t3 (u2 varchar(10) CHARACTER SET utf8);
PREPARE stmt FROM
  "SELECT t1.* FROM (t1 JOIN t2 ON (t2.u1 = t1.a2))
   WHERE (EXISTS (SELECT 1 FROM t3 WHERE t3.u2 = t1.a2))";

EXECUTE stmt;
EXECUTE stmt; <== Running this prepared statement the second time
                  results in server crash.

The reason of server crash is that an instance of the class
Item_func_conv_charset, that created for conversion of a column
from one character set to another, is allocated on execution
memory root but pointer to this instance is stored in an item
placed on prepared statement memory root. Below is calls trace to
the place where an instance of the class Item_func_conv_charset
is created.

setup_conds
 Item_func::fix_fields
  Item_bool_rowready_func2::fix_length_and_dec
   Item_func::setup_args_and_comparator
    Item_func_or_sum::agg_arg_charsets_for_comparison
     Item_func_or_sum::agg_arg_charsets
      Item_func_or_sum::agg_item_set_converter
       Item::safe_charset_converter

And the following trace shows the point where a pointer to
the instance of Item_func_conv_charset is passed to the class
Item_func_eq, that is created on a memory root of the prepared
statement.

Prepared_statement::execute
 mysql_execute_command
  execute_sqlcom_select
   handle_select
    mysql_select
     JOIN::optimize
      JOIN::optimize_inner
       convert_join_subqueries_to_semijoins
        convert_subq_to_sj
---
    Item_func_eq *item_eq=
      new (thd->mem_root) Item_func_eq(thd, subq_pred->left_expr_orig,
                                       subq_lex->ref_pointer_array[0]);

On completing execution of the statement EXECUTE stmt
the items placed on execution query arena are released but the instance
of Item_func_eq that was created on permanent memory root still does exist
and has one of its arguments (stored in args[0]) referencing to the released
object. Next time the prepared statement is executed access to dangling
pointer leads to server crash.

So, to fix the issue the item Item_func_conv_charset should be created
on permanent memory root in case a prepared statement or stored routine
is run.

Author: Oleg Smirnov
Author Date: 2022-04-06 12:41:50 UTC

Remove a garbage file from mysql-test

Author: Daniel Black
Author Date: 2022-04-07 04:15:08 UTC

MDEV-27955: main.func_json_notembedded postfix -make big

While attempting to reduce the iterations per Monty's suggestion in
MDEV-27955, I wasn't able to succeed in arriving at a predictable
value.

While some functions like json_merge_patch where predictably within
the limit on small values, other functions like json_array_append
where still significantly fast to generate a result and not
ER_STATEMENT_TIMEOUT.

So lets leave this as a big test and hope test machine have sufficient
ram for the lower, but still large results previously committed.

Author: Daniel Black
Author Date: 2022-03-25 06:25:11 UTC

MDEV-28153: Debian autobake- use absolute dependencies (Salsa postfix)

Autobake now includes a dependency on lsb-release.

As the BB CI images
(https://github.com/MariaDB/mariadb.org-tools/blob/master/buildbot.mariadb.org/ci_build_images/debian.Dockerfile)
have explicit dependencies, there's no point maintaining them badly in
two places. We don't want do the full autobake-deb.sh there, just enough
to have the control file containing the correct dependencies.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-04-06 07:31:59 UTC

MDEV-25638 Assertion `!result' failed in convert_const_to_int

The test suite triggers 3 bugs:

I. Absence of STATUS_NO_RECORD on an empty record
II. Mismatch signed and unsigned becaue parameter is not
     taking from the item but was a constant.
III. Problem with rollback changes made for Items in the TABLE object:

    A cycle of live of Items (expressions) in table is following:

    1) Parsed in the query arena of the table
    2) Prepared with the permanent query arena of the table be active as a
    runtime and stmt (permanent) arena (here is difference from usual
    execution)
    3) usage (can be sorter of one query live (CREATE TABLE for example) or
    longer then several queries (SELECT and using table cache), it is not a
    problem because all fields are local to the table).
    4) cleanup and deallocate with the TABLE object

    So looking on the above all changes in the expressions belonged to
    a TABLE object should be just ignored.

Author: Elena Stepanova
Author Date: 2022-04-04 11:48:11 UTC

FreeBSD build test

Author: Elena Stepanova
Author Date: 2022-04-04 11:47:12 UTC

FreeBSD build test

Author: Elena Stepanova
Author Date: 2022-04-04 11:44:55 UTC

FreeBSD build test

Author: Elena Stepanova
Author Date: 2022-04-04 10:57:06 UTC

FreeBSD build test

Author: Marko Mäkelä
Author Date: 2022-04-04 07:37:23 UTC

Merge

Author: Jan Lindström
Author Date: 2022-04-01 10:29:31 UTC

MDEV-28210 : SIGSEGV in the test galera.galera_sst_rsync2

We should make sure that wsrep exists before calling wsrep->post_rollback

Author: Nayuta Yanagisawa
Author Date: 2022-03-31 13:38:54 UTC

MDEV-21618 CREATE UNIQUE INDEX fails with "ERROR 1286 (42000): Unknown storage engine 'partition'"

The server doesn't use the enforced storage engine in ALTER TABLE
without ENGINE clause to avoid an unwanted engine change.

However, the server tries to use the enforced engine in CREATE
INDEX. As a result, the false positive error is raised. The server
should not apply the enforced engine in CREATE INDEX too.

Author: mkaruza
Author Date: 2021-12-16 08:39:26 UTC

MDEV-27263 Cluster bootstrap node shows duplicate wsrep allowlist IP warning messages on each restart.

We should clear `wsrep_allowlist` table on bootstrap before writing to
it.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Author: Oleg Smirnov
Author Date: 2022-03-30 07:42:20 UTC

MDEV-24560 Avoid possible use of uninitialized tab->table

This patch amends the previous fix for MDEV-24560.
Issue: moving initialization of tab->table to the end of the function
can cause dereferencing NULL in add_sorting_to_table().
The solution is to initialize tab->table as before but reset it to NULL
in case of an error during JOIN::create_postjoin_aggr_table() execution.

Author: Jan Lindström
Author Date: 2022-03-03 16:51:12 UTC

MDEV-24845 : Oddities around innodb_fatal_semaphore_wait_threshold and global.innodb_disallow_writes

We will remove variable innodb_disallow_writes because it is badly
designed and implemented. Parameter will be marked as removed.

Instead we will be using
* Galera provider is paused i.e. all commits will wait
* FLUSH TABLES WITH READ LOCK (FTWRL) to avoid any DDL during SST, this
global read lock i.e MDL_BACKUP_FTWRL1 will conflict MDL_BACKUP_DML
taken by InnoDB background threads doing writes. Similarly, it will conflict
with MBL_BACKUP_START used by mariabackup.
* We force flushing all dirty pages from buffer pool and force InnoDB checkpoint
* Encryption, purge, background statistics and FTS-optimize threads will
acquire MDL_BACKUP_DML before continuing. This will conflict with lock
acquired in FTWRL. Note that we will not use waiting. If MDL-lock can't be
acquired we will skip the operation.

handler.cc
handler.h
  Add new API function ha_force_checkpoint to force checkpoint
  inside InnoDB.

xtrabackup.cc
  Remove INNODB_DISALLOW_WRITES code

mdl.cc
  New functions to trylock and release global MDL lock.

wsrep_sst.cc
  Force flushing all dirty pages from buffer pool and force InnoDB checkpoint

mysqld.cc
  Mark innodb-disallow-writes variable as removed.

dict0stats.cc
dict_stats_func
  Acquire backup lock and release it after we have done

fil0crypt.cc
fil_crypt_start_encrypting_space()
fil_crypt_thread()
  Acquire backup lock and release it after we have done

fts0opt.cc
fts_optimize_sync_table()
fts_optimize_callback()
  Acquire backup lock and release it after we have done

ha_innodb.cc
  Remove all WITH_INNODB_DISALLOW_WRITES code
wsrep_force_checkpoint()
  New API function to flush dirty pages from buffer pool and
  force full checkpoint.

trx_purge,cc
trx_purge()
  Acquire and release global MDL_BACKUP_DML lock

try

r

Author: Daniel Black
Author Date: 2022-03-18 06:36:53 UTC

Deb: enable pmem on riscv64

Author: Nayuta Yanagisawa
Author Date: 2021-10-29 10:04:53 UTC

MDEV-26127 Assertion `err != DB_DUPLICATE_KEY' failed or InnoDB: Failing assertion: id != 0 on ALTER ... REBUILD PARTITION

During rebuild of partition, the partitioning engine calls
alter_close_table(), which does not unlock and close some table
instances of the target table.
Then, the engine fails to rename partitions because there are table
instances that are still locked.

Closing all the table instance of the target table fixes the bug.