Author: midenok
Author Date: 2022-12-10 11:25:55 UTC

MDEV-29831 Galera crashes when running CoR for a locked table after
    setting the minimum memory for a user session.

Failure happens when finalize_atomic_replace() was already finished
and we removed the table from locked tables list.
finalize_locked_tables() doesn't know about that, it doesn't add back
last deleted lock because operation_failed == true.
reopen_tables() doesn't reopen table and as a result we have NULL in
pos_in_locked_tables->table.

The fix adds the knowledge that the locked_tables_count changed since
the start of the command. And if that happened we
add_back_last_deleted_lock(). That makes MDEV-29544 fix with
locked_tables_decremented deprecated.

Alternative fix would add atomic_replace_finished to Atomic_info and
updated it on successful finalize_atomic_replace(). Then the condition
would look like this:

  if (atomic_replace_finished || !operation_failed)
  {
    /*
      Add back the deleted table and re-created table as a locked table
      This should always work as we have a meta lock on the table.
    */
    thd->locked_tables_list.add_back_last_deleted_lock(pos_in_locked_tables);
  }

Author: sjaakola
Author Date: 2022-06-30 08:39:05 UTC

MDEV-26391 BF abortable mariabackup execution

Forcing old style node desync and pause, despite of WSREP_MODE_BF_MARIABACKUP
if node is operating as SST donor.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Author: Daniel Black
Author Date: 2023-01-11 23:34:14 UTC

Use MariaDB as the project name in CMakeLists.txt (fix)

Partial revert of d29d915790f7da41cd89c469f0a4c6c9b8f8a5c3.

It seems from CI that lower case RPM names are generated without
explictly setting the package name to mixed case.

Author: Yuchen Pei
Author Date: 2023-01-13 07:23:09 UTC

Revert "MDEV-20502 Queries against spider tables return wrong values for columns following constant declarations."

This reverts commit 69c86abb646361c607a248f079f8fd4e600dcada.

Author: Jan Lindström
Author Date: 2023-01-15 10:55:57 UTC

Revert "Merge branch 10.4 into 10.5"

This reverts commit 179c2833721292a918280b7d114d94d81020105b, reversing
changes made to a44d896f98f2d2a3ebf0f1393bf84fd659ecd225.

Author: Jan Lindström
Author Date: 2023-01-15 10:01:50 UTC

Revert "Merge branch 10.4 into 10.5"

This reverts commit 179c2833721292a918280b7d114d94d81020105b, reversing
changes made to 981a6b704475176d032c13461baffcaf88802d34.

Author: Jan Lindström
Author Date: 2023-01-15 09:05:26 UTC

Merge 10.5 into 10.6

Author: Julius Goryavsky
Author Date: 2023-01-12 20:54:25 UTC

Merge 10.3 into 10.4

Author: Sergey Petrunia
Author Date: 2023-01-12 12:55:40 UTC

MDEV-30081: Crash with split_materialized over a VIEW

Consider a view and use of Split Materialized optimization to read it:

  create view v1 as (SELECT ... id1 FROM t1 GROUP BY id1);
  SELECT ... LEFT JOIN v1 ON v1.id1 = tbl2.col

Split Materialized will need to construct ref access inside the view:

  t1.id1= {outer reference to tbl2.col}

It will do so by cloning both parts of the equality (v1.id1 =tbl2.col)
and then "pushing it down" into subquery by calling

  item->walk(&Item::set_fields_as_dependent_processor, parent_join->select_lex)

for both parts of the equality.
The problem was that left part of the equality was "v1.id1", namely an
Item_direct_view_ref, which did not implement
set_fields_as_dependent_processor() function.

Failure to do that resulted in Item_direct_view_ref reporting invalid
used_tables() in some cases (the case we've hit required that
Item_direct_view_ref is inside an outer join, and the column it refers
to is marked as constant)

Fixed by implementing Item_direct_ref::set_fields_as_dependent_processor().

Author: Vladislav Vaintroub
Author Date: 2023-01-10 10:56:44 UTC

MDEV-26714 - fix mariadb-upgrade

mariadb-upgrade needs to accept credential-manager parameter.
At the moment, it would have no effect - all the credential manager logic
is encapsulated inside cli_connect, and mariadb-upgrade does not
connect to the server itself (instead invoking the cli)

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-01-06 11:44:23 UTC

MDEV-16232 Use fewer mini-transactions

- Fail to change the latch operation in row_search_mvcc.

Author: Vladislav Vaintroub
Author Date: 2023-01-04 17:52:18 UTC

optimizer_switch stored procedures in sys schema.

Split optimizer_switch into individual option names and values, and return
it as result set.

Author: Marko Mäkelä
Author Date: 2023-01-05 08:26:20 UTC

Merge 10.6

Author: Sergey Petrunia
Author Date: 2022-12-22 12:49:10 UTC

MDEV-20501: Assertion `maybe_null || !null_value' failed in Item_func_round::date_op

When the optimizer finds a constant (or system) table $TBL which is empty
or has no matching row, it would set table->null_row=true. This is done
even for tables with table->maybe_null==0.

Then, it would proceed to perform query optimization phases (what for?)
which will attempt to evaluate Item expressions referring to $TBL.

Eventually some Item expression will get the value of $TBL.not_null_field,
get SQL NULL and fail an assertion.

Fixed by not performing any query optimization steps after we've got
constant/empty tables with no matching rows.

Test result changes contain a lot of changes like
- ... Impossible WHERE noticed after reading const tables
+ ... no matching row in const table

as well as other changes caused by slightly-different processing of
the special case of empty constant tables.

Author: Sergey Petrunia
Author Date: 2022-12-29 14:46:40 UTC

Make main.innodb_icp test stable

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func

Porting commit 3836098c29ef1b7ff9d5fbde99b690eab73a0df1 (MDEV-26285)
to current versions 10.3+ to fix a problem (MDEV-29447) where field
items that are arguments of a func item may be used before created /
initialised.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func

Porting commit 3836098c29ef1b7ff9d5fbde99b690eab73a0df1 (MDEV-26285)
to current versions 10.3+ to fix a problem (MDEV-29447) where field
items that are arguments of a func item may be used before created /
initialised.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func

Porting commit 3836098c29ef1b7ff9d5fbde99b690eab73a0df1 (MDEV-26285)
to current versions 10.3+ to fix a problem (MDEV-29447) where field
items that are arguments of a func item may be used before created /
initialised.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func

Porting commit 3836098c29ef1b7ff9d5fbde99b690eab73a0df1 (MDEV-26285)
to current versions 10.3+ to fix a problem (MDEV-29447) where field
items that are arguments of a func item may be used before created /
initialised.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Marko Mäkelä
Author Date: 2023-01-03 12:38:27 UTC

WIP merge 10.3 into 10.4

Author: Jan Lindström
Author Date: 2023-01-03 10:18:38 UTC

MDEV-30318 : galera error messages in mariadb log without galera enabled

Do not compile wsrep_provider plugin if WITH_WSREP is not enabled.
We should not enable wsrep_provider plugin if WSREP_ON=OFF and
at that case we can only print information that Plugin
'wsrep-provider' is disabled.

Make sure tests require Galera library 26.4.14 if needed.

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func

Porting commit 3836098c29ef1b7ff9d5fbde99b690eab73a0df1 (MDEV-26285)
to current versions 10.3+ to fix a problem (MDEV-29447) where field
items that are arguments of a func item may be used before created /
initialised.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Yuchen Pei
Author Date: 2023-01-03 05:24:04 UTC

MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func

Porting commit 3836098c29ef1b7ff9d5fbde99b690eab73a0df1 (MDEV-26285)
to current versions 10.3+ to fix a problem (MDEV-29447) where field
items that are arguments of a func item may be used before created /
initialised.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Oleg Smirnov
Author Date: 2023-01-02 07:46:46 UTC

MDEV-16232: Use fewer mini-transactions

SQL Layer support #3.
Add checks of applicability for UPDATEs and DELETEs.

Rename "operations batch" to "mini transactions" to disambiguate
the term "batch" which is already used.

Temporary disable mini-transactions for SELECTs.

Author: Rucha Deodhar
Author Date: 2022-12-29 12:04:53 UTC

MDEV-29381: JSON paths containing dashes are reported as syntax errors in
procedures

Analysis:
While scanning json path, when '-' is encountered while scanning a key,
the the json_path_transition reports incorrect state that denotes
syntax error. Hence it incorrectly reports error.
Fix:
Change JE_SYN to PS_KNMX to keep scanning the key.

Author: Julius Goryavsky
Author Date: 2022-12-19 09:29:55 UTC

MDEV-28669 addendum: additional tests and mtr changes

This commit contains additions to mtr:

1) Running .sh scripts (which can be executed before running
   the test) moved to an earlier stage so that they can create
   directories or perform other useful actions before the boostrap
   phase of the test;
2) Changes have been added that allow the user to set a custom
   location for datadir during a test, both as an option on the
   command line and via a configuration parameter - so that tests
   can be created that use a custom datadir location, and also
   check work with this parameter;
3) Added more precise filters for options that require a new
   bootstrap with rebuilding the database before starting the test;
4) Now mtr sets an environment variable MTR_BOOTSTRAP_OPTS_<suffix>
   that can be used in scripts if these scripts need to know with
   what additional bootstrap parameters was used when server was
   started - which is relevant for those tests that require individual
   bootstrap and then re-start server from within - these tests need
   to know what additional the parameters must be passed to the
   server (except for those specific to this particular text itself),
   for example, this facilitates the launch of tests for galera
   recovery.

This commit also contains tests for Galera: for MDEV-28669,
MDEV-30157, MDEV-30220, MDEV-29591, which depend on the presence
of the above changes in mtr and are themselves a check for these
changes.

Author: Sergey Petrunia
Author Date: 2022-12-28 18:26:45 UTC

MDEV-30310: patch from Monty, a bit amended

Author: Sergei Golubchik
Author Date: 2022-12-28 11:10:38 UTC

MDEV-30102 file missing in development libraries

move mariadb_capi_rename.h out of private server headers,
because it's included by mysql.h which is not private.

Author: Rex Johnston
Author Date: 2022-12-22 17:28:59 UTC

MDEV-28602 Wrong result with outer join, merged derived table and view

    const item reference being mishandled in outer to inner join on
    filling in null values.

Author: Igor Babaev
Author Date: 2022-12-24 04:54:01 UTC

MDEV-29971 Re-design the upper level of handling SELECT statements

The initial patch.
All tests from the main test suite passed.

Author: midenok
Author Date: 2022-12-23 08:45:14 UTC

MDEV-30186 Use of uninitialized value $test_name in substitution

Revert "Format properly mtr report for the test case that is not completed"
This reverts commit 0c62b6d5652a66fa654e91dd27e0a16433920e5c.

Revert "MDEV-25857: MTR should report at least last test that was executed in case of shutdown and not-completed"
This reverts commit 29e8c154172bd6b9dccd9757fddace93be6ad194.

Author: Yuchen Pei
Author Date: 2022-12-01 06:30:10 UTC

MDEV-28526 Spider: remove conn_kind member variables

The conn_kind, which stands for "connection kind", is no longer useful
because the HandlerSocket support is deleted and Spider now has only
one connection kind, SPIDER_CONN_KIND_MYSQL. Remove conn_kind and
related code.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>
Reviewed-by: Nayuta Yanagisawa <nayuta.yanagisawa@mariadb.com>

Author: Yuchen Pei
Author Date: 2022-11-29 01:08:27 UTC

MDEV-29269 Spider: remove #ifdef ITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC

Produced using the following command

unifdef -UITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC -m storage/spider/spd_*

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>
Reviewed-by: nayuta.yanagisawa@mariadb.com

Author: Andrew Hutchings
Author Date: 2022-12-16 15:51:38 UTC

MDEV-30275: mariadb names rather than mysql names should be used

Fix bug in mariadb-service-convert

If mariadb-service-convert is run and the user variable is unset then
this sets `User=` in `[Service]`, which then tries to run mariadb as
root, which in-turn fails. This only happens when mysqld_safe is missing
which is all the time now. So...

1. Don't set `User=` if there is no user variable.
2. User mariadbd-safe instead.

Also
* galera_recovery to use mariadbd
* mtr - wsrep use mariadb executables
* debian/mariadb-server.mariadb.init use mariadbd-safe
* debian/tests/smoke uses mariadb instead of mysql as client.

Co-Author: Daniel Black <daniel@mariadb.org>

Author: Daniel Black
Author Date: 2022-12-12 03:20:08 UTC

MDEV-30205 Move /usr/share/mysql to /usr/share/mariadb

These are mainly internal files so is a low impact change.

The few scripts/mysql*sql where renames to mariadb_* prefix
on the name.

mysql-test renamed to mariadb-test in the final packages

Author: Daniel Black
Author Date: 2022-12-14 07:12:15 UTC

MDEV-30203: create compat packages

For both Deb and RPM, create mariadb-client-compat and
mariadb-server-compat containing the mysql links to the mariadb
named executables/scripts.

The mariadb-client-core mysqlcheck was moved to mariadb-client-compat.

resolve-stack-dump was moved from server to client (RPM), like
where the man page and Debian package put it.

The symlinks in MYSQL_ADD_EXECUTABLE is tagged as a
{Client,Server}Symlinks component and placed in
the symlinks packages.

Man pages are restructured be installed into compat package
if that matches the executable.

Plugins now can have their own manual pages, as has been
done for rocksdb and s3.

Columnstore has a workaround as it doesn't use the cmake/plugin.cmake.

Additionally, without WSREP, no galera man pages are
installed.

Scripts likewise have compatibility symlinks are in
the {server,client}-compat packages.

Co-author: Andrew Hutchings <andrew@linuxjedi.co.uk>

Author: Monty
Author Date: 2022-12-20 13:01:28 UTC

Fixed usage of unitialised value error in test_sql_service

This caused valgrind errors when using plugins.test_sql_service

Author: Sergey Petrunia
Author Date: 2022-12-20 15:26:13 UTC

MDEV-30218: Incorrect optimization for rowid_filtering

Patch from Igor Babaev:

- Revert the changes in commit 87eccd

- Instead:
- When considering rowid filter with eq_ref access,
  assume the cost of index-only eq_ref scan is 50% of
  non-index-only eq_ref scan.
- For other kinds of keys: Cap the cost of index-only read estimate
  to the cost of non-index-only read estimate. This is to fix cost
  model inconsistencies where cost of index-only estimate ends up
  being higher.

- When considering rowid filter with range access, do not forget to
  multiply quick_index_only_costs[key_no] by record_count.

Author: Jan Lindström
Author Date: 2022-12-20 10:21:38 UTC

Add info about BF-BF lock wait

Author: Marko Mäkelä
Author Date: 2022-12-17 17:42:01 UTC

fixup! bcf43d7690fac0611273f606803f175b0203c324

Leave innodb_undo_log_truncate=OFF unchanged, as was the intention.

Author: Sergei Golubchik
Author Date: 2022-12-08 19:12:01 UTC

MDEV-30128 remove support for 5.1- replication events

including patches from Andrei Elkin

Author: Nikita Malyavin
Author Date: 2022-12-13 11:56:45 UTC

MDEV-15990 REPLACE on a precise-versioned table returns ER_DUP_ENTRY

Author: Dmitry Shulga
Author Date: 2022-12-15 14:19:56 UTC

MDEV-5816: Stored programs: validation of stored program statements

This is the follow-up to the commit i943b463cfaa303845a298d50a5c757cba6ab05e4
that fixes the issue with NEW/OLD pseudo-column in re-comipling SP statement.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-12-14 13:36:27 UTC

MDEV-29988: (spider fix) Major performance regression with 10.6.11

Make Item_direct_ref_to_item transparent for Spider

Author: Yuchen Pei
Author Date: 2022-12-14 23:42:07 UTC

MDEV-21007 Do not assert auto_increment_value unless all parts open

Commit 6dce6aecebe6ef78a14cb5c5c5daa8a355551e40 breaks out of a loop
in ha_partition::info when some partitions aren't opened, in which
case auto_increment_value assertion will fail. This commit patches
that hole.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>
Reviewed-by: Alexey Botchkov <holyfoot@mariadb.com>

Author: Julius Goryavsky
Author Date: 2022-12-13 18:06:34 UTC

MDEV-28669 addendum: additional tests and mtr changes

This commit contains additions to mtr:

1) Running .sh scripts (which can be executed before running
   the test) moved to an earlier stage so that they can create
   directories or perform other useful actions before the boostrap
   phase of the test;
2) Changes have been added that allow the user to set a custom
   location for datadir during a test, both as an option on the
   command line and via a configuration parameter - so that tests
   can be created that use a custom datadir location, and also
   check work with this parameter;
3) Added more precise filters for options that require a new
   bootstrap with rebuilding the database before starting the test;
4) Now mtr sets an environment variable MTR_BOOTSTRAP_OPTS_<suffix>
   that can be used in scripts if these scripts need to know with
   what additional bootstrap parameters was used when server was
   started - which is relevant for those tests that require individual
   bootstrap and then re-start server from within - these tests need
   to know what additional the parameters must be passed to the
   server (except for those specific to this particular text itself),
   for example, this facilitates the launch of tests for galera
   recovery.

This commit also contains tests for Galera: for MDEV-28669,
MDEV-30157, MDEV-30220, MDEV-29591, which depend on the presence
of the above changes in mtr and are themselves a check for these
changes.

Author: Daniel Black
Author Date: 2022-12-13 23:07:56 UTC

MDEV-30222: display errno for popen failures

Author: Sergey Petrunia
Author Date: 2022-12-08 18:24:31 UTC

MDEV-21095: Make Optimizer Trace support Index Condition Pushdown

Fixes over previous patches: do tracing of attached conditions
close to where we generate them.

Fix the tracing code to print the right conditions.

Author: Alexey Botchkov
Author Date: 2022-12-13 11:11:27 UTC

fix to make the RENAME working.

Author: Daniel Black
Author Date: 2022-12-13 01:21:25 UTC

p3 -part revert in debian/rules

Author: Julius Goryavsky
Author Date: 2022-10-28 09:16:25 UTC

MDEV-29814: galera_var_notify_ssl_ipv6 causes testing system to hang

This commit fixes the test system hanging due to
the galera_var_notify_ssl_ipv6 test and also brings
the wsrep_notify[_ssl].sh files in line with each other
between the user template and the mtr suite.

Quotes are also added here to avoid problems if the
user specifies the value of one of the variables at the
beginning of the file containing shell-specific characters,
for example, if the password or username specified in the
PSWD and USER variables will contain the "$" character.

Also fixed an issue with automatic --ssl-verify-server-cert
option substitution when the corresponding value is set
by the user to "1" or "on".

Also fixed some tests here to avoid joining one of the nodes
to another cluster when the nodes are restarted from the mtr
side, which can lead to random failures when testing with
buildbot.

Author: Julius Goryavsky
Author Date: 2022-12-09 00:36:57 UTC

galera_3nodes tests corrected

Author: Brandon Nesterenko
Author Date: 2022-12-08 22:04:09 UTC

MDEV-21469 Test and error message fixups

Author: Sergey Petrunia
Author Date: 2022-12-08 18:24:31 UTC

MDEV-21095: Make Optimizer Trace support Index Condition Pushdown

Fixes over previous patches: do tracing of attached conditions
close to where we generate them.

Fix the tracing code to print the right conditions.

Author: Julius Goryavsky
Author Date: 2022-12-08 14:12:08 UTC

MDEV-6536 addendum: special handling of localhost

Linux and glibc return only the local ipv4 address
for localhost unless an alternative ipv6 address is
explicitly entered in /etc/hosts...

Author: Jan Lindström
Author Date: 2022-12-08 10:57:35 UTC

WIP

Author: Vladislav Vaintroub
Author Date: 2022-12-07 18:47:58 UTC

WIP MDEV-30136 Fix compile error.

Also make it more obvious, why fil_system.buffered settiong is
sometimes ignored.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-12-06 09:22:06 UTC

MDEV-26161: fix of view protocol

Author: midenok
Author Date: 2022-12-05 01:07:56 UTC

MDEV-20865 Store foreign key info in TABLE_SHARE

1. Access foreign keys via TABLE_SHARE::foreign_keys and
  TABLE_SHARE::referenced_keys;

foreign_keys and referenced_keys are objects in TABLE_SHARE.

2. Remove handler FK interface:

  - get_foreign_key_list()
  - get_parent_foreign_key_list()
  - referenced_by_foreign_key()

3. Invalidate referenced shares on:

  - RENAME TABLE
  - DROP TABLE
  - RENAME COLUMN
  - ADD FOREIGN KEY

When foreign table is created or altered by the above operations all
referenced shares are closed. This blocks the operation while any
referenced shares are used (when at least one its TABLE instance is
locked).

4. Update referenced shares on:

  - CREATE TABLE

On CREATE TABLE add items to referenced_keys of referenced shares.

5. Invalidate foreign shares on:

  - RENAME TABLE
  - RENAME COLUMN

The above-mentioned blocking takes effect.

6. Check foreign/referenced shares consistency on:

  - CHECK TABLE

7. Temporary change until MDEV-21051:

InnoDB fill foreign key info at handler open().

On first TABLE open FK info is loaded from storage engine into
TABLE_SHARE. All referenced shares (if any exist) are closed. This
leads to blocking of first time foreign table open while referenced
tables are used.

Restore states of referenced shares in case of errors.

FOREIGN_KEY_INFO refactored to FK_info holding Lex_cstring

(MDEV-21311) Converge Foreign_key and supplemental generated Key together

mysql_prepare_create_table() does data validation and such utilities
as automatic name generation. But it does that only for indexes and
ignores Foreign_key objects. Now as Foreign_key data needs to be
stored in FRM files as well this processing must be done for them like
for any other Key objects.

Replace Key::FOREIGN_KEY type with Key::foreign flag of type
Key::MULTIPLE and Key::generated set to true. Construct one object
with Key::foreign == true instead of two objects of type
Key::FOREIGN_KEY and Key::MULTIPLE.

(MDEV-21051) datadict refactorings

- Move read_extra2() to datadict.cc
- Refactored extra2_fields to Extra2_info
- build_frm_image() readability

(MDEV-21051) build_table_shadow_filename() refactoring

mysql_prepare_alter_table() leaks fixes

(MDEV-21051) amend system tables locking restriction

Table mysql.help_relation has foreign key to mysql.help_keyword. On
bootstrap when help_relation is opened, it preopens help_keyword for
READ and fails in lock_tables_check().

If system table is opened for write then fk references are opened for
write.

Related to: Bug#25422, WL#3984
Tests: main.lock

(MDEV-21051) Store and read foreign key info into/from FRM files

1. Introduce Foreign_key_io class which creates/parses binary stream
containing foreign key structures. Referenced tables store there only
hints about foreign tables (their db and name), they restore full info
from the corresponding tables.

Foreign_key_io is stored under new EXTRA2_FOREIGN_KEY_INFO field in
extra2 section of FRM file.

2. Modify mysql_prepare_create_table() to generate names for foreign
keys. Until InnoDB storage of foreign keys is removed, FK names must
be unique across the database: the FK name must be based on table
name.

3. Keep stored data in sync on DDL changes. Referenced tables update
their foreign hints after following operations on foreign tables:

  - RENAME TABLE
  - DROP TABLE
  - CREATE TABLE
  - ADD FOREIGN KEY
  - DROP FOREIGN KEY

Foreign tables update their foreign info after following operations on
referenced tables:

  - RENAME TABLE
  - RENAME COLUMN

4. To achieve 3. there must be ability to rewrite extra2 section of
FRM file without full reparse. FRM binary is built from primary
structures like HA_CREATE_INFO and cannot be built from TABLE_SHARE.

Use shadow write and rename like fast_alter_partition_table() does.

Create table workflow:

1. Foreign_key is constructed in parser, placed into alter_info->key_list;

2. mysql_prepare_create_table() translates them to FK_info, assigns
   foreign_id if needed;

3. build_frm_image() writes two FK_info lists into FRM's extra2
   section, for referenced keys it stores only table names (hints);

4. init_from_binary_frm_image() parses extra2 section and fills
   foreign_keys and referenced_keys of TABLE_SHARE.

   It restores referenced_keys by reading hint list of table names,
   opening corresponding shares and restoring FK_info from their
   foreign_keys. Hints resolution is done only when initializing
   non-temporary shares. Usually temporary share has different
   (temporary) name and it is impossible to resolve foreign keys by that
   name (as we identify them by both foreign and referenced table
   names). Another not unimportant reason is performance: this saves
   spare share acquisitions.

Alter table workflow:

1. Foreign_key is constructed in parser, placed into
   alter_info->key_list;

2. mysql_prepare_alter_table() prepares action lists and share list of
   foreigns/references;

3. mysql_prepare_alter_table() locks list of foreigns/references by
   MDL_INTENTION_EXCLUSIVE, acquires shares;

4. prepare_create_table() converts key_list into FK_list, assigns
   foreign_id;

5. shadow FRM of altered table is created;

6. data is copied;

7. altered table is locked by MDL_EXCLUSIVE;

8. fk_handle_alter() processes action lists, creates FK backups,
   modifies shares, writes shadow FRMs;

9. altered table is closed;

10. shadow FRMs are installed;

11. altered table is renamed, FRM backup deleted;

12. (TBD in MDEV-21053) shadow FRMs installation log closed, backups deleted;

On FK backup system:

In case of failed DDL operation all shares that was modified must be
restored into original state. This is done by FK_ddl_backup (CREATE,
DROP), FK_rename_backup (RENAME), FK_alter_backup (ALTER).

On STL usage:

STL is used for utility not performance-critical algorithms, core
structures hold native List. A wrapper was made to convert STL
exception into bool error status or NULL value.

MDEV-20865 fk_check_consistency() in CHECK TABLE

Self-refs fix

Test table_flags fix: "debug" deviation is now gone.

Author: Jan Lindström
Author Date: 2022-12-02 09:11:39 UTC

Galera test case cleanup

* Delete tests that are not supported and not going to be supported
  any time soon
* Fix result set on tests that are not run on bb
* Fix tests that fail because of auto increment offset
* Make sure that disabled tests have open bug report

Author: Jan Lindström
Author Date: 2022-11-29 05:21:08 UTC

Galera test case cleanup

 * Delete tests that are not supported and not going to be supported
   any time soon
 * Fix result set on tests that are not run on bb
 * Fix tests that fail because of auto increment offset
 * Make sure that disabled tests have open bug report

Author: Yuchen Pei
Author Date: 2022-12-01 06:30:10 UTC

MDEV-28526 Spider: remove conn_kind member variables

The conn_kind, which stands for "connection kind", is no longer useful
because the HandlerSocket support is deleted and Spider now has only
one connection kind, SPIDER_CONN_KIND_MYSQL. Remove conn_kind and
related code.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>
Reviewed-by: Nayuta Yanagisawa <nayuta.yanagisawa@mariadb.com>

Author: Angelique Sklavounos
Author Date: 2022-12-02 01:46:45 UTC

MDEV-16951: This was caused by a race condition where the log rotation occurs before the Binlog_checkpoint is written to the binary log indicated in the error message. Fixed by ensuring Binlog_checkpoint is written before the binary log rotation.

Author: Alexey Botchkov
Author Date: 2022-11-29 11:10:51 UTC

m_mysql removed.

Author: midenok
Author Date: 2022-12-01 13:34:17 UTC

MDEV-30112 ASAN errors in Item_ident::print / generate_partition_syntax

Like in MDEV-16110 we must release items allocated on thd->mem_root by
reopening the table.

MDEV-16290 relocated MDEV-16110 fix in 10.5 so it works for MDEV-28576
as well. 10.3 without MDEV-16290 now duplicates this fix.

Author: Vicențiu Ciorbaru
Author Date: 2022-11-24 17:50:14 UTC

MDEV-30023 Revoking Privilege on the Column Yields the Error

The change from MDEV-29465 exposed a flaw in replace_column_table
where again we were not properly updating the column-level bits.

replace_table_table was changed in MDEV-29465 to properly update
grant_table->init_cols, however replace_column_table still only
modified grant_column->rights when the GRANT_COLUMN already existed.

This lead to a missmatch between GRANT_COLUMN::init_rights and
GRANT_COLUMN::rights, *if* the GRANT_COLUMN already existed.

As an example:

GRANT SELECT (col1) ...
Here:
For col1
GRANT_COLUMN::init_rights and GRANT_COLUMN::rights are set to 1 (SELECT) in
replace_column_table.

GRANT INSERT (col1) ...
Here, without this patch GRANT_COLUMN::init_rights is still 1 and
GRANT_COLUMN::rights is 3 (SELECT_PRIV | INSERT_PRIV)

Finally, if before this patch, one does:

REVOKE SELECT (col1) ...

replace_table_table will see that init_rights loses bit 1 thus it
considers there are no more rights granted on that particular table.

This prompts the whole GRANT_TABLE to be removed via the first revoke,
when the GRANT_COLUMN corresponding to it should still have init_rights == 2.

By also updating replace_column_table to keep init_rights in sync
properly, the issue is resolved.

Reviewed by <serg@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2022-11-30 18:59:38 UTC

MDEV-29981: Replica stops with 'Found invalid event in binary log'

Prototype commit for a primary to print certain event's
headers before sending to replica to facilitate debugging
a release build. Specifically, it will output the header
of a fake rotate event, the following format description
event, and the next event.

Author: Nikita Malyavin
Author Date: 2022-11-29 19:40:07 UTC

MDEV-29069 follow-up: improve DEFAULT rules

previously, fields with DEFAULTs were allowed just when expression is
deterministic. In case of online alter, we should recursively check that
underlying fields of expression also either have explicit values, or
have DEFAULT following this validity rule.

Author: Daniele Sciascia
Author Date: 2022-11-11 12:08:20 UTC

MDEV-29878 Galera test failure on MDEV-26575

Test MDEV-26575 fails when it runs after MDEV-25389. This is because
the latter simulates a failure while an applier thread is
created in `start_wsrep_THD()`. The failure was not handled correctly
and would not cleanup the created THD from the global
`server_threads`. A subsequent shutdown would hang and eventually fail
trying to close this THD.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Author: sjaakola
Author Date: 2022-11-08 14:36:34 UTC

MDEV-29512 deadlock between commit monitor and THD::LOCK_thd_data mutex

This commit contains only a mtr test for reproducing the issue in MDEV-29512
The actual fix will be pushed in wsrep-lib repository

The hanging in MDEV-29512 happens when binlog purging is attempted, and there is
one local BF aborted transaction waiting for commit monitor.

The test will launch two node cluster and enable binlogging with expire log days,
to force binlog purging to happen.
A local transaction is executed so that will become BF abort victim, and has advanced
to replication stage waiting for commit monitor for final cleanup (to mark position in innodb)
after that, applier is released to complete the BF abort and due to binlog configuration,
starting the binlog purging. This is where the hanging would occur, if code is buggy

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Author: Yuchen Pei
Author Date: 2022-11-29 05:46:53 UTC

MDEV-28891 Spider: remove #ifdef SPIDER_FIELD_FIELDPTR_REQUIRES_THDPTR

unifdef -DSPIDER_FIELD_FIELDPTR_REQUIRES_THDPTR -m
storage/spider/*.{cc,h}

and removing the definition manually.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>
Reviewed-by: Nayuta Yanagisawa <nayuta.yanagisawa@mariadb.com>

Author: Yuchen Pei
Author Date: 2022-11-29 01:08:27 UTC

MDEV-29269 Spider: remove #ifdef ITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC

Produced using the following command

unifdef -UITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC -m storage/spider/spd_*

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>
Reviewed-by: nayuta.yanagisawa@mariadb.com

Author: Monty
Author Date: 2022-11-28 16:37:55 UTC

Fixed warning from innodb.create_isl_with_direct if have_symlink is disabled

Author: Lena Startseva
Author Date: 2022-11-21 09:59:48 UTC

MDEV-29390: Improve coverage for UPDATE and DELETE statements in MTR test suites

For the update_use_source.test tests, data recovery in the table has been changed
from a rollback transaction to a complete delete and re-insert of the data with
optimaze table. Cases are now being checked on three engines.

Author: Julius Goryavsky
Author Date: 2022-11-22 11:11:49 UTC

hashicorp vault plugin: added statistics variables

Author: Sergey Petrunia
Author Date: 2022-11-21 14:28:43 UTC

MDEV-30059: Optimizer Trace: plan_prefix should be a comma-separated-list

Author: Sergey Petrunia
Author Date: 2022-11-21 14:28:43 UTC

MDEV-30059: Optimizer Trace: plan_prefix should be a comma-separated-list

Author: Monty
Author Date: 2022-11-14 15:08:09 UTC

Fixed bug in Aria with aria_log files that are exactly 8K

In the case one has an old Aria log file that ands with a Aria checkpoint
and the server restarts after next recovery, just after created a
new Aria log file (of 8K), the Aria recovery code would abort.
If one would try to delete all Aria log files after this (but not the
aria_control_file), the server would crash during recovery.

The problem was that translog_get_last_page_addr() would regard a log file
of exactly 8K as illegal and the rest of the code could not handle this
case.

Another issue was that if there was a crash directly after the log file
head was written to the next page, the code in translog_get_next_chunk()
would crash.

This patch fixes most of the issues, but not all. For Sanja to look at!

Things fixed:
- Added code to ignore 8K log files.
- Removed ASSERT in translog_get_next_chunk() that checks if page only
  contains the log page header.

Author: Monty
Author Date: 2022-11-15 16:55:39 UTC

fixed test result

Author: Marko Mäkelä
Author Date: 2022-11-15 06:59:30 UTC

Merge 10.6

Interesting test failures:

./mtr --parallel=auto --mysqld=--debug-dbug=d,ib_log_checkpoint_avoid \
--max-test-fail=0 --big-test --force \
binlog{,_encryption}.binlog_incident rpl.rpl_skip_incident \
binlog.binlog_killed_simulate rpl.rpl_loaddata_fatal

Author: Nayuta Yanagisawa
Author Date: 2022-11-11 16:47:23 UTC

MDEV-29947 Spider doesn't return all rows when doing a join of two tables with no usable keys

The present commit only has a MTR test case because the bug has been
fixed by reverting the commits that caused it.

Author: Marko Mäkelä
Author Date: 2022-11-08 11:49:55 UTC

MDEV-29504/MDEV-29849 TRUNCATE breaks FOREIGN KEY locking

ha_innobase::referenced_by_foreign_key(): Protect the check with
dict_sys.freeze(), to prevent races with TRUNCATE TABLE.
The test innodb.instant_alter_crash has been adjusted for this
additional locking.

dict_table_is_referenced_by_foreign_key(): Removed (merged to
the only caller).

create_table_info_t::create_table(): Ignore missing indexes for
FOREIGN KEY constraints if foreign_key_checks=0.

create_table_info_t::create_table_update_dict(): Rewritten as
a static function. Do not return any error.

ha_innobase::create(): When trx!=nullptr and we are operating
on a persistent table, do not rollback, commit, or release the
data dictionary latch.

ha_innobase::truncate(): Protect the entire critical section
with an exclusive dict_sys.latch, so that
ha_innobase::referenced_by_foreign_key() on referenced tables
will return a consistent result. In case of a failure,
invoke dict_load_foreigns() to restore also any FOREIGN KEY
constraints.

ha_innobase::free_foreign_key_create_info(): Define inline.

lock_release(): Disregard innodb_evict_tables_on_commit_debug=ON
when dict_sys.locked() holds. It would hold when fts_load_stopword()
is invoked by create_table_info_t::create_table_update_dict().

dict_sys_t::locked(): Return whether the current thread is holding
the exclusive dict_sys.latch.

dict_sys_t::frozen_not_locked(): Return whether any thread is
holding a shared dict_sys.latch.

Author: Angelique Sklavounos
Author Date: 2022-11-07 07:21:29 UTC

MDEV-25611: Dummy commit to test again

Author: Daniel Black
Author Date: 2022-11-07 03:13:46 UTC

service_encryption.h HAVE_ALLOC_H was global

Rather than global include for header files, we use
cmake to make service_encryption.h a generated file
and populate it with the discovery of ALLOCA_H or not.

Fixes: #2289
Closes: #2321

Author: Georg Richter
Author Date: 2022-11-06 14:03:04 UTC

Updated libmariadb

Author: midenok
Author Date: 2022-10-30 23:03:34 UTC

MDEV-29831 Galera crashes when running CoR for a locked table after
    setting the minimum memory for a user session.

Failure happens when finalize_atomic_replace() was already finished
and we removed the table from locked tables list.
finalize_locked_tables() doesn't know about that, it doesn't add back
last deleted lock because operation_failed == true.
reopen_tables() doesn't reopen table and as a result we have NULL in
pos_in_locked_tables->table.

The fix adds the knowledge that the locked_tables_count changed since
the start of the command. And if that happened we
add_back_last_deleted_lock(). That makes MDEV-29544 fix with
locked_tables_decremented deprecated.

Alternative fix would add atomic_replace_finished to Atomic_info and
updated it on successful finalize_atomic_replace(). Then the condition
would look like this:

  if (atomic_replace_finished || !operation_failed)
  {
    /*
      Add back the deleted table and re-created table as a locked table
      This should always work as we have a meta lock on the table.
    */
    thd->locked_tables_list.add_back_last_deleted_lock(pos_in_locked_tables);
  }

Author: Marko Mäkelä
Author Date: 2022-10-31 07:38:59 UTC

MDEV-16232 preparation: Disable row_prebuilt_t::fetch_cache

The minimal change to disable the use of the fetch_cache is the 2 hunks
#if 1/#else/#endif in row_search_mvcc(). The rest is disabling dead code
and data.

Author: Vlad Lesin
Author Date: 2022-10-27 20:35:34 UTC

MDEV-28709 unexpected X lock on Supremum in READ COMMITTED

Post-push fix. The flag of transaction which indicates that it's necessary
to forbid gap lock inheritance after XA PREPARE could be inverted if
lock_release_on_prepare_try() is invoked several times. The fix is to
toggle it on lock_release_on_prepare() exit.

Author: Alexey Botchkov
Author Date: 2022-10-25 11:39:03 UTC

MDEV-29800 MDEV-29800 ERROR 1062 (23000): Duplicate entry 'test-t-0' for key 'PRIMARY' on CoR...

The CREATE TABLE command can appear in the ha_spider::rename_table(),
so should be handled.

Author: Daniel Black
Author Date: 2022-10-27 06:29:11 UTC

MDEV-24377: Fixed review nits

Author: Alexander Barkov
Author Date: 2022-10-26 10:26:21 UTC

MDEV-9777 MyISAM and InnoDB work differently when comparing a TIME column to an empty string

This problem was fixed earlier (presumably by MDEV-9604 in 10.1).
Adding MTR tests only.

Author: Igor Babaev
Author Date: 2022-10-26 01:07:27 UTC

MDEV-28846 Poor performance when rowid filter contains no elements

When a range rowid filter was used with an index ref access the cost of
accessing the index entries for the records rejected by the filter was not
taken into account. For a ref access by an index with big average number
of records per key this led to poor execution plans if selectivity of the
used filter was high.
The patch resolves this problem. It also introduces a minor optimization
that skips look-ups into a filter that turns out to be empty.
With this patch the output of ANALYZE stmt reports the number of look-ups
into used rowid filters.

The test cases that were supposed to use rowid filters have been adjusted
in order to use similar execution plans after this fix.

This patch prepared for 10.5 differs from the patch resolving the problem
for 10.4 because he code that calculates the cost of index only access has
been changed for 10.5 making usage of rowid filters more favorable.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Alexey Botchkov
Author Date: 2022-10-25 11:39:03 UTC

MDEV-29800 MDEV-29800 ERROR 1062 (23000): Duplicate entry 'test-t-0' for key 'PRIMARY' on CoR...

The CREATE TABLE command can appear in the ha_spider::rename_table(),
so should be handled.

Author: Daniel Black
Author Date: 2022-10-24 23:45:57 UTC

MDEV-XXX make my_free mark memory as MEM_UNDEFINED

Allow MSAN/Valgrind to find memory that is reused without
initialization.

Author: Nayuta Yanagisawa
Author Date: 2022-09-27 06:22:57 UTC

MDEV-29644 a potential bug of null pointer dereference in spider_db_mbase::print_warnings()

The function spider_db_mbase::print_warnings() can potentially result
in a null pointer dereference.

Remove the null pointer dereference by cleaning up the function.

Author: Oleg Smirnov
Author Date: 2022-10-23 13:48:43 UTC

MDEV-29640 FederatedX does not properly handle pushdown in case of difference in local and remote table names

FederatedX table on the local server may refer to a table having
another name on the remote server. The remote table may even reside
in a different database. For example:

-- Remote server
CREATE TABLE t1 (id int(32));

-- Local server
CREATE TABLE t2 ENGINE="FEDERATEDX"
CONNECTION="mysql://joe:joespass@192.168.1.111:9308/federatedx/t1";

It's not a problem while the federated_pushdown is disabled 'cause
the CONNECTION strings are being parsed for every table during
the execution, so the table names are translated from local to remote.
But in case of the federated_pushdown the whole query is pushed down
to the engine without any translation, so the remote server may try
to select data from a nonexistent table (for example, query
"SELECT * FROM t2" will try to retrieve data from nonexistent "t2").

Solution: do not allow pushing down queries with tables having
different names on local and remote names.

Author: Daniel Black
Author Date: 2022-10-19 07:07:17 UTC

Deb: remove lsb_release dep and use os-release

If nothing is there, assume its sid, because the
base debian configuration is written for the latest
sid.

Author: Igor Babaev
Author Date: 2022-10-19 02:11:47 UTC

MDEV-7487 Semi-join optimization for single-table update/delete statements

This patch allows to use semi-join optimization at the top level of
single-table update and delete statements.
The problem of supporting such optimization became easy to resolve after
processing a single-table update/delete statement started using JOIN
structure. This allowed to use JOIN::prepare() not only for multi-table
updates/deletes but for single-table ones as well. This was done in the
patch for mdev-28883:
Re-design the upper level of handling UPDATE and DELETE statements.

Note that JOIN::prepare() detects all subqueries that can be considered
as candidates for semi-join optimization. The code added by this patch
looks for such candidates at the top level and if such candidates are found
in the processed single-table update/delete the statement is handled in
the same way as a multi-table update/delete.

    Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Igor Babaev
Author Date: 2022-10-19 00:19:21 UTC

MDEV-7487 Semi-join optimization for single-table update/delete statements

This patch allows to use semi-join optimization at the top level of
single-table update and delete statements.
The problem of supporting such optimization became easy to resolve after
processing a single-table update/delete statement started using JOIN
structure. This allowed to use JOIN::prepare() not only for multi-table
updates/deletes but for single-table ones as well. This was done in the
patch for mdev-28883:
Re-design the upper level of handling UPDATE and DELETE statements.

Note that JOIN::prepare() detects all subqueries that can be considered
as candidates for semi-join optimization. The code added by this patch
looks for such candidates at the top level and if such candidates are found
in the processed single-table update/delete the statement is handled in
the same way as a multi-table update/delete.

    Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Monty
Author Date: 2022-10-18 16:57:23 UTC

Disable valgrind for rpl_start_alter_restart_master

This is needed as the test does restarts, which doesn't work that well
with valgrind.