lp:maria

Author: Jan Lindström
Author Date: 2022-01-11 12:37:30 UTC

MDEV-25977 : Warning: Memory not freed: 32 on SET GLOBAL wsrep_sst_auth=USER

Add missing wsrep_sst_auth_free call.

Author: Julius Goryavsky
Author Date: 2022-01-18 02:12:42 UTC

MDEV-27524: Incorrect binlogs after Galera SST using rsync and mariabackup

This commit adds correct handling of binlogs for SST using rsync
or mariabackup. Before this fix, binlogs were handled incorrectly -
- only one (last) binary log file was transferred during SST, which
then led to various failures, for example, when trying to list all
events from the binary log. These bugs were long masked by flaws in
the very primitive binlogs handling code in the SST scripts, which
caused binary logs files to be erased or not added to the index on
the joiner node. Now the correct transfer of all binlogs (not just
the last of the binary log file) has been added both for the rsync
(at the script level) and for the mariabackup (at the level of the
main utility code).

Also this commit adds a new sst_max_binlogs=<n> parameter, which
can be located in the [sst] section or in the [xtrabackup] section
(historically and for mariabackup only), or in one of the server
sections. This parameter specifies the number of binary logs to
be sent to the joiner node during SST. This option is added for
compatibility with old SST scripting behavior, which can be emulated
by setting the sst_max_binlogs=1 (although in general this can cause
problems for the reasons described above). In addition, setting
the sst_max_binlogs=0 mode can be used to suppress the transmission
of binary logs to the joiner nodes (although sometimes a single
current binary log can still be transmitted to the joiner even with
sst_max_binlogs=0, because this sometimes necessary in modes that
involve the use of GTIDs with Galera).

In addition, this commit contains the following enhancements:

 1) Added tests for mtr, which check the correct work with binlogs
    after SST (using rsync and mariabackup);
 2) Added correct handling of slashes at the end of all paths
    that the SST script receives as parameters;
 3) Improved parsing code for mysqld_args parameters, now it
    correctly processes the sequence "--" after the name of
    the one-letter option;
 4) Checking the secret code during joiner authentication is
    made independent of bash (as a unix shell);
 5) All directories that are necessary for the correct placement
    of various logs are created in advance (before running
    mariabackup on joiner);
 6) Removal of old binlogs on joiner is done by binlogs index
    (if it exists) and not by pattern with current binlog name;
 7) Paths for placing binlogs are correctly processed if they
    are set relative to datadir;
 8) Scripts are made even more resistant to spaces in filenames
    (now for binlogs);
 9) In case of errors, scripts should now always end with
    an exit code other than zero;
10) SST scripts also now correctly create a tar file with
    binlogs if the paths to them in the index file are specified
    as absolute paths, even if they do not match the last one.

Author: Andrei
Author Date: 2022-01-17 19:27:30 UTC

Corrects the replication alter state in "special" goto-end_inplace branches

Two mysql_alter_table execution branches missed to invoke
write_bin_log_start_alter and wait_for_master prior to process_master_state.

This is corrected to indentify the need and call them inside
process_master_state.
As these branches are fast thanks to the goto, so calling the two
functions at the receiving end of the goto is fine for performance
as well.

Author: Sergei Golubchik
Author Date: 2022-01-05 21:55:07 UTC

MDEV-27341 Use SET PASSWORD to change PAM service

SET PASSWORD = PASSWORD('foo') would fail for pam plugin with

ERROR HY000: SET PASSWORD is ignored for users authenticating via pam plugin

but SET PASSWORD = 'foo' would not.

Now it will.

Author: Sergei Golubchik
Author Date: 2022-01-05 21:55:07 UTC

MDEV-27341 Use SET PASSWORD to change PAM service

SET PASSWORD = PASSWORD('foo') would fail for pam plugin with

ERROR HY000: SET PASSWORD is ignored for users authenticating via pam plugin

but SET PASSWORD = 'foo' would not.

Now it will.

Author: Sergei Golubchik
Author Date: 2021-12-29 13:37:03 UTC

MDEV-26230 mysql_upgrade fails to load type_mysql_json due to insufficient maturity level

bump maturity to beta

Author: Sergei Golubchik
Author Date: 2021-12-29 13:37:03 UTC

MDEV-26230 mysql_upgrade fails to load type_mysql_json due to insufficient maturity level

bump maturity to beta

Author: Eugene
Author Date: 2022-01-17 15:18:30 UTC

MDEV-27272 Crash on EXPORT/IMPORT tablespace with column added in the middle

dict_index_t::reconstruct_fields(): add input validation by replacing some
assertions

handle_instant_metadata(): fix nullptr dereference

Author: Vladislav Vaintroub
Author Date: 2022-01-13 16:27:28 UTC

MDEV-27494 Rename .ic files to .inl

Author: Anel Husakovic
Author Date: 2022-01-17 15:26:47 UTC

MDEV-18284: JSON casting using JSON_COMPACT doesn't always work with values from subqueries

- Cherry-pick 2fcff310d024cc2201586c568391ba8b039f0bf3 (MDEV-21902)
- Closed PR #1145
Reviewed by: <>

Author: Marko Mäkelä
Author Date: 2022-01-17 14:56:07 UTC

MDEV-27499 Performance regression in log_checkpoint_margin()

In commit 4c3ad24413f234599eda27f4958dd3ff21df3203 (MDEV-27416)
an unnecessarily strict wait condition was introduced in the
function buf_flush_wait(). Most callers actually only care that
the pages have been flushed, not that a checkpoint has completed.

Only in the buf_flush_sync() call for log resizing, we might care
about the log checkpoint. But, in fact,
srv_prepare_to_delete_redo_log_file() is explicitly disabling
checkpoints. So, we can simply remove the unnecessary wait loop.

Thanks to Krunal Bauskar for reporting this performance regression
that we failed to repeat in our testing.

Author: Vladislav Vaintroub
Author Date: 2022-01-17 14:48:09 UTC

Appveyor : test with system OpenSSL3

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2022-01-13 10:02:50 UTC

MDEV-27424 mariabackup ignores physically corrupt first pages

- Store the deferred tablespace and check whether we have
FILE_CREATE redo log for the tablespace in backup_fix_ddl().
If there is no FILE_CREATE redo then Mariabackup should
display the error and backup should fail.

Author: Andrei
Author Date: 2022-01-17 12:46:09 UTC

Fixed

  Assertion `rgi->gtid_ev_flags_extra & Gtid_log_event::FL_COMMIT_ALTER_E1'

that unexectedly met RA which still was harmless.

Instead of the assert arg extending RA execution gets cut short
into binlogging, so the original assert indeed must not see RA
coming it anymore.

Author: Marko Mäkelä
Author Date: 2022-01-17 09:19:41 UTC

MDEV-27469: Assertion failure in defragment due to tx_read_only

In commit c5fd9aa562fb15e8d6ededceccbec0c9792a3243 (MDEV-25919)
we prevented the function dict_stats_save_index_stat()
from being called in read-only mode in dict_stats_save(),
but not elsewhere.

dict_stats_save_defrag_summary(), dict_stats_save_defrag_stats():
If the transaction is in read-only mode, return DB_READ_ONLY
and do not attempt to lock or modify anything.

Author: Alexander Barkov
Author Date: 2022-01-10 14:05:55 UTC

MDEV-27018 IF and COALESCE lose "json" property

Hybrid functions (IF, COALESCE, etc) did not preserve the JSON property
from their arguments. The same problem was repeatable for single row subselects.

The problem happened because the method Item::is_json_type() was inconsistently
implemented across the Item hierarchy. For example, Item_hybrid_func
and Item_singlerow_subselect did not override is_json_type().

Solution:

- Removing Item::is_json_type()

- Implementing specific JSON type handlers:
  Type_handler_string_json
  Type_handler_varchar_json
  Type_handler_tiny_blob_json
  Type_handler_blob_json
  Type_handler_medium_blob_json
  Type_handler_long_blob_json

- Reusing the existing data type infrastructure to pass JSON
  type handlers across all item types, including classes Item_hybrid_func
  and Item_singlerow_subselect. Note, these two classes themselves do not
  need any changes!

- Extending the data type infrastructure so data types can inherit
  their properties (e.g. aggregation rules) from their base data types.
  E.g. VARCHAR/JSON acts as VARCHAR, LONGTEXT/JSON acts as LONGTEXT
  when mixed to a non-JSON data type. This is done by:
    - adding virtual method Type_handler::type_handler_base()
    - adding class Recursive_type_pair_iterator
    - refactoring Type_handler_hybrid_field_type methods
      aggregate_for_result(), aggregate_for_min_max(),
      aggregate_for_num_op() to use Recursive_type_pair_iterator.

This change also fixes:

  MDEV-27361 Hybrid functions with JSON arguments do not send format metadata

Author: Vlad Lesin
Author Date: 2022-01-17 07:31:21 UTC

Crash on rows count mismatch during "CHECK TABLE".

This commit is only for debug purpose, remove it after bug fix.

Author: Marko Mäkelä
Author Date: 2022-01-17 07:17:08 UTC

MDEV-27519 2-argument CRC32 call upon Columnstore table returns a wrong value

Author: Julius Goryavsky
Author Date: 2022-01-17 02:00:42 UTC

Experimental change

Author: Otto Kekäläinen
Author Date: 2020-02-20 11:05:15 UTC

Deb: Track libmariadb3 ABI explicitly to detect future symbol changes

When this file exists, Debian builds will automatically compare the built
ABI and symbols to the ones defined in the list. If there is a mismatch,
developers need to update the libmariadb3.symbols file, otherwise the build
fails.

This ensures there are no accidental symbol changes. This also helps track
what symbols changed and in what versions.

Also add a README embedded in the sources to facilitate correct use of this
new file.

This change is made for branch 10.2, as it was the first server version to
introduce an embedded libmariadb3 client library, and from this version
the file with the same libmariadb3 specific content will be merged to
10.3, 10.4 and 10.5.

Related: MDEV-21732

Author: Eugene Kosov
Author Date: 2020-06-18 14:26:01 UTC

MDEV-22441 implement a generic way to change a value of a variable in a scope

Example:
{
  auto _= make_scope_value(var, tmp_value);
}

make_scope_value(): a function which returns RAII object which temporary
changes a value of a variable

detail::Scope_value: actual implementation of such RAII class.
It shouldn't be used directly! That's why it's inside a namespace detail.

Author: Sergey Petrunia
Author Date: 2022-01-16 17:38:35 UTC

Make tests stable

Author: Georg Richter
Author Date: 2022-01-16 07:16:02 UTC

updated connector/c

Author: Sergei Golubchik
Author Date: 2022-01-15 16:33:28 UTC

don't build with OpenSSL 3.0, it doesn't work before MDEV-25785

Author: midenok
Author Date: 2022-01-15 11:17:19 UTC

MDEV-27217 typo fix

Author: midenok
Author Date: 2022-01-15 11:17:19 UTC

MDEV-27217 typo fix

Author: Igor Babaev
Author Date: 2022-01-14 06:46:59 UTC

MDEV-27132 Wrong result from query when using split optimization

This bug could affect queries with a grouping derived table containing
equalities in the where clause of its specification if the optimizer
chose to apply split optimization to access the derived table. In such
cases wrong results could be returned from the queries.
When the optimizer considers a possibility of using split optimization
to a derived table it injects equalities joining the derived table with
other tables into the where condition of the derived table. After the join
order for the execution using split optimization has been chosen as the
cheapest the injected equalities that are not used to access the derived
table are removed from the where condition of the derived table.
For this removal the optimizer looks through the conjuncts of the where
condition of the derived table, fetches the equalities and checks whether
they belong to the list of injected equalities.
As the injection of the list was performed just by the insertion of it
into the list of top level AND condition of the where condition some extra
conjuncts from the where condition could be automatically attached to the
end of the list of injected equalities. If such attached conjunct happened
to be an equality predicate it was removed from the where condition of the
derived table and thus lost for checking at the execution phase.
The bug has been fixed by injecting of a shallow copy of the list of the
pushed equalities rather than the list itself leaving the latter intact.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Marko Mäkelä
Author Date: 2022-01-14 17:52:40 UTC

MDEV-27199: Remove FIL_PAGE_FILE_FLUSH_LSN

The only purpose of the field FIL_PAGE_FILE_FLUSH_LSN was to
store the log sequence number for a new ib_logfile0 when the
InnoDB redo log was missing at startup.

Because FIL_PAGE_FILE_FLUSH_LSN no longer serves any purpose,
we will stop updating it. The writes of that field were inherently
risky, because they were not covered by neither the redo log nor
the doublewrite buffer.

Author: Marko Mäkelä
Author Date: 2022-01-14 17:30:14 UTC

Merge 10.6 into 10.7

Author: Sergei Golubchik
Author Date: 2022-01-14 13:25:44 UTC

make crc32() behave consistently with other functions that take an INT

Author: Vlad Lesin
Author Date: 2022-01-11 15:19:39 UTC

MDEV-27025 insert-intention lock conflicts with waiting ORDINARY lock

When lock is checked for conflict, ignore other locks on the record if
they wait for the requesting transaction.

lock_rec_has_to_wait_in_queue() iterates not all locks for
the page, but only the locks located before the waiting lock in the
queue. So there is some invariant - any lock in the queue can wait only
lock which is located before the waiting lock in the queue.

In the case when conflicting lock waits for the transaction of
requesting lock, we need to place the requesting lock before the waiting
lock in the queue to preserve the invariant. That is why we are looking
for the first waiting for requesting transation lock and place the new
lock just after the last granted requesting transaction lock before the
first waiting for requesting transaction lock.

Example:

trx1 waiting lock, trx1 granted lock, ..., trx2 lock - waiting for trx1
place new lock here -----------------^

There are also implicit locks which are lazily converted to explicit
ones, and we need to place the newly created explicit lock to the correct
place in a queue. All explicit locks converted from implicit ones are
placed just after the last non-waiting lock of the same transaction before
the first waiting for the transaction lock.

trx_lock_t::wait_trx was backported from MDEV-24738 to provide the above
functionality. As there is no need to backport MDEV-24789 for MDEV-27025,
trx_lock_t::wait_trx is protected with the same mutexes as
trx_lock_t::wait_lock.

This fix should not break innodb-lock-schedule-algorithm=VATS. This
algorithm uses an Eldest-Transaction-First (ETF) heuristic, which prefers
older transactions over new ones. In this fix we just insert granted lock
just before the last granted lock of the same transaction, what does not
change transactions execution order.

The changes in lock_rec_create_low() should not break Galera Cluster,
there is a big "if" branch for WSREP. This branch is necessary to provide
the correct transactions execution order, and should not be changed for the
current bug fix.

Author: Nayuta Yanagisawa
Author Date: 2022-01-14 06:58:38 UTC

Revert "MDEV-26345 SELECT MIN on Spider table returns more rows than expected"

This reverts commit b9730226dce5bf34b87aa28963f1df68a695a93c.

Author: Julius Goryavsky
Author Date: 2022-01-14 03:10:48 UTC

Experimental change

Author: Alexander Barkov
Author Date: 2022-01-13 10:20:05 UTC

MDEV-25659 trigger name is empty after upgrade to 10.4

Problem:

At some point, we made stored rountines fail at CREATE time
instead of execution time in case of this syntax:

   IF unknown_variable
     ...
   END IF

As a result, a trigger created before this change and contained an unknown
variable worked in a bad way after upgrade:
- It was displayed with an empty trigger name by SHOW CREATE TRIGGER
- It was displayed with an empty trigger name by INFORMATION_SCHEMA.TRIGGERS
- An attempt to DROP this trigger returned errors - nothing happened.
- DROP TABLE did not remove the .TRN file corresponding to this broken trigger.

Underlying code observations:

The old code assumed that the trigger name resides in the current lex:

  if(thd->lex->spname)
    m_trigger_name= &thd->lex->spname->m_name;

This is not always the case. Some SP statements (e.g. IF)
do the following in their beginning:

- create a separate local LEX
- set thd->lex to this new local LEX
- push the new local LEX to the stack in sp_head::m_lex

and the following at the end of the statement:

- pop the previous LEX from the stack sp_head::m_lex
- set thd->lex back to the popped value

So when the parse error happens inside e.g. IF statement, thd->lex->spname
is a NULL pointer, because thd->lex points to the local LEX (without SP name)
rather than the top level LEX (with SP name).

Fix:
- Adding a new method sp_head::find_spname_recursive()
  which walks inside the LEX stack sp_head::m_lex from
  the top (the newest, most local) to the bottom (the oldest),
  and finds the one which contains a non-zero spname pointer.

- Using the new method inside
  Deprecated_trigger_syntax_handler::handle_condition():
  First it still tests thd->lex->spname (like before this change),
  and uses it in case it is not empty.
  Otherwise (if thd->lex->spname is empty), it calls
  sp_head::find_spname_recursive() to find the LEX with a
  non-empty spname inside the LEX stack of the current sphead.

Author: Vlad Lesin
Author Date: 2021-12-02 13:14:27 UTC

MDEV-27025 insert-intention lock conflicts with waiting ORDINARY lock

When lock is checked for conflict, ignore other locks on the record if
they wait for the requesting transaction.

lock_rec_has_to_wait_in_queue() iterates not all locks for
the page, but only the locks located before the waiting lock in the
queue. So there is some invariant - any lock in the queue can wait only
lock which is located before the waiting lock in the queue.

In the case when conflicting lock waits for the transaction of
requesting lock, we need to place the requesting lock before the waiting
lock in the queue to preserve the invariant. That is why we are looking
for the first waiting for requesting transation lock and place the new
lock just after the last granted requesting transaction lock before the
first waiting for requesting transaction lock.

Example:

trx1 waiting lock, trx1 granted lock, ..., trx2 lock - waiting for trx1
place new lock here -----------------^

There are also implicit locks which are lazily converted to explicit
ones, and we need to place the newly created explicit lock to the correct
place in a queue. All explicit locks converted from implicit ones are
placed just after the last non-waiting lock of the same transaction before
the first waiting for the transaction lock.

Code review and cleanup was made by Marko Mäkelä.

Author: Marko Mäkelä
Author Date: 2022-01-13 16:29:13 UTC

fixup! 6e055f134b290945a4efd842fdfec4d975de4f6f

Correct a debug assertion

Author: Oleg Smirnov
Author Date: 2022-01-13 16:24:22 UTC

MDEV-10000 Add support for FORMAT=JSON

Makes possible calls to SHOW EXPLAIN and EXPLAIN FOR CONNECTION
providing output in JSON format:
- SHOW EXPLAIN FORMAT=JSON FOR $con
- EXPLAIN FORMAT=JSON FOR CONNECTION $con

Author: Sergey Petrunia
Author Date: 2022-01-03 21:01:52 UTC

MDEV-26337: subquery with groupby and ROLLUP returns incorrect results on LEFT JOIN on INDEXED values

Disable LATERAL DERIVED optimization for subqueries that have WITH ROLLUP.

This bug could affect queries with grouping derived tables / views / CTEs
with ROLLUP. The bug could manifest itself if the corresponding
materialized derived tables are subject to split optimization.

The current implementation of the split optimization produces rows
from the derived table in an arbitrary order. So these rows must be
accumulated in another temporary table and sorted according to the
used GROUP BY clause in order to be able to generate the additional
ROLLUP rows.

This patch prohibits to use split optimization for grouping derived
tables / views / CTEs with ROLLUP.

Author: Sergey Petrunia
Author Date: 2022-01-13 12:53:44 UTC

MDEV-27382: OFFSET is ignored when combined with DISTINCT

A query in form

  SELECT DISTINCT expr_that_is_inferred_to_be_const LIMIT 0 OFFSET n

produces one row when it should produce none. The issue was in
JOIN_TAB::remove_duplicates() in the piece of logic that tried to
avoid duplicate removal for such cases but didn't account for possible
"LIMIT 0".

Fixed in two places:
- Make JOIN::optimize_inner be able to infer "Zero limit" for
   "LIMIT 0 OFFSET some_non_zero_value" (in addition to just "LIMIT 0")

- Make JOIN_TAB::remove_duplicates not apply its optimization for cases
  with non-zero OFFSET clause.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2022-01-13 10:02:50 UTC

MDEV-27424 mariabackup ignores physically corrupt first pages

- Store the deferred tablespace and check whether we have
FILE_CREATE redo log for the tablespace in backup_fix_ddl().
If there is no FILE_CREATE redo then Mariabackup should
display the error and backup should fail.

Author: midenok
Author Date: 2022-01-12 18:22:05 UTC

Cleanup: check_native_partitioned() removed

We should not add native partitioning in future.

Author: Alexey Botchkov
Author Date: 2022-01-12 21:08:44 UTC

cmake/pcre.cmake fixed so the server compilation doesn't fail.

Author: midenok
Author Date: 2022-01-12 12:13:58 UTC

MDEV-21650 Non-empty statement transaction on global rollback after TRT update error

TRT opens statement transaction. Cleanup it in case of error.

Author: midenok
Author Date: 2022-01-12 14:18:38 UTC

MDEV-26824 Can't add foreign key with empty referenced columns list

create_table_info_t::create_foreign_keys() expects equal number of
iterations through fk->columns and fk->ref_columns. If fk->ref_columns
is empty copy it from fk->columns.

Author: Rucha Deodhar
Author Date: 2022-01-04 20:53:54 UTC

MDEV-26878: Query failing with syntax error sets ROW_NUMBER to non-zero

Analysis: m_current_row_for_warning counter is set to 0 at the beginning
of parsing.
Fix: Reset it to 1 and increment it before starting to parse insert values,
during SET and SELECT for INSERT...SET statement and INSERT...SELECT
respectively.

Author: Marko Mäkelä
Author Date: 2022-01-11 15:54:49 UTC

Test: Merge buf_pool.flush_order_mutex to log_sys.mutex

This will actually reduce throughput by increasing contention
on log_sys.mutex.

Author: Monty
Author Date: 2022-01-11 12:41:37 UTC

Improve --help and remove not-free warnings for mysql_tzinfo_to_sql

Author: Rucha Deodhar
Author Date: 2021-08-13 10:19:16 UTC

MDEV-24920: Merge "old" SQL variable to "old_mode" sql variable

Analysis: There are 2 server variables- "old_mode" and "old". "old" is no
longer needed as "old_mode" has replaced it (however still used in some places
 in the code). "old_mode" and "old" has same purpose- emulate behavior from
previous MariaDB versions. So they can be merged to avoid confusion.
Fix: Deprecate "old" variable and create another mode for @@old_mode to mimic
behavior of previous "old" variable. Create specific modes for specifix task
that --old sql variable was doing earlier and use the new modes instead.

Author: Rucha Deodhar
Author Date: 2021-08-13 10:19:16 UTC

MDEV-24920: Merge "old" SQL variable to "old_mode" sql variable

Analysis: There are 2 server variables- "old_mode" and "old". "old" is no
longer needed as "old_mode" has replaced it (however still used in some places
 in the code). "old_mode" and "old" has same purpose- emulate behavior from
previous MariaDB versions. So they can be merged to avoid confusion.
Fix: Deprecate "old" variable and create another mode for @@old_mode to mimic
behavior of previous "old" variable. Create specific modes for specifix task
that --old sql variable was doing earlier and use the new modes instead.

Author: Sergei Golubchik
Author Date: 2022-01-03 14:17:04 UTC

C/C 3.3

Author: Jan Lindström
Author Date: 2022-01-09 07:37:44 UTC

MDEV-25201 : Assertion `thd->wsrep_trx_meta.gtid.seqno == (-1)' failed in int wsrep_to_isolation_begin(THD*, const char*, const char*, const TABLE_LIST*, Alter_info*)

Test case does not assert anymore but works incorrectly. We should
not replicate PREPARE using TOI.

Author: Jan Lindström
Author Date: 2022-01-09 07:37:44 UTC

MDEV-25201 : Assertion `thd->wsrep_trx_meta.gtid.seqno == (-1)' failed in int wsrep_to_isolation_begin(THD*, const char*, const char*, const TABLE_LIST*, Alter_info*)

Test case does not assert anymore but works incorrectly. We should
not replicate PREPARE using TOI.

Author: Dmitry Shulga
Author Date: 2022-01-11 04:50:29 UTC

MDEV-20325: Assertion `outer_context || !*from_field || *from_field == not_found_field' failed in Item_field::fix_outer_field | `!derived->is_excluded()' failed in TABLE_LIST::set_check_materialized | SIGEGV in st_select_lex::mark_as_dependent (optimized builds)

Re-execution of a query containing subquery in the FROM clause results
in assert failure in case the query is run as part of a stored routine or
as a prepared statement AND derived table merge optimization is off.
As an example, the following test case
  CREATE TABLE t1 (a INT) ;
  CREATE PROCEDURE sp() SELECT * FROM (SELECT a FROM t1) tb;
  CALL sp();
  SET optimizer_switch='derived_merge=off';
  CALL sp();
results in assert failure on the second invocation of the 'sp' stored routine.

The reason for assertion failure is that the expression
  derived->is_excluded()
returns the value true where the value false expected.

The method is_excluded() returns the value true for a derived table
that has been merged to a parent select. Such transformation happens as part
of Derived Table Merge Optimization that is performed on first invocation of
a stored routine or a prepared statement containing a query with subquery
in the FROM clause of the main SELECT.

When the same routine or prepared statement is run the second time and
Derived Table Merge Optimization is OFF the MariaDB server tries to materialize
a derived table specified by the subquery that fails since this subquery
has already been merged to the top-most SELECT. This transformation is permanent
and can't be reverted. That is the reason why the assert
  DBUG_ASSERT(!derived->is_excluded());
fails inside the function TABLE_LIST::set_check_materialized().

Similar behaviour can be observed in case a stored routine or prepared statement
containing a SELECT statement with subquery in the FROM clause, first is run
with the optimizer_switch option set to derived_merge=off and re-run after this
option has been switched to derived_merge=on. In this case a derived table for
subquery is materialized on the first execution and marked as merged derived
table on the second execution that results in error with misleading error
message:

MariaDB [test]> CALL sp1();
ERROR 1030 (HY000): Got error 1 "Operation not permitted" from storage engine MEMORY

To fix the issue, a derived table that has been already optimized shouldn't be
re-marked for one more round of optimization.

One significant consequence following from suggested change is that the data
member TABLE_LIST::derived_type is not updated once the table optimization
has been done. This fact should be taken into account when Prepared Statement
being handled since once a table listed in a query has been optimized on
execution of the statement PREPARE FROM it won't be touched anymore on handling
the statement EXECUTE.

One side effect caused by this change could be observed for the following
test case:
  CREATE TABLE t1 (s1 INT);
  CREATE VIEW v1 AS
    SELECT s1,s2 FROM (SELECT s1 as s2 FROM t1 WHERE s1 <100) x, t1 WHERE t1.s1=x.s2;
  INSERT INTO v1 (s1) VALUES (-300);

  PREPARE stmt FROM "INSERT INTO v1 (s1) VALUES (-300)";
  EXECUTE stmt;

Execution of the above EXECUTE statement results in issuing the error
ER_COLUMNACCESS_DENIED_ERROR since table_ref->is_merged_derived() is false
and check_column_grant_in_table_ref() called for a temporary table that
shouldn't be. To fix this issue the function find_field_in_tables has been
modified in such a way that the function check_column_grant_in_table_ref()
is not called for a temporary table.

Author: Brandon Nesterenko
Author Date: 2022-01-06 16:02:56 UTC

MDEV-25768: ALTER gtid_slave_pos table storage engine should not be allowed when slave is running.

Problem:
========
The mysql.gtid_slave_pos table can have its engine modified while
the SQL thread is alive.

Solution:
========
Update ALTER logic to ensure that the table can only have its
engine modified when no SQL threads are running.

Reviewed By:
============
<TODO>

Author: midenok
Author Date: 2022-01-10 22:16:57 UTC

MDEV-24176 cleanup_excluding_fields_processor() returned

cached_table_cleanup_processor() removed

This will be the message of the squashed commit:

vcol_info->expr is allocated on expr_arena at parsing stage. Since
expr item is allocated on expr_arena all its containee items must be
allocated on expr_arena too. Otherwise fix_session_expr() will
encounter prematurely freed item.

When table is reopened from cache vcol_info contains stale
expression. We refresh expression via TABLE::vcol_fix_exprs() but
first we must prepare a proper context (Vcol_expr_context) which meets
some requirements:

1. As noted above expr update must be done on expr_arena as there may
be new items created. It was a bug in fix_session_expr_for_read() and
was just not reproduced because of no second refix. Now refix is done
for more cases so it does reproduce. Tests affected: vcol.binlog

2. Also name resolution context must be narrowed to the single table
with all crutches in place. Tests affected: vcol.update

3. sql_mode must be clean and not fail expr update.

sql_mode such as MODE_NO_BACKSLASH_ESCAPES, MODE_NO_ZERO_IN_DATE, etc
must not affect vcol expression update. If the table was created
successfully any further evaluation must not fail. Tests affected:
main.func_like

4. Warnings are disabled during expr update. It is assumed that these
warnings were printed before during initialization phase or later
during execution phase. Tests affected: vcol.wrong_arena

Note: "refresh", "update" and "refix" are the synonyms for
vcol_fix_exprs() or fix_session_expr_for_read() calls.

Author: Igor Babaev
Author Date: 2022-01-06 08:31:42 UTC

MDEV-25631 Crash executing query with VIEW, aggregate and subquery

This bug could cause a crash of the server for queries with a derived table
whose specification contained the set function using a subquery over a view
as its only argument. The crash could happen if the specification of the
view contained an outer reference. In this case the aggregation select
could be determined incorrectly.
The crash also could be observed if a CTE is used instead of the view, but
only for queries having at least two references to the CTE.

Author: Rucha Deodhar
Author Date: 2022-01-10 15:25:19 UTC

MDEV-27462: Assertion `! is_set() || m_can_overwrite_status' failed in
Diagnostics_area::set_error_status

Analysis:
Some system variables can be set per query basis because flag NO_SET_STMT
is missing.
Fix: Added NO_SET_STMT flag. List of disallowed variables taken from
documentation:
autocommit, character_set_client, character_set_connection,
character_set_filesystem, collation_connection, default_master_connection,
debug_sync, interactive_timeout, gtid_domain_id, last_insert_id,
log_slow_filter, log_slow_rate_limit, log_slow_verbosity, long_query_time,
min_examined_row_limit, profiling, profiling_history_size, query_cache_type,
rand_seed1, rand_seed2, skip_replication, slow_query_log, sql_log_off,
tx_isolation, wait_timeout

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2022-01-10 15:44:44 UTC

MDEV-27445 Index inconsistency and assertion failure after renaming virtual column with DESC key

- Commit bc1140b43ce4717cd079814cb6804bbc0f52bc0f fails to keep
the correct the position for non-descending and non-prefix index.
This patch fixes the position value while renaming the column.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2022-01-10 13:45:39 UTC

MDEV-27640 trx_has_lock_x() gives wrong result if the table has pending table lock

trx_has_lock_x() fails to find whether the trx has X-lock on the table
when other transactions are waiting for an X or S lock on the table.

Author: Jan Lindström
Author Date: 2022-01-09 07:37:44 UTC

MDEV-25201 : Assertion `thd->wsrep_trx_meta.gtid.seqno == (-1)' failed in int wsrep_to_isolation_begin(THD*, const char*, const char*, const TABLE_LIST*, Alter_info*)

Test case does not assert anymore but works incorrectly. We should
not replicate PREPARE using TOI.

Author: Jan Lindström
Author Date: 2022-01-05 09:52:33 UTC

MDEV-18848 : Galera: 10.4 node crashed with Assertion `client_state.transaction().active()` after altering SEQUENCE table's engine to myisam and back to innodb

We need to start Galera transaction for select NEXT VALUE FOR
sequence if it is not yet started. Note that ALTER is handled
as TOI and transaction is already started.

Author: Vladislav Vaintroub
Author Date: 2022-01-09 22:52:24 UTC

Silence CMake warning from exteral cmake project (pcre2)

The warning reads:

CMake Deprecation Warning at CMakeLists.txt:101 (CMAKE_MINIMUM_REQUIRED):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Author: Elena Stepanova
Author Date: 2022-01-08 23:39:57 UTC

Test commit, not to be merged anywhere

Author: Igor Babaev
Author Date: 2022-01-07 19:52:25 UTC

MDEV-22846 Server crashes in handler_index_cond_check on SELECT

If the optimizer decides to rewrites a NOT IN predicand of the form
  outer_expr IN (SELECT inner_col FROM ... WHERE subquery_where)
into the EXISTS subquery
  EXISTS (SELECT 1 FROM ... WHERE subquery_where AND
        (outer_expr=inner_col OR inner_col IS NULL))
then the pushed equality predicate outer_expr=inner_col can be used for
ref[or_null] access if inner_col is a reference to an indexed column.
In this case if there is a selective range condition over this column then
a Rowid filter may be employed coupled the with ref[or_null] access. The
filter is 'pushed' into the engine and in InnoDB currently it cannot be
used with index look-ups by primary key. The ref[or_null] access can be
used only when outer_expr is not NULL. Otherwise the original predicand
is evaluated to TRUE only if the result set returned by the query
 SELECT 1 FROM ... WHERE subquery_where
is empty. When performing this evaluation the executor switches to the
table scan by primary key. Before this patch the pushed filter still
remained marked as active and the engine tried to apply the filter. This
was incorrect and in InnoDB this attempt to use the filter led to an
assertion failure.

This patch fixes the problem by disabling usage of the filter when
outer_expr is evaluated to NULL.

Author: Sergei Golubchik
Author Date: 2022-01-07 12:51:07 UTC

MDEV-14938 make buildbot to include galera into bintars

allow injecting extra files into the bintar

Author: Brandon Nesterenko
Author Date: 2021-10-21 02:13:45 UTC

MDEV-11853: semisync thread can be killed after sync binlog but before ACK in the sync state

Problem:
========
If a primary is shutdown during an active semi-sync connection
during the period when the primary is awaiting an ACK, the primary
hard kills the active communication thread and does not ensure the
transaction was received by a replica. This can lead to an
inconsistent replication state.

Solution:
========
During shutdown, the primary should wait for an ACK or timeout
before hard killing a thread which is awaiting a communication. We
extend the `SHUTDOWN WAIT FOR SLAVES` logic to identify and ignore
any threads waiting for a semi-sync ACK in phase 1. Then, before
stopping the ack receiver thread, the shutdown is delayed until all
waiting semi-sync connections receive an ACK or time out. The
connections are then killed in phase 2.

Reviewed By:
============
TODO

Author: Nayuta Yanagisawa
Author Date: 2022-01-04 04:21:14 UTC

MDEV-26583 SIGSEGV's in spider_get_select_limit_from_select_lex when DELAYED INSERT is used

Spider dereferences a freed select_lex and then results in SIGSEGV.

Author: Brandon Nesterenko
Author Date: 2022-01-05 15:35:05 UTC

MDEV-16091: Seconds_Behind_Master spikes to millions of seconds

Problem:
========
A slave’s relay log format description event is used when
calculating Seconds_Behind_Master (SBM). This forces the SBM
value to spike when processing these events, as their creation
date is set to the timestamp that the IO thread begins.

Solution:
========
When the slave generates a format description event, mark the
event as a relay log event so it does not update the
rli->last_master_timestamp variable.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Monty
Author Date: 2021-12-27 16:51:00 UTC

MDEV-14907 FEDERATEDX doesn't respect DISTINCT

Federated and Federatex cannot be used with ROR scans

Federated::position() and Federatex::position() is storing in 'ref' a
pointer into a local result set buffer. This means that one cannot
compare 'ref' from different handler instances to see if they point to the
same physical record.

This bug caused federated.federatedx to return wrong results when the
optimizer tried to use index_merge to resolve some queries.

Fixed by introducing table flag HA_NON_COMPARABLE_ROWID and using this
with the above handlers.

Todo:
- Fix multi_delete(), multi_update and read_records() to use primary key
  instead of 'ref' if case HA_NON_COMPARABLE_ROWID is set. The current
  code only works if we have only one range (like table scan) for the
  tables that will be updated in the second pass.
- Enable DBUG_ASSERT() in ha_federated::cmp_ref() and
  ha_federatedx::cmp_ref().

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2021-12-30 07:43:58 UTC

MDEV-27017 Assertion failure 'table->get_ref_count() == 0' on DDL that involves FULLTEXT INDEX

 purge_sys.stop_FTS() does not wait for purge operation
on FTS tables to finish. InnoDB DDL does purge_sys.stop_FTS()
and lock all fts tables. It eventually fails due to
n_ref_count value.

fts_stop_purge(): Stops the purge thread to process new FTS tables,
check n_ref_count of all fts index auxiliary, common tables.
This should make sure that consecutive fts_lock_tables()
is always successful.

Author: Marko Mäkelä
Author Date: 2022-01-05 13:08:02 UTC

MDEV-27208: Extend CRC32() and implement CRC32C()

We used to define a native unary function crc32() that computes the CRC-32
of a string using the ISO 3309 polynomial that is being used by zlib
and many others.

Often, CRC is computed in pieces. To faciliate this, we introduce a
2-ary variant of the function that inputs a previous CRC as the first
argument: CRC32('MariaDB')=CRC32(CRC32('Maria'),'DB').

InnoDB and MyRocks use a different polynomial, which was implemented
in SSE4.2 instructions that were introduced in the
Intel Nehalem microarchitecture. This is commonly called CRC-32C
(Castagnoli).

We introduce a native function that uses the Castagnoli polynomial:
CRC32C('MariaDB')=CRC32C(CRC32C('Maria'),'DB'). This allows
SELECT...INTO DUMPFILE to be used for the creation of files with
valid checksums, such as a logically empty InnoDB redo log file
ib_logfile0 corresponding to a particular log sequence number.

Author: Nayuta Yanagisawa
Author Date: 2022-01-05 13:04:48 UTC

tmp (doesn't work)

Author: Monty
Author Date: 2022-01-05 12:33:03 UTC

Another temporary commit to be combined with previous ones

Author: Vladislav Vaintroub
Author Date: 2021-12-21 11:24:35 UTC

MDEV-27335 Windows, MSI - Bring the datadir location into the instance config UI

Author: =?utf-8?b?SmVzw7pzIE1hcsOtbg==?=
Author Date: 2022-01-04 22:58:52 UTC

Update errmsg-utf8.txt (spa) part 2

Further changes to 3e0304884b45d143e4194b3244d6e77a663680fc

New changes in translation:

* Converted to LATAM countries treatment: tú for vd. This way it serves good for Spain and all LATAM countries.
* Minor changes

Author: Brandon Nesterenko
Author Date: 2021-12-27 18:53:14 UTC

MDEV-16091: Seconds_Behind_Master spikes to millions of seconds

Problem:
========
A slave’s relay log format description event is used when
calculating Seconds_Behind_Master (SBM). This forces the SBM
value to spike when processing these events, as their creation
date is set to the timestamp that the IO thread begins.

Solution:
========
When the slave generates a format description event, mark the
event as a relay log event so it does not update the
rli->last_master_timestamp variable.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Alexey Botchkov
Author Date: 2022-01-04 10:59:45 UTC

The SQL SERVICE backported into the 10.4 to be tested.

Author: Andrei
Author Date: 2021-12-26 16:13:49 UTC

MDEV-27365 CREATE-or-REPLACE SEQUENCE is binlogged without DDL flag

CREATE-OR-REPLACE SEQUENCE is not logged with Gtid event DDL flag
which affects its slave parallel execution.
Unlike other DDL:s it can occur in concurrent execution with following transactions
which can lead to various errors, including asserts like

(mdl_request->type != MDL_INTENTION_EXCLUSIVE && mdl_request->type != MDL_EXCLUSIVE) || !(get_thd()->rgi_slave && get_thd()->rgi_slave->is_parallel_exec && lock->check_if_conflicting_replication_locks(this)

in MDL_context::acquire_lock.

Fixed to wrap internal statement level commit with save-
and-restore of TRANS_THD::m_unsafe_rollback_flags.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-01-03 12:51:56 UTC

cc

Author: Andrei
Author Date: 2021-12-27 10:36:51 UTC

MDEV-27039 Trying to lock mutex ... when the mutex was already locked

The reason of the double lock was an extraneous ha_flush_logs().
Unlike the upstream it is unnecessary in Mariadb that exploits a binlog
checkpoint mechanism for not letting PURGE or RESET-MASTER to trouble
transaction recovery. That is in case should a trx
be prepared but its binlog file gone, the trx then is committed on disk too.
Those facts have been always verified by existing tests of

  binlog.binlog_{checkpoint,xa_recover}.test.

A regression test for the bug is included though.

Author: Oleg Smirnov
Author Date: 2021-12-24 14:27:03 UTC

MDEV-27021 Implement SHOW ANALYZE command

Author: Alexander Barkov
Author Date: 2021-12-29 19:04:13 UTC

MDEV-22256 Assertion `length == pack_length()' failed in Field_timestamp_with_dec::sort_string

Adding 10.2 specific tests only. No code changes.

The problem was earlier fixed by MDEV-21580 and MDEV-22715.

Author: Alexander Barkov
Author Date: 2021-12-29 18:50:10 UTC

MDEV-22256 Assertion `length == pack_length()' failed in Field_timestamp_with_dec::sort_string

Adding 10.5 specific tests only. No code change.

The patches for MDEV-21580 and MDEV-22715 fixed this problem.

Author: Andrei Elkin
Author Date: 2021-12-29 16:30:53 UTC

MDEV-27365 Incremental commit to address a test simplification note.

Author: midenok
Author Date: 2021-12-29 13:13:14 UTC

alias_name_used fix

Author: Alexander Barkov
Author Date: 2021-12-29 10:54:34 UTC

MDEV-21639 DEFAULT(col) evaluates to a bad value in WHERE clause

The problem happened because Item_default_value did not overload
properly the val_xxx_result() family methods.

This change backports the patch for:
MDEV-24958 Server crashes in my_strtod / Value_source::Converter_strntod::Converter_strntod with DEFAULT(blob)
which earlier fixed the problem in 10.3.

Author: Sergei Golubchik
Author Date: 2021-07-02 17:13:26 UTC

MDEV-24909 JSON functions don't respect KILL QUERY / max_statement_time limit

pass the pointer to thd->killed down to the json library,
check it while scanning,
use thd->check_killed() to generate the proper error message

Author: Alexander Barkov
Author Date: 2021-12-28 13:43:40 UTC

MDEV-18918 SQL mode EMPTY_STRING_IS_NULL breaks RBR upon CREATE TABLE .. SELECT

The 10.5 version of the patch.

Removing DEFAULT from INFORMATION_SCHEMA columns.
DEFAULT in read-only tables is rather meaningless.
Upgrade should go smoothly.

Also fixes:
 MDEV-20254 Problems with EMPTY_STRING_IS_NULL and I_S tables

Author: Alexander Barkov
Author Date: 2021-12-28 13:43:40 UTC

MDEV-18918 SQL mode EMPTY_STRING_IS_NULL breaks RBR upon CREATE TABLE .. SELECT

Removing DEFAULT from INFORMATION_SCHEMA columns.
DEFAULT in read-only tables is rather meaningless.
Upgrade should go smoothly.

Also fixes:
 MDEV-20254 Problems with EMPTY_STRING_IS_NULL and I_S tables

Author: Sergei Golubchik
Author Date: 2021-12-27 13:47:47 UTC

MDEV-27364 Build from 10.2-10.4 srpm fails on RHEL8 family due to discrepancy in jemalloc requirements

pass jemalloc requirement down to srpm build

Author: Julius Goryavsky
Author Date: 2021-12-26 15:35:34 UTC

Duplicate line deleted

Author: Brandon Nesterenko
Author Date: 2021-12-21 21:42:32 UTC

MDEV-26919 Fixed assert

Temporary branch used for testing on buildbot

Author: Alexander Barkov
Author Date: 2021-12-21 07:59:08 UTC

MDEV-27307 main.ctype_utf8mb4_uca_allkeys tests fail with Valgrind/MSAN

In case when filesort does not use addon field packing (because of
too small potential savings) and uses fixed width addon fields instead,
the field->pack() call can store less bytes when the field maximum
possible field length, e.g. in case of VARCHAR().
The memory between the packed length and addonf->length (the maximum length)
stayed uninitialized, which was reported by Valgrind/MSAN.

The problem was introduced by f52bf92014efae6a1da9c2f26a7e3792ed5f5396 in 10.5,
which removed the tail initialization (probably unintentionally).

Restoring the bzero() in the fixed length branch,
so in case when pack() stores less bytes than addonf->length says,
the trailing bytes gets initialized.

Note, before f52bf92014efae6a1da9c2f26a7e3792ed5f5396, the bzero()
was under HAVE_valgrind conditional compilation. Now it's being added
unconditionally:
- MSAN also reported the problem, so it's not only Valgrind specific.
- As Serg proposed, conditional initialization is bad - it can have
  potentional security problems as the non-initialized memory fragments
  can store various pieces of essential information, e.g. passwords.

Author: Sergei Golubchik
Author Date: 2021-12-20 17:14:36 UTC

bump up server's maturity

Author: midenok
Author Date: 2021-12-20 17:14:56 UTC

MDEV-27328 Change of SYSTEM_TIME partitioning options is not possible without data copy

When we need to add/remove or change LIMIT, INTERVAL, AUTO (10.8 and
more) we have to recreate partitioning from scratch. Such operations
should be done fast. To remove options like LIMIT or INTERVAL one
should write:

alter table t1 partition by system_time;

The command checks whether it is new or existing SYSTEM_TIME
partitioning. And in the case of new it behaves as CREATE would do:
adds default number of partitions (2). If SYSTEM_TIME partitioning
already existed it just changes its options: removes unspecified ones
and adds/changes those specified explicitly. In case when partitions
list was supplied it behaves as usual: does full repartitioning.

Author: Sergey Petrunia
Author Date: 2021-12-20 20:51:55 UTC

Reverse-ordered indexes: remove SEL_ARG::is_ascending

Get the information from the array of KEY_PART structures that describes
the [pseudo-]index that is being analyzed.

Author: Vicențiu Ciorbaru
Author Date: 2021-12-20 15:35:32 UTC

Merge remote-tracking branch 'upstream/10.2' into 10.3

Author: Sergei Krivonos
Author Date: 2021-12-01 01:11:14 UTC

MDEV-27036: re-enable my_json_writer-t unit test

Author: Sergei Krivonos
Author Date: 2021-12-19 21:26:26 UTC

MCOL-4944: fixed systemd/sd-daemon.h detection: may not use HAVE_SYSTEMD_SD_DAEMON_H name

Author: midenok
Author Date: 2021-11-02 08:13:21 UTC

MDEV-25004 FTS tests for System Versioning

MTR combination 'vers' for debug build sets sysvers_force and
sysvers_hide. sysvers_force makes every created table
system-versioned, sysvers_hide hides WITH SYSTEM VERSIONING for SHOW
CREATE.

To use 'vers' combination one must include maybe_versioning.inc into
his test.

Removed innodb-fts-stopword.test as it duplicates stopword.test