lp:maria

Author: Oleg Smirnov
Author Date: 2022-07-27 14:25:34 UTC

MDEV-24819 Optimizer uses wrong parameter evaluating TRP_GROUP_MIN_MAX

When evaluating GROUP_MIN_MAX access (also known as Loose Index Scan)
the optimizer considers applying QUICK_RANGE_SELECT object to the
constructed TRP_GROUP_MIN_MAX object. To do that the optimizer
evaluates param->quick_rows[cur_index] which has been set before
while evaluating Index Merge access. But Loose Index Scan cannot be
combined with Index Merge access so evaluating param->quick_rows is not
valid here. The correct parameter to evaluate possibility of
QUICK_RANGE_SELECT is param->table->opt_range[cur_index].rows.

Author: Nayuta Yanagisawa
Author Date: 2022-07-22 06:06:23 UTC

MDEV-26247 Spider: Valid LEFT JOIN results in ERROR 1064

The current implementation of Spider's GROUP BY handler cannot
handle some type of queries yet. Thus, we disable the handler for
the queries.

This is tentative fix and more and more queries should be handled
by smart handlers.

Author: Andrei
Author Date: 2022-07-26 16:09:46 UTC

MDEV-29078. Review contribution/notes.

1. The new test needs an absolute TZ for SELECTing
2. The client mysqlbinlog also is better off to be aware of OLD binlog
   so it won't generate automatic
    SET @@explicit_defaults_for_timestamp = 0
   in this case.
   The SET should be left out to the user (like it is for the OLD
   slave's global explicit_defaults_for_timestamp that should be
   configured accordingly to the master).

Author: Daniel Black
Author Date: 2022-07-26 08:41:37 UTC

MDEV-29166: reduce locking of innodb trx_sys_t::history_*

Acquire and release locks in trx_sys_t::history_* one by one so
we reduce the risk of deadlock with a process that acquires a
later segement and then aquires and earlier segment.

While doing this means that the accuracy of the returned history
length could be off by a factor, but by the time we release from
these microfunctions, the rseg locks are released, so the history
value is out of date by the time the calling function access it already.

reasons why not are https://jira.mariadb.org/browse/MDEV-29141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Pushing for test purposes only at the momement.

Author: Nikita Malyavin
Author Date: 2022-07-19 13:36:32 UTC

rename rpl/rpl_alter_instant -> rpl/rpl_alter_innodb

Author: Luis Eduardo Oliveira Lizardo
Author Date: 2022-07-18 15:48:01 UTC

MDEV-28926 Add time spent on query optimizer to JSON ANALYZE (#2193)

* Add query optimizer timer to ANALYZE FORMAT=JSON

* Adapt tests and results

* Change logic to always close the writer after printing query blocks

Author: Sergey Petrunia
Author Date: 2022-07-25 08:44:10 UTC

Revert the commit with MDEV-28926: Add time spent on query optimizer to JSON ANALYZE

It will go into 10.11.

Author: Luis Eduardo Oliveira Lizardo <108760288+mariadb-LuisLizardo@users.noreply.github.com>
Date: Mon Jul 18 17:48:01 2022 +0200

    MDEV-28926 Add time spent on query optimizer to JSON ANALYZE (#2193)

    * Add query optimizer timer to ANALYZE FORMAT=JSON

    * Adapt tests and results

    * Change logic to always close the writer after printing query blocks

Author: Nayuta Yanagisawa
Author Date: 2022-04-25 09:29:01 UTC

MDEV-27233 Server hangs when using --init-file which loads Spider and creates a Spider table

Spider waits until the server initialization is complete
(i.e., mysqld_server_started == 1) before creating the Spider system
tables like mysql.spider_XX. Spider also wait until the tables are
created before instantiating ha_spider.

This results in dead lock when Spider is loaded and a non-system
Spider table is created by --in-file because queries passed by
--in-file are executed during the server initialization
(i.e., mysqld_server_started == 0).

In the first place, there seems to be no need for Spider to wait until
the server initialization is completed before creating the system
tables. Thus, we simply remove that synchronization.

Author: Sergey Petrunia
Author Date: 2022-07-19 11:13:17 UTC

MDEV-28929: Plan selection takes forever with MDEV-28852 ...
Part #2: Extend heuristic pruning to use multiple tables as the
"Model tables".

Before the patch, heuristic pruning uses only one "Model table":
The table which had the best cost AND record became the "Model table".
After that, if a table's cost and record were both worse than
those of the Model Table, the table would be pruned away.

This didn't work well when the first table (the optimizer sorts them
by record_count) had low record_count but relatively high cost: nothing
could be pruned afterwards.

The patch adds the two additional "Model tables": one with the least
cost and the other with the least record_count.
(In both cases, a table can be pruned away if BOTH its cost and
record_count are worse than those of a Model table)

The new pruning is active when the number of tables to consider for
the prefix is higher than @@optimizer_extra_pruning_depth.

One can see the new pruning in the Optimizer Trace as
- "pruned_by_heuristic":"min_record_count", or
- "pruned_by_heuristic":"min_read_time".
Old heuristic pruning shows as "pruned_by_heuristic":1.

Author: Sergey Petrunia
Author Date: 2022-07-19 11:13:17 UTC

MDEV-28929: Plan selection takes forever with MDEV-28852 ...
Part #2: Extend heuristic pruning to use multiple tables as the
"Model tables".

Before the patch, heuristic pruning uses only one "Model table":
The table which had the best cost AND record became the "Model table".
After that, if a table's cost and record were both worse than
those of the Model Table, the table would be pruned away.

This didn't work well when the first table (the optimizer sorts them
by record_count) had low record_count but relatively high cost: nothing
could be pruned afterwards.

The patch adds the two additional "Model tables": one with the least
cost and the other with the least record_count.
(In both cases, a table can be pruned away if BOTH its cost and
record_count are worse than those of a Model table)

The new pruning is active when the number of tables to consider for
the prefix is higher than @@optimizer_extra_pruning_depth.

One can see the new pruning in the Optimizer Trace as
- "pruned_by_heuristic":"min_record_count", or
- "pruned_by_heuristic":"min_read_time".
Old heuristic pruning shows as "pruned_by_heuristic":1.

Author: Julius Goryavsky
Author Date: 2022-07-19 11:26:19 UTC

MDEV-28758: Mariabackup copies binary logs to backup directory

This commit restores defaults and functionality regarding binlogs
to the way it was prior to MDEV-27524. The mariabackup utility no
longer saves binlogs files as part of a backup without the --galera-info
option. However, since we use --galera-info during SST, the behavior
of mariabackup changes and, in combination with GTIDs support enabled,
mariabackup transfers one (most recent) binlog file obtained after
FLUSH BINARY LOGS. In other cases, binlogs are not transferred during
SST in mariabackup mode. As for SST in the rsync mode, it works the
same way as before MDEV-27524 - by default it transfers one last
binlog file.

The --sst-max-binlogs option for mariabackup and the sst_max_binlogs
parameter in the [sst] / server sections are no longer supported for
SST via mariabackup.

Author: midenok
Author Date: 2022-07-18 12:13:23 UTC

Tmp

Author: mariadb-LuisLizardo
Author Date: 2022-07-14 10:06:00 UTC

Change logic to always close the writer after printing query blocks

Author: Oleg Smirnov
Author Date: 2022-07-07 15:57:38 UTC

MDEV-28881 Fix memory leak caused by STL usage

Dep_analysis_context::create_unique_pseudo_key_if_needed() was using
std::set which allocated memory outside MEM_ROOT and so required
explicit deallocation. This has been solved by replacing std::set with
MY_BITMAP which is allocated on MEM_ROOT

Author: Daniel Black
Author Date: 2022-07-11 02:52:16 UTC

MDEV-29015 innodb os_file_set_size WSL workaround

WSL wasn't keeping stack of renamed files
(https://github.com/microsoft/WSL/issues/8443)
resulting in the fstat calls in os_file_set_size returning
ENOENT and no fallocate fallback being possible.

Users reported MySQL was ok, and it used my_seek to
determine the size. We copy this concept here to avoid
the WSL bug.

Author: Alexey Botchkov
Author Date: 2022-07-08 09:54:10 UTC

Fix to quiet the compiler on Windows.

Author: midenok
Author Date: 2022-07-04 12:08:01 UTC

MDEV-29023 MTR hangs after multiple failures

Passing $opt_parallel as $childs is wrong: child can be killed before
it connects and you will never decrement $childs for this.

Another problem is (and that is the cause of this bug): child can be
killed and never close server socket. This can happen f.ex. after
unmaskable KILL signal. In such case the socket is closed by reaping
the child but that never happens inside reading the socket loop in
run_test_server().

The proper design is the waitless reap of children inside the socket
loop and if there is no more children we finish the socket loop. Since
there is Windows variation where we don't control the children via
waitpid(), all the clients must normally close the socket and only
this can finish the socket loop. For Unix variation we reckon that
case as all children closed the socket but not all yet died and for
that we do final waiting waitpid() (was done before the patch as
well).

To be more complete, we now handle 3 end-of-game scenarios in Unix:

   1. all children closed socket, all children died: everything is
      handled by the socket loop;

   2. all children closed socket, not all yet died: we wait for alive
      children to die after exiting the socket loop;

   3. not all children closed socket, all children died: everything is
      handled by the socket loop.

For Windows end-of-game scenario is only one:

   All children close the socket.

Author: midenok
Author Date: 2022-07-07 17:06:48 UTC

Tmp

Author: midenok
Author Date: 2022-07-07 14:27:34 UTC

MDEV-20088 Cleanup: unused Ed_connection, Execute_sql_statement, Protocol_local, etc

Removed needless classes.

Author: Nayuta Yanagisawa
Author Date: 2022-07-05 13:56:26 UTC

MDEV-29027 ASAN errors in spider_db_free_result after partition DDL

Spider calls ha_spider::close() at least twice on ALTER TABLE ... ADD
PARTITION. The first call frees wide_handler->trx and the second call
accesses wide_handler->trx->thd (heap-use-after-free).

In general, there seems to be no problem with using THD obtained by
the macro current_thd() except in background threads. Thus, we simply
replace wide_handler->trx->thd with current_thd().

Author: Nikita Malyavin
Author Date: 2022-07-04 22:19:12 UTC

MDEV-29013 ER_KEY_NOT_FOUND/lock timeout upon online alter with long unique

1. ER_KEY_NOT_FOUND
Some virtual columns were not updated because were not included in read_set
to the moment of calculation.

In Row_logs_event::do_apply_event some fields are excluded based on m_cols
value, the number of replicated rows.

We can't rely on this. Basically, we'd be satisfied, if all columns were
just set, at least for now.

2. ER_LOCK_WAIT_TIMEOUT
This is a long unique specific problem.

Sometimes, lookup_handler is created for to->file. To properly free it,
ha_reset should be called. It is usually done by calling
close_thread_table, but ALTER TABLE makes it differently. Hence, a single
ha_reset call is added to mysql_alter_table.

Also, the lifetime of lookup_handler is corrected: it's been allocated on
thd->mem_root, which in case of online alter
is event's mem_root, and can vary other ways.
It is changed to table->mem_root, which is more appropriate:
ha_reset is called only when table is closed, or in a few additional cases,
which correspond to a statement end. So memory leaks are unlikely here.

Author: Vladislav Vaintroub
Author Date: 2022-07-05 08:47:03 UTC

MDEV-24071 - repro

fixed a couple of assertions or crashes in the server.
Now Innodb is acting up

Author: Oleg Smirnov
Author Date: 2022-06-27 05:37:59 UTC

MDEV-28869 Eliminated tables are not shown in EXPLAIN

Show tables that have been eliminated during optimization stage
in the output of EXPLAIN/ANALYZE [FORMAT=JSON] commands. Show both
regular and derived tables.

Author: Vicențiu Ciorbaru
Author Date: 2022-07-02 19:15:22 UTC

Implement cost_for_order_by function

The sort length is extracted similarly to how sortlength() function does
it. The function makes use of filesort_use_addons function to compute
the length of addon fields. Finally, by calling compute_sort_costs we
get the fastest_sort possible.

Other changes:
* Sort_param::using_addon_fields() assumes addon fields are already
  allocated. This makes the use of Sort_param unusable for
  compute_sort_costs *if* we don't want to allocate addon fields.

  As a preliminary fix, pass "with_addon_fields" as bool value to
  compute_sort_costs() and make the internal functions use that value
  instead of Sort_param::using_addon_fields() method.

  The ideal fix would be to define a "leaner" struct with only the
  necessary members, but this can be done as a separate commit.

* Remove unnecessary fields from Sort_costs::costs array.
  As we do not yet differentiate between merge sort with fixed length vs
  dynamic length fields, eliminate that differentiation from enum sort_type.
  It can be added at a later date when we indeed have a cost
  differentiation.

* Fixup comments

Author: Nikita Malyavin
Author Date: 2022-06-30 11:37:22 UTC

speed up main.alter_table_online_debug

Author: Julius Goryavsky
Author Date: 2022-06-30 01:44:50 UTC

MDEV-28758: Mariabackup copies binary logs to backup directory

This commit restores defaults and functionality regarding binlogs
to the way it was prior to MDEV-27524. The mariabackup utility no
longer saves binlogs files as part of a backup without the --galera-info
option. However, since we use --galera-info during SST, the behavior
of mariabackup changes and, in combination with GTIDs support enabled,
mariabackup transfers one (most recent) binlog file obtained after
FLUSH BINARY LOGS. In other cases, binlogs are not transferred during
SST in mariabackup mode. As for SST in the rsync mode, as it was
before MDEV-27524 - it transfers one last binlog file by default.

The --sst-max-binlogs option for mariabackup and the sst_max_binlogs
parameter in the [sst] / server sections are no longer supported for
SST via mariabackup.

Author: Nikita Malyavin
Author Date: 2022-06-29 17:16:19 UTC

MDEV-28930 ALTER TABLE Deadlocks with parallel TL_WRITE

ALTER ONLINE TABLE acquires table with TL_READ. Myisam normally acquires
TL_WRITE for DML, which makes it hang until table is freed.

We deadlock once ALTER upgrades its MDL lock.

Solution:
Unlock table earlier. We don't need to hold TL_READ once we finished
copying. Relay log replication requires no data locks on `from` table.

Author: Nayuta Yanagisawa
Author Date: 2021-10-13 13:45:26 UTC

MDEV-26544 Assertion `part_share->auto_inc_initialized' failed in ha_partition::get_auto_increment on INSERT

The partition storage engine ignores return (error) values of
handler::info(). As a result, a query that should be aborted is
not aborted and then the server violates the assertion.

Author: Julius Goryavsky
Author Date: 2022-06-29 03:05:57 UTC

MDEV-27524 addendum: additional tests

Author: Igor Babaev
Author Date: 2022-06-29 00:40:08 UTC

An attempt to fix MDEV-21633

Author: Julius Goryavsky
Author Date: 2022-06-28 11:55:42 UTC

MDEV-27524 addendum: additional tests

Author: Julius Goryavsky
Author Date: 2022-06-28 03:06:30 UTC

MDEV-28968: mariabackup SST doesn't properly handle --innodb-buffer-pool-filename

The mariabackup SST script cannot properly handle SST from
the joiner side if a non-standard filename is specified in
the "--innodb-buffer-pool-filename" option. Log analysis
shows that the received file is copied to the destination
directory under the original name "ib_buffer_pool", and not
under the name specified using the option on the command
line or in the configuration file.

This commit fixes this bug and adds an updated mtr tests
to check for correct handling of the non-standard filenames
specified using the "--innodb-buffer-pool-filename" option.

Author: Alexander Barkov
Author Date: 2022-06-08 11:12:03 UTC

MDEV-23287 The INET4 data type

Author: Brandon Nesterenko
Author Date: 2022-06-27 13:39:19 UTC

MDEV-27655 Fix UBSAN's Findings in Replication

Tagged mysys/my_compare.c::ha_key_cmp() with attribute to ignore
UBSAN runtime checks on shift operations. In particular, it is
expected that this function can perform shift of greater length
than the number of bits in the underlying field variables.

Author: Julius Goryavsky
Author Date: 2022-06-27 03:03:21 UTC

MDEV-28758: Mariabackup copies binary logs to backup directory

This commit changes the default settings so that, by default,
mariabackup will not copy binary logs files to the backup directory
and will transfer only one (most recet) binary log file between nodes
during SST. If the user wants to make a backup containing all binary
logs or wants to transfer them all between nodes, then user must add
the sst_max_binlogs=-1 option to the [sst] section, to one of the
server configuration sections, or must use the equivalent command
line option (when running mariabackup).

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-06-24 09:55:42 UTC

MDEV-22133 handle_fatal_signal (sig=11) on optimized builds in handle_grant_table instead of ERROR | Buffer overflow (on optimized builds)

Return an error if we can not read a table required for ACL.

Author: Alexey Botchkov
Author Date: 2022-02-15 09:37:59 UTC

MDEV-27832 disable binary logging for SQL SERVICE.

Binary logging is now disabled for the queries run by SQL SERVICE.
The binlogging can be turned on with the 'SET SQL_LOG_BIN=On' query.

Conflicts:
 sql/sql_prepare.cc

Author: Nikita Malyavin
Author Date: 2022-06-23 22:45:44 UTC

sampling analysis

file=open(...)
for l in file:
     if l.startswith('SAMPLE') and not l.startswith('SAMPLE root'):
         print(l)
         n = dict((p.split('=')
           for p in l.replace('SAMPLE ', '').replace(' REJECT', '').split()
           ))
         if 'REJECT' in l:
             n['rej']=True
         nodes.append(n)

max(int(n['size']) for n in nodes)
max((float(n['p']), serialize(n)) for n in nodes)
max((float(n['size']), serialize(n)) for n in nodes if n['height'] == '0')

Author: Nayuta Yanagisawa
Author Date: 2022-06-22 17:17:24 UTC

MDEV-19190 Assertion `part_share->auto_inc_initialized' failed in ha_partition::get_auto_increment

Author: Julius Goryavsky
Author Date: 2022-06-21 11:29:45 UTC

MDEV-26562: galera-sst-mariabackup is failing due to missing xtrabackup_checkpoints

This commit contains workaround for a bug known as 'Red Hat issue 1870279'
(connection reset by peer issue in socat versions 1.7.3.3 to 1.7.4.0) which
further causes crashes during SST using mariabackup (when openssl is used).

Also fixed broken logic of automatic generation of the Diffie-Hellman parameters
for socat version less than 1.7.3 (which defaults to 512-bit values instead of
2048-bit ones).

Author: Julius Goryavsky
Author Date: 2022-06-16 01:45:45 UTC

MDEV-28656: Inability to roll upgrade without stopping the Galera cluster

This commit fixes a divergence in the IST/SST script code between
the 10.3+ and 10.9+ branches, due to which the cluster cannot contain
nodes with different versions of the server (since in this case SST/IST
scripts may fail).

This commit also includes test fixes specific to 10.4+ branches and
a few manual fixes after automatic "git cherry-pick", as well as server
code change specific to 10.9+.

Author: Nikita Malyavin
Author Date: 2022-02-23 19:38:51 UTC

Add tests and code instrumentations

Author: Julius Goryavsky
Author Date: 2022-06-17 11:56:03 UTC

MDEV-28583 postfix: fixing .result files after automatic merge

Author: Igor Babaev
Author Date: 2022-06-20 06:43:17 UTC

MDEV-7487 Semi-join optimization for single-table update/delete statements

This patch allows to use semi-join optimization at the top level of
single-table update and delete statements.
The problem of supporting such optimization became easy to resolve after
processing a single-table update/delete statement started using JOIN
structure. This allowed to use JOIN::prepare() not only for multi-table
updates/deletes but for single-table ones as well. This was done in the
patch for mdev-28883:
 Re-design the upper level of handling UPDATE and DELETE statements.

Note that JOIN::prepare() detects all subqueries that can be considered
as candidates for semi-join optimization. The code added by this patch
looks for such candidates at the top level and if such candidates are found
in the processed single-table update/delete the statement is handled in
the same way as a multi-table update/delete.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Nayuta Yanagisawa
Author Date: 2022-06-20 09:02:30 UTC

MDEV-28739 Trying to lock uninitialized mutex or hang upon shutdown after using Spider with query_cache

Author: Nayuta Yanagisawa
Author Date: 2022-06-20 09:02:30 UTC

MDEV-28739 Trying to lock uninitialized mutex or hang upon shutdown after using Spider with query_cache

Author: Oleksandr "Sanja" Byelkin
Author Date: 2021-10-20 09:37:14 UTC

MDEV-26647 (simple_password_check) Include password validation plugin information in the error message if the SQL statement is not satisfied password policy

Make the plugin reporting cause of the error.

Author: Nayuta Yanagisawa
Author Date: 2022-06-20 11:05:32 UTC

MDEV-27575 SIGSEGV in intern_plugin_lock on SHUTDOWN when setting Spider as default storage engine (temporary or global)

When Spider is set as the default storage engine, it is unloaded later
compared with other plugins. That is because Spider is referenced by
@@global.default_storage_engine and cannot be unloaded before the
variable is set to NULL.

spider_db_done() creates a new THD but, in the above mentioned case,
the new THD needs to lock its default_storage_engine and it does not
expect that the server is almost shut down and the variable is NULL.

Luckily, the newly created THD is used for nothing. Thus, we can just
remove the THD creation.

Author: Nikita Malyavin
Author Date: 2022-02-23 19:38:51 UTC

Add tests and code instrumentations

Author: Sergey Petrunia
Author Date: 2022-06-18 16:52:14 UTC

Cleanups for: 0762dd928 Improve pruning in greedy_search by sorting tables during search

Author: Igor Babaev
Author Date: 2022-06-18 07:49:15 UTC

MDEV-27159 Re-design the upper level of handling DML commands

This patch allows to execute only single-table and multi-table
UPDATE and DELETE statements using the method Sql_cmd_dml::execute().
The code that handles INSERT statements has not been touched.

Author: Julius Goryavsky
Author Date: 2022-06-17 02:59:29 UTC

MDEV-26562: galera-sst-mariabackup is failing due to missing xtrabackup_checkpoints

This commit contains workaround for a bug known as 'Red Hat issue 1870279'
(connection reset by peer issue in socat versions 1.7.3.3 to 1.7.4.0) which
further causes crashes during SST using mariabackup (when openssl is used).

Author: Julius Goryavsky
Author Date: 2022-06-17 11:56:03 UTC

MDEV-28583 postfix: fixing .result files after automatic merge

Author: Igor Babaev
Author Date: 2022-06-17 06:45:38 UTC

MDEV-27159 Re-design the upper level of handling DML commands

This patch allows to execute only single-table and multi-table
UPDATE and DELETE statements using the method Sql_cmd_dml::execute().
The code that handles INSERT statements has not been touched.

Author: Julius Goryavsky
Author Date: 2022-06-15 23:12:15 UTC

MDEV-28656: Inability to roll upgrade without stopping the Galera cluster

This commit fixes a divergence in the IST/SST script code between
the 10.3+ and 10.9+ branches, due to which the cluster cannot contain
nodes with different versions of the server (since in this case SST/IST
scripts may fail).

This commit also includes test fixes specific to 10.4+ branches.

Author: midenok
Author Date: 2021-12-05 19:20:20 UTC

MDEV-27180 Fully atomic partitioning DDL operations

Atomic DDL for partitioning originally covers crash-safety but it does
not recover fully from failures. F.ex. if error happened during the
rename or drop of partitions the ALTER operation does not return the
table to its original state.

The patch solves the above problem similarly to MDEV-25292 (atomic
CREATE OR REPLACE): new partitions are created as temporary, old
partitions are backed up, everything is guarded by two DDL log chains:
rollback and cleanup. Rollback chain contains the actions that bring
back the original table. Cleanup chain deletes the backup files.

The generic operation of alter partition is as follows:

  1. Create new partitions as tmp partitions;
  2. Fill TMP partitions with data;
  3. Rename old and not needed partitions to backup partitions;
  4. Rename tmp partitions to original partititions;
  5. Do any required logging (mariabackup, binary);
  6. If everything is ok, drop the backup partitions;
  7. In the case any of pp.1-5 fails drop new partitions, rename back
     backup partitions.

Each rename operation is executed right after the corresponding DDL
logging. Originally they were done in different loops:
write_log_changed_partitions(), mysql_change_partitions(), etc. The
actual table operations were done in ha_partition handler:
ha_change_partitions(), ha_drop_partitions(),
ha_rename_partitions(). Now these calls are deprecated and removed.

Instead the above deprecated interfaces the following classes now
handle DDL logging and table operations:

      Alter_partition_logger
      Alter_partition_add
      Alter_partition_change

Alter_partition_logger handles the basic operation, DDL logging and
table renames. Alter_partition_add does the stuff for adding the
partitions. Alter_partition_change combines Alter_partition_logger and
Alter_partition_add for complex operations such as REORGANIZE.

ha_partition::change_partitions() call was not fully removed. Stages
1-4 from that call was renamed to allocate_partitions() and that is
used by Alter_partition_add for initializing file handler arrays
(m_added_file, m_reorged_file). ha_partition::prepare_new_partition()
is renamed to create_partition() because it actually does ha_create()
and the new name better describes what it does.

DDL_LOG_DELETE/DDL_LOG_RENAME/DDL_LOG_REPLACE are now pure file
operations. To process both .par and .frm files we now issue two
actions instead of one. That makes interface more simple and that will
be used by MDEV-16417 in future. All the table operations are done via
other actions, such as DDL_LOG_RENAME_TABLE, DDL_LOG_DROP_TABLE, etc.

DDL_LOG_RENAME_TABLE has new alter_partition option. With this option
it does simple ha_rename_table() which is needed for renaming the
partitions. The action for renaming the partitions must start from
phase DDL_RENAME_PHASE_TABLE.

The testing was refactored a bit. Many new fail/crash points was
added, the test was powered by binary and mariabackup logging, it
tests more ALTER commands with variations of partitions-only and
subpartitions, without and under LOCK TABLES.

EXCHANGE PARTITION was fixed: after failure under LOCK TABLES the
table became unlocked. Fixed by correcting place of
reopen_tables(). Tested by partition_debug.

Author: Shunpoco
Author Date: 2022-05-20 03:23:20 UTC

MDEV-21027: Assertion `part_share->auto_inc_initialized || !can_use_for_auto_inc_init()' failed in ha_partition::set_auto_increment_if_higher

ha_partition::set_auto_increment_if_higher expects
part_share->auto_inc_initialized is true or can_use_for_auto_inc_init()
is false (but as the comment of this method saids, it returns false
only if we use Spider engine with DROP TABLE or ALTER TABLE query).
However, part_share->auto_inc_initialized becomes true only after all
partitions are opened (since 6dce6aecebe6ef78a14cb5c5c5daa8a355551e40).

Therefore, I added a conditional expression in order to read all
partitions when we REPLACE a table that has an AUTO_INCREMENT column.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2021-12-13 15:15:21 UTC

MDEV-5215 Granted to PUBLIC

Author: Dmitry Shulga
Author Date: 2022-06-15 11:01:19 UTC

MDEV-5816: Stored programs: validation of stored program statements

Fixed the test main.sp

Author: Jan Lindström
Author Date: 2022-06-12 07:39:15 UTC

Fixed regression on SST tests. We incorrectly used my_thread_end(), which
sets mysys_var pointer to NULL, so the next THD::THD will crash.
Removed my_thread_init()/end() pairs and because
Wsrep_allowlist_service::allowlist_cb is not always called from a
new thread added a thread to do so.

Fix co-authored by Sergei Golubchik <serg@mariadb.org>
and mkaruza <mario.karuza@galeracluster.com>

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-06-14 11:35:15 UTC

more test fixes

Author: Igor Babaev
Author Date: 2022-06-13 22:31:53 UTC

MDEV-27159 Re-design the upper level of handling DML commands

This is the second commit for the task. This patch allows to execute only
single-table and multi-table DELETE statements using the method
Sql_cmd_dml::execute(). The code that handles INSERT statements has not
been touched.

This patch still does not have the final changes to handle UPDATE/DELETE
statements.
All tests from the main suite passed. With --ps-protocol one test from
opt_trace_security returns not the same result. This will be fixed soon.

Author: Oleg Smirnov
Author Date: 2022-06-13 15:28:21 UTC

MDEV-28817 Derived table elimination issue for multiple occurences

If the same field occurs more than once in the SELECT list
(using aliases) we need to store all those occurences
to be able to eliminate the derived table. For example,
  SELECT t1.* from t1 LEFT JOIN
    (SELECT a AS a1, MAX(c), b AS b1, b AS b2, a AS a2
      FROM t2 GROUP BY a, b) D
    ON D.a1=t1.a and D.b2=t1.b
Derived table D is joined using aliases a1 and b2 so the eliminating
algorithm must know that fields (a1,b2) are the same as (a,b)
which are on the GROUP BY list

Author: Oleg Smirnov
Author Date: 2022-02-17 15:53:37 UTC

MDEV-26278 Add functionality to eliminate derived tables from the query

Elimination of unnecessary tables from SQL queries is already present
in MariaDB. But it only works for regular tables and not for derived ones.

Imagine we have a view:
  CREATE VIEW v1 AS SELECT a, b, max(c) AS maxc FROM t1 GROUP BY a, b

Due to "GROUP BY a, b" the values of combinations {a, b} are unique,
and this fact can be treated as like derived table "v1" has a unique key
on fields {a, b}.

Suppose we have a SQL query:
  SELECT t2.* FROM t2 LEFT JOIN v1 ON t2.a=v1.a and t2.b=v1.b

1. Since {v1.a, v1.b} is unique and both these fields are bound to t2,
   "v1" is functionally dependent on t2.
   This means every record of "t2" will be either joined with
   a single record of "v1" or NULL-complemented.
2. No fields of "v1" are present on the SELECT list

These two facts allow the server to completely exclude (eliminate)
the derived table "v1" from the query.

Author: Nayuta Yanagisawa
Author Date: 2022-05-27 07:11:19 UTC

MDEV-27902 SIGSEGV's in spider_check_and_set_trx_isolation, spider_db_open_handler, and ha_spider::external_lock and __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed in psi_mutex_lock and inline_mysql_mutex_lock, and SIGABRT in safe_mutex_lock

Author: Jan Lindström
Author Date: 2022-06-11 09:31:58 UTC

Restore thread creation as we now do my_thread_init()/end()

Author: Dmitry Shulga
Author Date: 2022-06-09 08:27:29 UTC

MDEV-5816: Stored programs: validation of stored program statements

This is the prerequisite patch introducing the class sp_lex_instr
that encapsulates access to an instance of sp_lex_keeper. Every SP instruction
that does need access to a LEX object on its processing should inherit this class
and implement two abstract methods:
  is_invalid(),
  invalidate().

These method will be used in subsequent patches to implement recomplilation of
SP instructions on failure.

Currently, the following instructions are derived from the class sp_lex_instr:
  sp_instr_stmt,
  sp_instr_set,
  sp_instr_set_trigger_field,
  sp_instr_jump_if_not,
  sp_instr_freturn,
  sp_instr_cpush,
  sp_instr_cursor_copy_struct,
  sp_instr_set_case_expr

Additionally, this patch converts the class sp_instr_opt_meta
to the base abstract class (that is, not inhereted from
the class sp_instr).

Every jump SP instruction now must be inhereted directly from
the class sp_instr_opt_meta and additionally from either the class
sp_lex_instr or sp_instr depending on whether this SP instruction
does need access to LEX object or not.

Moreover, the class sp_cursor is no more owner of sp_lex_keeper.
Instead, the virtual method get_lex_keeper() has been added to the
class sp_cursor() that returns nullptr and this method is overriden
in the derived class sp_instr_cpush to provide a pointer to a real
instance of the class sp_lex_keeper.

Author: Nikita Malyavin
Author Date: 2022-06-09 14:13:47 UTC

fix long tpool execution

Author: Julius Goryavsky
Author Date: 2022-06-09 13:59:03 UTC

MDEV-28690 addendum: fixed regression in tests for qpress

Author: Daniel Black
Author Date: 2022-06-09 08:27:59 UTC

MDEV-28782 mariadb-tzinfo-to-sql to work in bootstrap mode

Work around MDEV-28718 for now, but also optimize the interation
of information_schema.SYSTEM_VARIABLES.

Add test case to show that tzinfo data into bootstrap is
desired functionality.

Bug report thanks to Dan Lenski of AWS.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-05-11 15:11:49 UTC

MDEV-5215 prerequisite: remove test and test_* database hacks in the test suite

Author: Roman Nozdrin
Author Date: 2022-06-08 15:05:31 UTC

MCOL-5114 Removing ctor specialization b/c it violates c++20 syntax needed in ColumnStorew

Author: Sergei Golubchik
Author Date: 2022-06-06 18:34:49 UTC

fix the test for FreeBSD

Author: Sergey Petrunia
Author Date: 2022-06-06 19:21:22 UTC

MDEV-28749: restore_prev_nj_state() doesn't update cur_sj_inner_tables correctly

Errors:
1. restore_prev_sj_state() assumed that
  "we assume remaining_tables doesnt contain @tab"
  which wasn't true.
2. Another bug in this function: it did remove bits from
   join->cur_sj_inner_tables but never added them.
3. greedy_search() adds tables into the join prefix but
   neglects to update the semi-join optimization state.
   It does update nested outer join state, see this call:

  check_interleaving_with_nj(best_table)

   but there's no matching call to update the semi-join state.
   (This wasn't visible because most of the state is in the POSITION
    structure which is updated. But there is also state in JOIN, too)

The patch:
- Fixes all of the above
- Adds JOIN::dbug_verify_sj_inner_tables() which is used to verify the
  state is correct at every step.
- Renamed advance_sj_state() to optimize_semi_joins().
  = Introduced update_sj_state() which ideally should be called
    "advance_sj_state" but I didn't reuse the name to avoid confusion.

Author: Igor Babaev
Author Date: 2022-06-04 03:05:31 UTC

MDEV-7487 Semi-join optimization for single-table UPDATE/DELETEs

This is a preliminary patch.

Author: Oleg Smirnov
Author Date: 2022-05-24 13:41:59 UTC

MDEV-27366 SIGSEGV in handler_index_cond_check with rowid_filter

Cause:
After the optimizer has generated a query plan using "ref access"
plus "rowid filter", the execution reaches "join_cache_level" check.
As the result the server may decide to employ full table scan and
hash join but does not discard the rowid filter. This leads to an
inconsistency in the table access.

Solution:
Discard "rowid filter" when full table scan | hash join are employed

Author: Oleg Smirnov
Author Date: 2022-05-31 09:47:13 UTC

MDEV-28598 Assertion 'got_name == named_item_expected()' failed

This assertion triggered on an attempt to write a piece of corrupt
JSON to the optimizer trace:
  "transformation": {
    "select_id": 2,
    "from": "IN (SELECT)",
    "to": "semijoin",
    {
       "join_optimization": {
       "select_id": 3,
       "steps": []
    }
  }

This happened because some parts of the query may be evaluated right
during the optimization stage. To handle such cases Json_writer will now
implicitly add named members with automatically assigned names
"implicitly_added_member_#1", "implicitly_added_member_#2", etc
to preserve JSON document correctness. Also the tail of the JSON document
will be printed to stderr before abort.
Those two improvements only apply for Debug builds, not for Release ones
as the Release builds skip JSON correctness checks

Author: Marko Mäkelä
Author Date: 2022-05-31 12:07:02 UTC

fixup! 424ad322ea096d6552c49e0bc268c730e897e915

Author: Marko Mäkelä
Author Date: 2022-05-31 11:43:46 UTC

Merge 10.5 into 10.6

Author: tmokmss
Author Date: 2022-05-30 08:57:26 UTC

use unlikely instead of likely

Author: Nikita Malyavin
Author Date: 2022-05-29 16:07:29 UTC

seems working kinda eh

Author: Nayuta Yanagisawa
Author Date: 2022-05-26 23:55:42 UTC

MDEV-28522 Delete constant SPIDER_SQL_TYPE_*_HS

The HandlerSocket support of Spider has been deleted by MDEV-26858.
Thus, the constants, SPIDER_SQL_TYPE_*_HS, are no longer necessary.

Author: Daniel Black
Author Date: 2022-05-19 23:31:18 UTC

MDEV-28597: double free in maria_open()

disk_cache is freed error handling functions.

Because we've freed this earlier in maria_open,
disk_cache is assigning to 0 (like the 10.3+ commit
 c127a1ba1450) to prevents the double free happing
in the error path that can be triggered later in
this function.

Author: Marko Mäkelä
Author Date: 2022-05-25 09:55:44 UTC

fixup! 036d091d2ef314bf5faab6c01e15c4030adee51e

Author: Marko Mäkelä
Author Date: 2022-05-14 08:45:50 UTC

Performance test: Disable some counter updates

Author: Jan Lindström
Author Date: 2022-05-24 04:17:16 UTC

Update galera disabled.def file

Author: Jan Lindström
Author Date: 2022-05-24 04:37:08 UTC

Update galera_sr disabled.def file

Author: Nikita Malyavin
Author Date: 2022-05-22 19:00:50 UTC

WIP resolve sql_type and field linkage

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-05-11 15:11:49 UTC

MDEV-5215 prerequisite: remove test and test_* database hacks in the test suite

Author: Julius Goryavsky
Author Date: 2022-05-19 02:53:32 UTC

MDEV-28423 continuation: Galera progress reporting + rolling updates

A new feature to support progress reporting (including progress
reporting in the JSON format) that was added in 10.9 can crash the
rolling update scenario if the old version donor node does not pass
the "total" tag in the MAGIC_FILE to the joiner node. The "grep"
utility that is used to extract this tag in this case will return
an empty string, which will then be substituted into the "if"
statement, resulting in a syntactically invalid construct.

This commit corrects this issue.

Also, when using the previous fixes (from MDEV-28423 and from
MDEV-28593), authentication errors are possible in case of mixing
versions at the opposite sides - if the donor is newer than joiner,
due to incorrect reading of the new MAGIC_FILE format by older node
versions (from the joiners side). This problem is also fixed here,
taking advantage of the fact that older versions are still able
to read GUID coordinates even if multiple strings are passed to
them at the end.

In addition, the new progress reporting exchanges the length of
the transmitted data between the donor and the joiner before the
start of the transfer, but does not use this information on the
joiner side, which can lead to incorrect calculation of progress
by the joiner (not synchronous with calculations from the donor
side) if from the joiner side reporting works in compatibility
mode with older versions. This commit moves the construction
of the command line for progress reporting to a point where
we already know the size and can use that knowledge.

Also, to facilitate SST script support in the future, the
script code now has been made unified between all versions
from 10.3 to 10.9.

Author: Julius Goryavsky
Author Date: 2022-05-19 01:45:19 UTC

MDEV-28423 continuation: Galera progress reporting + rolling updates

A new feature to support progress reporting (including progress
reporting in the JSON format) that was added in 10.9 can crash the
rolling update scenario if the old version donor node does not pass
the "total" tag in the MAGIC_FILE to the joiner node. The "grep"
utility that is used to extract this tag in this case will return
an empty string, which will then be substituted into the "if"
statement, resulting in a syntactically invalid construct.

This commit corrects this issue.

Also, when using the previous fixes (from MDEV-28423 and from
MDEV-28593), authentication errors are possible in case of mixing
versions at the opposite sides - if the donor is newer than joiner,
due to incorrect reading of the new MAGIC_FILE format by older node
versions (from the joiners side). This problem is also fixed here,
taking advantage of the fact that older versions are still able
to read GUID coordinates even if multiple strings are passed to
them at the end.

In addition, the new progress reporting exchanges the length of
the transmitted data between the donor and the joiner before the
start of the transfer, but does not use this information on the
joiner side, which can lead to incorrect calculation of progress
by the joiner (not synchronous with calculations from the donor
side) if from the joiner side reporting works in compatibility
mode with older versions. This commit moves the construction
of the command line for progress reporting to a point where
we already know the size and can use that knowledge.

Also, to facilitate SST script support in the future, the
script code now has been made unified between all versions
from 10.3 to 10.9.

Author: Oleg Smirnov
Author Date: 2022-05-18 14:36:32 UTC

MDEV-24837 HAVING clause yields wrong results

Set items references to point to the temporary table fields
when processing aggregate operations

Author: Kiyoshi Takeda
Author Date: 2022-05-18 12:49:24 UTC

MDEV-14642 Assertion 'table->s->db_create_options == part_table->s->db_create_options' failed in compare_table_with_partition

When trying to execute ALTER TABLE EXCHANGE PARTITION with different definitions, assertion

    table->s->db_create_options == part_table->s->db_create_options

failed in compare_table_with_partition().
However, this execution should not be allowed since executing 'exchange partition' requires the identical structure of the two tables.

To fix the problem, I deleted the assertion code and added code that returns an error that indicates tables have different definitions.

Author: Sergei Golubchik
Author Date: 2022-05-18 11:11:16 UTC

MDEV-28606 Server crashes in st_select_lex::add_table_to_list instead of error 1066: Not unique table/alias

10.2-only fix, 10.3+ uses LEX_STRING's and checks the length first

Author: Vlad Lesin
Author Date: 2022-05-18 11:09:40 UTC

MDEV-28607 add option to mtr to start mariabackup under rr

The new --rr-backup mtr option is added.

Author: Oleg Smirnov
Author Date: 2022-05-16 11:32:39 UTC

MDEV-27366 SIGSEGV in handler_index_cond_check on SELECT in connection with rowid_filter setting

Cause: when optimizer picks a full table scan it does not reset previously
chosen rowid filter.
Solution: reset rowid filter for full table scan in best_access_path()

Author: Julius Goryavsky
Author Date: 2022-05-17 21:49:13 UTC

MDEV-28583: Galera: binlogs disappear after rsync IST

This commit sends a flag indicating the presence of the "--bypass"
option from the donor node to the joiner nodes during rsync IST,
because without such a flag it is impossible to distinguish IST
from the SST on the joiner nodes (in IST/SST scripts, because the
"--bypass" option is still not passed to scripts from server code).
Specifically, this fixes an issue with binary logs disappearing
after IST (via rsync). There are also changes to diagnostic messages
here that will make it easier to diagnose script-related problems
in the future when debugging and when checking the logs. This commit
also adds more robust signal handlers - to handle exceptions during
script execution. These handlers won't mask some crashes and it
also unifies exit codes between different scripts. These changes
have already been helpful to debugging "bypass" flag handling.

Author: Julius Goryavsky
Author Date: 2022-05-17 21:43:32 UTC

MDEV-28583: Galera: binlogs disappear after rsync IST

This commit sends a flag indicating the presence of the "--bypass"
option from the donor node to the joiner nodes during rsync IST,
because without such a flag it is impossible to distinguish IST
from the SST on the joiner nodes (in IST/SST scripts, because the
"--bypass" option is still not passed to scripts from server code).
Specifically, this fixes an issue with binary logs disappearing
after IST (via rsync). There are also changes to diagnostic messages
here that will make it easier to diagnose script-related problems
in the future when debugging and when checking the logs. This commit
also adds more robust signal handlers - to handle exceptions during
script execution. These handlers won't mask some crashes and it
also unifies exit codes between different scripts. These changes
have already been helpful to debugging "bypass" flag handling.

Author: Julius Goryavsky
Author Date: 2022-05-16 12:17:15 UTC

rnew

Author: midenok
Author Date: 2022-05-16 13:11:20 UTC

MDEV-27328 MSAN fix