lp:maria

Author: Daniel Black
Author Date: 2022-09-19 00:37:26 UTC

MDEV-16194 stack not resolve (addr2line)

Remove the offset when invoking addr2line as tested by
Sergey Vojtovich.

Author: Igor Babaev
Author Date: 2022-09-16 21:59:56 UTC

MDEV-7487 Semi-join optimization for single-table update/delete statements

This patch allows to use semi-join optimization at the top level of
single-table update and delete statements.
The problem of supporting such optimization became easy to resolve after
processing a single-table update/delete statement started using JOIN
structure. This allowed to use JOIN::prepare() not only for multi-table
updates/deletes but for single-table ones as well. This was done in the
patch for mdev-28883:
Re-design the upper level of handling UPDATE and DELETE statements.

Note that JOIN::prepare() detects all subqueries that can be considered
as candidates for semi-join optimization. The code added by this patch
looks for such candidates at the top level and if such candidates are found
in the processed single-table update/delete the statement is handled in
the same way as a multi-table update/delete.

    Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Igor Babaev
Author Date: 2022-09-17 05:36:22 UTC

Another fix after the latest rebase of commits for MDEV-28883

Author: Oleg Smirnov
Author Date: 2022-09-14 11:11:19 UTC

MDEV-29535 Cleanup error handling in ha_federatedx

1. Class ha_federatedx_derived_handler overrides virtual function
derived_handler::print_error() with an empty implementation which
causes problems with error handling. Errors are being simply ignored
and occur during later processing.

2. Federatedx_handler_base::share can be NULL on unsuccessful attempt
of initialization, this must be checked.

Author: Vladislav Vaintroub
Author Date: 2022-08-25 22:38:00 UTC

MDEV-26715 Fix mysql_install_db_win tests.

It looks like C/C reconnect retains the database. It is correct behavior
However there is a problem that mysql_install_db.exe does not create
'test' database, thus using mysql_install_db to create datadir,
then restarting server and reconnecting does not succeed, if 'test' was
current db before reconnect.

The problem became apparent with MDEV-26715. Why it worked before,
with a simpler 'root' definition is a mystery.

Fix is to switch current connection to 'mysql' database, prior to
reconnect to the instance created via mysql_install_db.exe

Author: Lawrin Novitsky
Author Date: 2022-09-13 14:18:29 UTC

MDEV-29514 Extending plugins sql service with mysql_options,

mysql_fetch_lenghts, mysql_set_character_set, mysql_num_fields and
mysql_select_db.
sql service version has been changed to 0x0101
ABI files updated.

Author: Marko Mäkelä
Author Date: 2022-09-13 05:40:05 UTC

Merge 10.5 into 10.6

Author: Angelique Sklavounos
Author Date: 2022-09-12 23:17:52 UTC

MDEV-26633, MDEV-27950: intermediate commit to test changes

Author: Lena Startseva
Author Date: 2022-06-09 03:32:51 UTC

MDEV-27691: make working view-protocol

Tests with checking metadata or that cannot be run with
the view-protocol are excluded from --view-protocol.
For tests that do not allow the use of an additional connection,
the util connection is disabled with "--disable_service_connection".
Also cases with bugs for --view-protocol are disabled.

Author: Michael Widenius
Author Date: 2022-08-21 02:53:54 UTC

Added optimizer_costs.h which includes all optimizer costs

This makes it easier to see how costs changes over commits

Author: Michael Widenius
Author Date: 2022-08-21 02:53:54 UTC

Added optimizer_costs.h which includes all optimizer costs

This makes it easier to see how costs changes over commits

Author: Andrei
Author Date: 2022-08-16 17:03:15 UTC

MDEV-24660 MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0'

The shutdown time assert was caused by untimely deactivation of
the binlog background thread and related structs destruction.
It could specifically occur when a transaction is replication unsafe
and has to be completed with a ROLLBACK event in binlog.

This gets fixed with the binlog background thread stop relocation
to a point and user transactions have been completed.
A test case is added to binlog.binlog_checkpoint which
also receives as a bonus a minor correction to reactivate a MDEV-4322 test
case that originally required a shutdown phase (that ceased to do).

Author: Alexander Barkov
Author Date: 2022-09-07 06:50:40 UTC

More UCA collation performance improvements

Author: Vlad Lesin
Author Date: 2022-09-05 17:03:00 UTC

Log waiting and waited transaction info if there was lock wait timeout
and the waiting transaction is in prepared state and executed by slave
thread.

The commit is for custom build. If you want to push it to trunk, wrap
the code with "#ifdef UNIV_DEBUG". The same is true for the previous
commit.

Author: Marko Mäkelä
Author Date: 2022-09-06 09:44:58 UTC

MDEV-29440 InnoDB instant ALTER TABLE recovery must use READ UNCOMMITTED

In commit 8f8ba758559e473f643baa0a0601d321c42517b9 (MDEV-27234)
the data dictionary recovery was changed to use READ COMMITTED
so that table-rebuild operations (OPTIMIZE TABLE, TRUNCATE TABLE,
some forms of ALTER TABLE) would be recovered correctly.

However, for operations that avoid a table rebuild thanks to
being able to instantly ADD, DROP or reorder columns, recovery
must use the READ UNCOMMITTED isolation level so that changes to
the hidden metadata record can be rolled back.

We will detect instant operations by detecting uncommitted changes
to SYS_COLUMNS in case there is no uncommitted change of SYS_TABLES.ID
for the table. In any table-rebuilding DDL operation, the SYS_TABLES.ID
(and likely also the table name) will be updated.

As part of rolling back the instant ALTER TABLE operation, after the
operation on the hidden metadata record has been rolled back, a rollback
of an INSERT into SYS_COLUMNS in row_undo_ins_remove_clust_rec() will
invoke trx_t::evict_table() to discard the READ UNCOMMITTED definition
of the table. After that, subsequent recovery steps will load and use
the correct table definition.

Author: tanruixiang
Author Date: 2022-06-16 07:05:35 UTC

MDEV-26182: Implement JSON_INTERSECT()

First we introduce the compare_whole parameter. The intersection results of
different types of values vary according to compare_whole.
If compare_whole is true (When the array or object or scalar is not at the
outermost layer, compare_whole is true.):
  it means that the two jsons being compared, whether they are scalars, arrays
  or objects, must be exactly equal to have an intersection. Different types
  indicate that there is no intersection between jsons. If two scalars are
  compared, there is an intersection between them, which means that the two
  scalars are exactly the same. If two arrays are compared, there is an
  intersection between them, which means that the values at all indices of the
  two arrays are equal. If two objects are compared, there is an intersection
  between them, which means that all KV pairs of the two objects are equal.
if compare_whole is false:
  When a value is a subset of another value, there is an intersection. For
  example, taking the intersection of two objects is taking their common KV
  pairs. The intersection of array and array is to take their common elements.
  An object and a scalar take an intersection with an array. As long as the
  object or scalar exists in the array, the intersection is the object or the
  scalar.

We put a json into the hash, scan another json, and efficiently get the
intersection by reading and updating the hash information.

Author: Dmitry Shulga
Author Date: 2022-09-01 09:52:41 UTC

MDEV-5816: Stored programs: validation of stored program statements

Follow-up patch that fixes existing mtr tests.

Author: Vlad Lesin
Author Date: 2022-09-01 06:55:49 UTC

MDEV-29433 innodb.lock_delete_updated is unstable

Use suspend thread syncpoint instead of include/wait_condition.inc to
make sure DELETE created waiting lock before the next UPDATE begins
locking.

Author: Vlad Lesin
Author Date: 2022-09-01 06:55:49 UTC

MDEV-29433 innodb.lock_delete_updated is unstable

Use suspend thread syncpoint instead of include/wait_condition.inc to
make sure DELETE created waiting lock before the next UPDATE begins
locking.

Author: Sergey Petrunia
Author Date: 2022-08-29 20:12:27 UTC

MDEV-29382: Query returns wrong number of records

The issue occurred when DuplicateWeedout optimization was applied
together with Join Buffer, and also there was an SJ-Materialization-lookup
table in the join prefix.

DuplicateWeedout optimization includes the rowid of SJ-Materialization
temptable into its Duplicate checks.
However, Join Buffering code did not save/restore the rowids of
SJ-Materialization temptables. This meant the check could miss some
duplicates.

The fix makes the Join Buffering code to save/restore rowids for
SJ-Materialization temp.tables.

Author: Marko Mäkelä
Author Date: 2022-08-29 12:14:59 UTC

MDEV-27983: InnoDB hangs after loading a ROW_FORMAT=COMPRESSED page

If multiple threads invoke buf_page_get_low() on a ROW_FORMAT=COMPRESSED
page that does not reside in the buffer pool, then one of the threads
will end up acquiring an exclusive page latch (the "if" statement
right before the new wait_for_unzip: label) and other threads will
end up waiting for a shared latch while holding a buffer-fix.
The exclusive latch holder would then wait for the buffer-fixes to
be released while the buffer-fix holders are waiting for the shared latch.

buf_page_get_low(): Prevent the hang that was introduced
in commit 9436c778c3adba7c29dab5649668433d71e086f2 (MDEV-27058),
by releasing the buffer-fix, sleeping some time, and retrying the
page lookup.

Author: Elena Stepanova
Author Date: 2022-08-28 18:23:28 UTC

MDEV-29403 innodb.innodb_sys_semaphore_waits fails with wrong errno 5014

take into account C/C specific CR_ERR_NET_WRITE error

Author: Marko Mäkelä
Author Date: 2022-08-26 12:18:11 UTC

MDEV-28800 WIP: Avoid crashes on memory allocation failure

FIXME: Allocate locks upfront for page split or reorganize,
so that the operation can gracefully fail before any irreversible
persistent changes are performed. This affects lock_move_reorganize_page(),
lock_move_rec_list_end(), lock_move_rec_list_start(),
btr_root_raise_and_insert(), btr_insert_into_right_sibling(),
btr_page_split_and_insert().

buf_block_alloc(): Remove. This was an alias of
buf_LRU_get_free_block(false). Let us call that function directly.

buf_LRU_get_free_block(), buf_buddy_alloc_low(), buf_buddy_alloc():
If there is no free block in the buffer pool, return nullptr.

recv_sys_t::recover_low(), recv_sys_t::recover(): Return an error code,
which may be DB_OUT_OF_MEMORY.

lock_rec_create_low(): Return nullptr if the lock table is full.
This will be the only caller of buf_pool.running_out().

btr_search_check_free_space_in_heap(): Replaced with
btr_search_lock_and_alloc().

Author: Daniel Black
Author Date: 2022-07-29 14:11:08 UTC

MDEV-23607 MariaBackup - align required GRANTS to cmd options

Since the 10.5 split of the privileges, the required GRANTs
for various mariabackup operations has changed.

In the addition of tests, a number of mappings where incorrect:

The option --lock-ddl-per-table didn't require connection admin.

The option --safe-slave-backup requires SLAVE MONITOR even without
the --no-lock option.

Change MySQL server -> MariaDB server in code/information messages.

Author: Alexander Barkov
Author Date: 2022-08-02 12:23:08 UTC

Backporting MDEV-29159 from 10.7 to 10.6

MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability

1. Store assignment failures on incompatible data types now raise errors if:
- STRICT_ALL_TABLES or STRICT_TRANS_TABLES sql_mode is used, and
- IGNORE is not used

Otherwise, only a warning is raised and the statement continues.

2. Changing the error/warning test as follows:

-ERROR HY000: Illegal parameter data types inet6 and int for operation 'SET'
+ERROR HY000: Cannot cast 'int' as 'inet6' in assignment of `db`.`t`.`col`

so in case of a big table it's easier to see which column has the problem.
The new error text is also applied to SP variables.

Author: Jan Lindström
Author Date: 2022-08-25 06:31:08 UTC

Instrumentation

Author: Vicențiu Ciorbaru
Author Date: 2022-08-24 13:57:02 UTC

Cleanup mysqlbinlog

Author: Brandon Nesterenko
Author Date: 2022-04-27 01:51:42 UTC

MDEV-28294: set default role bypasses Replicate_Wild_Ignore_Table: mysql.%

Problem:
========
When replicating SET DEFAULT ROLE, the pre-update check (i.e. that
in set_var_default_role::check()) tries to validate the existence of
the given rules/user even when the targeted tables are ignored. When
previously issued CREATE USER/ROLE commands are ignored by the
replica because of the replication filtering rules, this results in
an error because the targeted data does not exist.

Solution:
========
Before checking that the given roles/user exist of a SET DEFAULT
ROLE command, first ensure that the mysql.user and
mysql.roles_mapping tables are not excluded by replication filters.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
Sergei Golubchik <serg@mariadb.com>

Author: Nikita Malyavin
Author Date: 2022-07-21 23:18:21 UTC

MDEV-29068 Cascade foreign key updates do not apply in online alter

Foreign key cascade operations are still implemented through innodb.

Add support for online logging in row_ins_foreign_check_on_constraint.

upd_node_t::sql_is_online_alter is added to reuse cascade->row and
cascade->upd_row.

Author: Nayuta Yanagisawa
Author Date: 2022-08-23 10:53:59 UTC

Fix wrong diff introduced by merge commit

Many Spider tests were broken by the merge commit, 36d173e.

Author: Marko Mäkelä
Author Date: 2022-08-22 13:24:10 UTC

Merge 10.6

Author: Marko Mäkelä
Author Date: 2022-08-18 14:39:46 UTC

MDEV-29043 mariabackup --compress hangs

compress_write(): Fix a race condition between threads that would
use the same worker thread object. Make thd->data_avail contain the
thread identifier of the submitter, and add thd->avail_cond to
notify other compress_write() threads that are waiting for a slot.

Author: Igor Babaev
Author Date: 2022-08-17 04:43:40 UTC

Correction after rebase of bb-10.10-MDEV-28883 against the latest 10.10.

Author: Monty
Author Date: 2022-08-11 10:05:23 UTC

TEMPORARY PUSH: Changing all cost calculation to be given in ms

- Added tests/check_costs.pl, a tool to verify optimizer cost calculations.
  - Most costs has been found with this program. All steps to calculate
    the new costs are documented in Docs/optimizer.costs
- User optimizer_cost variables are given in usec (as individual
  costs can be very small). Internally they are stored in ms.
- Changed DISK_READ_COST (was DISK_SEEK_BASE_COST) from a hard disk cost
  (9 ms) to common SSD cost (400MB/sec).
- Changed the following handler functions to return IO_AND_CPU_COST.
  This makes it easy to apply different cost modifiers in ha_..time()
  functions for io and cpu costs.
  - scan_time()
  - rndpos_time()
  - keyread_time()
- Enhanched keyread_time() to calculate the full cost of reading of a set
  of keys with a given number of ranges and optionall number of blocks that
  need to be accessed.
- Removed read_time() as keyread_time() + rndpos_time() is the same thing.
- Added the following new optimizer_variables:
  - optimizer_scan_lookup_cost
  - optimizer_row_lookup_cost
  - optimizer_index_lookup_cost
  - optimizer_disk_read_cost
- Added include/my_tracker.h ; Useful include file to quickly test costs
  of a function.
- Tuned sequence and heap engine costs (rest will be done in an updated
  commit)

Author: Andrei
Author Date: 2022-08-09 16:28:49 UTC

MDEV-24660 MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0'

The assert fired at shutdown time to indicate a missed decrement to
the count. The counter could not have been decremented because
the binlog background thread had already gone.

Fixed with relocation the binlog background thread exit in the server
shutdown module.

Author: Daniel Black
Author Date: 2022-08-09 08:41:30 UTC

Deb: uca-dump make as not-installed

Author: Andrei
Author Date: 2022-08-08 09:44:01 UTC

MDEV-29078. Merge to 10.10 to account MDEV-19801.

Also a new test comment line gets corrected.

Author: Oleg Smirnov
Author Date: 2022-08-01 14:54:32 UTC

MDEV-27624 Wrong result for left join, 'not exists' and join buffers

Cause:
When linked join buffers are employed to process nested outer joins
there is a problem with applying the 'not exists' optimization:
the match flag used for outer joins is not propagated among linked
join buffers. This leads to incorrect results when the 'not exists'
optimization is applied to a nested outer join.

Solution:
Disable applying the 'not exists' optimization for linked join buffers

Author: Julius Goryavsky
Author Date: 2022-07-19 11:26:19 UTC

MDEV-28758: Mariabackup copies binary logs to backup directory

This commit restores defaults and functionality regarding binlogs
to the way it was prior to MDEV-27524. The mariabackup utility no
longer saves binlogs files as part of a backup without the --galera-info
option. However, since we use --galera-info during SST, the behavior
of mariabackup changes and, in combination with GTIDs support enabled,
mariabackup transfers one (most recent) binlog file obtained after
FLUSH BINARY LOGS. In other cases, binlogs are not transferred during
SST in mariabackup mode. As for SST in the rsync mode, it works the
same way as before MDEV-27524 - by default it transfers one last
binlog file.

The --sst-max-binlogs option for mariabackup and the sst_max_binlogs
parameter in the [sst] / server sections are no longer supported for
SST via mariabackup.

Author: Norio Akagi
Author Date: 2022-07-06 09:23:09 UTC

MDEV-28315 Fix ASAN stack-buffer-overflow in String::copy_aligned

Starting since this commit 36cdd5c there is an ASAN
stack-buffer-overflow error because we append a NULL terminator
beyond the length of memory allocated.

Author: Sergei Golubchik
Author Date: 2022-07-31 08:19:06 UTC

wip version for explicit_def_ts

Author: Brandon Nesterenko
Author Date: 2022-07-29 22:38:12 UTC

Potential fix for rpl.rpl_old_master_29078.test

Author: MikeG
Author Date: 2022-07-29 06:07:42 UTC

MDEV-18702 mysqldump: add variable 'max-statement-time'

With a global non-default max-statement-time of a time interval that exceed
the query time mysqldump queries when doing a backup.

To solve both, add a max-statement-time option, defaulting to 0 (unlimited time).

Also like mariabackup, set the session wait_timeout=2147483. The
time/processing between mysqldump times isn't expected to get that
close ever, but let's adopt the standard of mariabackup as no-one has
challenged it has having a detrimental effect.

Author: Georg Richter
Author Date: 2022-07-29 05:40:00 UTC

Test fixes:

Since fix for CONC-603 (wrong error handling in TLS read/write) in case
of a read/write error client doesn't return always error 2013 (server
has gone away), so in addition we need to check for error 2026
(TLS/SSL error) and 5014 (write error).

Author: Oleg Smirnov
Author Date: 2022-07-27 14:25:34 UTC

MDEV-24819 Optimizer uses wrong parameter evaluating TRP_GROUP_MIN_MAX

When evaluating GROUP_MIN_MAX access (also known as Loose Index Scan)
the optimizer considers applying QUICK_RANGE_SELECT object to the
constructed TRP_GROUP_MIN_MAX object. To do that the optimizer
evaluates param->quick_rows[cur_index] which has been set before
while evaluating Index Merge access. But Loose Index Scan cannot be
combined with Index Merge access so evaluating param->quick_rows is not
valid here. The correct parameter to evaluate possibility of
QUICK_RANGE_SELECT is param->table->opt_range[cur_index].rows.

Author: Nayuta Yanagisawa
Author Date: 2022-07-22 06:06:23 UTC

MDEV-26247 Spider: Valid LEFT JOIN results in ERROR 1064

The current implementation of Spider's GROUP BY handler cannot
handle some type of queries yet. Thus, we disable the handler for
the queries.

This is tentative fix and more and more queries should be handled
by smart handlers.

Author: Andrei
Author Date: 2022-07-26 16:09:46 UTC

MDEV-29078. Review contribution/notes.

1. The new test needs an absolute TZ for SELECTing
2. The client mysqlbinlog also is better off to be aware of OLD binlog
   so it won't generate automatic
    SET @@explicit_defaults_for_timestamp = 0
   in this case.
   The SET should be left out to the user (like it is for the OLD
   slave's global explicit_defaults_for_timestamp that should be
   configured accordingly to the master).

Author: Daniel Black
Author Date: 2022-07-26 08:41:37 UTC

MDEV-29166: reduce locking of innodb trx_sys_t::history_*

Acquire and release locks in trx_sys_t::history_* one by one so
we reduce the risk of deadlock with a process that acquires a
later segement and then aquires and earlier segment.

While doing this means that the accuracy of the returned history
length could be off by a factor, but by the time we release from
these microfunctions, the rseg locks are released, so the history
value is out of date by the time the calling function access it already.

reasons why not are https://jira.mariadb.org/browse/MDEV-29141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Pushing for test purposes only at the momement.

Author: Nikita Malyavin
Author Date: 2022-07-19 13:36:32 UTC

rename rpl/rpl_alter_instant -> rpl/rpl_alter_innodb

Author: Luis Eduardo Oliveira Lizardo
Author Date: 2022-07-18 15:48:01 UTC

MDEV-28926 Add time spent on query optimizer to JSON ANALYZE (#2193)

* Add query optimizer timer to ANALYZE FORMAT=JSON

* Adapt tests and results

* Change logic to always close the writer after printing query blocks

Author: Sergey Petrunia
Author Date: 2022-07-25 08:44:10 UTC

Revert the commit with MDEV-28926: Add time spent on query optimizer to JSON ANALYZE

It will go into 10.11.

Author: Luis Eduardo Oliveira Lizardo <108760288+mariadb-LuisLizardo@users.noreply.github.com>
Date: Mon Jul 18 17:48:01 2022 +0200

    MDEV-28926 Add time spent on query optimizer to JSON ANALYZE (#2193)

    * Add query optimizer timer to ANALYZE FORMAT=JSON

    * Adapt tests and results

    * Change logic to always close the writer after printing query blocks

Author: Nayuta Yanagisawa
Author Date: 2022-04-25 09:29:01 UTC

MDEV-27233 Server hangs when using --init-file which loads Spider and creates a Spider table

Spider waits until the server initialization is complete
(i.e., mysqld_server_started == 1) before creating the Spider system
tables like mysql.spider_XX. Spider also wait until the tables are
created before instantiating ha_spider.

This results in dead lock when Spider is loaded and a non-system
Spider table is created by --in-file because queries passed by
--in-file are executed during the server initialization
(i.e., mysqld_server_started == 0).

In the first place, there seems to be no need for Spider to wait until
the server initialization is completed before creating the system
tables. Thus, we simply remove that synchronization.

Author: Sergey Petrunia
Author Date: 2022-07-19 11:13:17 UTC

MDEV-28929: Plan selection takes forever with MDEV-28852 ...
Part #2: Extend heuristic pruning to use multiple tables as the
"Model tables".

Before the patch, heuristic pruning uses only one "Model table":
The table which had the best cost AND record became the "Model table".
After that, if a table's cost and record were both worse than
those of the Model Table, the table would be pruned away.

This didn't work well when the first table (the optimizer sorts them
by record_count) had low record_count but relatively high cost: nothing
could be pruned afterwards.

The patch adds the two additional "Model tables": one with the least
cost and the other with the least record_count.
(In both cases, a table can be pruned away if BOTH its cost and
record_count are worse than those of a Model table)

The new pruning is active when the number of tables to consider for
the prefix is higher than @@optimizer_extra_pruning_depth.

One can see the new pruning in the Optimizer Trace as
- "pruned_by_heuristic":"min_record_count", or
- "pruned_by_heuristic":"min_read_time".
Old heuristic pruning shows as "pruned_by_heuristic":1.

Author: Sergey Petrunia
Author Date: 2022-07-19 11:13:17 UTC

MDEV-28929: Plan selection takes forever with MDEV-28852 ...
Part #2: Extend heuristic pruning to use multiple tables as the
"Model tables".

Before the patch, heuristic pruning uses only one "Model table":
The table which had the best cost AND record became the "Model table".
After that, if a table's cost and record were both worse than
those of the Model Table, the table would be pruned away.

This didn't work well when the first table (the optimizer sorts them
by record_count) had low record_count but relatively high cost: nothing
could be pruned afterwards.

The patch adds the two additional "Model tables": one with the least
cost and the other with the least record_count.
(In both cases, a table can be pruned away if BOTH its cost and
record_count are worse than those of a Model table)

The new pruning is active when the number of tables to consider for
the prefix is higher than @@optimizer_extra_pruning_depth.

One can see the new pruning in the Optimizer Trace as
- "pruned_by_heuristic":"min_record_count", or
- "pruned_by_heuristic":"min_read_time".
Old heuristic pruning shows as "pruned_by_heuristic":1.

Author: Julius Goryavsky
Author Date: 2022-07-19 11:26:19 UTC

MDEV-28758: Mariabackup copies binary logs to backup directory

This commit restores defaults and functionality regarding binlogs
to the way it was prior to MDEV-27524. The mariabackup utility no
longer saves binlogs files as part of a backup without the --galera-info
option. However, since we use --galera-info during SST, the behavior
of mariabackup changes and, in combination with GTIDs support enabled,
mariabackup transfers one (most recent) binlog file obtained after
FLUSH BINARY LOGS. In other cases, binlogs are not transferred during
SST in mariabackup mode. As for SST in the rsync mode, it works the
same way as before MDEV-27524 - by default it transfers one last
binlog file.

The --sst-max-binlogs option for mariabackup and the sst_max_binlogs
parameter in the [sst] / server sections are no longer supported for
SST via mariabackup.

Author: midenok
Author Date: 2022-07-18 12:13:23 UTC

Tmp

Author: mariadb-LuisLizardo
Author Date: 2022-07-14 10:06:00 UTC

Change logic to always close the writer after printing query blocks

Author: Oleg Smirnov
Author Date: 2022-07-07 15:57:38 UTC

MDEV-28881 Fix memory leak caused by STL usage

Dep_analysis_context::create_unique_pseudo_key_if_needed() was using
std::set which allocated memory outside MEM_ROOT and so required
explicit deallocation. This has been solved by replacing std::set with
MY_BITMAP which is allocated on MEM_ROOT

Author: Daniel Black
Author Date: 2022-07-11 02:52:16 UTC

MDEV-29015 innodb os_file_set_size WSL workaround

WSL wasn't keeping stack of renamed files
(https://github.com/microsoft/WSL/issues/8443)
resulting in the fstat calls in os_file_set_size returning
ENOENT and no fallocate fallback being possible.

Users reported MySQL was ok, and it used my_seek to
determine the size. We copy this concept here to avoid
the WSL bug.

Author: Alexey Botchkov
Author Date: 2022-07-08 09:54:10 UTC

Fix to quiet the compiler on Windows.

Author: midenok
Author Date: 2022-07-04 12:08:01 UTC

MDEV-29023 MTR hangs after multiple failures

Passing $opt_parallel as $childs is wrong: child can be killed before
it connects and you will never decrement $childs for this.

Another problem is (and that is the cause of this bug): child can be
killed and never close server socket. This can happen f.ex. after
unmaskable KILL signal. In such case the socket is closed by reaping
the child but that never happens inside reading the socket loop in
run_test_server().

The proper design is the waitless reap of children inside the socket
loop and if there is no more children we finish the socket loop. Since
there is Windows variation where we don't control the children via
waitpid(), all the clients must normally close the socket and only
this can finish the socket loop. For Unix variation we reckon that
case as all children closed the socket but not all yet died and for
that we do final waiting waitpid() (was done before the patch as
well).

To be more complete, we now handle 3 end-of-game scenarios in Unix:

   1. all children closed socket, all children died: everything is
      handled by the socket loop;

   2. all children closed socket, not all yet died: we wait for alive
      children to die after exiting the socket loop;

   3. not all children closed socket, all children died: everything is
      handled by the socket loop.

For Windows end-of-game scenario is only one:

   All children close the socket.

Author: midenok
Author Date: 2022-07-07 17:06:48 UTC

Tmp

Author: midenok
Author Date: 2022-07-07 14:27:34 UTC

MDEV-20088 Cleanup: unused Ed_connection, Execute_sql_statement, Protocol_local, etc

Removed needless classes.

Author: Nayuta Yanagisawa
Author Date: 2022-07-05 13:56:26 UTC

MDEV-29027 ASAN errors in spider_db_free_result after partition DDL

Spider calls ha_spider::close() at least twice on ALTER TABLE ... ADD
PARTITION. The first call frees wide_handler->trx and the second call
accesses wide_handler->trx->thd (heap-use-after-free).

In general, there seems to be no problem with using THD obtained by
the macro current_thd() except in background threads. Thus, we simply
replace wide_handler->trx->thd with current_thd().

Author: Nikita Malyavin
Author Date: 2022-07-04 22:19:12 UTC

MDEV-29013 ER_KEY_NOT_FOUND/lock timeout upon online alter with long unique

1. ER_KEY_NOT_FOUND
Some virtual columns were not updated because were not included in read_set
to the moment of calculation.

In Row_logs_event::do_apply_event some fields are excluded based on m_cols
value, the number of replicated rows.

We can't rely on this. Basically, we'd be satisfied, if all columns were
just set, at least for now.

2. ER_LOCK_WAIT_TIMEOUT
This is a long unique specific problem.

Sometimes, lookup_handler is created for to->file. To properly free it,
ha_reset should be called. It is usually done by calling
close_thread_table, but ALTER TABLE makes it differently. Hence, a single
ha_reset call is added to mysql_alter_table.

Also, the lifetime of lookup_handler is corrected: it's been allocated on
thd->mem_root, which in case of online alter
is event's mem_root, and can vary other ways.
It is changed to table->mem_root, which is more appropriate:
ha_reset is called only when table is closed, or in a few additional cases,
which correspond to a statement end. So memory leaks are unlikely here.

Author: Vladislav Vaintroub
Author Date: 2022-07-05 08:47:03 UTC

MDEV-24071 - repro

fixed a couple of assertions or crashes in the server.
Now Innodb is acting up

Author: Oleg Smirnov
Author Date: 2022-06-27 05:37:59 UTC

MDEV-28869 Eliminated tables are not shown in EXPLAIN

Show tables that have been eliminated during optimization stage
in the output of EXPLAIN/ANALYZE [FORMAT=JSON] commands. Show both
regular and derived tables.

Author: Vicențiu Ciorbaru
Author Date: 2022-07-02 19:15:22 UTC

Implement cost_for_order_by function

The sort length is extracted similarly to how sortlength() function does
it. The function makes use of filesort_use_addons function to compute
the length of addon fields. Finally, by calling compute_sort_costs we
get the fastest_sort possible.

Other changes:
* Sort_param::using_addon_fields() assumes addon fields are already
  allocated. This makes the use of Sort_param unusable for
  compute_sort_costs *if* we don't want to allocate addon fields.

  As a preliminary fix, pass "with_addon_fields" as bool value to
  compute_sort_costs() and make the internal functions use that value
  instead of Sort_param::using_addon_fields() method.

  The ideal fix would be to define a "leaner" struct with only the
  necessary members, but this can be done as a separate commit.

* Remove unnecessary fields from Sort_costs::costs array.
  As we do not yet differentiate between merge sort with fixed length vs
  dynamic length fields, eliminate that differentiation from enum sort_type.
  It can be added at a later date when we indeed have a cost
  differentiation.

* Fixup comments

Author: Nikita Malyavin
Author Date: 2022-06-30 11:37:22 UTC

speed up main.alter_table_online_debug

Author: Julius Goryavsky
Author Date: 2022-06-30 01:44:50 UTC

MDEV-28758: Mariabackup copies binary logs to backup directory

This commit restores defaults and functionality regarding binlogs
to the way it was prior to MDEV-27524. The mariabackup utility no
longer saves binlogs files as part of a backup without the --galera-info
option. However, since we use --galera-info during SST, the behavior
of mariabackup changes and, in combination with GTIDs support enabled,
mariabackup transfers one (most recent) binlog file obtained after
FLUSH BINARY LOGS. In other cases, binlogs are not transferred during
SST in mariabackup mode. As for SST in the rsync mode, as it was
before MDEV-27524 - it transfers one last binlog file by default.

The --sst-max-binlogs option for mariabackup and the sst_max_binlogs
parameter in the [sst] / server sections are no longer supported for
SST via mariabackup.

Author: Nikita Malyavin
Author Date: 2022-06-29 17:16:19 UTC

MDEV-28930 ALTER TABLE Deadlocks with parallel TL_WRITE

ALTER ONLINE TABLE acquires table with TL_READ. Myisam normally acquires
TL_WRITE for DML, which makes it hang until table is freed.

We deadlock once ALTER upgrades its MDL lock.

Solution:
Unlock table earlier. We don't need to hold TL_READ once we finished
copying. Relay log replication requires no data locks on `from` table.

Author: Nayuta Yanagisawa
Author Date: 2021-10-13 13:45:26 UTC

MDEV-26544 Assertion `part_share->auto_inc_initialized' failed in ha_partition::get_auto_increment on INSERT

The partition storage engine ignores return (error) values of
handler::info(). As a result, a query that should be aborted is
not aborted and then the server violates the assertion.

Author: Julius Goryavsky
Author Date: 2022-06-29 03:05:57 UTC

MDEV-27524 addendum: additional tests

Author: Igor Babaev
Author Date: 2022-06-29 00:40:08 UTC

An attempt to fix MDEV-21633

Author: Julius Goryavsky
Author Date: 2022-06-28 11:55:42 UTC

MDEV-27524 addendum: additional tests

Author: Julius Goryavsky
Author Date: 2022-06-28 03:06:30 UTC

MDEV-28968: mariabackup SST doesn't properly handle --innodb-buffer-pool-filename

The mariabackup SST script cannot properly handle SST from
the joiner side if a non-standard filename is specified in
the "--innodb-buffer-pool-filename" option. Log analysis
shows that the received file is copied to the destination
directory under the original name "ib_buffer_pool", and not
under the name specified using the option on the command
line or in the configuration file.

This commit fixes this bug and adds an updated mtr tests
to check for correct handling of the non-standard filenames
specified using the "--innodb-buffer-pool-filename" option.

Author: Alexander Barkov
Author Date: 2022-06-08 11:12:03 UTC

MDEV-23287 The INET4 data type

Author: Brandon Nesterenko
Author Date: 2022-06-27 13:39:19 UTC

MDEV-27655 Fix UBSAN's Findings in Replication

Tagged mysys/my_compare.c::ha_key_cmp() with attribute to ignore
UBSAN runtime checks on shift operations. In particular, it is
expected that this function can perform shift of greater length
than the number of bits in the underlying field variables.

Author: Julius Goryavsky
Author Date: 2022-06-27 03:03:21 UTC

MDEV-28758: Mariabackup copies binary logs to backup directory

This commit changes the default settings so that, by default,
mariabackup will not copy binary logs files to the backup directory
and will transfer only one (most recet) binary log file between nodes
during SST. If the user wants to make a backup containing all binary
logs or wants to transfer them all between nodes, then user must add
the sst_max_binlogs=-1 option to the [sst] section, to one of the
server configuration sections, or must use the equivalent command
line option (when running mariabackup).

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-06-24 09:55:42 UTC

MDEV-22133 handle_fatal_signal (sig=11) on optimized builds in handle_grant_table instead of ERROR | Buffer overflow (on optimized builds)

Return an error if we can not read a table required for ACL.

Author: Alexey Botchkov
Author Date: 2022-02-15 09:37:59 UTC

MDEV-27832 disable binary logging for SQL SERVICE.

Binary logging is now disabled for the queries run by SQL SERVICE.
The binlogging can be turned on with the 'SET SQL_LOG_BIN=On' query.

Conflicts:
 sql/sql_prepare.cc

Author: Nikita Malyavin
Author Date: 2022-06-23 22:45:44 UTC

sampling analysis

file=open(...)
for l in file:
     if l.startswith('SAMPLE') and not l.startswith('SAMPLE root'):
         print(l)
         n = dict((p.split('=')
           for p in l.replace('SAMPLE ', '').replace(' REJECT', '').split()
           ))
         if 'REJECT' in l:
             n['rej']=True
         nodes.append(n)

max(int(n['size']) for n in nodes)
max((float(n['p']), serialize(n)) for n in nodes)
max((float(n['size']), serialize(n)) for n in nodes if n['height'] == '0')

Author: Nayuta Yanagisawa
Author Date: 2022-06-22 17:17:24 UTC

MDEV-19190 Assertion `part_share->auto_inc_initialized' failed in ha_partition::get_auto_increment

Author: Julius Goryavsky
Author Date: 2022-06-21 11:29:45 UTC

MDEV-26562: galera-sst-mariabackup is failing due to missing xtrabackup_checkpoints

This commit contains workaround for a bug known as 'Red Hat issue 1870279'
(connection reset by peer issue in socat versions 1.7.3.3 to 1.7.4.0) which
further causes crashes during SST using mariabackup (when openssl is used).

Also fixed broken logic of automatic generation of the Diffie-Hellman parameters
for socat version less than 1.7.3 (which defaults to 512-bit values instead of
2048-bit ones).

Author: Julius Goryavsky
Author Date: 2022-06-16 01:45:45 UTC

MDEV-28656: Inability to roll upgrade without stopping the Galera cluster

This commit fixes a divergence in the IST/SST script code between
the 10.3+ and 10.9+ branches, due to which the cluster cannot contain
nodes with different versions of the server (since in this case SST/IST
scripts may fail).

This commit also includes test fixes specific to 10.4+ branches and
a few manual fixes after automatic "git cherry-pick", as well as server
code change specific to 10.9+.

Author: Nikita Malyavin
Author Date: 2022-02-23 19:38:51 UTC

Add tests and code instrumentations

Author: Julius Goryavsky
Author Date: 2022-06-17 11:56:03 UTC

MDEV-28583 postfix: fixing .result files after automatic merge

Author: Igor Babaev
Author Date: 2022-06-20 06:43:17 UTC

MDEV-7487 Semi-join optimization for single-table update/delete statements

This patch allows to use semi-join optimization at the top level of
single-table update and delete statements.
The problem of supporting such optimization became easy to resolve after
processing a single-table update/delete statement started using JOIN
structure. This allowed to use JOIN::prepare() not only for multi-table
updates/deletes but for single-table ones as well. This was done in the
patch for mdev-28883:
 Re-design the upper level of handling UPDATE and DELETE statements.

Note that JOIN::prepare() detects all subqueries that can be considered
as candidates for semi-join optimization. The code added by this patch
looks for such candidates at the top level and if such candidates are found
in the processed single-table update/delete the statement is handled in
the same way as a multi-table update/delete.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Nayuta Yanagisawa
Author Date: 2022-06-20 09:02:30 UTC

MDEV-28739 Trying to lock uninitialized mutex or hang upon shutdown after using Spider with query_cache

Author: Nayuta Yanagisawa
Author Date: 2022-06-20 09:02:30 UTC

MDEV-28739 Trying to lock uninitialized mutex or hang upon shutdown after using Spider with query_cache

Author: Oleksandr "Sanja" Byelkin
Author Date: 2021-10-20 09:37:14 UTC

MDEV-26647 (simple_password_check) Include password validation plugin information in the error message if the SQL statement is not satisfied password policy

Make the plugin reporting cause of the error.

Author: Nayuta Yanagisawa
Author Date: 2022-06-20 11:05:32 UTC

MDEV-27575 SIGSEGV in intern_plugin_lock on SHUTDOWN when setting Spider as default storage engine (temporary or global)

When Spider is set as the default storage engine, it is unloaded later
compared with other plugins. That is because Spider is referenced by
@@global.default_storage_engine and cannot be unloaded before the
variable is set to NULL.

spider_db_done() creates a new THD but, in the above mentioned case,
the new THD needs to lock its default_storage_engine and it does not
expect that the server is almost shut down and the variable is NULL.

Luckily, the newly created THD is used for nothing. Thus, we can just
remove the THD creation.

Author: Nikita Malyavin
Author Date: 2022-02-23 19:38:51 UTC

Add tests and code instrumentations

Author: Sergey Petrunia
Author Date: 2022-06-18 16:52:14 UTC

Cleanups for: 0762dd928 Improve pruning in greedy_search by sorting tables during search

Author: Igor Babaev
Author Date: 2022-06-18 07:49:15 UTC

MDEV-27159 Re-design the upper level of handling DML commands

This patch allows to execute only single-table and multi-table
UPDATE and DELETE statements using the method Sql_cmd_dml::execute().
The code that handles INSERT statements has not been touched.

Author: Julius Goryavsky
Author Date: 2022-06-17 02:59:29 UTC

MDEV-26562: galera-sst-mariabackup is failing due to missing xtrabackup_checkpoints

This commit contains workaround for a bug known as 'Red Hat issue 1870279'
(connection reset by peer issue in socat versions 1.7.3.3 to 1.7.4.0) which
further causes crashes during SST using mariabackup (when openssl is used).

Author: Julius Goryavsky
Author Date: 2022-06-17 11:56:03 UTC

MDEV-28583 postfix: fixing .result files after automatic merge

Author: Igor Babaev
Author Date: 2022-06-17 06:45:38 UTC

MDEV-27159 Re-design the upper level of handling DML commands

This patch allows to execute only single-table and multi-table
UPDATE and DELETE statements using the method Sql_cmd_dml::execute().
The code that handles INSERT statements has not been touched.

Author: Julius Goryavsky
Author Date: 2022-06-15 23:12:15 UTC

MDEV-28656: Inability to roll upgrade without stopping the Galera cluster

This commit fixes a divergence in the IST/SST script code between
the 10.3+ and 10.9+ branches, due to which the cluster cannot contain
nodes with different versions of the server (since in this case SST/IST
scripts may fail).

This commit also includes test fixes specific to 10.4+ branches.

Author: midenok
Author Date: 2021-12-05 19:20:20 UTC

MDEV-27180 Fully atomic partitioning DDL operations

Atomic DDL for partitioning originally covers crash-safety but it does
not recover fully from failures. F.ex. if error happened during the
rename or drop of partitions the ALTER operation does not return the
table to its original state.

The patch solves the above problem similarly to MDEV-25292 (atomic
CREATE OR REPLACE): new partitions are created as temporary, old
partitions are backed up, everything is guarded by two DDL log chains:
rollback and cleanup. Rollback chain contains the actions that bring
back the original table. Cleanup chain deletes the backup files.

The generic operation of alter partition is as follows:

  1. Create new partitions as tmp partitions;
  2. Fill TMP partitions with data;
  3. Rename old and not needed partitions to backup partitions;
  4. Rename tmp partitions to original partititions;
  5. Do any required logging (mariabackup, binary);
  6. If everything is ok, drop the backup partitions;
  7. In the case any of pp.1-5 fails drop new partitions, rename back
     backup partitions.

Each rename operation is executed right after the corresponding DDL
logging. Originally they were done in different loops:
write_log_changed_partitions(), mysql_change_partitions(), etc. The
actual table operations were done in ha_partition handler:
ha_change_partitions(), ha_drop_partitions(),
ha_rename_partitions(). Now these calls are deprecated and removed.

Instead the above deprecated interfaces the following classes now
handle DDL logging and table operations:

      Alter_partition_logger
      Alter_partition_add
      Alter_partition_change

Alter_partition_logger handles the basic operation, DDL logging and
table renames. Alter_partition_add does the stuff for adding the
partitions. Alter_partition_change combines Alter_partition_logger and
Alter_partition_add for complex operations such as REORGANIZE.

ha_partition::change_partitions() call was not fully removed. Stages
1-4 from that call was renamed to allocate_partitions() and that is
used by Alter_partition_add for initializing file handler arrays
(m_added_file, m_reorged_file). ha_partition::prepare_new_partition()
is renamed to create_partition() because it actually does ha_create()
and the new name better describes what it does.

DDL_LOG_DELETE/DDL_LOG_RENAME/DDL_LOG_REPLACE are now pure file
operations. To process both .par and .frm files we now issue two
actions instead of one. That makes interface more simple and that will
be used by MDEV-16417 in future. All the table operations are done via
other actions, such as DDL_LOG_RENAME_TABLE, DDL_LOG_DROP_TABLE, etc.

DDL_LOG_RENAME_TABLE has new alter_partition option. With this option
it does simple ha_rename_table() which is needed for renaming the
partitions. The action for renaming the partitions must start from
phase DDL_RENAME_PHASE_TABLE.

The testing was refactored a bit. Many new fail/crash points was
added, the test was powered by binary and mariabackup logging, it
tests more ALTER commands with variations of partitions-only and
subpartitions, without and under LOCK TABLES.

EXCHANGE PARTITION was fixed: after failure under LOCK TABLES the
table became unlocked. Fixed by correcting place of
reopen_tables(). Tested by partition_debug.