lp:maria

Author: Yuchen Pei
Author Date: 2023-04-26 02:40:47 UTC

MDEV-22979 [do not push] remove wait in spider_create_handler()

This will fail mdev_22979_bootstrap due to race condition.

Author: Brandon Nesterenko
Author Date: 2023-04-05 16:43:28 UTC

MDEV-30430: Enabling system versioning on tables without primary key breaks replication

When replicating MDL events for a table that uses system versioning
without primary keys, ensure that for data sets with duplicate
records, the updates to these records with duplicates are enacted on
the correct row. That is, there was a bug (reported in MDEV-30430)
such that the function to find the row to update would stop after
finding the first matching record. However, in the absence of
primary keys, the version of the record is needed to compare the row
to ensure we are updating the correct one.

The fix, therefore, updates the record comparison functionality to
use system version columns when there are no primary keys on the
table.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2022-09-28 18:34:44 UTC

MDEV-28798: Previously Binlog Encrypted Master Segfaults on Binlog Dump with Using_Gtid=Slave_Pos

Problem:
========
A master can segfault if it can't set up decryption for its binary
log during a binlog dump with Using_Gtid=Slave_Pos. If slave
connects using GTID mode, the master will call into
log.cc::get_gtid_list_event(), which iterate through binlog events
looking for a Gtid_list_log_event. On an encrypted binlog that the
master cannot decrypt, the first event will be a
START_ENCRYPTION_EVENT which will call into the following decryption branch

if (fdle->start_decryption((Start_encryption_log_event*) ev))
  errormsg= ‘Could not set up decryption for binlog.’;

The event iteration however, does not stop in spite of this error.
The master will try to read the next event, but segfault while
trying to decrypt it because decryption failed to initialize.

Solution:
========
Break the event iteration if decryption cannot be set up.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2023-04-05 16:43:28 UTC

MDEV-30430: Enabling system versioning on tables without primary key breaks replication

When replicating MDL events for a table that uses system versioning
without primary keys, ensure that for data sets with duplicate
records, the updates to these records with duplicates are enacted on
the correct row. That is, there was a bug (reported in MDEV-30430)
such that the function to find the row to update would stop after
finding the first matching record. However, in the absence of
primary keys, the version of the record is needed to compare the row
to ensure we are updating the correct one.

The fix, therefore, updates the record comparison functionality to
use system version columns when there are no primary keys on the
table.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Brandon Nesterenko
Author Date: 2023-04-05 16:43:28 UTC

MDEV-30430: Enabling system versioning on tables without primary key breaks replication

When replicating MDL events for a table that uses system versioning
without primary keys, ensure that for data sets with duplicate
records, the updates to these records with duplicates are enacted on
the correct row. That is, there was a bug (reported in MDEV-30430)
such that the function to find the row to update would stop after
finding the first matching record. However, in the absence of
primary keys, the version of the record is needed to compare the row
to ensure we are updating the correct one.

The fix, therefore, updates the record comparison functionality to
use system version columns when there are no primary keys on the
table.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

Author: Vladislav Vaintroub
Author Date: 2023-04-25 12:24:39 UTC

MDEV-31095 tpool - do not create new worker, if thread creation is pending.

Use an std::atomic_flag to track thread creation in progress.
This is mainly a cleanup, the effect of this change was not measureable
in my tests.

Author: Sergey Petrunia
Author Date: 2023-04-24 14:57:45 UTC

MDEV-31121: ANALYZE statement produces 0 for all timings in embedded server

One must initialize the timer sybsystem with my_timer_init() call.
It was made only in mysqld_main(). Call it also from init_embedded_server().

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-04-17 13:06:52 UTC

MDEV-30889: 3 - Item_in_optimizer leak

Keep Item_in_optimizer cache always (but only once) in statement memory.

Author: Brandon Nesterenko
Author Date: 2023-04-24 15:52:28 UTC

MDEV-30430: Needed primary-replica sync in test

Author: Brandon Nesterenko
Author Date: 2022-09-29 19:40:51 UTC

MDEV-28798: Cosmetic Changes Only

Removed trailing whitespaces

Author: Igor Babaev
Author Date: 2023-04-21 20:46:14 UTC

MDEV-31102 Crash when pushing condition into view defined as union

This bug could manifest itself at the first execution of prepared statement
created for queries using a materialized view defined as union. A crash
could happen for sure if the query contained a condition pushable into
the view and this condition was over the column defined via a complex string
expression requiring implicit conversion from one charset to another for
some of its sub-expressions. The bug could cause crashes when executing
PS for some other queries whose optimization needed building clones for
such expressions.

This bug was introduced in the patch for MDEV-29988 where the class
Item_direct_ref_to_item was added. The implementations of the virtual
methods get_copy() and build_clone() were invalid for the class and this
could cause crashes after the method build_clone() was called for
expressions containing objects of the Item_direct_ref_to_item type.

Approved by Sergei Golubchik <serg@mariadb.com>

Author: Marko Mäkelä
Author Date: 2023-04-21 11:14:30 UTC

fixup! 1451bc6522dec0d2a64f398577d3c4c3956c4dd0

Author: Yuchen Pei
Author Date: 2023-04-21 06:13:52 UTC

Comparing test results under a broken spider

- mdev_22979: pass with failed post-test check
- mdev_22979_2: pass
- mdev_22979_bootstrap: ERROR: 12524 Can't open system table mysql.spider_table_sts

Author: Sergei Golubchik
Author Date: 2023-04-20 15:05:38 UTC

Merge branch 'merge-zlib' into 10.3

Author: midenok
Author Date: 2023-04-20 09:38:34 UTC

Tmp

Author: Yuchen Pei
Author Date: 2023-04-20 06:45:23 UTC

MDEV-27233-22979-30370 Add and update tests for spider init bugs

For development only - will squash this into another commit later - do
not push.

mdev_27233_v2: a variant of mdev-27233 where the spider plugin is
loaded in during server startup.

mdev_30370: we update this test to use mysqld_cmd instead of the
noisy mysqld_bootstrap_cmd. We have to kill the server first to avoid
"Can't lock aria control file"

With the change in commit c160a115b8b that was pushed to fix
mdev-27233 but later reverted because of mdev-29904, mdev_27233 and
the updated mdev_30370 would pass, but not mdev_27233_v2 ("[ERROR]
SPIDER plugin initialization failed at 'drop procedure if exists
mysql.spider_fix_one_table' by 'Unknown storage engine 'Aria''") or
mdev_30370 if it uses mysqld_bootstrap_cmd (post-test check will fail
when running this case with some other case, e.g. mtr mdev_30370
mdev_29562).

Author: Marko Mäkelä
Author Date: 2023-03-27 10:47:07 UTC

Cleanup: MONITOR_EXISTING trx_undo_slots_used, trx_undo_slots_cached

Let us remove explicit updates of MONITOR_NUM_UNDO_SLOT_USED
and MONITOR_NUM_UNDO_SLOT_CACHED, and let us compute the rough values
from trx_sys.rseg_array[] on demand.

Author: Sergey Petrunia
Author Date: 2023-04-19 12:15:27 UTC

MDEV-31067: selectivity_from_histogram >1.0 for a DOUBLE_PREC_HB histogram

Variant #2.

When Histogram::point_selectivity() sees that the point value of interest
falls into one bucket, it tries to guess whether the bucket has many
different (unpopular) values or a few popular values. (The number of
rows is fixed, as it's a Height-balanced histogram).
The basis for this guess is the "width" of the value range the bucket
covers. Buckets covering wider value ranges are assumed to contain
values with proportionally lower frequencies.

This is just a [brave] guesswork. For a very narrow bucket, it may
produce an estimate that's larger than total #rows in the bucket
or even in the whole table.

Remove the guesswork and replace it with basic logic: return
either the per-table average selectivity of col=const, or selectivity
of one bucket, whichever is lower.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-04-04 05:17:20 UTC

MDEV-30996 insert.. select in presence of full text index freezes all other commits at commit time

- When binlog is enabled, trasaction takes a lot of time to
do sync operation on innodb fts table. This leads to block
of other transaction commit. To avoid this failure, remove
the fulltext sync operation during transaction commit.

- This patch does the revert of MDEV-25581 and also introduces
the multiple fts background threads to process and optimize
the fts table.

- Introduced the new variable innodb_fulltext_bg_threads
which does background processing of message and optimization.
Minimum value is 1 and Maximum value can be 255

- By introducing the above variable, InnoDB can do sync on
multiple tables parallel

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2023-04-19 09:43:20 UTC

MDEV-30996 INSERT..SELECT in presence of fulltext index
    freezes all other commits at commit time

- Introduced new variable innodb_fts_threads which processing
fulltext message, optimization of fulltext table.
Minimum value is 1, default value is 2 and maximum value is 255

- By having multiple fts threads, InnoDB can do sync of multiple
table at the same time.

- Introduce the class fts_slots_t, which can be used to store
the fts table in the slot.

Author: Marko Mäkelä
Author Date: 2023-04-19 09:12:54 UTC

MDEV-26827 fixup: Do not hog buf_pool.mutex

buf_flush_LRU_list_batch(): When evicting clean pages,
release and reacquire the buf_pool.mutex after every 32 pages.
Also, eliminate some conditional branches.

Author: Dmitry Shulga
Author Date: 2023-04-18 16:40:45 UTC

MDEV-30073: Stored Procedure Returns Corrupt Results When Run a Second Time

When name resolution is performed for a view involved in a query
executed as Prepared Statement, changes in an item list of view's select_lex
is resgistered for rollback with Item_change_list::rollback_item_tree_changes.
Later, EXISTS-to-IN transformation takes place and previously resolved items
are replace with new items created as part of EXISTS-to-IN transformation.
This change is supposed to be permanent, so it is not recorded for rollback.
On fininishing execution of a prepared statement clean-up is performed
that does rollback of changes made on phase of name resolutions.
In result, changes made by EXISTS-to-IN transformation is rolled back
that lead to output of incorrect results set.

To fix the original issue it is proposed to make permanent name resolution
for view's fields.

Unfortunately, this fix introduces another issue that takes plce when
join of a regular table and a view is run and there is condition
  'where exist (subquery)'
(see test case in the patch)

Author: Marko Mäkelä
Author Date: 2023-04-18 13:19:39 UTC

MDEV-30986 hybrid build with 10.6 InnoDB

Author: Sergey Petrunia
Author Date: 2023-04-18 10:25:27 UTC

MDEV-31067: selectivity_from_histogram >1.0 for a DOUBLE_PREC_HB histogram

When Histogram::point_selectivity() sees that the point value of interest
falls into one bucket, it tries to guess whether the bucket has many
different (unpopular) values or a few popular values. (The number of
rows is fixed, as it's a Height-balanced histogram).

The basis for this guess is the "width" of the value range the bucket
covers. Buckets covering wider value ranges are assumed to contain
values with proportionally lower frequencies.

Note that this is just a [brave] guesswork. For a very narrow bucket,
it may produce an estimate that's larger than total #rows in the table
or even in the whole table.

Make a conservative fix: make sure the produced estimate doesn't exceed
the number of rows in the histogram's bucket.

Author: Daniele Sciascia
Author Date: 2023-03-29 11:55:30 UTC

MDEV-30955 Explicit locks released too early in rollback path

Assertion `thd->mdl_context.is_lock_owner()` fires when a client is
disconnected, while transaction and and a table is opened through
`HANDLER` interface.
Reason for the assertion is that when a connection closes, its ongoing
transaction is eventually rolled back in
`Wsrep_client_state::bf_rollback()`. This method also releases explicit
which are expected to survive beyond the transaction lifetime.
This patch also removes calls to `mysql_ull_cleanup()`. User level
locks are not supported in combination with Galera, making these calls
unnecessary.

Author: Denis Protivensky
Author Date: 2023-03-14 11:08:12 UTC

MDEV-30804 Rollback multi-engine transaction requiring 2PC but committing in one phase

Author: Rex Johnston
Author Date: 2023-04-17 19:33:00 UTC

MDEV-31022 SIGSEGV in maria_create from create_internal_tmp_table

keydef incorrectly allocated on the stack in create_internal_tmp_table()

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-04-17 13:06:52 UTC

MDEV-14959: 3 - Item_in_optimizer leak

Keep Item_in_optimizer cache always (but only once) in statement memory.

Author: Yuchen Pei
Author Date: 2023-04-13 13:10:52 UTC

MDEV-28363 remove #ifdef SPIDER_use_LEX_CSTRING_for_Field_blob_constructor

Author: Vlad Lesin
Author Date: 2023-04-13 14:56:38 UTC

MDEV-31049 fil_delete_tablespace() returns wrong file handle if tablespace was closed by parrallel thread

Author: midenok
Author Date: 2022-12-13 06:03:10 UTC

MDEV-27180 Fully atomic partitioning DDL operations

Atomic DDL for partitioning originally covers crash-safety but it does
not recover fully from failures. F.ex. if error happened during the
rename or drop of partitions the ALTER operation does not return the
table to its original state.

The patch solves the above problem similarly to MDEV-25292 (atomic
CREATE OR REPLACE): new partitions are created as temporary, old
partitions are backed up, everything is guarded by two DDL log chains:
rollback and cleanup. Rollback chain contains the actions that bring
back the original table. Cleanup chain deletes the backup files.

The generic operation of alter partition is as follows:

  1. Create new partitions as tmp partitions;
  2. Fill TMP partitions with data;
  3. Rename old and not needed partitions to backup partitions;
  4. Rename tmp partitions to original partititions;
  5. Do any required logging (mariabackup, binary);
  6. If everything is ok, drop the backup partitions;
  7. In the case any of pp.1-5 fails drop new partitions, rename back
     backup partitions.

Each rename operation is executed right after the corresponding DDL
logging. Originally they were done in different loops:
write_log_changed_partitions(), mysql_change_partitions(), etc. The
actual table operations were done in ha_partition handler:
ha_change_partitions(), ha_drop_partitions(),
ha_rename_partitions(). Now these calls are removed.

Instead the above deprecated interfaces the following classes now
handle DDL logging and table operations:

      Alter_partition_logger
      Alter_partition_add
      Alter_partition_change

Alter_partition_logger handles the basic operation, DDL logging and
table renames. Alter_partition_add does the stuff for adding the
partitions. Alter_partition_change combines Alter_partition_logger and
Alter_partition_add for complex operations such as REORGANIZE.

ha_partition::change_partitions() call was not fully removed. Stages
1-4 from that call was renamed to allocate_partitions() and that is
used by Alter_partition_add for initializing file handler arrays
(m_added_file, m_reorged_file). ha_partition::prepare_new_partition()
is renamed to create_partition() because it actually does ha_create()
and the new name better describes what it does (see previous commit
"ha_partition refactoring").

DDL_LOG_DELETE/DDL_LOG_RENAME/DDL_LOG_REPLACE are now pure file
operations. To process both .par and .frm files we now issue two
actions instead of one. That makes interface more simple and that will
be used by MDEV-16417 in future. All the table operations are done via
other actions, such as DDL_LOG_RENAME_TABLE, DDL_LOG_DROP_TABLE, etc.

DDL_LOG_RENAME_TABLE has new alter_partition option. With this option
it does simple ha_rename_table() which is needed for renaming the
partitions. The action for renaming the partitions must start from
phase DDL_RENAME_PHASE_TABLE.

The testing was refactored a bit. Many new fail/crash points was
added, the test was powered by binary and mariabackup logging, it
tests more ALTER commands with variations of partitions-only and
subpartitions, without and under LOCK TABLES.

EXCHANGE PARTITION was fixed: after failure under LOCK TABLES the
table became unlocked. Fixed by correcting place of
reopen_tables(). Tested by partition_debug.

HA_PARTITION_ONE_PHASE handling was removed from
fast_alter_partition() as native partitioning is not supported.

Author: Marko Mäkelä
Author Date: 2023-04-13 10:26:18 UTC

squash! f6fa068076419395cc82d3757cfa8a0bd4b0cf56

recv_sys_t::is_initialised(): Refer to end_lsn.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-04-12 13:08:23 UTC

MDEV-14959: 2 - Allocation in TABLE_SHARE::init_from_sql_statement_string

 Fix leack in TABLE_SHARE::init_from_sql_statement_string
 by removing uneeded switching arenas.

Author: Daniel Black
Author Date: 2021-03-30 06:32:36 UTC

MDEV-25282 Auto-shutdown on idle when socket-activated

Adds max_idle_execution system variable that corresponds
to the time in seconds under which the mariadbd executable will
run in an idle state (since last query) with no connections.

Under systemd socket activation this variable will get a 10 minute
default value, otherwise 0, representing no timeout. This will
enable a service to be activated on connection and fall back to
a shutdown state after 10 minutes of no queries. The systemd
socket activation can restart the service on the next connection
transparently.

The maximum max_idle_execution is given by the following
factors given its dependence on the OS system calls.

Windows WaitForMultipleObjects takes a DWORD (unsigned)
measure in milliseconds. Poll takes a signed int milliseconds,
and negative values are treated as infinite so we can't overflow.
Select, the fall back if poll isn't there, takes a seconds value
in a timeval.time_t structure. As such the interface maximums are:
Windows: UINT_MAX / 1000
Poll: INT_MAX / 1000
Select: UINT_MAX (or higher)

As even the smallest value here, INT_MAX(32) / 1000 is ~25 days,
sufficient for the typical use case, its used in all environment for
simplicity of documentation and test cases.

A (non-exposed) global variable of server_last_activity is updated
on accepted connections (when max_idle_execution !=0)
and when the connection count (standard or extra) is down
to <= 1 to keep the number of updates on a single variable
low.

When the main accept loop times out on the max_idle_execution
seconds, and then the server_last_activity is checked along
with if current connection count (standard + extra) is 0
(in case a recently started connection hasn't finished
a query).

To make this neater, in non-Windows main accept loop moved
code to handle_new_socket_connection that encompasses accepting
a connection and the timeout mechanism has been separated too.

Changed when looping though possible connections, loop until
the end of the connection list and hereby assume two connection can
occur on the same poll/select call and both will be accepted.

The interactive_timeout and wait_timeout inherit the
max_idle_execution time (if set) compared to their previous defaults.

Thanks Sergei for the review.

Author: =?utf-8?q?Jan_Lindstr=C3=B6m?=
Author Date: 2023-03-27 08:33:11 UTC

MDEV-28641 : Query cache entries not invalidated on slave of a Galera cluster

Query cache should be invalidated if we are not in applier. For some
reason this condition was incorrect starting from 10.5 but it is
correct in 10.4.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Sergey Petrunia
Author Date: 2023-03-30 13:18:49 UTC

MDEV-30830: ANALYZE FORMAT=JSON: r_unpack_time_ms is empty for the hashed joins

Do the time counting in the right place.

Author: Yuchen Pei
Author Date: 2023-03-21 23:31:26 UTC

MDEV-26137 Improve import tablespace workflow.

Allow ALTER TABLE ... IMPORT TABLESPACE without creating the table
followed by discarding the tablespace.

That is, assuming we want to import table t1 to t2, instead of

CREATE TABLE t2 LIKE t1;
ALTER TABLE t2 DISCARD TABLESPACE;
FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We can simply do

FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.frm $MYSQLD_DATADIR/test/t2.frm
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We achieve this by creating a "stub" table in the second scenario
while opening the table, where t2 does not exist but needs to import
from t1. The "stub" table is similar to a table that is created but
then instructed to discard its tablespace.

We include tests with various row formats, encryption, with indexes
and auto-increment.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Yuchen Pei
Author Date: 2023-03-29 23:25:02 UTC

separated out changes to handle_instant_metadata() from 9501981c

Author: Sergey Petrunia
Author Date: 2023-03-29 16:42:21 UTC

MDEV-30964: MAX_SEL_ARG memory exhaustion is not visible in the optimizer trace

Add printing

Author: midenok
Author Date: 2023-03-05 10:21:02 UTC

Review

Author: Yuchen Pei
Author Date: 2023-03-23 23:38:20 UTC

MDEV-30920 Remove need_lock and table from spider_close_sys_table()

They became obsolete after commit cfd145faedd:

commit cfd145faedd68e27d4ac8f1e5f95ec1a6dcd8c8d
Author: Nayuta Yanagisawa <nayuta.yanagisawa@hey.com>
Date: Fri Jan 28 01:03:06 2022 +0900

    MDEV-27641 Spider: remove #if MYSQL_VERSION_ID < ${VERSION}

Author: Marko Mäkelä
Author Date: 2023-03-27 16:38:38 UTC

WIP fix MDEV-30936 clang 15.0.7 -fsanitize=memory fails massively

FIXME: innodb.full_crc32_import fails due to CRC-32C computation in
buf_page_is_corrupted().

Author: Vlad Lesin
Author Date: 2023-03-23 13:26:17 UTC

MDEV-29050 mariabackup issues error messages during InnoDB tablespaces export on partial backup preparing

The solution is to suppress error messages for missing tablespaces if
mariabackup is launched with "--prepare --export" options.

"mariabackup --prepare --export" invokes itself with --mysqld parameter.
If the parameter is set, then it starts server to feed "FLUSH TABLES ...
FOR EXPORT;" queries for exported tablespaces. This is "normal" server
start, that's why new srv_operation value is introduced.

Reviewed by Marko Makela.

Author: Vlad Lesin
Author Date: 2023-03-23 13:26:17 UTC

MDEV-29050 mariabackup issues error messages during InnoDB tablespaces export on partial backup preparing

The solution is to suppress error messages for missing tablespaces if
mariabackup is launched with "--prepare --export" options.

"mariabackup --prepare --export" invokes itself with --mysqld parameter.
If the parameter is set, then it starts server to feed "FLUSH TABLES ...
FOR EXPORT;" queries for exported tablespaces. This is "normal" server
start, that's why new srv_operation value is introduced.

Reviewed by Marko Makela.

Author: midenok
Author Date: 2023-03-05 10:21:02 UTC

Review

Author: midenok
Author Date: 2023-01-15 21:01:35 UTC

MDEV-30421 Allow administrators to enable or disable parallel replication on a per-table basis

Per-domain dedicated thread for processing ordered transactions. The
thread is reserved from the total number of domain threads (controlled
by slave_parallel_threads and slave_domain_parallel_threads). Whether
the event goes to ordered thread depends on FL_ALLOW_PARALLEL flag as
well as several other conditions. FL_ALLOW_PARALLEL is passed from
master and is set for the event depending on master configuration
directives. To allow dedicated slave on server one must enable it
explicitly with configuration directive:

  set global slave_ordered_thread= 1;

Originally it was controlled by skip_parallel_replication session
variable which can be changed per-statement. This patch adds several
more directives to control it on per-schema and per-table levels:

  parallel_do_db
  parallel_do_table
  parallel_ignore_db
  parallel_ignore_table
  parallel_wild_do_table
  parallel_wild_ignore_table

Each directive is comma-separated list of fully-qualified table
names. Spaces after comma are ignored (but not before).

"Table" directives take precedence over "db" directives. "Do"
directives take precedence over "ignore" directives. "Wild" directives
are checked if "do" and "ignore" directives did not match.

If none of the above directives present everything is considered
parallel. If any of the above directives present and the table did not
match anything in the lists it is considered ordered.

Examples:

  set @@global.parallel_do_db= "db_parallel";
  set @@global.parallel_ignore_db= "db_serial";
  set global parallel_do_table= "db_serial.t3, db_serial.t1";
  set global parallel_wild_ignore_table= "db_parallel.non_parallel_%"

Normal behaviour of ordered transaction is before start to wait any of
prior transactions to commit: they get into different commit
groups. But since all the ordered transactions (within one domain) go
to a single thread we may avoid that restriction with this directive
on slave:

  set global slave_ordered_dont_wait= 1;

When set events without explicit FL_WAITED flag going to ordered
thread nonetheless accept optimistic speculation. I.e. they get into
same commit group with parallel events: ordered event is executed in
parallel with parallel events.

Author: Yuchen Pei
Author Date: 2023-03-23 06:32:09 UTC

MDEV-30265 Document some sts related functions in spider.

Author: Yuchen Pei
Author Date: 2023-03-23 23:30:22 UTC

MDBF-535 Breaking some spider tests to check which ci's cover spider

Author: Andrei
Author Date: 2023-03-23 16:45:34 UTC

MDEV-30620 Trying to lock uninitialized LOCK_parallel_entry

The error was seen by a number of mtr tests being caused
by overdue initialization of rpl_parallel::LOCK_parallel_entry.
Specifically, SHOW-SLAVE-STATUS might find in
rpl_parallel::workers_idle() a gtid domain hash entry
already inserted whose mutex had not done
mysql_mutex_init().

Fixed with swapping the mutex init and the its entry's stack insertion.

Tested with a generous number of `mtr --repeat` of a few of the reported
to fail tests, incl rpl.parallel_backup.

Author: Nikita Malyavin
Author Date: 2023-03-23 13:29:21 UTC

wake up from sleep

Author: Yuchen Pei
Author Date: 2023-03-23 06:32:09 UTC

MDEV-30265 Document some sts related functions in spider.

Author: Igor Babaev
Author Date: 2023-03-23 04:27:51 UTC

MDEV-29971 Re-design the upper level of handling SELECT statements

The initial patch.
All tests from the main test suite passed.

Author: Nayuta Yanagisawa
Author Date: 2022-05-26 23:55:42 UTC

MDEV-28522 Delete constant SPIDER_SQL_TYPE_*_HS

The HandlerSocket support of Spider has been deleted by MDEV-26858.
Thus, the constants, SPIDER_SQL_TYPE_*_HS, are no longer necessary.

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: midenok
Author Date: 2023-03-22 09:53:58 UTC

MDEV-30421 more tests cleaned up

All the .inc files that included from binlog_encryption are refactored.

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-03-21 23:33:45 UTC

Addressing review comments

Author: Yuchen Pei
Author Date: 2023-02-08 01:29:52 UTC

MDEV-28152 Features for sequences

- Add `as <int_type>` to sequence creation options
  - int_type can be signed or unsigned integer types, including
    tinyint, smallint, mediumint, int and bigint
  - Limitation: when alter sequence as <new_int_type>, cannot have any
    other alter options in the same statement
  - Limitation: increment remains signed longlong, and the hidden
    constraint (cache_size x abs(increment) < longlong_max) stays for
    unsigned types. This means for bigint unsigned, neither
    abs(increment) nor (cache_size x abs(increment)) can be between
    longlong_max and ulonglong_max
- Truncating maxvalue and minvalue from user input to the nearest max
  or min value of the type, plus or minus 1. When the truncation
  happens, a warning is emitted
- Information schema table for sequences

Author: Yuchen Pei
Author Date: 2023-03-20 01:41:30 UTC

MDEV-28152 Addressing round 1 review comments

... at https://lists.launchpad.net/maria-developers/msg13297.html

Also implemented setval and auto increment for unsigned sequences.

Author: Yuchen Pei
Author Date: 2023-02-08 01:29:52 UTC

MDEV-28152 Features for sequences

- Add `as <int_type>` to sequence creation options
  - int_type can be signed or unsigned integer types, including
    tinyint, smallint, mediumint, int and bigint
  - Limitation: when alter sequence as <new_int_type>, cannot have any
    other alter options in the same statement
  - Limitation: increment remains signed longlong, and the hidden
    constraint (cache_size x abs(increment) < longlong_max) stays for
    unsigned types. This means for bigint unsigned, neither
    abs(increment) nor (cache_size x abs(increment)) can be between
    longlong_max and ulonglong_max
- Truncating maxvalue and minvalue from user input to the nearest max
  or min value of the type, plus or minus 1. When the truncation
  happens, a warning is emitted
- Information schema table for sequences

Author: Oleg Smirnov
Author Date: 2023-02-17 12:21:30 UTC

MDEV-8320 Allow index usage for DATE(col) <=> const and YEAR <=> const

Rewrite datetime comparison conditions into sargeable. For example,
    YEAR(col) <= val -> col <= YEAR_END(val)
    YEAR(col) < val -> col < YEAR_START(val)
    YEAR(col) >= val -> col >= YEAR_START(val)
    YEAR(col) > val -> col > YEAR_END(val)
    YEAR(col) = val -> col BETWEEN YEAR_START(val) AND YEAR_END(val)
Do the same with DATE(col), for example:
    DATE(col) <= val -> col <= DAY_END(val)

After such a rewrite index lookup on column "col" can be employed

Author: Marko Mäkelä
Author Date: 2023-03-17 14:26:16 UTC

Merge 11.1

Author: Yuchen Pei
Author Date: 2023-03-15 03:50:22 UTC

wip please ignore

Author: Yuchen Pei
Author Date: 2023-03-15 03:50:22 UTC

wip please ignore

Author: Julius Goryavsky
Author Date: 2023-03-15 17:41:51 UTC

MDEV-30855: disable unnecessary test

Author: Lena Startseva
Author Date: 2022-09-19 05:26:59 UTC

MDEV-29390: Improve coverage for UPDATE and DELETE statements in MTR test suites

Created tests for "delete" based on update_use_source.test

For the update_use_source.test tests, data recovery in the table has been changed
from a rollback transaction to a complete delete and re-insert of the data with
optimize table. Cases are now being checked on three engines.

Added tests for update/delete with LooseScan and DuplicateWeedout optimization strategies
Added tests for engine MEMORY on delete and update
Added tests for multi-update with JSON_TABLE
Added tests for multi-update and multi-delete for engine Connect

Author: Rex Johnston
Author Date: 2023-03-06 21:29:56 UTC

MDEV-30605 Wrong result while using index for group-by

 incorrect result when group function applied on a primary key

Author: Igor Babaev
Author Date: 2023-01-31 03:42:27 UTC

MDEV-28958 Crash when checking whether condition can be pushed into view

Do not set any flags in the items for constant subformulas TRUE/FALSE when
checking pushability of a formula into a view. Occurrences of these
subformulas can be ignored when checking pushability of the formula.
At the same time the items used for these constants became immutable
starting from version 10.7.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Julius Goryavsky
Author Date: 2023-03-13 16:41:06 UTC

Post-MDEV-30700: moving alloca() definitions from all *.h files to new header file

Included config file for proper compilation without <my_global.h>

Author: Monty
Author Date: 2023-03-13 16:38:02 UTC

fixup! 2cc16653817dd80df1c57f7d8499352fc1eeaa5b

Author: Dmitry Shulga
Author Date: 2023-03-13 08:01:53 UTC

MDEV-5816: Stored programs: validation of stored program statements

Follow-up patch that fixes issue with missing set up of versioning
condition for query involving a versioning partitioned table.

Author: Igor Babaev
Author Date: 2023-03-10 05:48:58 UTC

Adjusted test results after rebase against 11.0.1

Author: Monty
Author Date: 2023-03-09 09:07:10 UTC

Test commit to see if reverting 'unscrambling' of table order changes things

This is an experimental commit, only to be used for testing

Author: Rex Johnston
Author Date: 2023-03-09 01:41:07 UTC

MDEV-30815 Printing a stack trace leaks file descriptors.

 In situations where mariadbd emits a lot of stack traces it can
        run out of file descriptors. Each time the pipe to the external
        utility addr2line is opened, those file descriptor are never closed.

Author: Yuchen Pei
Author Date: 2023-03-01 06:34:05 UTC

MDEV-28152 wip wip wip alter sequence as <new_value_type>

Basic case:
- start with a narrow value type, the alter to a wider type without
  changing any other metadata

Advanced cases to fix once the basic case works:
- start with a wider value type, then alter to a narrower type with
  smaller metadata values in range for the narrower value type
- start with a narrower value type, then alter to a wider type with
  bigger metadata values out of range for the narrower type but in
  range for the wider type

Author: Rex Johnston
Author Date: 2023-03-06 00:58:27 UTC

MDEV-30771 Optimizer trace: table_scan.rows is traced as integer, change to double

 Alter formatting of printed doubles written to the optimizer trace
 to ensure it is clear they ARE floating point numbers.

Author: Rex Johnston
Author Date: 2023-02-01 03:59:39 UTC

MDEV-23384 Server crashes in subselect_indexsubquery_engine::print with optimizer trace enabled

 unitialized data structures
 causing subselect_indexsubquery_engine::print
 and subselect_uniquesubquery_engine::print to crash
 when optimizer_trace is enabled

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-11-25 12:37:07 UTC

MDEV-26831 fallout: name resolution cache simple test

Author: Sergey Petrunia
Author Date: 2023-02-22 09:11:55 UTC

MDEV-30693: Assertion `dbl_records <= s->records' failed in apply_selectivity_for_table on SELECT

The crash happened due to rows=2 vs rows=1 difference between how the
estimate of number of rows in a derived table is computed in
TABLE_LIST::fetch_number_of_rows() and JOIN::add_keyuses_for_splitting().

Made JOIN::add_keyuses_for_splitting() use the result of computations in
TABLE_LIST::fetch_number_of_rows().

Author: Yuchen Pei
Author Date: 2023-02-22 02:42:38 UTC

wip wip wip Adding unsigned by making fields Longlong_hybrid

Author: Michael Widenius
Author Date: 2023-02-19 18:05:13 UTC

Added mysql-log-rotate to .gitignore

Author: Michael Widenius
Author Date: 2023-02-20 10:28:44 UTC

squash! 4cb69791aaa59e786859405c3c3826a6a15c4a09

- Ensure that TEMP_TABLE_PARAM.func_count includes all items that may
  need a copy function.
- Fixed that Aria allocates enough space for key copies.
- Fixed that Aria does not check empty_bits if not allocated.

The first issue could cause crashes, the other issues should not
affect anything.

Author: Yuchen Pei
Author Date: 2023-02-21 01:21:28 UTC

updating some comments

Author: Teemu Ollakka
Author Date: 2023-02-13 16:14:50 UTC

MDEV-27317 wsrep_checkpoint order violation due to certification failure

With binlogs enabled, debug assertion ut_ad(xid_seqno > wsrep_seqno)
fired in trx_rseg_update_wsrep_checkpoint() when an applier thread
synced the seqno out of order for write set which had failed
certification. This was caused by releasing commit
order too early when binlogs were on, allowing group
commit to run in parallel and commit following transactions
too early.

Fixed by extending the commit order critical section to cover
call to wsrep_set_SE_checkpoint() also when binlogs are on.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Nikita Malyavin
Author Date: 2023-02-20 11:05:46 UTC

fix Run-Time Check Failure on Windows

Author: Yuchen Pei
Author Date: 2023-02-20 02:03:53 UTC

add check for discarded in h_i::open()

Author: Nikita Malyavin
Author Date: 2023-02-17 16:58:41 UTC

fix after rebasing onto MDEV-30378 fix

Author: Lena Startseva
Author Date: 2022-09-19 05:26:59 UTC

MDEV-29390: Improve coverage for UPDATE and DELETE statements in MTR test suites

Created tests for "delete" based on update_use_source.test

For the update_use_source.test tests, data recovery in the table has been changed
from a rollback transaction to a complete delete and re-insert of the data with
optimize table. Cases are now being checked on three engines.

Added tests for update/delete with LooseScan and DuplicateWeedout optimization strategies
Added tests for engine MEMORY on delete and update
Added tests for multi-update with JSON_TABLE
Added tests for multi-update and multi-delete for engine Connect

Author: Yuchen Pei
Author Date: 2023-02-15 06:25:01 UTC

WIP try creating a table using cfg and cloning exiting table

Author: Rex Johnston
Author Date: 2023-02-16 20:28:38 UTC

MDEV-29179 Condition pushdown from HAVING into WHERE is not shown in optimizer trace

 JOIN::optimize_inner(), Condition pushdown from HAVING into WHERE
   not shown in optimizer trace.

Author: Nikita Malyavin
Author Date: 2022-10-21 12:08:26 UTC

MDEV-29068 Cascade foreign key updates do not apply in online alter

Author: Alexander Barkov
Author Date: 2023-02-14 09:27:46 UTC

MDEV-15751 CURRENT_TIMESTAMP should return a TIMESTAMP [WITH TIME ZONE?]

Author: Alexander Barkov
Author Date: 2023-02-14 07:52:10 UTC

MDEV-15750 preserve MYSQL_TYPE_TIMESTAMP in temporal arithmetic

    In progress.