Author: Marko Mäkelä
Author Date: 2023-04-18 13:19:39 UTC

MDEV-30986 hybrid build with 10.6 InnoDB

Author: Sergey Petrunia
Author Date: 2023-04-18 10:25:27 UTC

MDEV-31067: selectivity_from_histogram >1.0 for a DOUBLE_PREC_HB histogram

When Histogram::point_selectivity() sees that the point value of interest
falls into one bucket, it tries to guess whether the bucket has many
different (unpopular) values or a few popular values. (The number of
rows is fixed, as it's a Height-balanced histogram).

The basis for this guess is the "width" of the value range the bucket
covers. Buckets covering wider value ranges are assumed to contain
values with proportionally lower frequencies.

Note that this is just a [brave] guesswork. For a very narrow bucket,
it may produce an estimate that's larger than total #rows in the table
or even in the whole table.

Make a conservative fix: make sure the produced estimate doesn't exceed
the number of rows in the histogram's bucket.

Author: Daniele Sciascia
Author Date: 2023-03-29 11:55:30 UTC

MDEV-30955 Explicit locks released too early in rollback path

Assertion `thd->mdl_context.is_lock_owner()` fires when a client is
disconnected, while transaction and and a table is opened through
`HANDLER` interface.
Reason for the assertion is that when a connection closes, its ongoing
transaction is eventually rolled back in
`Wsrep_client_state::bf_rollback()`. This method also releases explicit
which are expected to survive beyond the transaction lifetime.
This patch also removes calls to `mysql_ull_cleanup()`. User level
locks are not supported in combination with Galera, making these calls
unnecessary.

Author: Denis Protivensky
Author Date: 2023-03-14 11:08:12 UTC

MDEV-30804 Rollback multi-engine transaction requiring 2PC but committing in one phase

Author: Rex Johnston
Author Date: 2023-04-17 19:33:00 UTC

MDEV-31022 SIGSEGV in maria_create from create_internal_tmp_table

keydef incorrectly allocated on the stack in create_internal_tmp_table()

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-04-17 13:06:52 UTC

MDEV-14959: 3 - Item_in_optimizer leak

Keep Item_in_optimizer cache always (but only once) in statement memory.

Author: Yuchen Pei
Author Date: 2023-04-13 13:10:52 UTC

MDEV-28363 remove #ifdef SPIDER_use_LEX_CSTRING_for_Field_blob_constructor

Author: Vlad Lesin
Author Date: 2023-04-13 14:56:38 UTC

MDEV-31049 fil_delete_tablespace() returns wrong file handle if tablespace was closed by parrallel thread

Author: midenok
Author Date: 2022-12-13 06:03:10 UTC

MDEV-27180 Fully atomic partitioning DDL operations

Atomic DDL for partitioning originally covers crash-safety but it does
not recover fully from failures. F.ex. if error happened during the
rename or drop of partitions the ALTER operation does not return the
table to its original state.

The patch solves the above problem similarly to MDEV-25292 (atomic
CREATE OR REPLACE): new partitions are created as temporary, old
partitions are backed up, everything is guarded by two DDL log chains:
rollback and cleanup. Rollback chain contains the actions that bring
back the original table. Cleanup chain deletes the backup files.

The generic operation of alter partition is as follows:

  1. Create new partitions as tmp partitions;
  2. Fill TMP partitions with data;
  3. Rename old and not needed partitions to backup partitions;
  4. Rename tmp partitions to original partititions;
  5. Do any required logging (mariabackup, binary);
  6. If everything is ok, drop the backup partitions;
  7. In the case any of pp.1-5 fails drop new partitions, rename back
     backup partitions.

Each rename operation is executed right after the corresponding DDL
logging. Originally they were done in different loops:
write_log_changed_partitions(), mysql_change_partitions(), etc. The
actual table operations were done in ha_partition handler:
ha_change_partitions(), ha_drop_partitions(),
ha_rename_partitions(). Now these calls are removed.

Instead the above deprecated interfaces the following classes now
handle DDL logging and table operations:

      Alter_partition_logger
      Alter_partition_add
      Alter_partition_change

Alter_partition_logger handles the basic operation, DDL logging and
table renames. Alter_partition_add does the stuff for adding the
partitions. Alter_partition_change combines Alter_partition_logger and
Alter_partition_add for complex operations such as REORGANIZE.

ha_partition::change_partitions() call was not fully removed. Stages
1-4 from that call was renamed to allocate_partitions() and that is
used by Alter_partition_add for initializing file handler arrays
(m_added_file, m_reorged_file). ha_partition::prepare_new_partition()
is renamed to create_partition() because it actually does ha_create()
and the new name better describes what it does (see previous commit
"ha_partition refactoring").

DDL_LOG_DELETE/DDL_LOG_RENAME/DDL_LOG_REPLACE are now pure file
operations. To process both .par and .frm files we now issue two
actions instead of one. That makes interface more simple and that will
be used by MDEV-16417 in future. All the table operations are done via
other actions, such as DDL_LOG_RENAME_TABLE, DDL_LOG_DROP_TABLE, etc.

DDL_LOG_RENAME_TABLE has new alter_partition option. With this option
it does simple ha_rename_table() which is needed for renaming the
partitions. The action for renaming the partitions must start from
phase DDL_RENAME_PHASE_TABLE.

The testing was refactored a bit. Many new fail/crash points was
added, the test was powered by binary and mariabackup logging, it
tests more ALTER commands with variations of partitions-only and
subpartitions, without and under LOCK TABLES.

EXCHANGE PARTITION was fixed: after failure under LOCK TABLES the
table became unlocked. Fixed by correcting place of
reopen_tables(). Tested by partition_debug.

HA_PARTITION_ONE_PHASE handling was removed from
fast_alter_partition() as native partitioning is not supported.

Author: Marko Mäkelä
Author Date: 2023-04-13 10:26:18 UTC

squash! f6fa068076419395cc82d3757cfa8a0bd4b0cf56

recv_sys_t::is_initialised(): Refer to end_lsn.

Author: Oleksandr "Sanja" Byelkin
Author Date: 2023-04-12 13:08:23 UTC

MDEV-14959: 2 - Allocation in TABLE_SHARE::init_from_sql_statement_string

 Fix leack in TABLE_SHARE::init_from_sql_statement_string
 by removing uneeded switching arenas.

Author: Daniel Black
Author Date: 2021-03-30 06:32:36 UTC

MDEV-25282 Auto-shutdown on idle when socket-activated

Adds max_idle_execution system variable that corresponds
to the time in seconds under which the mariadbd executable will
run in an idle state (since last query) with no connections.

Under systemd socket activation this variable will get a 10 minute
default value, otherwise 0, representing no timeout. This will
enable a service to be activated on connection and fall back to
a shutdown state after 10 minutes of no queries. The systemd
socket activation can restart the service on the next connection
transparently.

The maximum max_idle_execution is given by the following
factors given its dependence on the OS system calls.

Windows WaitForMultipleObjects takes a DWORD (unsigned)
measure in milliseconds. Poll takes a signed int milliseconds,
and negative values are treated as infinite so we can't overflow.
Select, the fall back if poll isn't there, takes a seconds value
in a timeval.time_t structure. As such the interface maximums are:
Windows: UINT_MAX / 1000
Poll: INT_MAX / 1000
Select: UINT_MAX (or higher)

As even the smallest value here, INT_MAX(32) / 1000 is ~25 days,
sufficient for the typical use case, its used in all environment for
simplicity of documentation and test cases.

A (non-exposed) global variable of server_last_activity is updated
on accepted connections (when max_idle_execution !=0)
and when the connection count (standard or extra) is down
to <= 1 to keep the number of updates on a single variable
low.

When the main accept loop times out on the max_idle_execution
seconds, and then the server_last_activity is checked along
with if current connection count (standard + extra) is 0
(in case a recently started connection hasn't finished
a query).

To make this neater, in non-Windows main accept loop moved
code to handle_new_socket_connection that encompasses accepting
a connection and the timeout mechanism has been separated too.

Changed when looping though possible connections, loop until
the end of the connection list and hereby assume two connection can
occur on the same poll/select call and both will be accepted.

The interactive_timeout and wait_timeout inherit the
max_idle_execution time (if set) compared to their previous defaults.

Thanks Sergei for the review.

Author: Vicențiu Ciorbaru
Author Date: 2023-04-04 14:27:40 UTC

Patch#4 Sysschema fix

Author: =?utf-8?q?Jan_Lindstr=C3=B6m?=
Author Date: 2023-03-27 08:33:11 UTC

MDEV-28641 : Query cache entries not invalidated on slave of a Galera cluster

Query cache should be invalidated if we are not in applier. For some
reason this condition was incorrect starting from 10.5 but it is
correct in 10.4.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Sergey Petrunia
Author Date: 2023-03-30 13:18:49 UTC

MDEV-30830: ANALYZE FORMAT=JSON: r_unpack_time_ms is empty for the hashed joins

Do the time counting in the right place.

Author: Yuchen Pei
Author Date: 2023-03-21 23:31:26 UTC

MDEV-26137 Improve import tablespace workflow.

Allow ALTER TABLE ... IMPORT TABLESPACE without creating the table
followed by discarding the tablespace.

That is, assuming we want to import table t1 to t2, instead of

CREATE TABLE t2 LIKE t1;
ALTER TABLE t2 DISCARD TABLESPACE;
FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We can simply do

FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.frm $MYSQLD_DATADIR/test/t2.frm
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We achieve this by creating a "stub" table in the second scenario
while opening the table, where t2 does not exist but needs to import
from t1. The "stub" table is similar to a table that is created but
then instructed to discard its tablespace.

We include tests with various row formats, encryption, with indexes
and auto-increment.

Signed-off-by: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Yuchen Pei
Author Date: 2023-03-29 23:25:02 UTC

separated out changes to handle_instant_metadata() from 9501981c

Author: Sergey Petrunia
Author Date: 2023-03-29 16:42:21 UTC

MDEV-30964: MAX_SEL_ARG memory exhaustion is not visible in the optimizer trace

Add printing

Author: midenok
Author Date: 2023-03-05 10:21:02 UTC

Review

Author: Yuchen Pei
Author Date: 2023-03-23 23:38:20 UTC

MDEV-30920 Remove need_lock and table from spider_close_sys_table()

They became obsolete after commit cfd145faedd:

commit cfd145faedd68e27d4ac8f1e5f95ec1a6dcd8c8d
Author: Nayuta Yanagisawa <nayuta.yanagisawa@hey.com>
Date: Fri Jan 28 01:03:06 2022 +0900

    MDEV-27641 Spider: remove #if MYSQL_VERSION_ID < ${VERSION}

Author: Marko Mäkelä
Author Date: 2023-03-27 16:38:38 UTC

WIP fix MDEV-30936 clang 15.0.7 -fsanitize=memory fails massively

FIXME: innodb.full_crc32_import fails due to CRC-32C computation in
buf_page_is_corrupted().

Author: Vlad Lesin
Author Date: 2023-03-23 13:26:17 UTC

MDEV-29050 mariabackup issues error messages during InnoDB tablespaces export on partial backup preparing

The solution is to suppress error messages for missing tablespaces if
mariabackup is launched with "--prepare --export" options.

"mariabackup --prepare --export" invokes itself with --mysqld parameter.
If the parameter is set, then it starts server to feed "FLUSH TABLES ...
FOR EXPORT;" queries for exported tablespaces. This is "normal" server
start, that's why new srv_operation value is introduced.

Reviewed by Marko Makela.

Author: Vlad Lesin
Author Date: 2023-03-23 13:26:17 UTC

MDEV-29050 mariabackup issues error messages during InnoDB tablespaces export on partial backup preparing

The solution is to suppress error messages for missing tablespaces if
mariabackup is launched with "--prepare --export" options.

"mariabackup --prepare --export" invokes itself with --mysqld parameter.
If the parameter is set, then it starts server to feed "FLUSH TABLES ...
FOR EXPORT;" queries for exported tablespaces. This is "normal" server
start, that's why new srv_operation value is introduced.

Reviewed by Marko Makela.

Author: midenok
Author Date: 2023-03-05 10:21:02 UTC

Review

Author: midenok
Author Date: 2023-01-15 21:01:35 UTC

MDEV-30421 Allow administrators to enable or disable parallel replication on a per-table basis

Per-domain dedicated thread for processing ordered transactions. The
thread is reserved from the total number of domain threads (controlled
by slave_parallel_threads and slave_domain_parallel_threads). Whether
the event goes to ordered thread depends on FL_ALLOW_PARALLEL flag as
well as several other conditions. FL_ALLOW_PARALLEL is passed from
master and is set for the event depending on master configuration
directives. To allow dedicated slave on server one must enable it
explicitly with configuration directive:

  set global slave_ordered_thread= 1;

Originally it was controlled by skip_parallel_replication session
variable which can be changed per-statement. This patch adds several
more directives to control it on per-schema and per-table levels:

  parallel_do_db
  parallel_do_table
  parallel_ignore_db
  parallel_ignore_table
  parallel_wild_do_table
  parallel_wild_ignore_table

Each directive is comma-separated list of fully-qualified table
names. Spaces after comma are ignored (but not before).

"Table" directives take precedence over "db" directives. "Do"
directives take precedence over "ignore" directives. "Wild" directives
are checked if "do" and "ignore" directives did not match.

If none of the above directives present everything is considered
parallel. If any of the above directives present and the table did not
match anything in the lists it is considered ordered.

Examples:

  set @@global.parallel_do_db= "db_parallel";
  set @@global.parallel_ignore_db= "db_serial";
  set global parallel_do_table= "db_serial.t3, db_serial.t1";
  set global parallel_wild_ignore_table= "db_parallel.non_parallel_%"

Normal behaviour of ordered transaction is before start to wait any of
prior transactions to commit: they get into different commit
groups. But since all the ordered transactions (within one domain) go
to a single thread we may avoid that restriction with this directive
on slave:

  set global slave_ordered_dont_wait= 1;

When set events without explicit FL_WAITED flag going to ordered
thread nonetheless accept optimistic speculation. I.e. they get into
same commit group with parallel events: ordered event is executed in
parallel with parallel events.

Author: Yuchen Pei
Author Date: 2023-03-23 06:32:09 UTC

MDEV-30265 Document some sts related functions in spider.

Author: Yuchen Pei
Author Date: 2023-03-23 23:30:22 UTC

MDBF-535 Breaking some spider tests to check which ci's cover spider

Author: Andrei
Author Date: 2023-03-23 16:45:34 UTC

MDEV-30620 Trying to lock uninitialized LOCK_parallel_entry

The error was seen by a number of mtr tests being caused
by overdue initialization of rpl_parallel::LOCK_parallel_entry.
Specifically, SHOW-SLAVE-STATUS might find in
rpl_parallel::workers_idle() a gtid domain hash entry
already inserted whose mutex had not done
mysql_mutex_init().

Fixed with swapping the mutex init and the its entry's stack insertion.

Tested with a generous number of `mtr --repeat` of a few of the reported
to fail tests, incl rpl.parallel_backup.

Author: Nikita Malyavin
Author Date: 2023-03-23 13:29:21 UTC

wake up from sleep

Author: Yuchen Pei
Author Date: 2023-03-23 06:32:09 UTC

MDEV-30265 Document some sts related functions in spider.

Author: Igor Babaev
Author Date: 2023-03-23 04:27:51 UTC

MDEV-29971 Re-design the upper level of handling SELECT statements

The initial patch.
All tests from the main test suite passed.

Author: Nayuta Yanagisawa
Author Date: 2022-05-26 23:55:42 UTC

MDEV-28522 Delete constant SPIDER_SQL_TYPE_*_HS

The HandlerSocket support of Spider has been deleted by MDEV-26858.
Thus, the constants, SPIDER_SQL_TYPE_*_HS, are no longer necessary.

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: midenok
Author Date: 2023-03-22 09:53:58 UTC

MDEV-30421 more tests cleaned up

All the .inc files that included from binlog_encryption are refactored.

Author: Yuchen Pei
Author Date: 2023-01-18 00:51:28 UTC

MDEV-27912 Fixing inconsistency w.r.t. expect files in tests.

mtr uses group suffix, but some existing inc and test files use
server_id for expect files. This patch aims to fix that.

For spider:

With this change we will not have to maintain a separate version of
restart_mysqld.inc for spider, that duplicates code, just because
spider tests use different names for expect files, and shutdown_mysqld
requires magical names for them.

With this change spider tests will also be able to use other features
provided by restart_mysqld.inc without code duplication, like the
parameter $restart_parameters (see e.g. the testcase mdev_29904.test
in commit ef1161e5d4f).

Tests run after this change: default, spider, rocksdb, galera, using
the following command

mtr --parallel=auto --force --max-test-fail=0 --skip-core-file
mtr --suite spider,spider/*,spider/*/* \
    --skip-test="spider/oracle.*|.*/t\..*" --parallel=auto --big-test \
    --force --max-test-fail=0 --skip-core-file
mtr --suite galera --parallel=auto
mtr --suite rocksdb --parallel=auto

Author: Yuchen Pei
Author Date: 2023-03-21 23:33:45 UTC

Addressing review comments

Author: Yuchen Pei
Author Date: 2023-02-08 01:29:52 UTC

MDEV-28152 Features for sequences

- Add `as <int_type>` to sequence creation options
  - int_type can be signed or unsigned integer types, including
    tinyint, smallint, mediumint, int and bigint
  - Limitation: when alter sequence as <new_int_type>, cannot have any
    other alter options in the same statement
  - Limitation: increment remains signed longlong, and the hidden
    constraint (cache_size x abs(increment) < longlong_max) stays for
    unsigned types. This means for bigint unsigned, neither
    abs(increment) nor (cache_size x abs(increment)) can be between
    longlong_max and ulonglong_max
- Truncating maxvalue and minvalue from user input to the nearest max
  or min value of the type, plus or minus 1. When the truncation
  happens, a warning is emitted
- Information schema table for sequences

Author: Yuchen Pei
Author Date: 2023-03-20 01:41:30 UTC

MDEV-28152 Addressing round 1 review comments

... at https://lists.launchpad.net/maria-developers/msg13297.html

Also implemented setval and auto increment for unsigned sequences.

Author: Yuchen Pei
Author Date: 2023-02-08 01:29:52 UTC

MDEV-28152 Features for sequences

- Add `as <int_type>` to sequence creation options
  - int_type can be signed or unsigned integer types, including
    tinyint, smallint, mediumint, int and bigint
  - Limitation: when alter sequence as <new_int_type>, cannot have any
    other alter options in the same statement
  - Limitation: increment remains signed longlong, and the hidden
    constraint (cache_size x abs(increment) < longlong_max) stays for
    unsigned types. This means for bigint unsigned, neither
    abs(increment) nor (cache_size x abs(increment)) can be between
    longlong_max and ulonglong_max
- Truncating maxvalue and minvalue from user input to the nearest max
  or min value of the type, plus or minus 1. When the truncation
  happens, a warning is emitted
- Information schema table for sequences

Author: Oleg Smirnov
Author Date: 2023-02-17 12:21:30 UTC

MDEV-8320 Allow index usage for DATE(col) <=> const and YEAR <=> const

Rewrite datetime comparison conditions into sargeable. For example,
    YEAR(col) <= val -> col <= YEAR_END(val)
    YEAR(col) < val -> col < YEAR_START(val)
    YEAR(col) >= val -> col >= YEAR_START(val)
    YEAR(col) > val -> col > YEAR_END(val)
    YEAR(col) = val -> col BETWEEN YEAR_START(val) AND YEAR_END(val)
Do the same with DATE(col), for example:
    DATE(col) <= val -> col <= DAY_END(val)

After such a rewrite index lookup on column "col" can be employed

Author: Marko Mäkelä
Author Date: 2023-03-17 14:26:16 UTC

Merge 11.1

Author: Yuchen Pei
Author Date: 2023-03-15 03:50:22 UTC

wip please ignore

Author: Yuchen Pei
Author Date: 2023-03-15 03:50:22 UTC

wip please ignore

Author: Julius Goryavsky
Author Date: 2023-03-15 17:41:51 UTC

MDEV-30855: disable unnecessary test

Author: Lena Startseva
Author Date: 2022-09-19 05:26:59 UTC

MDEV-29390: Improve coverage for UPDATE and DELETE statements in MTR test suites

Created tests for "delete" based on update_use_source.test

For the update_use_source.test tests, data recovery in the table has been changed
from a rollback transaction to a complete delete and re-insert of the data with
optimize table. Cases are now being checked on three engines.

Added tests for update/delete with LooseScan and DuplicateWeedout optimization strategies
Added tests for engine MEMORY on delete and update
Added tests for multi-update with JSON_TABLE
Added tests for multi-update and multi-delete for engine Connect

Author: Rex Johnston
Author Date: 2023-03-06 21:29:56 UTC

MDEV-30605 Wrong result while using index for group-by

 incorrect result when group function applied on a primary key

Author: Igor Babaev
Author Date: 2023-01-31 03:42:27 UTC

MDEV-28958 Crash when checking whether condition can be pushed into view

Do not set any flags in the items for constant subformulas TRUE/FALSE when
checking pushability of a formula into a view. Occurrences of these
subformulas can be ignored when checking pushability of the formula.
At the same time the items used for these constants became immutable
starting from version 10.7.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Author: Julius Goryavsky
Author Date: 2023-03-13 16:41:06 UTC

Post-MDEV-30700: moving alloca() definitions from all *.h files to new header file

Included config file for proper compilation without <my_global.h>

Author: Monty
Author Date: 2023-03-13 16:38:02 UTC

fixup! 2cc16653817dd80df1c57f7d8499352fc1eeaa5b

Author: Dmitry Shulga
Author Date: 2023-03-13 08:01:53 UTC

MDEV-5816: Stored programs: validation of stored program statements

Follow-up patch that fixes issue with missing set up of versioning
condition for query involving a versioning partitioned table.

Author: Igor Babaev
Author Date: 2023-03-10 05:48:58 UTC

Adjusted test results after rebase against 11.0.1

Author: Monty
Author Date: 2023-03-09 09:07:10 UTC

Test commit to see if reverting 'unscrambling' of table order changes things

This is an experimental commit, only to be used for testing

Author: Rex Johnston
Author Date: 2023-03-09 01:41:07 UTC

MDEV-30815 Printing a stack trace leaks file descriptors.

 In situations where mariadbd emits a lot of stack traces it can
        run out of file descriptors. Each time the pipe to the external
        utility addr2line is opened, those file descriptor are never closed.

Author: Yuchen Pei
Author Date: 2023-03-01 06:34:05 UTC

MDEV-28152 wip wip wip alter sequence as <new_value_type>

Basic case:
- start with a narrow value type, the alter to a wider type without
  changing any other metadata

Advanced cases to fix once the basic case works:
- start with a wider value type, then alter to a narrower type with
  smaller metadata values in range for the narrower value type
- start with a narrower value type, then alter to a wider type with
  bigger metadata values out of range for the narrower type but in
  range for the wider type

Author: Rex Johnston
Author Date: 2023-03-06 00:58:27 UTC

MDEV-30771 Optimizer trace: table_scan.rows is traced as integer, change to double

 Alter formatting of printed doubles written to the optimizer trace
 to ensure it is clear they ARE floating point numbers.

Author: Rex Johnston
Author Date: 2023-02-01 03:59:39 UTC

MDEV-23384 Server crashes in subselect_indexsubquery_engine::print with optimizer trace enabled

 unitialized data structures
 causing subselect_indexsubquery_engine::print
 and subselect_uniquesubquery_engine::print to crash
 when optimizer_trace is enabled

Author: Oleksandr "Sanja" Byelkin
Author Date: 2022-11-25 12:37:07 UTC

MDEV-26831 fallout: name resolution cache simple test

Author: Sergey Petrunia
Author Date: 2023-02-22 09:11:55 UTC

MDEV-30693: Assertion `dbl_records <= s->records' failed in apply_selectivity_for_table on SELECT

The crash happened due to rows=2 vs rows=1 difference between how the
estimate of number of rows in a derived table is computed in
TABLE_LIST::fetch_number_of_rows() and JOIN::add_keyuses_for_splitting().

Made JOIN::add_keyuses_for_splitting() use the result of computations in
TABLE_LIST::fetch_number_of_rows().

Author: Yuchen Pei
Author Date: 2023-02-22 02:42:38 UTC

wip wip wip Adding unsigned by making fields Longlong_hybrid

Author: Michael Widenius
Author Date: 2023-02-19 18:05:13 UTC

Added mysql-log-rotate to .gitignore

Author: Michael Widenius
Author Date: 2023-02-20 10:28:44 UTC

squash! 4cb69791aaa59e786859405c3c3826a6a15c4a09

- Ensure that TEMP_TABLE_PARAM.func_count includes all items that may
  need a copy function.
- Fixed that Aria allocates enough space for key copies.
- Fixed that Aria does not check empty_bits if not allocated.

The first issue could cause crashes, the other issues should not
affect anything.

Author: Yuchen Pei
Author Date: 2023-02-21 01:21:28 UTC

updating some comments

Author: Teemu Ollakka
Author Date: 2023-02-13 16:14:50 UTC

MDEV-27317 wsrep_checkpoint order violation due to certification failure

With binlogs enabled, debug assertion ut_ad(xid_seqno > wsrep_seqno)
fired in trx_rseg_update_wsrep_checkpoint() when an applier thread
synced the seqno out of order for write set which had failed
certification. This was caused by releasing commit
order too early when binlogs were on, allowing group
commit to run in parallel and commit following transactions
too early.

Fixed by extending the commit order critical section to cover
call to wsrep_set_SE_checkpoint() also when binlogs are on.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Author: Nikita Malyavin
Author Date: 2023-02-20 11:05:46 UTC

fix Run-Time Check Failure on Windows

Author: Yuchen Pei
Author Date: 2023-02-20 02:03:53 UTC

add check for discarded in h_i::open()

Author: Nikita Malyavin
Author Date: 2023-02-17 16:58:41 UTC

fix after rebasing onto MDEV-30378 fix

Author: Lena Startseva
Author Date: 2022-09-19 05:26:59 UTC

MDEV-29390: Improve coverage for UPDATE and DELETE statements in MTR test suites

Created tests for "delete" based on update_use_source.test

For the update_use_source.test tests, data recovery in the table has been changed
from a rollback transaction to a complete delete and re-insert of the data with
optimize table. Cases are now being checked on three engines.

Added tests for update/delete with LooseScan and DuplicateWeedout optimization strategies
Added tests for engine MEMORY on delete and update
Added tests for multi-update with JSON_TABLE
Added tests for multi-update and multi-delete for engine Connect

Author: Yuchen Pei
Author Date: 2023-02-15 06:25:01 UTC

WIP try creating a table using cfg and cloning exiting table

Author: Rex Johnston
Author Date: 2023-02-16 20:28:38 UTC

MDEV-29179 Condition pushdown from HAVING into WHERE is not shown in optimizer trace

 JOIN::optimize_inner(), Condition pushdown from HAVING into WHERE
   not shown in optimizer trace.

Author: Nikita Malyavin
Author Date: 2022-10-21 12:08:26 UTC

MDEV-29068 Cascade foreign key updates do not apply in online alter

Author: Alexander Barkov
Author Date: 2023-02-14 09:27:46 UTC

MDEV-15751 CURRENT_TIMESTAMP should return a TIMESTAMP [WITH TIME ZONE?]

Author: Alexander Barkov
Author Date: 2023-02-14 07:52:10 UTC

MDEV-15750 preserve MYSQL_TYPE_TIMESTAMP in temporal arithmetic

    In progress.

Author: Rex Johnston
Author Date: 2023-02-01 18:29:05 UTC

MDEV-30334 Optimizer trace produces invalid JSON with WHERE subquery

 Simple code rearrangement to stop it displaying an unsigned int in a String.

Author: Sergey Petrunia
Author Date: 2023-02-13 13:54:13 UTC

MDEV-30596: Assertion 'pushed_rowid_filter != __null ...' failed

ha_partition doesn't forward Rowid Filter API calls to the storage
engines handling partitions.
An attempt to use rowid filtering with caused either
- Rowid Filter being shown in EXPLAIN but not actually used
- Assertion failure when subquery code tried to disable/enable rowid
  filter, which was not present.

Fixed by returning correct flags from ha_partition::index_flags()

Author: Sergey Petrunia
Author Date: 2023-02-13 13:54:13 UTC

MDEV-30596: Assertion 'pushed_rowid_filter != __null ...' failed

ha_partition doesn't forward Rowid Filter API calls to the storage
engines handling partitions.
An attempt to use rowid filtering with caused either
- Rowid Filter being shown in EXPLAIN but not actually used
- Assertion failure when subquery code tried to disable/enable rowid
  filter, which was not present.

Fixed by returning correct flags from ha_partition::index_flags()

Author: Alexander Barkov
Author Date: 2023-02-13 11:14:40 UTC

MDEV-30633 DATETIME to TIMESTAMP conversion to return maximum timestamp on overflow

Author: Vicențiu Ciorbaru
Author Date: 2023-02-10 09:59:36 UTC

Merge remote-tracking branch '10.4' into 10.5

Author: Daniel Black
Author Date: 2023-02-10 07:40:43 UTC

MDEV-30571 MariaDB-client does not pull MariaDB-compat anymore, conflicts with RHEL packages

Make MariaDB-client explictly pull MariaDB-common. I can't see
a reason for the change between 10.9.

Use SETA rather than set for common/compat dependencies out of caution.

Last commit seems to have resolved the common dependency on compat
(though its not obvious why).

Author: Igor Babaev
Author Date: 2023-02-10 06:42:29 UTC

Adjusted test results after rebase

Author: Rex Johnston
Author Date: 2023-02-09 22:38:28 UTC

MDEV-30629 Add optimizer trace to main/order_by_innodb.test to diagnose debian build issue

 Debian S390 build issue. Add optimizer trace to failing test
 (main.order_by_innodb) to help diagnose issue.

Author: Brandon Nesterenko
Author Date: 2023-02-08 17:32:35 UTC

MDEV-30608: rpl.rpl_delayed_parallel_slave_sbm sometimes fails with Seconds_Behind_Master should not have used second transaction timestamp

One of the constraints added in the MDEV-29639 patch, is that only
the first event after idling should update last_master_timestamp;
and as long as the replica has more events to execute, the variable
should not be updated. The corresponding test,
rpl_delayed_parallel_slave_sbm.test, aims to verify this; however,
if the IO thread takes too long to queue events, the SQL thread can
appear to catch up too fast.

This fix ensures that the relay log has been fully written before
executing the events.

Note that the underlying cause of this test failure needs to be
addressed as a bug-fix, this is a temporary fix to stop test
failures. To track work on the bug-fix for the underlying issue,
please see MDEV-30619.

Author: Fabrice Fontaine
Author Date: 2023-01-15 18:12:05 UTC

include/ssl_compat.h: fix build with libressl >= 3.5.0

Fix the following build failure with libressl >= 3.5.0:

In file included from /tmp/instance-10/output-1/build/mariadb-10.3.36/vio/viosslfactories.c:18:
/tmp/instance-10/output-1/build/mariadb-10.3.36/vio/viosslfactories.c: In function 'get_dh2048':
/tmp/instance-10/output-1/build/mariadb-10.3.36/include/ssl_compat.h:68:45: error: invalid use of incomplete typedef 'DH' {aka 'struct dh_st'}
   68 | #define DH_set0_pqg(D,P,Q,G) ((D)->p= (P), (D)->g= (G))
      | ^~

Fixes:
 - http://autobuild.buildroot.org/results/524198344aafca58d214537af64c5961c407b0f8

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>

Author: Daniel Black
Author Date: 2023-02-06 07:43:36 UTC

MDEV-30574 MariaDB-server conflicts with MariaDB-common from previous versions

Tie these two version to 11.0+.

Author: Marko Mäkelä
Author Date: 2023-02-06 05:23:43 UTC

MDEV-30567 rec_get_offsets() is not optimal

rec_init_offsets_comp_ordinary(), rec_init_offsets(),
rec_get_offsets_reverse(), rec_get_nth_field_offs_old():
Simplify some bitwise arithmetics to avoid conditional jumps,
and add branch prediction hints with the assumption that most
variable-length columns are short.

Author: Sergey Petrunia
Author Date: 2023-02-03 13:23:38 UTC

MDEV-30529: Assertion `rnd_records <= s->found_records' failed in best_access_path

best_access_path() has an assertion:

   DBUG_ASSERT(rnd_records <= s->found_records);

make it rounding-safe.

Author: Sergey Petrunia
Author Date: 2023-02-03 12:20:22 UTC

MDEV-30525: Assertion `ranges > 0' fails in IO_AND_CPU_COST handler::keyread_time

Make get_best_group_min_max() exit early if the table has
table->records()=0. Attempting to compute loose scan over 0
groups eventually causes an assert when trying to get the
cost of reading 0 ranges.

Author: Julius Goryavsky
Author Date: 2023-02-03 02:00:14 UTC

wsrep-lib submodule update

Author: Nayuta Yanagisawa
Author Date: 2022-09-27 06:22:57 UTC

MDEV-29644 a potential bug of null pointer dereference in spider_db_mbase::print_warnings()

The function spider_db_mbase::print_warnings() can potentially result
in a null pointer dereference.

Remove the null pointer dereference by cleaning up the function.

Some small changes to the original commit
422fb63a9bbee35c50b6c7be19d199afe0bc98fa.

Co-Authored-By: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Nayuta Yanagisawa
Author Date: 2022-09-27 06:22:57 UTC

MDEV-29644 a potential bug of null pointer dereference in spider_db_mbase::print_warnings()

The function spider_db_mbase::print_warnings() can potentially result
in a null pointer dereference.

Remove the null pointer dereference by cleaning up the function.

Some small changes to the original commit
422fb63a9bbee35c50b6c7be19d199afe0bc98fa.

Co-Authored-By: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Nayuta Yanagisawa
Author Date: 2022-09-27 06:22:57 UTC

MDEV-29644 a potential bug of null pointer dereference in spider_db_mbase::print_warnings()

The function spider_db_mbase::print_warnings() can potentially result
in a null pointer dereference.

Remove the null pointer dereference by cleaning up the function.

Some small changes to the original commit
422fb63a9bbee35c50b6c7be19d199afe0bc98fa.

Co-Authored-By: Yuchen Pei <yuchen.pei@mariadb.com>

Author: Sergey Petrunia
Author Date: 2023-02-02 16:19:36 UTC

Merge 'bb-10.5-release-mdev30218' into bb-10.6-release-mdev30218

Author: Sergey Petrunia
Author Date: 2023-02-02 16:17:51 UTC

Merge 'bb-10.4-release-mdev30218' into bb-10.5-release-mdev30218

Author: Sergey Petrunia
Author Date: 2023-02-02 15:12:39 UTC

MDEV-30218: Incorrect optimization for rowid_filtering, correction

Final corrections:
- Remove incorrect tracing, "rowid_filter_skipped"
- Put the worst_seeks sanity check back