Author: Jan Lindström
Author Date: 2020-01-30 10:20:54 UTC

Fix compile error on Windows.

Author: Marko Mäkelä
Author Date: 2020-01-29 10:42:45 UTC

Cleanup: More logical names and member functions

recv_sys_t::free_corrupted_page(): Renamed from recv_recover_corrupt_page().

recv_sys_t::is_memory_exhausted(): Renamed from recv_sys_heap_check().

recv_sys_t::max_blocks(): Accessor for max_log_blocks.

recv_sys_t::blocks: Buffer pool blocks reserved for redo log records.

For the debug functions related to recv_sys_t::alloc(), use the
logical name free().

Rename some variables 'store_to_hash' to 'store', because recv_sys.pages
is not a hash table.

Author: Vicentiu Ciorbaru
Author Date: 2020-01-28 15:59:47 UTC

Move mysqltest.1 man page to test package

Author: mkaruza
Author Date: 2019-04-01 11:23:05 UTC

Galera GTID support

Support for galera GTID consistency thru cluster. All nodes in cluster
should have same GTID for replicated events which are originating from cluster.
Cluster originating commands need to contain sequential WSREP GTID seqno
Ignore manual setting of gtid_seq_no=X.

In master-slave scenario where master is non galera node replicated GTID is
replicated and is preserved in all nodes.

To have this - domain_id, server_id and seqnos should be same on all nodes.
Node which bootstraps the cluster, to achieve this, sends domain_id and
server_id to other nodes and this combination is used to write GTID for events
that are replicated inside cluster.

Cluster nodes that are executing non replicated events are going to have different
GTID than replicated ones, difference will be visible in domain part of gtid.

With wsrep_gtid_domain_id you can set domain_id for WSREP cluster.

Functions WSREP_LAST_WRITTEN_GTID, WSREP_LAST_SEEN_GTID and
WSREP_SYNC_WAIT_UPTO_GTID now works with "native" GTID format.

Fixed galera tests to reflect this chances.

Add variable to manually update WSREP GTID seqno in cluster

Add variable to manipulate and change WSREP GTID seqno. Next command
originating from cluster and on same thread will have set seqno and
cluster should change their internal counter to it's value.
Behavior is same as using @@gtid_seq_no for non WSREP transaction.

Author: Nikita Malyavin
Author Date: 2020-01-13 07:53:40 UTC

add frm corruption check for extra2.application_period

Author: Vlad Lesin
Author Date: 2020-01-22 18:13:14 UTC

MDEV-14183: aria_pack segfaults in compress_maria_file

Post-push fix. aria_pack_mdev14183 test is unstable.

The fix is the following:
1. Disable the test for embedded server.
2. Create non-"transactional" Aria table in the test, as aria_pack does not
support "transactional" Aria tables.

Author: Jan Lindström
Author Date: 2020-01-22 09:58:50 UTC

MDEV-17571 : Make systemd timeout behavior more compatible with long Galera SSTs

This is 10.4 version.

Idea is to create monitor thread for both donor and joiner that will
periodically if needed extend systemd timeout while SST is being
processed. In 10.4 actual SST is executed by running SST script
and exchanging messages on pipe using blocking fgets. This fix
starts monitoring thread before SST script is started and
we stop monitoring thread when SST has been completed.

Author: Nikita Malyavin
Author Date: 2020-01-21 09:48:47 UTC

Period: add versioning combination

Author: Jan Lindström
Author Date: 2020-01-16 11:18:44 UTC

MDEV-17062 : Test failure on galera.MW-336

Add mutex protection while we calculate required slave
thread change and create them. Add error handling.

Author: Jan Lindström
Author Date: 2020-01-15 07:06:32 UTC

MDEV-18464 : Killing thread can cause mutex deadlock if done concurrently with Galera/replication victim kill

Following issues here:

Whenever Galera BF (brute force) transaction decides to abort conflicting transaction it will kill that thread using thd::awake()
Whenever replication selects a thread as a victim it will call thd::awake()
User KILL [QUERY|CONNECTION] ... for a thread it will also call thd::awake()
Whenever one of these actions is executed we will hold number of InnoDB internal mutexes and thd mutexes.
Sometimes these mutexes are taken in different order causing mutex deadlock.

In this patch we will fix Galera BF and user kill cases so that we enqueue
victim thread to a list while we hold InnoDB mutexes and we then release them.
A new background thread will pick victim thread from this new list and uses
thd::awake() with no InnoDB mutexes. Idea is similar to replication background
kill. This fix enforces that we take LOCK_thd_data -> lock sys mutex -> trx mutex
always in this order.

wsrep_mysqld.cc
 Here we introduce a list where victim threads are stored,
 condition variable to be used to wake up background thread
 and mutex to protect list.

wsrep_thd.cc
 Create a new background thread to handle victim thread
 abort. We may take wsrep_thd_LOCK mutex here but not any
 InnoDB mutexes.

wsrep_innobase_kill_one_trx
 Remove all the wsrep code that was moved to wsrep_thd.cc
 We just enqueue required information to background kill
 list and cancel victim trx lock wait if there is such.
 Here we have InnoDB lock sys mutex and trx mutex so here
 we can't take wsrep_thd_LOCK mutex.

wsrep_abort_transaction
 Cleanup only.

Author: Jan Lindström
Author Date: 2020-01-14 06:05:05 UTC

Merge branch 'codership-10.4-MDEV-19803' into 10.4

Author: Jan Lindström
Author Date: 2020-01-13 13:34:08 UTC

Merge branch 'codership-10.4-wsrep-position-after-cert-failure' into 10.4

Author: Jan Lindström
Author Date: 2020-01-02 06:06:23 UTC

MDEV-13569: wsrep_info.plugin failed in buildbot with "no nodes coming from prim view

Modify configuration so that all nodes are part of galera cluster
i.e. wsrep_on=ON. Add missing wait conditions.

test changes only.

Author: Jan Lindström
Author Date: 2019-12-31 06:26:46 UTC

Merge branch 'codership-10.4-MDEV-20793' into 10.4

Author: Vicențiu Ciorbaru
Author Date: 2019-12-27 21:26:25 UTC

Cleanup and more comments

Author: Vladislav Vaintroub
Author Date: 2019-12-24 17:22:45 UTC

Support AF_UNIX socket on Windows

Author: Vladislav Vaintroub
Author Date: 2019-12-20 11:49:37 UTC

WIP

Author: Sergey Petrunia
Author Date: 2019-12-22 22:13:38 UTC

MDEV-21383: Possible range plan is not used under certain conditions

(10.3's variant of the fix)

make_join_select() has a section of code which starts with
 "We plan to scan all rows. Check again if we should use an index."

the code in that section will [unnecessarily] re-run the range
optimizer using this condition:

  condition_attached_to_current_table AND current_table's_ON_expr

Note that the original invocation of range optimizer in
make_join_statistics was done using the whole select's WHERE condition.

taking the whole select's WHERE condition and using multiple-equalities
allowed the range optimizer to do more inferences.

This fix uses a very conservative approach: we still re-run the range
optimizer, but we save the quick select we already have, and take the
best of the two quick selects.

Author: Jan Lindström
Author Date: 2019-12-18 06:22:07 UTC

MDEV-21335 : Galera test failure on suite wsrep

Problem was that wsrep_on was OFF.

This is 10.4 version.

Author: Jan Lindström
Author Date: 2019-12-17 10:10:36 UTC

MDEV-21335 : Galera test failure on suite wsrep

Problem was that wsrep_on was OFF.

Author: Kentoku SHIBA
Author Date: 2019-12-05 19:32:56 UTC

fix compiler warnings

Author: Vicențiu Ciorbaru
Author Date: 2019-12-04 21:37:04 UTC

cleanup

Author: Jan Lindström
Author Date: 2019-12-03 13:02:17 UTC

Merge branch 'codership-10.4-MDEV-20848' into 10.4

Author: Sachin Setiya
Author Date: 2019-11-28 18:09:22 UTC

more info

Author: Vlad Lesin
Author Date: 2019-11-27 22:16:22 UTC

Add some extra logging to catch the case when page is written with wrong offset.

Author: Vlad Lesin
Author Date: 2019-11-27 22:16:22 UTC

Add some extra logging to catch the case when page is written with wrong offset.

Author: Vlad Lesin
Author Date: 2019-11-27 22:16:22 UTC

Add some extra logging to catch the case when page is written with wrong offset.

Author: Rasmus Johansson
Author Date: 2019-11-17 08:39:47 UTC

MDEV-19781 Create MariaDB named commands on Windows

Added CreateSymlinks and DeleteSymlinks functions to
CustomAction.cpp. Extra.wxs.in calls them.

Author: Nikita Malyavin
Author Date: 2019-10-31 23:38:22 UTC

move gotos to funcs

Author: Nikita Malyavin
Author Date: 2019-08-16 12:12:05 UTC

check referenced table period name in FK

Author: Sergey Vojtovich
Author Date: 2018-05-23 14:52:55 UTC

Use std::isfinite in C++ code

This is addition to parent revision fixing build failures.

(cherry picked from commit 54999f4e75f42baca484ae436b382ca8817df1dd)

Author: Sergey Vojtovich
Author Date: 2019-11-14 13:45:19 UTC

Don't use plugin->data for storage engine plugins

Use plugin->plugin->info->hton instead.
plugin_data() replaced with plugin_hton().
plugin_hton() must never return NULL anymore and is only good to be called
against plugins in PLUGIN_IS_READY state.

Part of
MDEV-20044 - Replace dynamic storage engine initialisation with declarative
             approach

Author: Sergey Petrunia
Author Date: 2019-11-09 18:03:23 UTC

MDEV-20854: ANALYZE for statements: not clear where the time is spent

Count the "gap" time between table accesses and display it as
r_other_time_ms in the "table" element.

* The advantage of this approach is that it doesn't add any new
  my_timer_cycles() calls.
* The disadvantage is that the definition of what is done during
  "other time" is not that clear: it includes checking the WHERE
  (for this table), constructing index lookup tuple (for the next table)
  writing to GROUP BY temporary table (as we dont account for that time
  separately [yet], etc)

Author: Kentoku SHIBA
Author Date: 2019-11-08 01:27:07 UTC

MDEV-18973 CLIENT_FOUND_ROWS wrong in spider

Get count from last_used_con->info
Contributed by willhan at Tencent Games

Author: Vicențiu Ciorbaru
Author Date: 2019-10-22 06:21:39 UTC

Second Parallel Build

Author: Vicențiu Ciorbaru
Author Date: 2019-10-22 06:21:06 UTC

First parallel build

Author: Vicențiu Ciorbaru
Author Date: 2019-10-22 05:54:29 UTC

Force rebuild

Author: Vlad Lesin
Author Date: 2019-10-20 18:42:32 UTC

MENT-344: Store Backups in S3

Mtr test is required, but there must be ability to remove backup from S3
to test it with MTR.

Author: Nikita Malyavin
Author Date: 2019-07-30 12:39:55 UTC

MDEV-18546 ASAN heap-use-after-free in innobase_get_computed_value / row_purge

the bug was already fixed in MDEV-17005, so now only test is added

Author: Nikita Malyavin
Author Date: 2019-10-09 06:48:50 UTC

MDEV-19130 Assertion `next_insert_id >= auto_inc_interval_for_cur_row.minimum()' failed in handler::update_auto_increment after error 167

Author: Julius Goryavsky
Author Date: 2019-10-08 08:25:23 UTC

MDEV-20626: MariaDB crash if wsrep_sst_method enabled without wsrep_cluster_address

Author: Julius Goryavsky
Author Date: 2019-10-08 08:47:30 UTC

MDEV-20625: MariaDB asserting when enabling wsrep=on

Author: Marko Mäkelä
Author Date: 2019-10-04 08:48:45 UTC

MDEV-19344 InnoDB purge buffering may corrupt a page

MySQL 5.5 introduced the ability of InnoDB to buffer delete-mark
and delete operations (the insert buffer was generalized to the
change buffer). These operations were only buffered on DELETE,
UPDATE (of a key) and the purge of history of committed transactions.

We never buffered anything on ROLLBACK; it could have been beneficial
for rolling back a large recovered transaction.

The delete-mark buffering appears to work fine, but there are problems
with the purge buffering.

MySQL Bug #61104 InnoDB: Failing assertion: page_get_n_recs(page) > 1
reported a problem with the purge buffering: an index page could
become empty, which essentially means that the secondary index becomes
corrupted. At MariaDB, we got closer to the root cause, by making
a failure repeatable with innodb.innodb-change-buffer-recovery-innodb
when an additional debug assertion is present.

page_header_set_field(): Enable the assertion that PAGE_N_RECS
must never be set to 0. (This function will not be invoked when
initializing an empty page. If an index page is empty, it must
be the root page, and the table must be empty. No changes to
the root page are ever buffered.)

A combination of two asynchronous, inherently nondeterministic operations
(purge and change buffering) is difficult to cover in tests or to reason
about. The purge buffering required a complex mechanism in the buffer pool,
the buffer pool watch. If we no longer buffer purge operations, we
can remove the watch as well.

We fix this by ceasing to buffer delete (purge) operations,
that is, by treating innodb_change_buffering=all (the default)
in the same way as innodb_change_buffering=changes
and treating innodb_change_buffering=purges
in the same way as innodb_change_buffering=deletes.

MDEV-16260 will attempt to improve the performance of purge in a more
controlled fashion by scaling the effort according to the workload.

We will retain the code that merges buffered purge operations,
so that upgrades from older versions will be possible.

BTR_DELETE_OP, BTR_DELETE, BUF_BLOCK_POOL_WATCH,
BUF_GET_IF_IN_POOL_OR_WATCH, BUF_POOL_WATCH_SIZE,
ROW_NOT_DELETED_REF: Remove.

btr_cur_t::purge_node, buf_pool_t::watch: Remove.

ibuf_get_volume_buffered_hash(): Remove. It is no longer necessary
to estimate whether the page could become empty.

Author: Sachin Setiya
Author Date: 2019-09-26 09:35:55 UTC

MDEV-20574 Position of events reported by mysqlbinlog is wrong with encrypted binlogs, SHOW BINLOG EVENTS reports the correct one.

Analysis

Mysqlbinlog output for encrypted binary log
#Q> insert into tab1 values (3,'row 003')
#190912 17:36:35 server id 10221 end_log_pos 980 CRC32 0x53bcb3d3 Table_map: `test`.`tab1` mapped to number 19
# at 940
#190912 17:36:35 server id 10221 end_log_pos 1026 CRC32 0xf2ae5136 Write_rows: table id 19 flags: STMT_END_F

Here we can see Table_map_log_event ends at 980 but Next event starts at 940.
And the reason for that is we do not send START_ENCRYPTION_EVENT to the slave

Solution:-
Send Start_encryption_log_event as Ignorable_log_event to slave(mysqlbinlog),
So that mysqlbinlog can update its log_pos.
Since Slave can request multiple FORMAT_DESCRIPTION_EVENT while master does not
have so We only update slave master pos when master actually have the
FORMAT_DESCRIPTION_EVENT. Similar logic should be applied for START_ENCRYPTION_EVENT.

Also added the test case when new server reads the data from old server which
does not send START_ENCRYPTION_EVENT to slave.

Author: Sachin Setiya
Author Date: 2019-09-26 09:35:55 UTC

MDEV-20574 Position of events reported by mysqlbinlog is wrong with encrypted binlogs, SHOW BINLOG EVENTS reports the correct one.

Analysis

Mysqlbinlog output for encrypted binary log
#Q> insert into tab1 values (3,'row 003')
#190912 17:36:35 server id 10221 end_log_pos 980 CRC32 0x53bcb3d3 Table_map: `test`.`tab1` mapped to number 19
# at 940
#190912 17:36:35 server id 10221 end_log_pos 1026 CRC32 0xf2ae5136 Write_rows: table id 19 flags: STMT_END_F

Here we can see Table_map_log_event ends at 980 but Next event starts at 940.
And the reason for that is we do not send START_ENCRYPTION_EVENT to the slave

Solution:-
Send Start_encryption_log_event as Ignorable_log_event to slave(mysqlbinlog),
So that mysqlbinlog can update its log_pos.
Since Slave can request multiple FORMAT_DESCRIPTION_EVENT while master does not
have so We only update slave master pos when master actually have the
FORMAT_DESCRIPTION_EVENT. Similar logic should be applied for START_ENCRYPTION_EVENT.

Also added the test case when new server reads the data from old server which
does not send START_ENCRYPTION_EVENT to slave.

Author: Julius Goryavsky
Author Date: 2019-09-25 12:00:39 UTC

MDEV-20614: Syntax error, and option put in wrong place

A syntax error in the mysqld_multi.sh script has been fixed
here + a "--defaults-group-suffix" option has been moved to
the top of the mysqld options list.

Author: Nikita Malyavin
Author Date: 2019-07-30 12:39:55 UTC

MDEV-18546 ASAN heap-use-after-free in innobase_get_computed_value / row_purge

the bug was already fixed in MDEV-17005, so now only test is added

Author: Galina Shalygina
Author Date: 2019-09-25 17:57:09 UTC

MDEV-11107 Use table check constraints in optimizer

Project code on 10.5 branch

Author: Galina Shalygina
Author Date: 2019-09-25 16:18:29 UTC

Cleaned tests, added new tests.
Removed additional is_deterministic() checks.
Added check: subquery used in ON expression.

Author: Sergey Vojtovich
Author Date: 2019-09-24 02:06:45 UTC

lf_hash_iterate() no zero-bucket initialization

Return immediately if hash is empty.

Part of MDEV-20630 - lf_hash get performance regression since the bucket
                     size won't decrease

Author: Monty
Author Date: 2019-09-20 11:09:26 UTC

MDEV-17084 - Optimize append only files for NVDIMM

Minor fixes.

Author: Vlad Lesin
Author Date: 2019-09-18 16:50:28 UTC

Some extra logging is added

(cherry picked from commit 7ff6e33d7ff7104fefb4df4a33c6f3fa54c0093d)

Author: Sergey Petrunia
Author Date: 2019-09-16 11:42:24 UTC

MDEV-20371: Invalid reads at plan refinement stage: join->positions...

(Backported to 10.2-compatiblity, build 22257)

best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).

Author: Eugene
Author Date: 2019-08-28 10:52:51 UTC

MDEV-20194 Warnings inconsistently issued upon CHECK on table from older versions

This patch changes conditions for HA_ERR_TO_BIG_ROW and users will complain!
But I hope now we really have a correct row size check.

This patch will be hard to merge because of changed tests.

Ideally, all row size checks code should live now in innodb.max_record_size
or innodb.max_record_size_compressed.

Now check for too big row size performed for every index on CREATE TABLE,
and for added index on ALTER TABLE. This is how the topic issue was fixed.

Error message for COMPRESSED tables might contain an incorrect maximum
possible row size if overflowed index was not clustered. Fixing it will
require some more effort.

create_table_info_t::row_size_is_acceptable(): high level too big row size
check interface.

record_size_info_t dict_index_t::record_size_info(): this is a low level
interface to perform too big row checks. Also can be used to get maximum
potential row size.

get_max_record_size_leaf_page()
get_max_record_size_non_leaf_page(): these are uniform way to get maximum
allows row size for a given page size and index row format. One function
instead of 5 slightly different computations.

format_too_big_row_error_message(): one function for formatting too big
row error messages instead of several.

Author: Sachin Setiya
Author Date: 2019-09-13 10:37:26 UTC

Fix compile error in macOS caused by MDEV-20477

Author: Sergey Petrunia
Author Date: 2019-09-11 17:22:08 UTC

MDEV-20371: Invalid reads at plan refinement stage: join->positions...

(re-committing in 10.4)
best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).

Author: Nikita Malyavin
Author Date: 2018-06-22 13:26:43 UTC

cherry-pick MDEV-16490

Author: Sergey Petrunia
Author Date: 2019-09-10 20:51:42 UTC

MDEV-20371: Invalid reads at plan refinement stage: join->positions...

best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).

Author: Marko Mäkelä
Author Date: 2019-09-09 11:03:03 UTC

MDEV-20487 Remove the InnoDB adaptive hash index

Based on the performance testing that was conducted in MDEV-17492,
the InnoDB adaptive hash index could only help performance in specific,
almost-read-only workloads. It could slow down all kinds of workloads
(especially DROP TABLE, TRUNCATE TABLE, ALTER TABLE, or DROP INDEX
operations), and it can become corrupted, causing crashes
and possibly data corruption. Furthermore, the adaptive hash index
consumes space from the InnoDB buffer pool, which could hurt performance
when the working set would almost fit in the buffer pool.

Given all this, it is best to remove innodb_adaptive_hash_index.
For now, we will change the default value of the build option
WITH_INNODB_AHI to OFF and adjust tests.

The test rpl.rpl_failed_drop_tbl_binlog depends on DROP TABLE being
able to be interrupted while it is dropping pages from the
adaptive hash index. It will be skipped from now on by default,
but it will be tested for builds WITH_INNODB_AHI=ON.

Author: Jan Lindström
Author Date: 2019-09-04 06:21:07 UTC

MDEV-20485: Galera test failure on galera.galera_var_node_address

Test changes only.

Author: Sachin Setiya
Author Date: 2019-09-01 14:35:07 UTC

19708 compatibility

Author: Kentoku SHIBA
Author Date: 2019-05-28 07:40:59 UTC

MDEV-6268 SPIDER table with no COMMENT clause causes queries to wait forever

Add loop checks by using user variables

Author: Teemu Ollakka
Author Date: 2019-07-25 13:41:45 UTC

MDEV-19826 10.4 seems to crash with "pool-of-threads"

MariaDB 10.4 was crashing when thread-handling was set to
pool-of-threads and wsrep was enabled.

There were two apparent reasons for the crash:
- Connection handling in threadpool_common.cc was missing calls to
  control wsrep client state.
- Thread specific storage which contains thread variables (THR_KEY_mysys)
  was not handled appropriately by wsrep patch when pool-of-threads
  was configured.

This patch addresses the above issues in the following way:
- Wsrep client state open/close was moved in thd_prepare_connection() and
  end_connection() to have common handling for one-thread-per-connection
  and pool-of-threads.
- Thread local storage handling in wsrep patch was reworked by introducing
  set of wsrep_xxx_threadvars() calls which replace calls to
  THD store_globals()/reset_globals() and deal with thread handling
  specifics internally.

Wsrep-lib was updated to version which relaxes internal concurrency
related sanity checks.

Rollback code from wsrep_rollback_process() was extracted to separate calls
for better readability.

Post rollback thread was removed as it was completely unused.

Author: Sergey Petrunia
Author Date: 2019-08-29 13:07:15 UTC

MDEV-20109: Optimizer ignores distinct key created for materialized...

Adjust the testcase according to the review input

Author: Jan Lindström
Author Date: 2019-08-27 14:04:59 UTC

Merge branch 'codership-vote-10.5' into 10.5

Author: Oleksandr "Sanja" Byelkin
Author Date: 2019-08-27 13:54:45 UTC

cc

Author: Andrei Elkin
Author Date: 2019-08-26 11:05:33 UTC

MDEV-742 (intermediate commit)

typ == XID_EVENT || typ == XA_PREPARE_LOG_EVENT augmentation.

Author: Sergey Petrunia
Author Date: 2019-08-22 21:57:40 UTC

MDEV-20109: Optimizer ignores distinct key created for materialized...

Sj_materialization_picker::check_qep(): fix error in cost/fanout
calculations:
- for each join prefix, add #prefix_rows / TIME_FOR_COMPARE to the cost,
  like best_extension_by_limited_search does
- Remove the fanout produced by the subquery tables.

optimize_wo_join_buffering() (used by LooseScan and FirstMatch)
- also add #prefix_rows / TIME_FOR_COMPARE to the cost of each prefix.

Author: Sergey Petrunia
Author Date: 2019-08-09 21:53:28 UTC

Optimizer trace: print cost and #rows of the join prefix

The names rows_for_plan and cost_for_plan follow MySQL
Also added post-join-operation selectivity cost

Author: Sachin Setiya
Author Date: 2018-05-17 09:10:55 UTC

MDEV-16239 Many test in rpl suite fails

Fix rpl_skip_error test.
  We cant reset Slave_skipped_errors(even with FLUSH STATUS), So instead
of absolute slave_skipped_errors we look for delta of slave_skipped_errors
Fix rpl.rpl_binlog_errors and binlog_encryption.rpl_binlog_errors
  We create the $load_file and $load_file2 but we never remove them.
Fix rpl_000011.test
  Instead of real value use delta value , Since flush status wont flush
LONGLONG variable.
Fix rpl_row_find_row_debug
  Instead of searching whole log_error_ file we will use search_pattern_in_file
which runs pattern search only on latest test run , instead of full file.
Fix rpl_ip_mix rpl_ip_mix2
  We should call reset slave all because we also want to reset master_host
otherwise show slave status wont be empty and making repeat N a failure.
Fix rpl_rotate_logs
  First we have to remove master.info file (cleanup) and second we have to
call reset slave all because if we do not call reset slave all then we wont
read master.info file beacuse we already have master config in memory.
And this makes start slave to pass , which shoud fail becuase its permision
is 000
Fix circular_serverid0 test
  The reason is that ++dbug_rows_event_count == 2 in queue_event does
not take --repeat into account. So I have reseted the dbug_rows_event_count
in if body.

Author: Nikita Malyavin
Author Date: 2019-08-08 14:56:44 UTC

MDEV-20297 sql_list: add begin() and end() for List<T>

Author: Sergey Petrunia
Author Date: 2019-08-03 20:15:44 UTC

Post-merge fixes for rocksdb.group_min_max test

- Fix the LooseScan code to support storage engines that return
  HA_ERR_END_OF_FILE if the index scan goes out of provided range
  bounds
- Add a DBUG_EXECUTE_IF("force_group_by",...) to allow a test to
  force a LooseScan
- Adjust rocksdb.group_min_max test not to use features not present
  in MariaDB 10.2 (e.g. optimizer_trace. In MariaDB 10.4 it's present
  but it doesn't meet the assumptions that the test makes about it
- Adjust the test result file:
  = MariaDB doesn't support "Enhanced Loose Scan" that FB/MySQL has
  = MariaDB has different cost calculations.

Author: Julius Goryavsky
Author Date: 2019-07-30 15:18:31 UTC

MDEV-20185: Windows: Use of uninitialized value $bpath in string eq

The execution of mtr in the Windows environment fails due to
the fact that the new code from MDEV-18565 does not take into
account the need to add the ".exe" extension to the names of
executable files when searching for pre-requisites that are
needed to run SST scripts (especially when using mariabackup)
and when searching paths to some other Galera utilities.
This patch fixes this flaw.

Also adding paths to the PATH environment variable is now
done with the correct delimiter character.

Author: Julius Goryavsky
Author Date: 2019-07-26 00:55:28 UTC

Fix for MW-336 test

Author: Julius Goryavsky
Author Date: 2019-07-16 09:33:11 UTC

MDEV-18565: Galera mtr-suite fails if galera library is not installed

Currently, running mtr with an incorrect (for example, new or
obsolete) version of wsrep_provider (for example, with the 26
version of libgalera_smm.so) leads to the failure of tests in
several suites with vague error diagnostics.

As for the galera_3nodes suite, the mtr also does not effectively
check all the prerequisites after merge with MDEV-18426 fixes.
For example, tests that using mariabackup do not check for presence
of ss and socat/nc. This is due to improper handling of relative
paths in mtr scripts.

In addition, some tests in different suites can be run without
setting the environment variables such as MTR_GALERA_TFMT, XBSTREAM,
and so on.

To eliminate all these issues, this patch makes the following changes:

1. Added auxiliary wsrep_mtr_check utility (which located in the
mysql-test/lib/My/SafeProcess subdirectory), which compares the
versions of the wsrep API that used by the server and by the wsrep
provider library, and it does this comparison safely, without
accessing the API if the versions do not match.

2. All checks related to the presence of mariabackup and utilities
that necessary for its operation transferred from the local directories
of different mtr suites (from the suite.pm files) to the main suite.pm
file. This not only reduces the amount of code and eliminates duplication
of identical code fragments, but also avoids problems due to the inability
of mtr to consider relative paths to include files when checking skip
combinations.

3. Setting the values of auxiliary environment variables that
are necessary for Galera, SST scripts and mariabackup (to work
properly) is moved to the main mysql-test-run.pl script, so as
not to duplicate this code in different suites, and to avoid
partial corrections of the same errors for different suites
(while other suites remain uncorrected).

4. Fixed duplication of the have_file_key_management.inc and
have_filekeymanagement.inc files between different suites,
these checks are also transferred to the top level.

5. Added garbd presence check and garbd path variable.

https://jira.mariadb.org/browse/MDEV-18565

Author: Julius Goryavsky
Author Date: 2019-07-16 10:48:04 UTC

MDEV-18565: Galera mtr-suite fails if galera library is not installed

Currently, running mtr with an incorrect (for example, new or
obsolete) version of wsrep_provider (for example, with the 26
version of libgalera_smm.so) leads to the failure of tests in
several suites with vague error diagnostics.

As for the galera_3nodes suite, the mtr also does not effectively
check all the prerequisites after merge with MDEV-18426 fixes.
For example, tests that using mariabackup do not check for presence
of ss and socat/nc. This is due to improper handling of relative
paths in mtr scripts.

In addition, some tests in different suites can be run without
setting the environment variables such as MTR_GALERA_TFMT, XBSTREAM,
and so on.

To eliminate all these issues, this patch makes the following changes:

1. Added auxiliary wsrep_mtr_check utility (which located in the
mysql-test/lib/My/SafeProcess subdirectory), which compares the
versions of the wsrep API that used by the server and by the wsrep
provider library, and it does this comparison safely, without
accessing the API if the versions do not match.

2. All checks related to the presence of mariabackup and utilities
that necessary for its operation transferred from the local directories
of different mtr suites (from the suite.pm files) to the main suite.pm
file. This not only reduces the amount of code and eliminates duplication
of identical code fragments, but also avoids problems due to the inability
of mtr to consider relative paths to include files when checking skip
combinations.

3. Setting the values of auxiliary environment variables that
are necessary for Galera, SST scripts and mariabackup (to work
properly) is moved to the main mysql-test-run.pl script, so as
not to duplicate this code in different suites, and to avoid
partial corrections of the same errors for different suites
(while other suites remain uncorrected).

4. Fixed duplication of the have_file_key_management.inc and
have_filekeymanagement.inc files between different suites,
these checks are also transferred to the top level.

5. Added garbd presence check and garbd path variable.

https://jira.mariadb.org/browse/MDEV-18565

Author: Jan Lindström
Author Date: 2019-07-15 04:02:18 UTC

Try #2

Author: Julius Goryavsky
Author Date: 2019-07-09 14:54:08 UTC

Improved error messages and added another path to the version check utility

Author: Marko Mäkelä
Author Date: 2019-07-03 14:31:20 UTC

MDEV-18582: Extend SHOW STATUS LIKE 'Innodb_%'

Many InnoDB internal variables and counters were only exposed
in an unstructured fashion via SHOW ENGINE INNODB STATUS.

Expose more variables via SHOW STATUS. Many of these were
exported in XtraDB.

Also, introduce SHOW_SIZE_T and use the proper size for
exporting the InnoDB variables.

Remove some unnecessary indirection via export_vars, and
bind some variables directly.

dict_sys_t::rough_size(): Replaces dict_sys_get_size()
and includes the hash table sizes.

This is based on a contribution by Tony Liu from ServiceNow.

Author: Marko Mäkelä
Author Date: 2019-06-26 18:57:42 UTC

MDEV-19845: Adaptive spin loops

Starting with the Intel Skylake microarchitecture, the PAUSE
instruction latency is about 140 clock cycles instead of earlier 10.
On AMD processors, the latency could be 10 or 50 clock cycles,
depending on microarchitecture.

Because of this big range of latency, let us scale the loops around
the PAUSE instruction based on timing results at server startup.

my_cpu_relax_multiplier: New variable: How many times to invoke PAUSE
in a loop. Only defined for IA-32 and AMD64.

my_cpu_init(): Determine with RDTSC the time to run 16 PAUSE instructions
in two unrolled loops according, and based on the quicker of the two
runs, initialize my_cpu_relax_multiplier. This form of calibration was
suggested by Mikhail Sinyavin from Intel.

LF_BACKOFF(), ut_delay(): Use my_cpu_relax_multiplier when available.

ut_delay(): Define inline in my_cpu.h.

UT_COMPILER_BARRIER(): Remove. This does not seem to have any effect,
because in our ut_delay() implementation, no computations are being
performed inside the loop. The purpose of UT_COMPILER_BARRIER() was to
prohibit the compiler from reordering computations. It was not
emitting any code.

Author: Sachin Setiya
Author Date: 2019-06-19 05:05:39 UTC

MDEV-19049 Server crashes in check_duplicate_long_entry_key, ASAN stack-buffer-overflow in Field_blob::get_key_image

Long Unique keys should always be last unique key.

Author: Rasmus Johansson
Author Date: 2019-05-18 06:11:01 UTC

MDEV-17592 Create MariaDB named commands/symlinks

Author: Georg Richter
Author Date: 2019-06-14 04:39:27 UTC

Fix for embedded test

Author: Julius Goryavsky
Author Date: 2019-06-13 11:59:54 UTC

MDEV-18863: Galera SST scripts can't read [mysqldN] option groups

Some users and some scripts (for example, mysqld_multi.sh) use special
option groups with names like [mysqld1], [mysqld2], ..., [mysqldN].

But SST scripts can't currently fully support these option groups.
The only option group-related value it gets from the server is
--defaults-group-suffix from the server, if that option was set
for mysqld when the server was started.

However, the SST script does not get told by the server to read
these option groups, so this means that the SST script will fail to
read options like innodb-data-home-dir when it is in a option group
like [mysqld1]...[mysqldN].

Moreover, SST scripts ignore many parameters that can be passed
to them explicitly and cannot transfer them further, for example,
to the input of mariabackup utility. Ideally, we want to transfer
all the parameters of the original mysqld call to utilities such
as mariabackup, however the SST script does not receive these
parameters and therefore cannot transfer them to mariabackup.

To correct these shortcomings, we need to add a transfer to the
script of all the parameters of the original mysqld call, and in
the SST scripts themselves provide for the transfer of these
parameters to utilities such as mariabackup. To prevent these
parameters from mixing with the script's own parameters, they
should be transferred to SST script after the special option
"--mysqld-args", followed by the line of the original parameters,
as received by mysqld call at the time of launch (further all
these parameters will be passed to mariabackup, for example).

In addition, the SST scripts themselves must be refined so that
they can read the parameters from the user-selected group, not just
from the global mysqld configuration group. And also so that they
can receive the parameters (which important for their work) as
command-line arguments.

Author: Georg Richter
Author Date: 2019-06-11 10:44:16 UTC

MDEV-14101 Provide an option to select TLS protocol version

Server and command line tools now support option --tls_version to specify the
TLS version between client and server. Valid values are TLSv1.0, TLSv1.1, TLSv1.2, TLSv1.3
or a combination of them. E.g.

--tls_version=TLSv1.3
--tls_version=TLSv1.2,TLSv1.3

In case there is a gap between versions, the lowest version will be used:
--tls_version=TLSv1.1,TLSv1.3 -> Only TLSv1.1 will be available.

If the used TLS library doesn't support the specified TLS version, it will use
the default configuration.

Limitations:

SSLv3 is not supported. The default configuration doesn't support TLSv1.0 anymore.
TLSv1.3 protocol currently is only supported by OpenSSL 1.1.0 (client and server) and
GnuTLS 3.6.5 (client only).

Overview of TLS implementations and protocols

Server:

+-----------+-----------------------------------------+
| Library | Supported TLS versions |
+-----------+-----------------------------------------+
| WolfSSL | TLSv1.1, TLSv1,2 |
+-----------+-----------------------------------------+
| OpenSSL | (TLSv1.0), TLSv1.1, TLSv1,2, TLSv1.3 |
+-----------+-----------------------------------------+
| LibreSSL | (TLSv1.0), TLSv1.1, TLSv1,2, TLSv1.3 |
+-----------+-----------------------------------------+

Client (MariaDB Connector/C)
+-----------+-----------------------------------------+
| Library | Supported TLS versions |
+-----------+-----------------------------------------+
| GnuTLS | (TLSv1.0), TLSv1.1, TLSv1.2, TLSv1.3 |
+-----------+-----------------------------------------+
| Schannel | (TLSv1.0), TLSv1.1, TLSv1.2 |
+-----------+-----------------------------------------+
| OpenSSL | (TLSv1.0), TLSv1.1, TLSv1,2, TLSv1.3 |
+-----------+-----------------------------------------+
| LibreSSL | (TLSv1.0), TLSv1.1, TLSv1,2, TLSv1.3 |
+-----------+-----------------------------------------+

Author: Sergei Golubchik
Author Date: 2019-06-11 15:51:09 UTC

MDEV-16249 CHECKSUM TABLE for a spider table is not parallel and saves all data in memory in the spider head by default (#1328)

followup for be5c432a42e

ha_partition::calculate_checksum() has to invoke calculate_checksum()
for partitions unconditionally, not under (HA_HAS_OLD_CHECKSUM | HA_HAS_NEW_CHECKSUM).
Because the server uses ::info() to ask for a live checksum, while
calculate_checksum() must, precisely, calculate it the slow way,
also for tables that don't have the live checksum at all.

Also, fix the compilation on Windows (ha_checksum/ulonglong type mix).

Author: Georg Richter
Author Date: 2019-06-10 07:15:49 UTC

Fixed windows build

Author: Sergey Petrunia
Author Date: 2019-05-27 14:43:13 UTC

MDEV-19608: MyRocks should check if it is compiled with CRC32 support

- An attempt at checking that MyRocks ins compiled with fast CRC32

Author: Alexey Botchkov
Author Date: 2019-05-24 16:48:09 UTC

Custom build.

Author: Kentoku SHIBA
Author Date: 2019-05-02 19:10:35 UTC

MDEV-18994 Fix the bug encountered when the size of (v1,v2,v3...) value list in “select * from tb where id in (v1, v2, v3...)” is greater than 1000
Some engines need to avoid in optimization.

Author: Monty
Author Date: 2019-05-17 13:34:35 UTC

Make it trivial to get stack traces from external programs.

To get a stacktrace in a program like aria_chk, one only have to do:

#include <my_stacktrace.h>

call my_init_stacktrace(1) in main().

Author: Kentoku SHIBA
Author Date: 2019-05-19 17:05:22 UTC

MDEV-19002 Partition performance optimization
Add the partition options of the following function for batch call instead o
- store_lock
- external_lock
- start_stmt
- extra
- cond_push
- info_push
- top_table

Author: Sergey Petrunia
Author Date: 2019-05-10 09:47:44 UTC

MDEV-19134: EXISTS() slower if ORDER BY is defined

Step #2: "[ORDER BY ...] LIMIT n" should not prevent EXISTS-to-IN
conversion, as long as
- the LIMIT clause doesn't have OFFSET
- the LIMIT is not "LIMIT 0".

Author: Monty
Author Date: 2019-04-15 15:16:02 UTC

MDEV-17841 S3 storage engine

A read-only storage engine that stores it's data in (aws) S3

To store data in S3 one could use ALTER TABLE:
ALTER TABLE table_name ENGINE=S3

libmarias3 integration done by Sergei Golubchik
libmarias3 created by Andrew Hutchings

Author: Sachin Setiya
Author Date: 2019-05-13 06:29:11 UTC

empty commit 3

Author: Sergey Vojtovich
Author Date: 2019-05-10 19:19:05 UTC

MDEV-19439 - Revert orig_thd hack

This patch is to be squashed with previous one when MDEV-19439 is fixed.
Causes assertion failurs currently.

Author: Kentoku SHIBA
Author Date: 2019-05-01 05:46:23 UTC

MDEV-18990 Wrong result when binary column is used as a condition in hexadecimal mode

Author: Vlad Lesin
Author Date: 2019-04-25 20:36:42 UTC

One more compilation error fix for c++11 in runtime code

Author: Marko Mäkelä
Author Date: 2019-04-26 14:55:35 UTC

WIP: MDEV-17603 REPLACE and INSERT…ON DUPLICATE KEY UPDATE are deadlock-prone

Implement an alternative fix for the bug whose original fix
mysql/mysql-server@c93b0d9a972cb6f98fd445f2b69d924350f9128a
in MySQL 5.7.4 caused problems.

This is based on
mysql/mysql-server@e0e4bacddf421550baca3578bc0db13693874fdb
in MySQL 5.7.26.

When performing a rollback to the start of the current row
operation in REPLACE or INSERT...ON DUPLICATE KEY UPDATE
we were not maintaining serializability, because we would
release implicit locks that could already have been acquired
for some of the indexes.

lock_rec_convert_impl_to_expl_for_trx(): Declare globally.

undo_node_t::convert_impl_to_expl(): Convert an implicit lock
to an explicit one during a partial rollback.

row_insert_for_mysql(): Set trx->duplicates=ULINT_UNDEFINED
for rolling back the current row operation. This will allow
undo_node_t::convert_impl_to_expl() to be effective only
for this use case, not for other scenarios, such as
rolling back to the start of the statement, or
ROLLBACK TO SAVEPOINT.

FIXME: Neither innodb.auto_increment_dup,log-bin nor the
upstream fix (which we did not add) innodb.iodku pass.
While the undo_node_t::convert_impl_to_expl() is working
as intended, what happens in innodb.auto_increment_dup,log-bin
is that the newly created explicit record lock for the record
heap number 6 on the PRIMARY key root page (3) will be released
when that record is deleted moments later, with the following
stack trace:

lock_rec_reset_nth_bit
lock_rec_reset_and_release_wait_low
lock_rec_reset_and_release_wait
lock_update_delete
btr_cur_optimistic_delete_func
row_undo_ins_remove_clust_rec
row_undo_ins
row_undo
row_undo_step
que_thr_step
que_run_threads_low
que_run_threads
trx_rollback_to_savepoint_low
trx_rollback_to_savepoint
row_mysql_handle_errors
row_insert_for_mysql

The idea might work with predicate locks, which we do not have.
This entire scenario could also be fixed by MDEV-16232, which
could allow the entire operation to be protected with page latches.

Author: Jani
Author Date: 2019-04-24 15:09:36 UTC

Merge branch '10.3' of github.com:MariaDB/server into bb-10.3-jani