Author: Alexander Barkov
Author Date: 2019-09-20 06:36:20 UTC

MDEV-20634 Report disallowed subquery errors as such (instead of parse error)

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-20 06:19:56 UTC

MDEV-16678 Use MDL lock for innodb background threads instead of dict_operation_lock

- Renaming #sql2 file to some temporary name if the table has
opened ref_count to it. That case is possibly with this patch.
Purge can wait for dict_sys->mutex to close the table. But alter can
hold dict_sys->mutex and try to rename the table during copy algorithm.

Author: Daniel Black
Author Date: 2017-05-08 06:36:37 UTC

MDEV-12646: systemd service file changes from Fedora

Include comment header that describes overrides.

Unit description now includes @VERSION@.

After=syslog.target removed - redunant

Add --basedir=@prefix to prevent /root/.my.cnf lookups. This is
placed after $MYSQLD_OPTIONS in case a user sets a --{no,}default
type options which has to be first in the mysqld arguements.

Additional changes to multi instance (support-files/mariadb@.service.in):
* added @SYSTEMD_EXECSTARTPRE@ / @SYSTEMD_EXECSTARTPOST@
* removed mariadb@bootstrap reference as galera_new_cluster as
  it's a little too proment.
* use_galera_new_cluster.conf updated to override pre/post steps
  to ensure it has no side effects

Signed-off-by: Daniel Black <daniel@linux.vnet.ibm.com>

Author: Ian Gilfillan
Author Date: 2019-09-12 10:09:02 UTC

Fix spelling mistakes in MyISAM code comments

Author: Vladislav Vaintroub
Author Date: 2019-09-19 21:59:57 UTC

Fix -Werror=unused-variable in GCC

Author: varun
Author Date: 2019-09-19 15:23:53 UTC

Adding more comments and cleanup

Author: Anel Husakovic
Author Date: 2019-05-08 13:40:37 UTC

MDEV 19205 Sphinx unable to connect using a host name

- Enable the test `sphinx.sphinx` which was disabled by MDEV 10986,
  comit ee0094d2fd48dac0c
- Add test case to `sphinx.sphinx` to cover host as localhost instead of `127.0.0.1`
- Add result file for single test

Author: Simon Lipp
Author Date: 2019-05-27 11:52:27 UTC

MDEV-18438 Don't stream xtrabackup_info of extra-lsndir

Author: Marko Mäkelä
Author Date: 2019-09-19 05:35:21 UTC

MDEV-12353 preparation: page_mem_alloc_heap()

Define the function in the same compilation unit as its only callers
page_cur_insert_rec_low() and page_cur_insert_rec_zip().

Author: Vlad Lesin
Author Date: 2019-09-18 16:50:28 UTC

Some extra logging is added

(cherry picked from commit 7ff6e33d7ff7104fefb4df4a33c6f3fa54c0093d)

Author: Sergei Golubchik
Author Date: 2019-09-18 12:17:26 UTC

MDEV-19713 Remove big_tables system variable

mark big_tables deprecated, the server can put temp tables on disk
as needed avoiding "table full" errors.

in case someone would really need to force a tmp table to be created
on disk from the start and for testing allow tmp_memory_table_size
to be set to 0.

fix tests to use that instead (and add a test that it actually
works).

make sure in-memory TREE size limit is never 0 (it's [ab]using
tmp_memory_table_size at the moment)

remove few sys_vars.*_basic tests

Author: Sergei Golubchik
Author Date: 2019-09-18 12:00:32 UTC

gis2 test no longer exists after 3fe38574fbdd

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-18 11:52:06 UTC

- Fixed innodb.innodb-defragment test case. Added the condition to
check page_is_leaf(page) before merging the change buffer.

Author: Otto Kekäläinen
Author Date: 2019-06-02 09:05:53 UTC

Deb: Implement proper version detection in maintainer scripts

Fixes bug introduced in commit 5415002.

Using script run-time filename does not always work. One cannot assume
that the filename is always the same as there might be temporary file
names used by dpkg in certain situations. See Debian #920415.

The same fix has been successfully in use in Debian official packages
since February 2019:
https://salsa.debian.org/mariadb-team/mariadb-10.3/commit/6440c0d6e75

Author: Sujatha
Author Date: 2019-09-18 07:14:56 UTC

MDEV-18648: slave_parallel_mode= optimistic default in 10.5

Description:
============
To change CONSERVATIVE @@global.slave_parallel_mode default to OPTIMISTIC in
10.5.

@sql/sys_vars.cc
Changed default parallel_mode to 'OPTIMISTIC'

@sql/sql_class.h
Moved 'wait_for_prior_commit(THD *thd)' method inside sql_class.cc

@sql/sql_class.cc
Added code to check for 'stop_on_error_sub_id' for event groups which get skipped
and don't have any preceding group to wait for.

@sql/rpl_filter.cc
Changed default parallel_mode to 'OPTIMISTIC'

@sql/mysqld.cc
Removed the initialization of 'SLAVE_PARALLEL_CONSERVATIVE' to 'opt_slave_parallel_mode' variable.

@mysql-test/suite/rpl/t/rpl_parallel_mdev6589.test
@mysql-test/suite/rpl/t/rpl_mdev6386.test
Added mtr suppression to ignore 'ER_PRIOR_COMMIT_FAILED'. In case of
OPTIMISTIC mode if a transaction gets killed during "wait_for_prior_commit" it
results in above error "1964". Hence suppression needs to be added for this
error.

@mysql-test/suite/rpl/t/rpl_parallel_conflicts.test
Test has a 'slave.opt' which explicitly sets
slave_parallel_mode='conservative'. When the test ends this mode conflicts
with new default mode. Hence check test case reports an error. The 'slave.opt'
is removed and options are set and reset within test.

@mysql-test/suite/multi_source/info_logs.result
@mysql-test/suite/multi_source/reset_slave.result
@mysql-test/suite/multi_source/simple.result
Result content mismatch in "show slave status" output. This is expected as new
slave_parallel_mode='OPTIMISTIC'.

@mysql-test/include/check-testcase.test
Updated default 'slave_parallel_mode' to 'optimistic'.

@mysql-test/suite/rpl/t/rpl_parallel_analyze_table_hang.test
@mysql-test/suite/rpl/t/rpl_parallel_deadlock_corrupt_binlog.test
@mysql-test/suite/rpl/t/rpl_parallel_domain.test
@mysql-test/suite/rpl/t/rpl_parallel_domain_slave_single_grp.test
@mysql-test/suite/rpl/t/rpl_parallel_free_deferred_event.test
@mysql-test/suite/rpl/t/rpl_parallel_gco_wait_kill.test
@mysql-test/suite/rpl/t/rpl_parallel_gtid_slave_pos_update_fail.test
@mysql-test/suite/rpl/t/rpl_parallel_ignored_errors.test
@mysql-test/suite/rpl/t/rpl_parallel_incorrect_relay_pos.test
@mysql-test/suite/rpl/t/rpl_parallel_innodb_lock_conflict.test
@mysql-test/suite/rpl/t/rpl_parallel_missed_error_handling.test
@mysql-test/suite/rpl/t/rpl_parallel_mode.test
@mysql-test/suite/rpl/t/rpl_parallel_partial_binlog_trans.test
@mysql-test/suite/rpl/t/rpl_parallel_record_gtid_wakeup.test
@mysql-test/suite/rpl/t/rpl_parallel_retry_deadlock.test
@mysql-test/suite/rpl/t/rpl_parallel_rollback_assert.test
@mysql-test/suite/rpl/t/rpl_parallel_single_grpcmt.test
@mysql-test/suite/rpl/t/rpl_parallel_slave_bgc_kill.test
@mysql-test/suite/rpl/t/rpl_parallel_stop_on_con_kill.test
@mysql-test/suite/rpl/t/rpl_parallel_stop_slave.test
@mysql-test/suite/rpl/t/rpl_parallel_wrong_binlog_order.test
@mysql-test/suite/rpl/t/rpl_parallel_wrong_exec_master_pos.test

Refactored rpl_parallel.test into above individual test cases.

rpl_parallel_ignored_errors.test had sporadic failures with following
symptoms.

Problem:
=======
CURRENT_TEST: rpl.rpl_parallel_ignored_errors
--- /test-10.5/mysql-test/suite/rpl/r/rpl_parallel_ignored_errors.result
+++ /test-10.5/mysql-test/suite/rpl/r/rpl_parallel_ignored_errors.reject
@@ -53,6 +53,7 @@
 a
 31
 32
+34
 SET sql_slave_skip_counter= 1;
 ERROR HY000: When using parallel replication and GTID with multiple
 replication domains, @@sql_slave_skip_counter can not be used. Instead,
 setting @@gtid_slave_pos explicitly can be used to skip to after a given GTID
 position
 include/stop_slave_io.inc
@@ -62,7 +63,6 @@
 a
 31
 32
-33
 34
 connection server_2;
 include/stop_slave.inc

Analysis:
=========
In general if there are three groups.
1 - Inserts 32 which fails due to local entry '32' on slave.
2 - Inserts 33
3 - Inserts 34

Each group considers itself as a waiter and it waits for prior group 'waitee'.
This is done in 'register_wait_for_prior_event_group_commit'. If there is no
other parallel group being scheduled then no waitee will be there.

Let us assume 3 groups are being scheduled in parallel.

3-> waits for 2-> waits for->1

'1' upon completion it checks is there any registered subsequent waiter. If
so it wakes up the subsequent waiter with its execution status. This execution
status is stored in wakeup_error.

If '1' failed then it sends corresponding wakeup_error to 2. Then '2' aborts
and it propagates error to '3'. So all further commits are aborted. This
mechanism works only when all transactions reach a stage where they are
waiting for their prior commit to complete.

In case of optimistic following scenario occurs.

1,2,3 are scheduled in parallel.

3 - Reaches group_commit_code waits for 2 to complete.
1 - errors out sets stop_on_error_sub_id=1.

When a group execution results in error its corresponding sub_id is set to
'stop_on_error_sub_id'. Any new groups queued for execution will check if
their sub_id is > stop_on_error_sub_id. If it is true their execution will be
skipped as prior group execution failed. 'skip_event_group=1' will be set.
Since the execution of SQL thread is about to stop we just skip execution of
all the following event groups. We still do all the normal waiting and wakeup
processing between the event groups as a simple way to ensure that everything
is stopped and cleaned up correctly.

Upon error '1' transaction checks for registered waiters. Since no one is
there it simply goes away.

2 - Starts the execution. It checks do I have a waitee.

Since wait_commit_sub_id == entry->last_committed_sub_id no waitee is set.

Secondly: 'entry->stop_on_error_sub_id' is set by '1'st execution. Now
'handle_parallel_thread' code checks if the current group 'sub_id' is greater
than the 'sub_id' set within 'stop_on_error_sub_id'.

Since the above is true 'skip_event_group=true' is set. Simply call
'wait_for_prior_commit' to wakeup all waiters. Group '2' didn't had any
waitee and its execution is skipped. Hence its wakeup_error=0.It sends a
positive wakeup signal to '3'. Which commits. This results in a missed
transaction. i.e 33 is missed.

Fix is to check for 'ston_on_error_sub_id' during wakeup. Please check
sql_class.cc specific changes.

Author: varun
Author Date: 2019-09-17 20:29:29 UTC

MDEV-20589: Server still crashes in Field::set_warning_truncated_wrong_value

The flag is_stat_field is not set for the min_value and max_value of field items
inside table share. This is a must requirement as we don't want to throw
warnings of truncation when we read values from the statistics table to the column
statistics of table share fields.

Author: Simon Lipp
Author Date: 2019-05-27 11:52:27 UTC

MDEV-18438 Don't stream xtrabackup_info of extra-lsndir

Author: Jan Lindström
Author Date: 2019-09-17 04:38:29 UTC

Merge branch 'codership-10.4-thd_stuck_fix' into 10.4

Author: Alexander Barkov
Author Date: 2019-07-10 13:15:52 UTC

MDEV-20016 Add MariaDB_DATA_TYPE_PLUGIN

Author: varun
Author Date: 2019-09-17 01:43:52 UTC

MDEV-20519: Query plan regression with optimizer_use_condition_selectivity > 1

The issue here is the wrong estimate of the cardinality of a partial join,
the cardinality is too high because the function table_cond_selectivity()
returns an absurd number 100 while selectivity cannot be greater than 1.

When accessing table t by outer reference t1.a via index we do not perform any
range analysis for t. Yet we see TABLE::quick_key_parts[key] and
TABLE->quick_rows[key] contain a non-zero value though these should have been
remained untouched and equal to 0.

Thus real cause of the problem is that TABLE::init does not clean the arrays
TABLE::quick_key_parts[] and TABLE::>quick_rows[].
It should have done it because the TABLE structure created for any
instance of a table can be reused for many queries.

Author: Sergey Petrunia
Author Date: 2019-09-16 11:42:24 UTC

MDEV-20371: Invalid reads at plan refinement stage: join->positions...

(Backported to 10.2-compatiblity, build 22257)

best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).

Author: Galina Shalygina
Author Date: 2019-09-11 14:22:38 UTC

Changed is_deterministic() function implementation.
Changed tests.

Author: Sujatha
Author Date: 2019-09-16 10:15:24 UTC

MDEV-20217: Semi_sync: Last_IO_Error: Fatal error: Failed to run 'after_queue_event' hook

Fix:
===
Implemented upstream fix.

commit 7d3d0fc303183ef50a343680ce70df71d5675cd1
Author: He Zhenxing <zhenxing.he@sun.com>

Backport Bug#45852 Semisynch: Last_IO_Error: Fatal error: Failed
to run 'after_queue_event' hook

Errors when send reply to master should never cause the IO thread
to stop, because master can fall back to async replication if it
does not get reply from slave.

The problem is fixed by deliberately ignoring the return value of
slave_reply.

Author: Sujatha
Author Date: 2019-09-16 10:15:24 UTC

MDEV-20217: Semi_sync: Last_IO_Error: Fatal error: Failed to run 'after_queue_event' hook

Fix:
===
Implemented upstream fix.

commit 7d3d0fc303183ef50a343680ce70df71d5675cd1
Author: He Zhenxing <zhenxing.he@sun.com>

Backport Bug#45852 Semisynch: Last_IO_Error: Fatal error: Failed
to run 'after_queue_event' hook

Errors when send reply to master should never cause the IO thread
to stop, because master can fall back to async replication if it
does not get reply from slave.

The problem is fixed by deliberately ignoring the return value of
slave_reply.

Author: Sujatha
Author Date: 2019-09-16 10:15:24 UTC

MDEV-20217: Semi_sync: Last_IO_Error: Fatal error: Failed to run 'after_queue_event' hook

Fix:
===
Implemented upstream fix.

commit 7d3d0fc303183ef50a343680ce70df71d5675cd1
Author: He Zhenxing <zhenxing.he@sun.com>

Backport Bug#45852 Semisynch: Last_IO_Error: Fatal error: Failed
to run 'after_queue_event' hook

Errors when send reply to master should never cause the IO thread
to stop, because master can fall back to async replication if it
does not get reply from slave.

The problem is fixed by deliberately ignoring the return value of
slave_reply.

Author: Eugene
Author Date: 2019-08-28 10:52:51 UTC

MDEV-20194 Warnings inconsistently issued upon CHECK on table from older versions

This patch changes conditions for HA_ERR_TO_BIG_ROW and users will complain!
But I hope now we really have a correct row size check.

This patch will be hard to merge because of changed tests.

Ideally, all row size checks code should live now in innodb.max_record_size
or innodb.max_record_size_compressed.

Now check for too big row size performed for every index on CREATE TABLE,
and for added index on ALTER TABLE. This is how the topic issue was fixed.

Error message for COMPRESSED tables might contain an incorrect maximum
possible row size if overflowed index was not clustered. Fixing it will
require some more effort.

create_table_info_t::row_size_is_acceptable(): high level too big row size
check interface.

record_size_info_t dict_index_t::record_size_info(): this is a low level
interface to perform too big row checks. Also can be used to get maximum
potential row size.

get_max_record_size_leaf_page()
get_max_record_size_non_leaf_page(): these are uniform way to get maximum
allows row size for a given page size and index row format. One function
instead of 5 slightly different computations.

format_too_big_row_error_message(): one function for formatting too big
row error messages instead of several.

Author: Igor Babaev
Author Date: 2019-09-15 00:23:20 UTC

MDEV-19956 Queries with subqueries containing UNION are not parsed

Shift-Reduce conflicts prevented parsing some queries with subqueries that
used set operations when the subqueries occurred in expressions or in IN
predicands.
The grammar rules for query expression were transformed in order to avoid
these conflicts. New grammar rules employ an idea taken from MySQL 8.0.

Author: Sachin Setiya
Author Date: 2019-09-14 07:23:36 UTC

MDEV-20591 Wrong Number of rows in mysqlbinlog output

calc_field_event_length should accurately calculate the size of BLOB type
fields, Instead of returning just the bytes taken by length it should return
length bytes + actual length.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-13 17:39:06 UTC

MDEV-19529 InnoDB hang on DROP FULLTEXT INDEX

Problem:
=======
  During dropping of fts index, InnoDB waits for fts_optimize_remove_table()
and it holds dict_sys->mutex and dict_operaiton_lock even though the
table id is not present in the queue. But fts_optimize_thread does wait
for dict_sys->mutex to process the unrelated table id from the slot.

Solution:
========
  Whenever table is added to fts_optimize_wq, update the fts_status
of in-memory fts subsystem to TABLE_IN_QUEUE. Whenever drop index
wants to remove table from the queue, it can check the fts_status
to decide whether it should send the MSG_DELETE_TABLE to the queue.

Removed the following functions because these are all deadcode.
dict_table_wait_for_bg_threads_to_exit(),
fts_wait_for_background_thread_to_start(),fts_start_shutdown(), fts_shudown().

Author: Sachin Setiya
Author Date: 2019-09-13 10:37:26 UTC

Fix compile error in macOS caused by MDEV-20477

Author: Sergei Golubchik
Author Date: 2019-09-13 07:52:30 UTC

MDEV-9546 mysqlaccess script shows an old version (which was vulnerable to CVE-2005-0004)

update mysqlaccess version

Author: Alexander Barkov
Author Date: 2019-09-13 07:04:23 UTC

MDEV-18153 Assertion `0' or Assertion `btr_validate_index(index, 0)' failed in row_upd_sec_index_entry or error code 126: Index is corrupted upon UPDATE with TIME_ROUND_FRACTIONAL

Conversion to a temporal data type resulting into a lower precision
depends on TIME_ROUND_FRACTIONAL. Taking into account this dependency in:
- indexed generated virtual column expressions
- persistent virtual column expressions

A warning is now issued if conversion from the generation expression
to the column data type depends on TIME_ROUND_FRACTIONAL.

The warning will be changed to error in 10.5

Author: Sergey Petrunia
Author Date: 2019-09-11 17:22:08 UTC

MDEV-20371: Invalid reads at plan refinement stage: join->positions...

(re-committing in 10.4)
best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).

Author: Jan Lindström
Author Date: 2019-09-12 04:23:24 UTC

Force rebuild.

Author: Nikita Malyavin
Author Date: 2018-06-22 13:26:43 UTC

cherry-pick MDEV-16490

Author: Marko Mäkelä
Author Date: 2019-09-11 13:09:18 UTC

Remove page_copy_rec_list_end_to_created_page()

Also remove the log records MLOG_LIST_END_COPY_CREATED,
MLOG_COMP_LIST_END_COPY_CREATED and
page_parse_copy_rec_list_to_created_page().

This will improve the fill factor of some pages.
Adjust some tests accordingly.

FIXME: Remove remaining references to the function and deduplicate code.

TODO: Implement new reusable function (template?) for insert+reorganize.

Author: Sergey Petrunia
Author Date: 2019-09-10 20:51:42 UTC

MDEV-20371: Invalid reads at plan refinement stage: join->positions...

best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).

Author: Nikita Malyavin
Author Date: 2019-09-10 11:54:31 UTC

MDEV-20143 innodb.innodb-virtual-columns-debug failed in buildbot with wrong result

there was a SELECT/DELETE race

Author: varun
Author Date: 2019-09-10 08:31:31 UTC

Moved the function trace_plan_prefix to the optimizer_trace file

Also improved the comments to the function and did some minor cleanup

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-10 06:04:38 UTC

MDEV-20320 Tablespace flags mismatch for full_crc32 format

Problem:
=======
   Tablespace flag mismatch happen when innodb_compression_algorithm set to different
value during restart for full_crc32 format.

Solution:
========
  Remove the innodb_compression_algorithm check while comparing the table flags and
tablespace flags stored in page0.

Author: Nikita Malyavin
Author Date: 2019-09-03 18:29:03 UTC

MDEV-16039 Crash when selecting virtual columns generated using functions with DAYNAME()

Due to collation specifics an Item tree could change while fixing. Virtual fields.

This should be done on the correct query arena, which is table->expr_arena.

Author: Marko Mäkelä
Author Date: 2019-09-09 11:03:03 UTC

MDEV-20487 Remove the InnoDB adaptive hash index

Based on the performance testing that was conducted in MDEV-17492,
the InnoDB adaptive hash index could only help performance in specific,
almost-read-only workloads. It could slow down all kinds of workloads
(especially DROP TABLE, TRUNCATE TABLE, ALTER TABLE, or DROP INDEX
operations), and it can become corrupted, causing crashes
and possibly data corruption. Furthermore, the adaptive hash index
consumes space from the InnoDB buffer pool, which could hurt performance
when the working set would almost fit in the buffer pool.

Given all this, it is best to remove innodb_adaptive_hash_index.
For now, we will change the default value of the build option
WITH_INNODB_AHI to OFF and adjust tests.

The test rpl.rpl_failed_drop_tbl_binlog depends on DROP TABLE being
able to be interrupted while it is dropping pages from the
adaptive hash index. It will be skipped from now on by default,
but it will be tested for builds WITH_INNODB_AHI=ON.

Author: Alexey Botchkov
Author Date: 2019-09-08 21:35:03 UTC

MDEV-16594 ALTER DATA DIRECTORY in PARTITIONS of InnoDB storage does nothing silently.

Author: Elena Stepanova
Author Date: 2019-09-08 20:58:53 UTC

List of unstable tests for 10.4.8 release (updated)

Author: Sergey Petrunia
Author Date: 2019-09-06 19:11:29 UTC

MDEV-17171: RocksDB Tables do not have "Creation Date"

Support Create_time and Update_time for MyRocks tables.
- Create_time is stored in the MyRocks' internal data dictionary.
- Update_time is in-memory only (like in InnoDB).

Author: midenok
Author Date: 2018-09-18 11:20:09 UTC

SQL: followup misc rename on versioning

vers_sys_type_t -> vers_kind_t

Author: midenok
Author Date: 2019-09-05 13:33:25 UTC

MDEV-18057 fix reverted

FIXME: https://github.com/midenok/mariadb/issues/60

Author: Vladislav Vaintroub
Author Date: 2019-09-05 12:26:38 UTC

Merge branch '10.5' into bb-10.5-wlad

# Conflicts:
# storage/innobase/CMakeLists.txt
# storage/innobase/os/os0file.cc

Author: Jan Lindström
Author Date: 2019-09-04 07:00:29 UTC

Fix incorrect reference to view on truncate.

Author: Jan Lindström
Author Date: 2019-09-04 06:21:07 UTC

MDEV-20485: Galera test failure on galera.galera_var_node_address

Test changes only.

Author: Monty
Author Date: 2019-09-03 22:59:35 UTC

Fix of query cache bug in Aria

MDEV-5817 query cache bug (returning inconsistent/old result
set) with aria table parallel inserts, row format = page

The problem is that for transactional aria tables
(row_type=PAGE and transactional=1), maria_lock_database()
didn't flush the state or the query cache.
Not flushing the state is correct for transactional tables as
this is done by checkpoint, but not flushing the query cache
was wrong and could cause concurrent SELECT queries to not
be deleted from the cache.

Fixed by introducing a flush of the query cache as part of commit, if the table has changed.
t for transactional aria tables (row_type=PAGE and transactional=1), maria_lock_table() didn't flush their state or the query cache.

Author: Eugene
Author Date: 2019-09-03 13:31:10 UTC

[NFC] range-forify loops

Author: Sergei Golubchik
Author Date: 2019-09-03 13:28:32 UTC

MDEV-20079 When setting back the system time while mysqld is running, NOW() and UNIX_TIMESTAMP() results get stuck

typo. system_time.start wasn't updated when system_time.sec
and system_time.sec_part were.

Author: Vladislav Vaintroub
Author Date: 2019-09-03 12:48:36 UTC

Connect : Remove debug cmake output, find packages QUIETly

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-03 12:12:53 UTC

- Fixes main.tc_heuristic_recover test case hang.

Author: Monty
Author Date: 2019-09-02 11:06:56 UTC

Merge branch '10.2' into 10.3

Author: Anel Husakovic
Author Date: 2018-04-16 20:49:27 UTC

Enable the auto parametar of the flag `default-character-set`

When invoking option `--default-character-set=auto` character set
from underlying OS settings should be detected for mysqldump.

Author: Alexander Barkov
Author Date: 2019-08-29 08:35:19 UTC

Part2: MDEV-18156 Assertion `0' failed or `btr_validate_index(index, 0, false)' in row_upd_sec_index_entry or error code 126: Index is corrupted upon DELETE with PAD_CHAR_TO_FULL_LENGTH

This patch allows the server to open old tables that have
"bad" generated columns (i.e. indexed virtual generated columns,
persistent generated columns) that depend on sql_mode,
for general things like SELECT, INSERT, DROP, etc.
Warning are issued in such cases.

Only these commands are now disallowed and return an error:
- CREATE TABLE introducing a "bad" generated column
- ALTER TABLE introducing a "bad" generated column
- CREATE INDEX introdicing a "bad" generated column
  (i.e. adding an index on a virtual generated column
   that depends on sql_mode).

Note, these commands are allowed:
- ALTER TABLE removing a "bad" generate column
- ALTER TABLE removing an index from a "bad" virtual generated column
- DROP INDEX removing an index from a "bad" virtual generated column
but only if the table does not have any "bad" columns as a result.

Author: Sachin Setiya
Author Date: 2019-09-01 14:35:07 UTC

19708 compatibility

Author: Kentoku SHIBA
Author Date: 2019-05-28 07:40:59 UTC

MDEV-6268 SPIDER table with no COMMENT clause causes queries to wait forever

Add loop checks by using user variables

Author: Kentoku SHIBA
Author Date: 2019-09-01 20:09:43 UTC

MDEV-6268 SPIDER table with no COMMENT clause causes queries to wait forever

Author: Igor Babaev
Author Date: 2019-09-01 05:44:58 UTC

Merge branch 'mdev-18844' of https://github.com/waynexia/server into mdev-18844

Author: Igor Babaev
Author Date: 2019-08-30 22:49:07 UTC

MDEV-15777 Use inferred IS NOT NULL predicates in the range optimizer

This patch introduces the optimization that allows range optimizer to
consider index range scans that are built employing NOT NULL predicates
inferred from WHERE conditions and ON expressions.
The patch adds a new optimizer switch not_null_range_scan.

Author: Eugene
Author Date: 2019-08-30 13:00:27 UTC

range-forify loops

Author: Nikita Malyavin
Author Date: 2019-08-30 06:22:30 UTC

fix cur_select_lex->where

Author: Teemu Ollakka
Author Date: 2019-07-25 13:41:45 UTC

MDEV-19826 10.4 seems to crash with "pool-of-threads"

MariaDB 10.4 was crashing when thread-handling was set to
pool-of-threads and wsrep was enabled.

There were two apparent reasons for the crash:
- Connection handling in threadpool_common.cc was missing calls to
  control wsrep client state.
- Thread specific storage which contains thread variables (THR_KEY_mysys)
  was not handled appropriately by wsrep patch when pool-of-threads
  was configured.

This patch addresses the above issues in the following way:
- Wsrep client state open/close was moved in thd_prepare_connection() and
  end_connection() to have common handling for one-thread-per-connection
  and pool-of-threads.
- Thread local storage handling in wsrep patch was reworked by introducing
  set of wsrep_xxx_threadvars() calls which replace calls to
  THD store_globals()/reset_globals() and deal with thread handling
  specifics internally.

Wsrep-lib was updated to version which relaxes internal concurrency
related sanity checks.

Rollback code from wsrep_rollback_process() was extracted to separate calls
for better readability.

Post rollback thread was removed as it was completely unused.

Author: Sergey Petrunia
Author Date: 2019-08-29 13:07:15 UTC

MDEV-20109: Optimizer ignores distinct key created for materialized...

Adjust the testcase according to the review input

Author: varun
Author Date: 2019-08-28 20:44:15 UTC

fix for rows of range scan

Author: Sergei Golubchik
Author Date: 2019-08-28 17:21:41 UTC

cleanup: don't invoke dh_installinit from debian/rules

as there's no SysV init script to install anymore

followup for MDEV-15526

Author: Julius Goryavsky
Author Date: 2019-08-28 11:59:34 UTC

MDEV-20420: SST failed after MDEV-18863 in some test configurations

After applying MDEV-18863, in some test configurations, SST
may fails due to duplication of some parameters (in particular
"--port") in the main part of the command line and after
"--mysqld-args", as well as due to incorrect interpretation
of the parameter "--port" passed after "--mysqld-args" when
the SST script is invoked without explicitly specifying a port
for SST. In addition, it is necessary to correctly handle spaces,
quotation marks and special characters when copying original
arguments from the argv[] array to a new command line (after
"--mysqld-args"). This patch resolves these shortcomings.

Author: Julius Goryavsky
Author Date: 2019-08-28 02:49:01 UTC

Improved handling of subdirectories in the xtrabackup-v2 SST scripts (similar to MDEV-18863) for more predictable test results (related to xtrabackup-v2 SST)

Author: Sergey Vojtovich
Author Date: 2019-07-31 08:51:03 UTC

MDEV-17084 - Optimize append only files for NVDIMM

Integration of persistent memory append cache with MariaDB binary log.

Author: Jan Lindström
Author Date: 2019-08-27 14:04:59 UTC

Merge branch 'codership-vote-10.5' into 10.5

Author: Jan Lindström
Author Date: 2019-08-27 05:30:26 UTC

Enable galera_sst_mysqldump_with_key test case.

Author: Eugene
Author Date: 2019-08-26 16:19:12 UTC

fix clang warnings

Author: Sergei Golubchik
Author Date: 2019-08-26 12:49:42 UTC

restore the correct InnoDB auto-inc behavior after the merge

Author: Sergei Golubchik
Author Date: 2019-08-26 12:49:42 UTC

restore the correct InnoDB auto-inc behavior after the merge

Author: Sujatha
Author Date: 2019-08-26 11:43:16 UTC

MDEV-19925: Column ... cannot be converted from type 'varchar(20)' to type 'varchar(20)'

Cherry picking:
Bug#25135304: RBR: WRONG FIELD LENGTH IN ERROR MESSAGE
commit 47bd3f7cf3c8518f62b1580ec65af2ba7ac13b95

Description:
============
In row based replication, when replicating from a table with a field with
character set set to UTF8mb3 to the same table with the same field set to
character set UTF8mb4 I get a confusing error message:

For VARCHAR: VARCHAR(1) 'utf8mb3' to VARCHAR(1) 'utf8mb4'
"Column 0 of table 'test.t1' cannot be converted from type 'varchar(3)' to
type 'varchar(1)'"

Similar issue with CHAR type as well.

Issue with respect to BLOB types:

For BLOB: LONGBLOB to TINYBLOB - Error message displays incorrect blob type.
"Column 0 of table 'test.t1' cannot be converted from type 'tinyblob' to type
'tinyblob'"

For BINARY to BINARY - Error message displays incorrect type for master side
field.
"Column 0 of table 'test.t' cannot be converted from type 'char(1)' to type
'binary(10)'"
Similar issue exists for VARBINARY type. It is displayed as 'VARCHAR'.

Analysis:
=========
In Row based replication charset information is not sent as part of metadata
from master to slave.

For VARCHAR field its character length is converted into equivalent
octets/bytes and stored internally. At the time of displaying the data to user
it is converted back to original character length.

For example:
VARCHAR(2)- utf8mb3 is stored as:2*3 = VARCHAR(6)
At the time of displaying it to user
VARCHAR(6)- charset utf8mb3:6/3= VARCHAR(2).

At present the internally converted octect length is sent from master to slave
with out providing the charset information. On slave side if the type
conversion fails 'show_sql_type' function is used to get the type specific
information from metadata. Since there is no charset information is available
the filed type is displayed as VARCHAR(6).

This results in confused error message.

For CHAR fields
CHAR(1)- utf8mb3 - CHAR(3)
CHAR(1)- utf8mb4 - CHAR(4)

'show_sql_type' function which retrieves type information from metadata uses
(bytes/local charset length) to get actual character length. If slave's chaset
is 'utf8mb4' then

CHAR(3/4)-->CHAR(0)
CHAR(4/4)-->CHAR(1).

This results in confused error message.

Analysis for BLOB type issue:

BLOB's length is represented in two forms.
1. Actual length
i.e
  (length < 256) type= MYSQL_TYPE_TINY_BLOB;
  (length < 65536) type= MYSQL_TYPE_BLOB; ...

2. packlength - The number of bytes used to represent the length of the blob
  1- tinyblob
  2- blob ...

In row based replication only the packlength is written in the binary log. On
the slave side this packlength is interpreted as actual length of the blob.
Hence the length is always < 256 and the type is displayed as tiny blob.

Analysis for BINARY to BINARY type issue:
The character set information is needed to identify a filed's type as char or
binary. Since master side character set information is not available on the
slave side both binary and char fields are displayed as char.

Fix:
===
For CHAR and VARCHAR fields display their length in octets for both source and
target fields. For target field display the charset information if it is
relevant.

For blob type changed the code to use the packlength and display appropriate
blob type in error message.

For binary and varbinary fields use the slave side character set as reference
to map them to binary or varbinary fields.

Author: Andrei Elkin
Author Date: 2019-08-26 11:05:33 UTC

MDEV-742 (intermediate commit)

typ == XID_EVENT || typ == XA_PREPARE_LOG_EVENT augmentation.

Author: Sergei Golubchik
Author Date: 2019-08-25 16:36:59 UTC

fix EXPLAIN output

1. avoid ha_rows->double->ha_rows cast, for ~0ULL it behaves
   differently on Linux and Windows. This fixes the differences
   in derived_view and mdev13607 results between Linux and Windows.
2. EXPLAIN should show the number of rows as an unsigned number.

Author: Roman Nozdrin
Author Date: 2019-08-24 15:42:35 UTC

MDEV-19831 find_select_handler() now tries its best to find a handlerton that is able to processes the whole query. For that it traverses tables from subqueries.

    Select_handler now cleans up temporary table structures on dctor call.

Author: Sergey Petrunia
Author Date: 2019-08-22 21:57:40 UTC

MDEV-20109: Optimizer ignores distinct key created for materialized...

Sj_materialization_picker::check_qep(): fix error in cost/fanout
calculations:
- for each join prefix, add #prefix_rows / TIME_FOR_COMPARE to the cost,
  like best_extension_by_limited_search does
- Remove the fanout produced by the subquery tables.

optimize_wo_join_buffering() (used by LooseScan and FirstMatch)
- also add #prefix_rows / TIME_FOR_COMPARE to the cost of each prefix.

Author: Monty
Author Date: 2019-08-18 09:43:41 UTC

Updated result for rocksdb tests after merge

Author: Anel Husakovic
Author Date: 2019-08-21 17:51:04 UTC

MDEV-20382 SHOW PRIVILEGES displays "Delete versioning rows" rather than "Delete History"

Author: Sujatha
Author Date: 2019-08-21 15:53:11 UTC

MDEV-20188: binlog.binlog_stm_drop_tmp_tbl fails in buildbot with Unknown table on exec

Analysis:
========
As part of BUG#28642318 fix, two new test cases were added. The first test
case tests a scenario where two sessions are present, in which the first
session has a regular table named 't1' and another session has a temporary
table named 't1'. Test executes a DELETE statement on regular table. These
statements are captured from binary log and replayed back on new client
connection to prove that DELETE statement is applied successfully. Note that
the binlog contains only CREATE TEMPORARY TABLE part hence a temporary table
gets created in new connection. This replaying logic is implemented by using
'--exec $MYSQL' command. If the new connection gets disconnected within the
scope of first test case the test passes, i.e the temporary table gets dropped
as part thread cleanup. But on slow platforms the connection gets closed at
the time of execution of test case 2. When the temporary table is dropped as
part thread cleanup a "DROP TEMPORARY TABLE t1" is written into the binary
log. In test case two the same sessions continue to exist and and table names
are reused to test a new bug scenario. The additional "DROP TEMPORARY TABLE"
command drops second test specific tables which results in "Unknown table"
error.

Fix:
====
Rename the second case specific table to 't2'. Even if the close connection
from test case one happens later the drop command with has
'DROP /*!40005 TEMPORARY */ TABLE IF EXISTS `t1`' will not result in an error.

Author: Jan Lindström
Author Date: 2019-08-20 08:55:43 UTC

Merge branch 'codership-vote-10.5' into 10.5

Author: Nikita Malyavin
Author Date: 2019-02-12 11:16:02 UTC

MDEV-16978 Application-time periods: WITHOUT OVERLAPS

* The overlaps check is implemented on a handler level per row command. It creates a separate cursor (actually, another handler instance) and caches it inside the original handler, when ha_update_row or ha_insert_row is issued. Cursor closes on unlocking the handler.

* Containing the same key in index means unique constraint violation even in usual terms. So we fetch left and right neighbours and check that they have same key prefix, excluding from the key only the period part. If it doesnt match, then there's no such neighbour, and the check passes. Otherwise, we check if this neighbour intersects with the considered key.

* the check does introduce new error and fails with ER_DUPP_KEY error. This might break REPLACE workflow and should be fixed separately

Author: Alexander Barkov
Author Date: 2019-08-19 18:42:56 UTC

MDEV-19034 ASAN unknown-crash in get_date_time_separator with PAD_CHAR_TO_FULL_LENGTH

Author: midenok
Author Date: 2019-08-19 11:02:13 UTC

MDEV-20360 Assertion bitmap_is_set(read_partitions) upon updating view of join

Author: Rucha Deodhar
Author Date: 2019-08-17 05:02:13 UTC

new test files for checking datatypes

Author: Robert Bindar
Author Date: 2019-08-16 11:35:54 UTC

MDEV-20298 lidbd-mariadb-perl is not always available

Author: Rasmus Johansson
Author Date: 2019-06-17 07:43:01 UTC

MDEV-19781 Create MariaDB named commands on Windows

Author: Alexander Barkov
Author Date: 2019-08-15 09:16:00 UTC

MDEV-20353 Add separate type handlers for unsigned integer data types

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-08-15 09:43:15 UTC

MDEV-18128 Simplify .ibd file creation

Problem:
========
During ibd file creation, InnoDB flushes the page0 without crypt
information. During recovery, InnoDB encounters encrypted page read
before initialising the crypt data of the tablespace. So it leads t
corruption of page and doesn't allow innodb to start.

Solution:
=========
Write crypt_data information in page0 while creating .ibd file creation.
During recovery, crypt_data will be initialised while processing
MLOG_FILE_NAME redo log record.

Author: Anel Husakovic
Author Date: 2019-07-31 10:28:38 UTC

MDEV-20210 If you have an INVISIBLE VIRTUAL column, SHOW CREATE TABLE doesn't list it as INVISIBLE

Author: Sergei Golubchik
Author Date: 2019-08-12 16:30:19 UTC

ensure that pam plugin is present in release packages

Author: Sergey Petrunia
Author Date: 2019-08-09 21:53:28 UTC

Optimizer trace: print cost and #rows of the join prefix

The names rows_for_plan and cost_for_plan follow MySQL
Also added post-join-operation selectivity cost

Author: Sachin Setiya
Author Date: 2018-05-17 09:10:55 UTC

MDEV-16239 Many test in rpl suite fails

Fix rpl_skip_error test.
  We cant reset Slave_skipped_errors(even with FLUSH STATUS), So instead
of absolute slave_skipped_errors we look for delta of slave_skipped_errors
Fix rpl.rpl_binlog_errors and binlog_encryption.rpl_binlog_errors
  We create the $load_file and $load_file2 but we never remove them.
Fix rpl_000011.test
  Instead of real value use delta value , Since flush status wont flush
LONGLONG variable.
Fix rpl_row_find_row_debug
  Instead of searching whole log_error_ file we will use search_pattern_in_file
which runs pattern search only on latest test run , instead of full file.
Fix rpl_ip_mix rpl_ip_mix2
  We should call reset slave all because we also want to reset master_host
otherwise show slave status wont be empty and making repeat N a failure.
Fix rpl_rotate_logs
  First we have to remove master.info file (cleanup) and second we have to
call reset slave all because if we do not call reset slave all then we wont
read master.info file beacuse we already have master config in memory.
And this makes start slave to pass , which shoud fail becuase its permision
is 000
Fix circular_serverid0 test
  The reason is that ++dbug_rows_event_count == 2 in queue_event does
not take --repeat into account. So I have reseted the dbug_rows_event_count
in if body.

Author: Sachin Setiya
Author Date: 2019-07-17 10:26:29 UTC

MDEV-17614 INSERT on dup key update is replication unsafe

Problem:-
When mysql executes INSERT ON DUPLICATE KEY INSERT, the storage engine checks
if the inserted row would generate a duplicate key error. If yes, it returns
the existing row to mysql, mysql updates it and sends it back to the storage
engine.When the table has more than one unique or primary key, this statement
is sensitive to the order in which the storage engines checks the keys.
Depending on this order, the storage engine may determine different rows
to mysql, and hence mysql can update different rows.The order that the
storage engine checks keys is not deterministic. For example, InnoDB checks
keys in an order that depends on the order in which indexes were added to
the table. The first added index is checked first. So if master and slave
have added indexes in different orders, then slave may go out of sync.

Solution:-
Make INSERT...ON DUPLICATE KEY UPDATE unsafe while using stmt or mixed format
When there is more then one unique key.
Although there is two exception.
  1. Auto Increment key is not counted because Innodb will get gap lock for
    failed Insert and concurrent insert will get a next increment value. But if
    user supplies auto inc value it can be unsafe.
  2. Count only unique keys for which insertion is performed.

So this patch also addresses the bug id #72921

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-08-08 16:06:37 UTC

MDEV-19348 MariaBackup prepare fails with corruption on disk

Problem:
========
 For alter table.. add index command, MLOG_INDEX_LOAD log ensures that only
index pages are flushed to disk. It doesn't ensure about page 0, page 1, page 2.
During recovery, redo of page0 applies after any index page read. It leads to
failure of decryption of index page.

Solution:
=========
- While parsing of redo log, store the space ids of failed crypt_data initialization
and ignore the pages of tablespace if the key version and space->crypt_data doesn't
match.
- Try to apply the redo log for ignored page after initializing the crypt_data
for the tablespace.