lp:maria

Author: Marko Mäkelä
Author Date: 2019-09-23 06:04:51 UTC

After-merge fix: Adjust a result

Author: Eugene
Author Date: 2019-09-22 18:04:00 UTC

MDEV-19189 ASAN memcpy-param-overlap in fill_alter_inplace_info upon adding indexes

memmove() should be used instead of memcpy() for overlapping memory regions.

Overlapping memory regions itself here are fine, because code simply removes
one element from arbitrary position of an array.

Author: Sergei Golubchik
Author Date: 2019-09-18 12:17:26 UTC

MDEV-19713 Remove big_tables system variable

mark big_tables deprecated, the server can put temp tables on disk
as needed avoiding "table full" errors.

in case someone would really need to force a tmp table to be created
on disk from the start and for testing allow tmp_memory_table_size
to be set to 0.

fix tests to use that instead (and add a test that it actually
works).

make sure in-memory TREE size limit is never 0 (it's [ab]using
tmp_memory_table_size at the moment)

remove few sys_vars.*_basic tests

Author: Michael Widenius
Author Date: 2019-09-22 01:08:48 UTC

Proper fix for disabling warnings in read_statistics_for_table().

MDEV-20589: Server still crashes in Field::set_warning_truncated_wrong_value

- Use dbug_tmp_use_all_columns() to mark that all fields can be used
- Remove field->is_stat_field (not needed)
- Remove extra arguments to Field::clone() that should not be there
- Safety fix for Field::set_warning_truncated_wrong_value() to not crash
  if table is zero in production builds (We have got crashes several times
  here so better to be safe than sorry).
- Threat wrong character string warnings identical to other field
  conversion warnings. This removes some warnings we before got from
  internal conversion errors. There is no good reason why a user would
  get an error in case of 'key_field='wrong-utf8-string' but not for
  'field=wrong-utf8-string'. The old code could also easily give
  thousands of no-sence warnings for one single statement.

Author: varun
Author Date: 2019-09-21 19:47:30 UTC

MDEV-20468: Allocating more space than required for JOIN_TAB array for a query with SJM table

Author: Sergei Golubchik
Author Date: 2019-09-20 21:44:43 UTC

cleanup: INSERT...RETURNING

* remove redundant class members and function arguments
* make DELETE...RETURNING to use setup_returning_fields()
* initialize Lex->returning_list only when needed
* update sql_yacc_ora.yy

Author: Sujatha
Author Date: 2019-09-21 17:26:14 UTC

MDEV-18648: Testing worker_error propogation.

Author: Vladislav Vaintroub
Author Date: 2019-09-21 12:27:46 UTC

MDEV-16264 - replace srv_mopnitor thread with timer task

Author: midenok
Author Date: 2018-05-23 19:15:04 UTC

MDEV-16130 wrong error message adding AS ROW START to versioned table

Author: varun
Author Date: 2019-09-21 08:39:24 UTC

Tests added when ordering is done by index on the first non-const table

Author: varun
Author Date: 2019-09-21 06:44:05 UTC

MDEV-18094: Query with order by limit picking index scan over filesort

In the function test_if_cheaper_ordering we make a decision if using an index is better than
using filesort for ordering. If we chose to do range access then in test_quick_select we
should make sure that cost for table scan is set to DBL_MAX so that it is not picked.

Author: rantal
Author Date: 2019-09-16 19:36:19 UTC

Update mysqld_safe.sh - introduce defaults-group-suffix handling

"--defaults-group-suffix" must be be given as the first argument on the command-line of mysqld

Author: David Carlier
Author Date: 2019-06-12 12:33:34 UTC

Removing obsolete register keyword

Author: Sergey Vojtovich
Author Date: 2019-09-20 18:52:00 UTC

Cleanup session tracker

- call current_schema::mark_as_changed() directly
- call state_change::mark_as_changed() directly
- replaced SESSION_TRACKER_CHANGED with dummy tracker
- replaced Session_tracker::mark_as_changed() with
  State_tracker::mark_as_changed()
- hide and devirtualize original State_tracker::mark_as_changed(),
  rename it to set_changed()
- all implementations of mark_as_changed() now check is_enabled() for
  consistency
- no argument casts anymore

Author: varun
Author Date: 2019-09-20 19:37:51 UTC

MDEV-18094: Query with order by limit picking index scan over filesort

In the function test_if_cheaper_ordering we make a decision if using an index is better than
using filesort for ordering. If we chose to do range access then in test_quick_select we
should make sure that cost for table scan is set to DBL_MAX so that it is not picked.

Author: Igor Babaev
Author Date: 2019-09-20 16:03:38 UTC

MDEV-19956 Queries with subqueries containing UNION are not parsed

Shift-Reduce conflicts prevented parsing some queries with subqueries that
used set operations when the subqueries occurred in expressions or in IN
predicands.
The grammar rules for query expression were transformed in order to avoid
these conflicts. New grammar rules employ an idea taken from MySQL 8.0.

Author: Monty
Author Date: 2019-09-20 11:09:26 UTC

MDEV-17084 - Optimize append only files for NVDIMM

Minor fixes.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-20 06:19:56 UTC

MDEV-16678 Use MDL lock for innodb background threads instead of dict_operation_lock

- Renaming #sql2 file to some temporary name if the table has
opened ref_count to it. That case is possibly with this patch.
Purge can wait for dict_sys->mutex to close the table. But alter can
hold dict_sys->mutex and try to rename the table during copy algorithm.

Author: Simon Lipp
Author Date: 2019-05-27 11:52:27 UTC

MDEV-18438 Don't stream xtrabackup_info of extra-lsndir

Author: Marko Mäkelä
Author Date: 2019-09-19 05:35:21 UTC

MDEV-12353 preparation: page_mem_alloc_heap()

Define the function in the same compilation unit as its only callers
page_cur_insert_rec_low() and page_cur_insert_rec_zip().

Author: Vlad Lesin
Author Date: 2019-09-18 16:50:28 UTC

Some extra logging is added

(cherry picked from commit 7ff6e33d7ff7104fefb4df4a33c6f3fa54c0093d)

Author: Sergei Golubchik
Author Date: 2019-09-18 12:00:32 UTC

gis2 test no longer exists after 3fe38574fbdd

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-18 11:52:06 UTC

- Fixed innodb.innodb-defragment test case. Added the condition to
check page_is_leaf(page) before merging the change buffer.

Author: Sujatha
Author Date: 2019-09-18 07:14:56 UTC

MDEV-18648: slave_parallel_mode= optimistic default in 10.5

Description:
============
To change CONSERVATIVE @@global.slave_parallel_mode default to OPTIMISTIC in
10.5.

@sql/sys_vars.cc
Changed default parallel_mode to 'OPTIMISTIC'

@sql/sql_class.h
Moved 'wait_for_prior_commit(THD *thd)' method inside sql_class.cc

@sql/sql_class.cc
Added code to check for 'stop_on_error_sub_id' for event groups which get skipped
and don't have any preceding group to wait for.

@sql/rpl_filter.cc
Changed default parallel_mode to 'OPTIMISTIC'

@sql/mysqld.cc
Removed the initialization of 'SLAVE_PARALLEL_CONSERVATIVE' to 'opt_slave_parallel_mode' variable.

@mysql-test/suite/rpl/t/rpl_parallel_mdev6589.test
@mysql-test/suite/rpl/t/rpl_mdev6386.test
Added mtr suppression to ignore 'ER_PRIOR_COMMIT_FAILED'. In case of
OPTIMISTIC mode if a transaction gets killed during "wait_for_prior_commit" it
results in above error "1964". Hence suppression needs to be added for this
error.

@mysql-test/suite/rpl/t/rpl_parallel_conflicts.test
Test has a 'slave.opt' which explicitly sets
slave_parallel_mode='conservative'. When the test ends this mode conflicts
with new default mode. Hence check test case reports an error. The 'slave.opt'
is removed and options are set and reset within test.

@mysql-test/suite/multi_source/info_logs.result
@mysql-test/suite/multi_source/reset_slave.result
@mysql-test/suite/multi_source/simple.result
Result content mismatch in "show slave status" output. This is expected as new
slave_parallel_mode='OPTIMISTIC'.

@mysql-test/include/check-testcase.test
Updated default 'slave_parallel_mode' to 'optimistic'.

@mysql-test/suite/rpl/t/rpl_parallel_analyze_table_hang.test
@mysql-test/suite/rpl/t/rpl_parallel_deadlock_corrupt_binlog.test
@mysql-test/suite/rpl/t/rpl_parallel_domain.test
@mysql-test/suite/rpl/t/rpl_parallel_domain_slave_single_grp.test
@mysql-test/suite/rpl/t/rpl_parallel_free_deferred_event.test
@mysql-test/suite/rpl/t/rpl_parallel_gco_wait_kill.test
@mysql-test/suite/rpl/t/rpl_parallel_gtid_slave_pos_update_fail.test
@mysql-test/suite/rpl/t/rpl_parallel_ignored_errors.test
@mysql-test/suite/rpl/t/rpl_parallel_incorrect_relay_pos.test
@mysql-test/suite/rpl/t/rpl_parallel_innodb_lock_conflict.test
@mysql-test/suite/rpl/t/rpl_parallel_missed_error_handling.test
@mysql-test/suite/rpl/t/rpl_parallel_mode.test
@mysql-test/suite/rpl/t/rpl_parallel_partial_binlog_trans.test
@mysql-test/suite/rpl/t/rpl_parallel_record_gtid_wakeup.test
@mysql-test/suite/rpl/t/rpl_parallel_retry_deadlock.test
@mysql-test/suite/rpl/t/rpl_parallel_rollback_assert.test
@mysql-test/suite/rpl/t/rpl_parallel_single_grpcmt.test
@mysql-test/suite/rpl/t/rpl_parallel_slave_bgc_kill.test
@mysql-test/suite/rpl/t/rpl_parallel_stop_on_con_kill.test
@mysql-test/suite/rpl/t/rpl_parallel_stop_slave.test
@mysql-test/suite/rpl/t/rpl_parallel_wrong_binlog_order.test
@mysql-test/suite/rpl/t/rpl_parallel_wrong_exec_master_pos.test

Refactored rpl_parallel.test into above individual test cases.

rpl_parallel_ignored_errors.test had sporadic failures with following
symptoms.

Problem:
=======
CURRENT_TEST: rpl.rpl_parallel_ignored_errors
--- /test-10.5/mysql-test/suite/rpl/r/rpl_parallel_ignored_errors.result
+++ /test-10.5/mysql-test/suite/rpl/r/rpl_parallel_ignored_errors.reject
@@ -53,6 +53,7 @@
 a
 31
 32
+34
 SET sql_slave_skip_counter= 1;
 ERROR HY000: When using parallel replication and GTID with multiple
 replication domains, @@sql_slave_skip_counter can not be used. Instead,
 setting @@gtid_slave_pos explicitly can be used to skip to after a given GTID
 position
 include/stop_slave_io.inc
@@ -62,7 +63,6 @@
 a
 31
 32
-33
 34
 connection server_2;
 include/stop_slave.inc

Analysis:
=========
In general if there are three groups.
1 - Inserts 32 which fails due to local entry '32' on slave.
2 - Inserts 33
3 - Inserts 34

Each group considers itself as a waiter and it waits for prior group 'waitee'.
This is done in 'register_wait_for_prior_event_group_commit'. If there is no
other parallel group being scheduled then no waitee will be there.

Let us assume 3 groups are being scheduled in parallel.

3-> waits for 2-> waits for->1

'1' upon completion it checks is there any registered subsequent waiter. If
so it wakes up the subsequent waiter with its execution status. This execution
status is stored in wakeup_error.

If '1' failed then it sends corresponding wakeup_error to 2. Then '2' aborts
and it propagates error to '3'. So all further commits are aborted. This
mechanism works only when all transactions reach a stage where they are
waiting for their prior commit to complete.

In case of optimistic following scenario occurs.

1,2,3 are scheduled in parallel.

3 - Reaches group_commit_code waits for 2 to complete.
1 - errors out sets stop_on_error_sub_id=1.

When a group execution results in error its corresponding sub_id is set to
'stop_on_error_sub_id'. Any new groups queued for execution will check if
their sub_id is > stop_on_error_sub_id. If it is true their execution will be
skipped as prior group execution failed. 'skip_event_group=1' will be set.
Since the execution of SQL thread is about to stop we just skip execution of
all the following event groups. We still do all the normal waiting and wakeup
processing between the event groups as a simple way to ensure that everything
is stopped and cleaned up correctly.

Upon error '1' transaction checks for registered waiters. Since no one is
there it simply goes away.

2 - Starts the execution. It checks do I have a waitee.

Since wait_commit_sub_id == entry->last_committed_sub_id no waitee is set.

Secondly: 'entry->stop_on_error_sub_id' is set by '1'st execution. Now
'handle_parallel_thread' code checks if the current group 'sub_id' is greater
than the 'sub_id' set within 'stop_on_error_sub_id'.

Since the above is true 'skip_event_group=true' is set. Simply call
'wait_for_prior_commit' to wakeup all waiters. Group '2' didn't had any
waitee and its execution is skipped. Hence its wakeup_error=0.It sends a
positive wakeup signal to '3'. Which commits. This results in a missed
transaction. i.e 33 is missed.

Fix is to check for 'ston_on_error_sub_id' during wakeup. Please check
sql_class.cc specific changes.

Author: varun
Author Date: 2019-09-17 20:29:29 UTC

MDEV-20589: Server still crashes in Field::set_warning_truncated_wrong_value

The flag is_stat_field is not set for the min_value and max_value of field items
inside table share. This is a must requirement as we don't want to throw
warnings of truncation when we read values from the statistics table to the column
statistics of table share fields.

Author: Simon Lipp
Author Date: 2019-05-27 11:52:27 UTC

MDEV-18438 Don't stream xtrabackup_info of extra-lsndir

Author: Jan Lindström
Author Date: 2019-09-17 04:38:29 UTC

Merge branch 'codership-10.4-thd_stuck_fix' into 10.4

Author: Alexander Barkov
Author Date: 2019-07-10 13:15:52 UTC

MDEV-20016 Add MariaDB_DATA_TYPE_PLUGIN

Author: Sergey Petrunia
Author Date: 2019-09-16 11:42:24 UTC

MDEV-20371: Invalid reads at plan refinement stage: join->positions...

(Backported to 10.2-compatiblity, build 22257)

best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).

Author: Galina Shalygina
Author Date: 2019-09-11 14:22:38 UTC

Changed is_deterministic() function implementation.
Changed tests.

Author: Sujatha
Author Date: 2019-09-16 10:15:24 UTC

MDEV-20217: Semi_sync: Last_IO_Error: Fatal error: Failed to run 'after_queue_event' hook

Fix:
===
Implemented upstream fix.

commit 7d3d0fc303183ef50a343680ce70df71d5675cd1
Author: He Zhenxing <zhenxing.he@sun.com>

Backport Bug#45852 Semisynch: Last_IO_Error: Fatal error: Failed
to run 'after_queue_event' hook

Errors when send reply to master should never cause the IO thread
to stop, because master can fall back to async replication if it
does not get reply from slave.

The problem is fixed by deliberately ignoring the return value of
slave_reply.

Author: Sujatha
Author Date: 2019-09-16 10:15:24 UTC

MDEV-20217: Semi_sync: Last_IO_Error: Fatal error: Failed to run 'after_queue_event' hook

Fix:
===
Implemented upstream fix.

commit 7d3d0fc303183ef50a343680ce70df71d5675cd1
Author: He Zhenxing <zhenxing.he@sun.com>

Backport Bug#45852 Semisynch: Last_IO_Error: Fatal error: Failed
to run 'after_queue_event' hook

Errors when send reply to master should never cause the IO thread
to stop, because master can fall back to async replication if it
does not get reply from slave.

The problem is fixed by deliberately ignoring the return value of
slave_reply.

Author: Eugene
Author Date: 2019-08-28 10:52:51 UTC

MDEV-20194 Warnings inconsistently issued upon CHECK on table from older versions

This patch changes conditions for HA_ERR_TO_BIG_ROW and users will complain!
But I hope now we really have a correct row size check.

This patch will be hard to merge because of changed tests.

Ideally, all row size checks code should live now in innodb.max_record_size
or innodb.max_record_size_compressed.

Now check for too big row size performed for every index on CREATE TABLE,
and for added index on ALTER TABLE. This is how the topic issue was fixed.

Error message for COMPRESSED tables might contain an incorrect maximum
possible row size if overflowed index was not clustered. Fixing it will
require some more effort.

create_table_info_t::row_size_is_acceptable(): high level too big row size
check interface.

record_size_info_t dict_index_t::record_size_info(): this is a low level
interface to perform too big row checks. Also can be used to get maximum
potential row size.

get_max_record_size_leaf_page()
get_max_record_size_non_leaf_page(): these are uniform way to get maximum
allows row size for a given page size and index row format. One function
instead of 5 slightly different computations.

format_too_big_row_error_message(): one function for formatting too big
row error messages instead of several.

Author: Sachin Setiya
Author Date: 2019-09-14 07:23:36 UTC

MDEV-20591 Wrong Number of rows in mysqlbinlog output

calc_field_event_length should accurately calculate the size of BLOB type
fields, Instead of returning just the bytes taken by length it should return
length bytes + actual length.

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-13 17:39:06 UTC

MDEV-19529 InnoDB hang on DROP FULLTEXT INDEX

Problem:
=======
  During dropping of fts index, InnoDB waits for fts_optimize_remove_table()
and it holds dict_sys->mutex and dict_operaiton_lock even though the
table id is not present in the queue. But fts_optimize_thread does wait
for dict_sys->mutex to process the unrelated table id from the slot.

Solution:
========
  Whenever table is added to fts_optimize_wq, update the fts_status
of in-memory fts subsystem to TABLE_IN_QUEUE. Whenever drop index
wants to remove table from the queue, it can check the fts_status
to decide whether it should send the MSG_DELETE_TABLE to the queue.

Removed the following functions because these are all deadcode.
dict_table_wait_for_bg_threads_to_exit(),
fts_wait_for_background_thread_to_start(),fts_start_shutdown(), fts_shudown().

Author: Sachin Setiya
Author Date: 2019-09-13 10:37:26 UTC

Fix compile error in macOS caused by MDEV-20477

Author: Sergei Golubchik
Author Date: 2019-09-13 07:52:30 UTC

MDEV-9546 mysqlaccess script shows an old version (which was vulnerable to CVE-2005-0004)

update mysqlaccess version

Author: Alexander Barkov
Author Date: 2019-09-13 07:04:23 UTC

MDEV-18153 Assertion `0' or Assertion `btr_validate_index(index, 0)' failed in row_upd_sec_index_entry or error code 126: Index is corrupted upon UPDATE with TIME_ROUND_FRACTIONAL

Conversion to a temporal data type resulting into a lower precision
depends on TIME_ROUND_FRACTIONAL. Taking into account this dependency in:
- indexed generated virtual column expressions
- persistent virtual column expressions

A warning is now issued if conversion from the generation expression
to the column data type depends on TIME_ROUND_FRACTIONAL.

The warning will be changed to error in 10.5

Author: Sergey Petrunia
Author Date: 2019-09-11 17:22:08 UTC

MDEV-20371: Invalid reads at plan refinement stage: join->positions...

(re-committing in 10.4)
best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).

Author: Jan Lindström
Author Date: 2019-09-12 04:23:24 UTC

Force rebuild.

Author: Nikita Malyavin
Author Date: 2018-06-22 13:26:43 UTC

cherry-pick MDEV-16490

Author: Marko Mäkelä
Author Date: 2019-09-11 13:09:18 UTC

Remove page_copy_rec_list_end_to_created_page()

Also remove the log records MLOG_LIST_END_COPY_CREATED,
MLOG_COMP_LIST_END_COPY_CREATED and
page_parse_copy_rec_list_to_created_page().

This will improve the fill factor of some pages.
Adjust some tests accordingly.

FIXME: Remove remaining references to the function and deduplicate code.

TODO: Implement new reusable function (template?) for insert+reorganize.

Author: Sergey Petrunia
Author Date: 2019-09-10 20:51:42 UTC

MDEV-20371: Invalid reads at plan refinement stage: join->positions...

best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).

Author: Nikita Malyavin
Author Date: 2019-09-10 11:54:31 UTC

MDEV-20143 innodb.innodb-virtual-columns-debug failed in buildbot with wrong result

there was a SELECT/DELETE race

Author: varun
Author Date: 2019-09-10 08:31:31 UTC

Moved the function trace_plan_prefix to the optimizer_trace file

Also improved the comments to the function and did some minor cleanup

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-10 06:04:38 UTC

MDEV-20320 Tablespace flags mismatch for full_crc32 format

Problem:
=======
   Tablespace flag mismatch happen when innodb_compression_algorithm set to different
value during restart for full_crc32 format.

Solution:
========
  Remove the innodb_compression_algorithm check while comparing the table flags and
tablespace flags stored in page0.

Author: Nikita Malyavin
Author Date: 2019-09-03 18:29:03 UTC

MDEV-16039 Crash when selecting virtual columns generated using functions with DAYNAME()

Due to collation specifics an Item tree could change while fixing. Virtual fields.

This should be done on the correct query arena, which is table->expr_arena.

Author: Marko Mäkelä
Author Date: 2019-09-09 11:03:03 UTC

MDEV-20487 Remove the InnoDB adaptive hash index

Based on the performance testing that was conducted in MDEV-17492,
the InnoDB adaptive hash index could only help performance in specific,
almost-read-only workloads. It could slow down all kinds of workloads
(especially DROP TABLE, TRUNCATE TABLE, ALTER TABLE, or DROP INDEX
operations), and it can become corrupted, causing crashes
and possibly data corruption. Furthermore, the adaptive hash index
consumes space from the InnoDB buffer pool, which could hurt performance
when the working set would almost fit in the buffer pool.

Given all this, it is best to remove innodb_adaptive_hash_index.
For now, we will change the default value of the build option
WITH_INNODB_AHI to OFF and adjust tests.

The test rpl.rpl_failed_drop_tbl_binlog depends on DROP TABLE being
able to be interrupted while it is dropping pages from the
adaptive hash index. It will be skipped from now on by default,
but it will be tested for builds WITH_INNODB_AHI=ON.

Author: Alexey Botchkov
Author Date: 2019-09-08 21:35:03 UTC

MDEV-16594 ALTER DATA DIRECTORY in PARTITIONS of InnoDB storage does nothing silently.

Author: Elena Stepanova
Author Date: 2019-09-08 20:58:53 UTC

List of unstable tests for 10.4.8 release (updated)

Author: Sergey Petrunia
Author Date: 2019-09-06 19:11:29 UTC

MDEV-17171: RocksDB Tables do not have "Creation Date"

Support Create_time and Update_time for MyRocks tables.
- Create_time is stored in the MyRocks' internal data dictionary.
- Update_time is in-memory only (like in InnoDB).

Author: midenok
Author Date: 2018-09-18 11:20:09 UTC

SQL: followup misc rename on versioning

vers_sys_type_t -> vers_kind_t

Author: Vladislav Vaintroub
Author Date: 2019-09-05 12:26:38 UTC

Merge branch '10.5' into bb-10.5-wlad

# Conflicts:
# storage/innobase/CMakeLists.txt
# storage/innobase/os/os0file.cc

Author: Jan Lindström
Author Date: 2019-09-04 07:00:29 UTC

Fix incorrect reference to view on truncate.

Author: Jan Lindström
Author Date: 2019-09-04 06:21:07 UTC

MDEV-20485: Galera test failure on galera.galera_var_node_address

Test changes only.

Author: Monty
Author Date: 2019-09-03 22:59:35 UTC

Fix of query cache bug in Aria

MDEV-5817 query cache bug (returning inconsistent/old result
set) with aria table parallel inserts, row format = page

The problem is that for transactional aria tables
(row_type=PAGE and transactional=1), maria_lock_database()
didn't flush the state or the query cache.
Not flushing the state is correct for transactional tables as
this is done by checkpoint, but not flushing the query cache
was wrong and could cause concurrent SELECT queries to not
be deleted from the cache.

Fixed by introducing a flush of the query cache as part of commit, if the table has changed.
t for transactional aria tables (row_type=PAGE and transactional=1), maria_lock_table() didn't flush their state or the query cache.

Author: Eugene
Author Date: 2019-09-03 13:31:10 UTC

[NFC] range-forify loops

Author: Sergei Golubchik
Author Date: 2019-09-03 13:28:32 UTC

MDEV-20079 When setting back the system time while mysqld is running, NOW() and UNIX_TIMESTAMP() results get stuck

typo. system_time.start wasn't updated when system_time.sec
and system_time.sec_part were.

Author: Vladislav Vaintroub
Author Date: 2019-09-03 12:48:36 UTC

Connect : Remove debug cmake output, find packages QUIETly

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-09-03 12:12:53 UTC

- Fixes main.tc_heuristic_recover test case hang.

Author: Monty
Author Date: 2019-09-02 11:06:56 UTC

Merge branch '10.2' into 10.3

Author: Anel Husakovic
Author Date: 2018-04-16 20:49:27 UTC

Enable the auto parametar of the flag `default-character-set`

When invoking option `--default-character-set=auto` character set
from underlying OS settings should be detected for mysqldump.

Author: Alexander Barkov
Author Date: 2019-08-29 08:35:19 UTC

Part2: MDEV-18156 Assertion `0' failed or `btr_validate_index(index, 0, false)' in row_upd_sec_index_entry or error code 126: Index is corrupted upon DELETE with PAD_CHAR_TO_FULL_LENGTH

This patch allows the server to open old tables that have
"bad" generated columns (i.e. indexed virtual generated columns,
persistent generated columns) that depend on sql_mode,
for general things like SELECT, INSERT, DROP, etc.
Warning are issued in such cases.

Only these commands are now disallowed and return an error:
- CREATE TABLE introducing a "bad" generated column
- ALTER TABLE introducing a "bad" generated column
- CREATE INDEX introdicing a "bad" generated column
  (i.e. adding an index on a virtual generated column
   that depends on sql_mode).

Note, these commands are allowed:
- ALTER TABLE removing a "bad" generate column
- ALTER TABLE removing an index from a "bad" virtual generated column
- DROP INDEX removing an index from a "bad" virtual generated column
but only if the table does not have any "bad" columns as a result.

Author: Sachin Setiya
Author Date: 2019-09-01 14:35:07 UTC

19708 compatibility

Author: Kentoku SHIBA
Author Date: 2019-05-28 07:40:59 UTC

MDEV-6268 SPIDER table with no COMMENT clause causes queries to wait forever

Add loop checks by using user variables

Author: Kentoku SHIBA
Author Date: 2019-09-01 20:09:43 UTC

MDEV-6268 SPIDER table with no COMMENT clause causes queries to wait forever

Author: Igor Babaev
Author Date: 2019-09-01 05:44:58 UTC

Merge branch 'mdev-18844' of https://github.com/waynexia/server into mdev-18844

Author: Igor Babaev
Author Date: 2019-08-30 22:49:07 UTC

MDEV-15777 Use inferred IS NOT NULL predicates in the range optimizer

This patch introduces the optimization that allows range optimizer to
consider index range scans that are built employing NOT NULL predicates
inferred from WHERE conditions and ON expressions.
The patch adds a new optimizer switch not_null_range_scan.

Author: Eugene
Author Date: 2019-08-30 13:00:27 UTC

range-forify loops

Author: Nikita Malyavin
Author Date: 2019-08-30 06:22:30 UTC

fix cur_select_lex->where

Author: Teemu Ollakka
Author Date: 2019-07-25 13:41:45 UTC

MDEV-19826 10.4 seems to crash with "pool-of-threads"

MariaDB 10.4 was crashing when thread-handling was set to
pool-of-threads and wsrep was enabled.

There were two apparent reasons for the crash:
- Connection handling in threadpool_common.cc was missing calls to
  control wsrep client state.
- Thread specific storage which contains thread variables (THR_KEY_mysys)
  was not handled appropriately by wsrep patch when pool-of-threads
  was configured.

This patch addresses the above issues in the following way:
- Wsrep client state open/close was moved in thd_prepare_connection() and
  end_connection() to have common handling for one-thread-per-connection
  and pool-of-threads.
- Thread local storage handling in wsrep patch was reworked by introducing
  set of wsrep_xxx_threadvars() calls which replace calls to
  THD store_globals()/reset_globals() and deal with thread handling
  specifics internally.

Wsrep-lib was updated to version which relaxes internal concurrency
related sanity checks.

Rollback code from wsrep_rollback_process() was extracted to separate calls
for better readability.

Post rollback thread was removed as it was completely unused.

Author: Sergey Petrunia
Author Date: 2019-08-29 13:07:15 UTC

MDEV-20109: Optimizer ignores distinct key created for materialized...

Adjust the testcase according to the review input

Author: varun
Author Date: 2019-08-28 20:44:15 UTC

fix for rows of range scan

Author: Sergei Golubchik
Author Date: 2019-08-28 17:21:41 UTC

cleanup: don't invoke dh_installinit from debian/rules

as there's no SysV init script to install anymore

followup for MDEV-15526

Author: Julius Goryavsky
Author Date: 2019-08-28 11:59:34 UTC

MDEV-20420: SST failed after MDEV-18863 in some test configurations

After applying MDEV-18863, in some test configurations, SST
may fails due to duplication of some parameters (in particular
"--port") in the main part of the command line and after
"--mysqld-args", as well as due to incorrect interpretation
of the parameter "--port" passed after "--mysqld-args" when
the SST script is invoked without explicitly specifying a port
for SST. In addition, it is necessary to correctly handle spaces,
quotation marks and special characters when copying original
arguments from the argv[] array to a new command line (after
"--mysqld-args"). This patch resolves these shortcomings.

Author: Julius Goryavsky
Author Date: 2019-08-28 02:49:01 UTC

Improved handling of subdirectories in the xtrabackup-v2 SST scripts (similar to MDEV-18863) for more predictable test results (related to xtrabackup-v2 SST)

Author: Jan Lindström
Author Date: 2019-08-27 14:04:59 UTC

Merge branch 'codership-vote-10.5' into 10.5

Author: Jan Lindström
Author Date: 2019-08-27 05:30:26 UTC

Enable galera_sst_mysqldump_with_key test case.

Author: Eugene
Author Date: 2019-08-26 16:19:12 UTC

fix clang warnings

Author: Sergei Golubchik
Author Date: 2019-08-26 12:49:42 UTC

restore the correct InnoDB auto-inc behavior after the merge

Author: Sergei Golubchik
Author Date: 2019-08-26 12:49:42 UTC

restore the correct InnoDB auto-inc behavior after the merge

Author: Sujatha
Author Date: 2019-08-26 11:43:16 UTC

MDEV-19925: Column ... cannot be converted from type 'varchar(20)' to type 'varchar(20)'

Cherry picking:
Bug#25135304: RBR: WRONG FIELD LENGTH IN ERROR MESSAGE
commit 47bd3f7cf3c8518f62b1580ec65af2ba7ac13b95

Description:
============
In row based replication, when replicating from a table with a field with
character set set to UTF8mb3 to the same table with the same field set to
character set UTF8mb4 I get a confusing error message:

For VARCHAR: VARCHAR(1) 'utf8mb3' to VARCHAR(1) 'utf8mb4'
"Column 0 of table 'test.t1' cannot be converted from type 'varchar(3)' to
type 'varchar(1)'"

Similar issue with CHAR type as well.

Issue with respect to BLOB types:

For BLOB: LONGBLOB to TINYBLOB - Error message displays incorrect blob type.
"Column 0 of table 'test.t1' cannot be converted from type 'tinyblob' to type
'tinyblob'"

For BINARY to BINARY - Error message displays incorrect type for master side
field.
"Column 0 of table 'test.t' cannot be converted from type 'char(1)' to type
'binary(10)'"
Similar issue exists for VARBINARY type. It is displayed as 'VARCHAR'.

Analysis:
=========
In Row based replication charset information is not sent as part of metadata
from master to slave.

For VARCHAR field its character length is converted into equivalent
octets/bytes and stored internally. At the time of displaying the data to user
it is converted back to original character length.

For example:
VARCHAR(2)- utf8mb3 is stored as:2*3 = VARCHAR(6)
At the time of displaying it to user
VARCHAR(6)- charset utf8mb3:6/3= VARCHAR(2).

At present the internally converted octect length is sent from master to slave
with out providing the charset information. On slave side if the type
conversion fails 'show_sql_type' function is used to get the type specific
information from metadata. Since there is no charset information is available
the filed type is displayed as VARCHAR(6).

This results in confused error message.

For CHAR fields
CHAR(1)- utf8mb3 - CHAR(3)
CHAR(1)- utf8mb4 - CHAR(4)

'show_sql_type' function which retrieves type information from metadata uses
(bytes/local charset length) to get actual character length. If slave's chaset
is 'utf8mb4' then

CHAR(3/4)-->CHAR(0)
CHAR(4/4)-->CHAR(1).

This results in confused error message.

Analysis for BLOB type issue:

BLOB's length is represented in two forms.
1. Actual length
i.e
  (length < 256) type= MYSQL_TYPE_TINY_BLOB;
  (length < 65536) type= MYSQL_TYPE_BLOB; ...

2. packlength - The number of bytes used to represent the length of the blob
  1- tinyblob
  2- blob ...

In row based replication only the packlength is written in the binary log. On
the slave side this packlength is interpreted as actual length of the blob.
Hence the length is always < 256 and the type is displayed as tiny blob.

Analysis for BINARY to BINARY type issue:
The character set information is needed to identify a filed's type as char or
binary. Since master side character set information is not available on the
slave side both binary and char fields are displayed as char.

Fix:
===
For CHAR and VARCHAR fields display their length in octets for both source and
target fields. For target field display the charset information if it is
relevant.

For blob type changed the code to use the packlength and display appropriate
blob type in error message.

For binary and varbinary fields use the slave side character set as reference
to map them to binary or varbinary fields.

Author: Andrei Elkin
Author Date: 2019-08-26 11:05:33 UTC

MDEV-742 (intermediate commit)

typ == XID_EVENT || typ == XA_PREPARE_LOG_EVENT augmentation.

Author: Sergei Golubchik
Author Date: 2019-08-25 16:36:59 UTC

fix EXPLAIN output

1. avoid ha_rows->double->ha_rows cast, for ~0ULL it behaves
   differently on Linux and Windows. This fixes the differences
   in derived_view and mdev13607 results between Linux and Windows.
2. EXPLAIN should show the number of rows as an unsigned number.

Author: Roman Nozdrin
Author Date: 2019-08-24 15:42:35 UTC

MDEV-19831 find_select_handler() now tries its best to find a handlerton that is able to processes the whole query. For that it traverses tables from subqueries.

    Select_handler now cleans up temporary table structures on dctor call.

Author: Sergey Petrunia
Author Date: 2019-08-22 21:57:40 UTC

MDEV-20109: Optimizer ignores distinct key created for materialized...

Sj_materialization_picker::check_qep(): fix error in cost/fanout
calculations:
- for each join prefix, add #prefix_rows / TIME_FOR_COMPARE to the cost,
  like best_extension_by_limited_search does
- Remove the fanout produced by the subquery tables.

optimize_wo_join_buffering() (used by LooseScan and FirstMatch)
- also add #prefix_rows / TIME_FOR_COMPARE to the cost of each prefix.

Author: Monty
Author Date: 2019-08-18 09:43:41 UTC

Updated result for rocksdb tests after merge

Author: Anel Husakovic
Author Date: 2019-08-21 17:51:04 UTC

MDEV-20382 SHOW PRIVILEGES displays "Delete versioning rows" rather than "Delete History"

Author: Jan Lindström
Author Date: 2019-08-20 08:55:43 UTC

Merge branch 'codership-vote-10.5' into 10.5

Author: Nikita Malyavin
Author Date: 2019-02-12 11:16:02 UTC

MDEV-16978 Application-time periods: WITHOUT OVERLAPS

* The overlaps check is implemented on a handler level per row command. It creates a separate cursor (actually, another handler instance) and caches it inside the original handler, when ha_update_row or ha_insert_row is issued. Cursor closes on unlocking the handler.

* Containing the same key in index means unique constraint violation even in usual terms. So we fetch left and right neighbours and check that they have same key prefix, excluding from the key only the period part. If it doesnt match, then there's no such neighbour, and the check passes. Otherwise, we check if this neighbour intersects with the considered key.

* the check does introduce new error and fails with ER_DUPP_KEY error. This might break REPLACE workflow and should be fixed separately

Author: Alexander Barkov
Author Date: 2019-08-19 18:42:56 UTC

MDEV-19034 ASAN unknown-crash in get_date_time_separator with PAD_CHAR_TO_FULL_LENGTH

Author: midenok
Author Date: 2019-08-19 11:02:13 UTC

MDEV-20360 Assertion bitmap_is_set(read_partitions) upon updating view of join

Author: Rucha Deodhar
Author Date: 2019-08-17 05:02:13 UTC

new test files for checking datatypes

Author: Robert Bindar
Author Date: 2019-08-16 11:35:54 UTC

MDEV-20298 lidbd-mariadb-perl is not always available

Author: Rasmus Johansson
Author Date: 2019-06-17 07:43:01 UTC

MDEV-19781 Create MariaDB named commands on Windows

Author: Alexander Barkov
Author Date: 2019-08-15 09:16:00 UTC

MDEV-20353 Add separate type handlers for unsigned integer data types

Author: THIRUNARAYANAN BALATHANDAYUTHAPANI
Author Date: 2019-08-15 09:43:15 UTC

MDEV-18128 Simplify .ibd file creation

Problem:
========
During ibd file creation, InnoDB flushes the page0 without crypt
information. During recovery, InnoDB encounters encrypted page read
before initialising the crypt data of the tablespace. So it leads t
corruption of page and doesn't allow innodb to start.

Solution:
=========
Write crypt_data information in page0 while creating .ibd file creation.
During recovery, crypt_data will be initialised while processing
MLOG_FILE_NAME redo log record.

Author: Anel Husakovic
Author Date: 2019-07-31 10:28:38 UTC

MDEV-20210 If you have an INVISIBLE VIRTUAL column, SHOW CREATE TABLE doesn't list it as INVISIBLE

Author: Sergei Golubchik
Author Date: 2019-08-12 16:30:19 UTC

ensure that pam plugin is present in release packages

Author: Sergey Petrunia
Author Date: 2019-08-09 21:53:28 UTC

Optimizer trace: print cost and #rows of the join prefix

The names rows_for_plan and cost_for_plan follow MySQL
Also added post-join-operation selectivity cost