MDEV-21053 Crash safety of foreign key DDL
DDL log extensions (and other stuff):
- New `.file` handler which means treat ddl_log_entry->name and
ddl_log_entry->from_name as file names. Previously they were used
without .frm extension.
- New ddl_log_entry_code: DDL_TRY_LOG_ENTRY_CODE ('t') which means try
log action but not fail the whole chain if action fails. Previously
it failed action chain unconditionally dispite the comment:
/* Write to error log and continue with next log entry */
- Better ddl_log tracing.
New classes:
- ddl_log_info: tracks DDL logging state;
- FK_backup: contains update FRM procedures;
- FK_share_backup, FK_ddl_backup: track (and possibly rollback)
foreign key changes for TABLE_SHARE;
- FK_backup_storage: collection of FK_ddl_backup, can rollback the
whole bunch of shares in memory and FRMs on disk.
Other:
- Test case for multi-rename.
MDEV-21652 Foreign key migration from old version
== fk_check_legacy_storage(), fk_upgrade_legacy_storage() ==
fk_check_legacy_storage() checks whether upgrade is required for the
given table name by looking SYS_FOREIGN table for corresponding record
existence.
fk_upgrade_legacy_storage() does the upgrade routine which includes
getting the foreign keys from SYS_FOREIGN[_COLS], updating the
foreign/referenced shares as well as their FRM files, deleting the
records from SYS_FOREIGN[_COLS] tables.
Both routines utilize the internal SQL for SYS_FOREIGN[_COLS]
processing.
== Upgrade foreign keys via backoff action ==
When table is opened fk_check_legacy_storage() detects whether upgrade
is required and HA_ERR_FK_UPGRADE is returned to SQL layer which then
handles this error by backoff action from Open_table_context where
table is opened again with HA_OPEN_FOR_REPAIR flag which indicates
that fk_upgrade_legacy_storage() is required. After
fk_upgrade_legacy_storage() is done fk_check_legacy_storage() is
checked again to ensure that SYS_FOREIGN[_COLS] are empty for the
given table.
== Check foreign/referenced indexes existence ==
fk_upgrade_legacy_storage() via fk_upgrade_push_fk() fails if there
are no indexes in foreign/referenced tables for the given data
acquired from SYS_FOREIGN[_COLS].
== Internal SQL: select into both func and vars extension ==
fk_upgrade_legacy_storage() utilizes syntax extension in internal SQL:
FETCH c INTO fk_upgrade_create_fk() fk_id, unused;
Thus the data is fetched into both fk_upgrade_create_fk() function and
fk_id variable.
== Rename table, rename column, drop table, drop column handling ==
When foreign table is opened it is automatically upgraded by backoff
action. But if the referenced table is altered or dropped first there
is no chance for the foreign table to get the correct data. So the
SYS_FOREIGN_[COLS] must be kept in sync with the above DDL operations
in respect of referenced names. DROP TABLE for the referenced table is
disabled as usual. DROP TABLE, DROP COLUMN relied in 10.5 on
dict_foreign_t cached data for the referenced tables. Now there is no
such possibility for the legacy data so we have to look at
SYS_FOREIGN_[COLS] directly.
== Reverted some SYS_FOREIGN(_COLS) routines ==
Rename table and rename column handling was done in sync with
SYS_FOREIGN_[COLS] in 10.5. To retain the above DDL consistency for
the referenced tables we still use that old synchronization code.
== ALGORITHM=COPY handling ==
Since we cannot faingrain ALGORITHM=COPY in innobase handler it is
disabled for the referenced tables unless the foreign tables are upgraded.
The check is done in create_table_info_t::create_table() and is
equivalent to DROP TABLE check as we are actually dropping the old
table after the copy routine is done.
== WITH_INNODB_FOREIGN_UPGRADE macro ==
Every SYS_FOREIGN_[COLS] handling is wrapped inside
WITH_INNODB_FOREIGN_UPGRADE compilation macro. When this macro is
disabled the foreign key upgrade is not possible. Future versions will
obsolete the upgrade procedure completely and the wrapped code will be
removed.
== innodb_eval_sql debug interface ==
Test cases must fill SYS_FOREIGN_[COLS] with data. This is done with
setting the new innodb_eval_sql debug variable. The value of that
variable is processed by que_eval_sql().
== Some syntax error-friendly parser handling ==
que_eval_sql() was unfriendly on syntax errors: it just failed with
SIGABRT exception. To keep the server alive some frequent syntax
errors are now returned as DB_ERROR from que_eval_sql().
== Drop empty SYS_FOREIGN[_COLS] after import ==
fk_create_legacy_storage debug flag to create SYS_FOREIGN[_COLS] on
innodb_eval_sql.
fk_release_locks() iterates through all indexes and pages and releases
locks on SUPREMUM records placed there previously by SELECT FOR
UPDATE. Also it releases all table locks.
MDEV-21052 InnoDB foreign key refactoring for TABLE_SHARE::foreign_keys
Refactor dict_load_foreigns() for synchronising TABLE_SHARE foreign
data with dict_table_t cache.
Remove a number of routines working with SYS_FOREIGNS and
SYS_FOREIGN_COLS. innobase_update_foreign_try() is now used solely for
ER_FK_INCORRECT_OPTION check.
Prelock parent tables as well as child tables. This is done for the
case when parent table doesn't know about its children when they
created before parent with foreign_key_checks=0. Opening the parent
table initiates fk_resolve_referenced_keys() which updates its
referenced_keys. Due to CREATE TABLE doesn't not know about "illegal"
children it can not check for foreign consistency. F.ex. this would
succeed:
set foreign_key_checks= 0;
create table child (fk int references parent (id)) engine=innodb;
set foreign_key_checks= 1;
create table parent (id bigint primary key) engine=innodb;
In the above case dict_load_foreigns() deduces which tables are
unknown to opened parent (tables_missing) and reloads their foreign
data via recursion. Infinite recursion is not possible via test case:
a table cannot be "parent after child" and "child before parent"
simultaneously. Though infinite recursion is possible via malicously
crafted FRM file, there is no protection from that at InnoDB level but
there is protection at SQL level: thd->fk_circular_check.
Later though it would not allow DML on child as well as on parent (see
innodb.foreign_key MDEV-10083). So this is pretty acceptable:
foreign_key_checks is unnormal setting, checking parent on CREATE
TABLE would impose all frms scanning which is not acceptable.
ha_innobase::open() then synchronizes these referenced_keys with its
referenced_set cache by calling dict_load_foreigns().
Disable self-references on same column. The Bug 12902967 restricted
them on some condition of "same column/index" (see
innodb_bug12902967.test), though such self-references were not
completely disabled (see other self-ref cases changed in this
patch). It is not clear why they worked if they are "self-refs on same
column/index".
