90485df...
by
Julius Goryavsky <email address hidden>
MDEV-27737: FreeBSD compatibility changes for SST scripts
This commit fixes problems due to bugs and quirks in bsdtar
(the FreeBSD version of tar). Separate tests are not required,
because without these fixes, many other tests fail when tested
in the FreeBSD environment.
Also, the grep patterns for reading utility version numbers
has been made more robust. The notation of some options of
the "cut" utility has been changed.
2d536e2...
by
Julius Goryavsky <email address hidden>
MDEV-27740: Incorrect SST with non-standard directories for innodb files
Prior to the MDEV-27524 change, a joiner node could not join the cluster
if it stored redo logs in a directory other than the default data directory.
The final version of MDEV-27524 should fix this shortcoming, but we need
new tests to verify correct operation in such scenarios.
1) This commit adds tests to verify that explicitly specifying
all directories for innodb works fine when using Galera with
SST via mariabackup and rsync, i.e. to verify that the previous
changes from the final version of the MDEV-27524 fixed all bugs
that caused incorrect operation of the SST scripts and the
mariabackup utility with innodb directories.
2) To check all this, a new control command has been added
to the mtr, which is specified as a pragma in any of the sections
(groups) in the .cnf configuration files:
#!bootstrap-opts <option1>[, <option2>]...
This feature allows you to list the names of the parameters
located in this section of the .cnf file, which must be converted
into command line options and passed to the input of the bootstrap
process during the construction of the initial base for the next
test run.
This allows you to specify the parameters that should be passed
to the bootstrap process directly in the .cnf files, without creating
separate .opt files (this is especially useful for the multi-node
configurations).
b5c5482...
by
Julius Goryavsky <email address hidden>
MDEV-27524: Incorrect binlogs after Galera SST using rsync and mariabackup
This commit adds correct handling of binlogs for SST using rsync
or mariabackup. Before this fix, binlogs were handled incorrectly -
- only one (last) binary log file was transferred during SST, which
then led to various failures (for example, when trying to list all
events from the binary log). These bugs were long masked by flaws
in the primitive binlogs handling code in the SST scripts, which
causing binary logs files to be erased after transfer or not added
to the binlog index on the joiner node. Now the correct transfer
of all binary logs (not just the last of the binary log files) has
been implemented both for the rsync (at the script level) and for
the mariabackup (at the level of the main utility code).
This commit also adds a new sst_max_binlogs=<n> parameter, which
can be located in the [sst] section or in the [xtrabackup] section
(historically, supported for mariabackup only, not for rsync), or
in one of the server sections. This parameter specifies the number
of binary log files to be sent to the joiner node during SST. This
option is added for compatibility with old SST scripting behavior,
which can be emulated by setting the sst_max_binlogs=1 (although
in general this can cause problems for the reasons described above).
In addition, setting the sst_max_binlogs=0 can be used to suppress
the transmission of binary logs to the joiner nodes during SST
(although sometimes a single file with the current binary log can
still be transmitted to the joiner, even with sst_max_binlogs=0,
because this sometimes necessary in modes that involve the use of
GTIDs with Galera).
Also, this commit ensures correct handling of paths to various
innodb files and directories in the SST scripts, and fixes some
problems with this that existed in mariabackup utility (which
were associated with incorrect handling of the innodb_data_dir
parameter in some scenarios).
In addition, this commit contains the following enhancements:
1) Added tests for mtr, which check the correct work with binlogs
after SST (using rsync and mariabackup);
2) Added correct handling of slashes at the end of all paths that
the SST script receives as parameters;
3) Improved parsing code for --mysqld-args parameters. Now it
correctly processes the sequence "--" after the name of the
one-letter option;
4) Checking the secret signature during joiner authentication
is made independent of presence of bash (as a unix shell)
in the system and diff utility no longer needed to check
certificates compliance;
5) All directories that are necessary for the correct placement
of various logs are automatically created by SST scripts in
advance (before running mariabackup on the joiner node);
6) Removal of old binary logs on joiner is done using the binlog
index (if it exists) (not only by fixed pattern that based
on the current binlog name, as before);
7) Paths for placing binary logs are correctly processed if they
are set as relative paths (to the datadir);
8) SST scripts are made even more resistant to spaces in filenames
(now for binlogs);
9) In case of failure, SST scripts now always end with an exit
code other than zero;
10) SST script for rsync now correctly create a tar file with
the binlogs, even if the paths to them (in the binlog index
file) are specified as a mix of absolute and relative paths,
and even if they do not match with the datadir path specified
in the current configuration settings.
a2d564b...
by
Julius Goryavsky <email address hidden>
MDEV-27602: Automatic creation of working directories in mtr
Our server has many settings for different directory paths:
--datadir=<path>, --log-bin=<path>, --innodb-data-home-dir=<path>
and so on. But we do not have a sufficient number of tests that
would check the operation of these settings in combination with
such complex subsystems as, for example, Galera + mariabackup
(for SST).
Creating such tests can be a non-trivial task. For example,
for a Galera cluster, to change the path for --innodb-data-home-dir,
we need to stop the first node, then create a new directory structure
for the database, restart the first node - but not as the main node
of the cluster, but as a joiner, then repeat the same manipulations
for the second node. Then, after the test, we must destroy the source
files and restart both nodes in sequence with default settings,
otherwise mtr will detect a configuration discrepancy or subsequent
tests will fail. And after that, we must carefully remove all the
"junk" directories that were created during the test.
This not only generates dozens of lines of code in .test files
that need to be maintained when changes are made (for example, if
new database state files or new types of logs are added to innodb),
but also a source of problems when restarting tests.
If such a test fails, its "junk" directory structure is not
completely destroyed. If some directory with the same name is
used in another similar test, this will lead to false failures.
Similarly, when restarting tests. And mtr does not have enough
facilities to detect and destroy such temporary working structures
before starting the test - unless we use system-specific inserts
in the form of .sh scripts, --exec or perl inserts.
Also, mtr does a good job of bootstrap the initial base from
the initial state (from the install.db directory), however if
we are testing a cluster (for example Galera) with non-standard
directories, then we have to actually redo all the work that mtr
does for bootstrap, plus with manually restarting nodes to change
directories.
Therefore, this commit introduces a change to mtr that allows us to
automatically create temporary directories before starting individual
tests and automatically destroy them after the end of the test,
including if the test fails. Also, if such temporary directories
are left over from previous failed test runs, but they are reused
in other tests, they will be automatically destroyed and recreated
before starting a new test.
For each test, a ".dirs" file can now be created that lists the
directories that must be created before starting the test. Then mtr
will automatically create these directories before starting the first
server (if necessary, destroying previous versions left over from
failed previous tests). After the test completes (whether normal or
due to failure), this directory will be automatically deleted by mtr
itself before running the next tests. And all test servers will be
automatically restarted again.
In this case, paths to automatically created directories can use
variable substitution through the standard ${variable_name} construct,
for example, like this: ${MYSQLTEST_VARDIR}/mysqld.1/innodb_data
If the paths to these temporary directories are specified in .cnf
files or in .opt files, then the bootstrap procedure will immediately
create all the database files in the working directories.
For example, this allows you to create tests that check work when
changing paths to some innodb working directories by copying code from
regular tests (which work with standard paths) - without interfering
with the structure of the .test file, only by changing the settings in
the .cnf file copied from regular test. Say, to test the operation of
the Galera cluster with SST through mariabackup in combination with
--innodb-data-home-dir=<non-default-path>, it is enough to copy the
existing test (for example, "galera_sst_mariabackup.test"), add a
".dirs" file for it with two lines (for two nodes):
and then we need to write these paths to .cnf file or .opt file.
And automatically we get a test that will check the operation of the
cluster with a non-standard data directory for innodb. That being said,
we don't need to manually write code to restart the cluster with the
new settings in the .test file, we don't need to repeat the bootstrap
work there, and we don't need to worry about destroying those working
directories at the end of the test.
In addition, lines in the .dirs file may contain an "=" sign after
the name of the temporary directory, followed by the path to another
already existing directory. In this case, mtr will itself copy the
entire subtree of files from this directory to the working directory
(and destroy everything after the test is completed). This will make
it easier to create tests that take some reference files (say from
std_data) and then modify them during the test.
0849012...
by
Julius Goryavsky <email address hidden>
mariabackup: cosmetic changes (whitespaces and indentation)
dfc72c0...
by
Julius Goryavsky <email address hidden>
MDEV-27534: mariabackup: missing line in the compress_qpress.result
This commit adds the missing line to the .result file for compress_qpress
(in the mariabackup test suite).
e777645...
by
Daniel Bartholomew <email address hidden>
MDEV-27746 Wrong comparision of BLOB's empty preffix with non-preffixed BLOB causes rows count mismatch for clustered and secondary indexes during non-locking read
row_sel_sec_rec_is_for_clust_rec() treats empty BLOB prefix field in
secondary index as a field equal to any external BLOB field in clustered
index. Row_sel_get_clust_rec_for_mysql::operator() doesn't zerro out
clustered record pointer in row_search_mvcc(), and row_search_mvcc()
thinks that delete-marked secondary index record has visible for
"CHECK TABLE"'s read view old-versioned clustered index record, and
row_scan_index_for_mysql() counts it as a row.
The fix is to execute row_sel_sec_rec_is_for_blob() in
row_sel_sec_rec_is_for_clust_rec() if clustered field contains BLOB's
reference.
MDEV-25636: Bug report: abortion in sql/sql_parse.cc:6294
The asserion failure was caused by this query
select /*id=1*/ from t1
where
col= ( select /*id=2*/ from ... where corr_cond1
union
select /*id=4*/ from ... where corr_cond2)
Here,
- select with id=2 was correlated due to corr_cond1.
- select with id=4 was initially correlated due to corr_cond2, but then
the optimizer optimized away the correlation, making the select with id=4
uncorrelated.
However, since select with id=2 remained correlated, the execution had to
re-compute the whole UNION. When it tried to execute select with id=4, it
hit an assertion (join buffer already free'd).
This is because select with id=4 has freed its execution structures after
it has been executed once. The select is uncorrelated, so it did not expect
it would need to be executed for the second time.
Fixed this by adding this logic in
st_select_lex::optimize_unflattened_subqueries():
If a member of a UNION is correlated, mark all its members as
correlated, so that they are prepared to be executed multiple times.
