Merge lp:~ebergen/maria/mysqld_safe_fix into lp:maria

[Eric Bergen]

This contains two changes to mysqld_safe.sh

The first adds an option --crash-script for a script to call when mysqld crashes. This can be useful for gathering extra information, notifying operators, or triggering a failover when using HA.

The second removes a rm command that deleted the sock file. This could delete the sock file for another instance when MYSQL_UNIX_PORT environment variable is set.

This caused problems for me because I use a non-standard installation. Upgrading the MariaDB-Server rpm would delete the sock file from my relocated mysqld leaving it orphaned.

[Michael Widenius (monty)]

Hi!

>>>>> "Eric" == Eric Bergen <email address hidden> writes:

Eric> Eric Bergen has proposed merging lp:~ebergen/maria/mysqld_safe_fix into lp:maria.
Eric> Requested reviews:
Eric> Maria-captains (maria-captains)

Eric> For more details, see:
Eric> https://code.launchpad.net/~ebergen/maria/mysqld_safe_fix/+merge/150210

Eric> This contains two changes to mysqld_safe.sh

Eric> The first adds an option --crash-script for a script to call when mysqld crashes. This can be useful for gathering extra information, notifying operators, or triggering a failover when using HA.

This part is ok.

Eric> The second removes a rm command that deleted the sock file. This could delete the sock file for another instance when MYSQL_UNIX_PORT environment variable is set.

What you removed was:

- rm -f $safe_mysql_unix_port "$pid_file" # Some extra safety

You can't remove the delation of "$pid_file";

Without that mysqld_safe.sh will not properly.

There is one problem with not removing the socket port:
- If you don't do that, mysqld will not restart from a crash as the
old socket file will stop the server from starting.

Eric> This caused problems for me because I use a non-standard installation. Upgrading the MariaDB-Server rpm would delete the sock file from my relocated mysqld leaving it orphaned.

If you specify "socket" in your my.cnf, this should never go wrong.

If you don't specify "socket" and you have another MySQL server
running, your new server should not start as the socket file would be
conflicting.

Can you please describe in litte more detail how your installation
looked like where you got conflicting socket names?

I will wait with applying your crash-script option until we find a
solution for the above.

Regards,
Monty

[Eric Bergen (ebergen)]

On Mon, Feb 25, 2013 at 5:40 AM, Michael Widenius <email address hidden> wrote:
>
> Hi!
>
>>>>>> "Eric" == Eric Bergen <email address hidden> writes:
>
> Eric> Eric Bergen has proposed merging lp:~ebergen/maria/mysqld_safe_fix into lp:maria.
> Eric> Requested reviews:
> Eric> Maria-captains (maria-captains)
>
> Eric> For more details, see:
> Eric> https://code.launchpad.net/~ebergen/maria/mysqld_safe_fix/+merge/150210
>
> Eric> This contains two changes to mysqld_safe.sh
>
> Eric> The first adds an option --crash-script for a script to call when mysqld crashes. This can be useful for gathering extra information, notifying operators, or triggering a failover when using HA.
>
> This part is ok.
>
> Eric> The second removes a rm command that deleted the sock file. This could delete the sock file for another instance when MYSQL_UNIX_PORT environment variable is set.
>
> What you removed was:
>
> - rm -f $safe_mysql_unix_port "$pid_file" # Some extra safety
>
> You can't remove the delation of "$pid_file";
>
> Without that mysqld_safe.sh will not properly.

mysqld_safe does a proper pid file check earlier and fails with an
error message if the stale pid file can't be cleaned up. The server
also opens the pid file with O_TRUNC so it will also get cleaned up.

>
> There is one problem with not removing the socket port:
> - If you don't do that, mysqld will not restart from a crash as the
> old socket file will stop the server from starting.

The server does a blind unlink call on the socket file prior to
creating a new one so it won't fail on a stale socket file.

>
> Eric> This caused problems for me because I use a non-standard installation. Upgrading the MariaDB-Server rpm would delete the sock file from my relocated mysqld leaving it orphaned.
>
> If you specify "socket" in your my.cnf, this should never go wrong.
>
> If you don't specify "socket" and you have another MySQL server
> running, your new server should not start as the socket file would be
> conflicting.
>
> Can you please describe in litte more detail how your installation
> looked like where you got conflicting socket names?

This happens when relocating a mysql instance from the default paths
provided in rpms. It is fairly common to relocate mysql for developers
to use a server with their own instances.

The problem showed up for me when a user was setting
MYSQL_UNIX_PORT=/home/user/path/to/relocated/mysql.sock and then
upgraded the MariaDB-Server package. The mysqld_safe executed during
the rpm installation deleted the relocated mysql_unix_port while
running the installation.

The real fix from my end is to manage MYSQL_UNIX_PORT better or to get
rid of it in favor of ~/.my.cnf. I found it odd that there was a line
with the comment "Some extra safety" implying that it didn't fix any
known issue. It was also doing cleanup that was harmful in my case and
not necessary.

>
> I will wait with applying your crash-script option until we find a
> solution for the above.
>
> Regards,
> Monty
>
> --
> https://code.launchpad.net/~ebergen/maria/mysqld_safe_fix/+merge/150210
> You are the owner of lp:~ebergen/maria/mysqld_safe_fix.

--
Eric Bergen
<email address hidden>
http://www.eb...

[Michael Widenius (monty)]

Hi!

>>>>> "Eric" == Eric Bergen <email address hidden> writes:

Eric> On Mon, Feb 25, 2013 at 5:40 AM, Michael Widenius <email address hidden> wrote:
>>
>> Hi!
>>
>>>>>>> "Eric" == Eric Bergen <email address hidden> writes:
>>
Eric> Eric Bergen has proposed merging lp:~ebergen/maria/mysqld_safe_fix into lp:maria.
Eric> Requested reviews:
Eric> Maria-captains (maria-captains)
>>
Eric> For more details, see:
Eric> https://code.launchpad.net/~ebergen/maria/mysqld_safe_fix/+merge/150210
>>
Eric> This contains two changes to mysqld_safe.sh
>>
Eric> The first adds an option --crash-script for a script to call when mysqld crashes. This can be useful for gathering extra information, notifying operators, or triggering a failover when using HA.
>>
>> This part is ok.
>>
Eric> The second removes a rm command that deleted the sock file. This could delete the sock file for another instance when MYSQL_UNIX_PORT environment variable is set.
>>
>> What you removed was:
>>
>> - rm -f $safe_mysql_unix_port "$pid_file" # Some extra safety
>>
>> You can't remove the delation of "$pid_file";
>>
>> Without that mysqld_safe.sh will not properly.

Eric> mysqld_safe does a proper pid file check earlier and fails with an
Eric> error message if the stale pid file can't be cleaned up. The server
Eric> also opens the pid file with O_TRUNC so it will also get cleaned up.

Yes, however the above 'rm -f' is in the loop that is executed on
restart. So the 'rm' needs to be there so that mysqld_safe can
know if the server did start and/or crash again.

>> There is one problem with not removing the socket port:
>> - If you don't do that, mysqld will not restart from a crash as the
>> old socket file will stop the server from starting.

Eric> The server does a blind unlink call on the socket file prior to
Eric> creating a new one so it won't fail on a stale socket file.

I have seen in the past InnoDB refusing to start because the socket
file existed from before.

This is probably with older InnoDB versions, as the newest one instead
uses a lock on the idbata file.

This means that it's indeed safe to remove the removal of the socket
file in mysqld_safe for newer servers.

Eric> This caused problems for me because I use a non-standard installation. Upgrading the MariaDB-Server rpm would delete the sock file from my relocated mysqld leaving it orphaned.
>>
>> If you specify "socket" in your my.cnf, this should never go wrong.
>>
>> If you don't specify "socket" and you have another MySQL server
>> running, your new server should not start as the socket file would be
>> conflicting.
>>
>> Can you please describe in litte more detail how your installation
>> looked like where you got conflicting socket names?

Eric> This happens when relocating a mysql instance from the default paths
Eric> provided in rpms. It is fairly common to relocate mysql for developers
Eric> to use a server with their own instances.

Eric> The problem showed up for me when a user was setting
Eric> MYSQL_UNIX_PORT=/home/user/path/to/relocated/mysql.sock and then
Eric> upgraded the MariaDB-Server package. The mysqld_safe executed during
Eric> the rpm installation deleted the relocated my...

[Michael Widenius (monty)]

Hi!

>>>>> "Eric" == Eric Bergen <email address hidden> writes:

Eric> On Mon, Feb 25, 2013 at 5:40 AM, Michael Widenius <email address hidden> wrote:
>>
>> Hi!
>>
>>>>>>> "Eric" == Eric Bergen <email address hidden> writes:
>>
Eric> Eric Bergen has proposed merging lp:~ebergen/maria/mysqld_safe_fix into lp:maria.
Eric> Requested reviews:
Eric> Maria-captains (maria-captains)
>>
Eric> For more details, see:
Eric> https://code.launchpad.net/~ebergen/maria/mysqld_safe_fix/+merge/150210
>>
Eric> This contains two changes to mysqld_safe.sh
>>
Eric> The first adds an option --crash-script for a script to call when mysqld crashes. This can be useful for gathering extra information, notifying operators, or triggering a failover when using HA.

<cut>

I have now merged this to my MariaDB 5.5 tree. Will push it as soon as
I have fixed another bug.

Regards,
Monty

Unmerged revisions

3504. By Eric Bergen on 2013-02-24

Add --crash-script and remove a rm call that causes problems when MYSQL_UNIX_PORT is set

=== modified file 'scripts/mysqld_safe.sh'
--- scripts/mysqld_safe.sh	2013-01-15 18:13:32 +0000
+++ scripts/mysqld_safe.sh	2013-02-24 07:05:28 +0000
@@ -71,6 +71,7 @@
   --defaults-extra-file=FILE Also use defaults from the specified file
   --ledir=DIRECTORY          Look for mysqld in the specified directory
   --open-files-limit=LIMIT   Limit the number of open files
+  --crash-script=FILE        Script to call when mysqld crashes
   --timezone=TZ              Set the system timezone
   --malloc-lib=LIB           Preload shared library LIB if available
   --mysqld=FILE              Use the specified file as mysqld
@@ -202,6 +203,7 @@
     optname_subst=`echo "$optname" | sed 's/_/-/g'`
     arg=`echo $arg | sed "s/^$optname/$optname_subst/"`
     case "$arg" in
+      --crash-script=*) CRASH_SCRIPT="$val" ;;
       # these get passed explicitly to mysqld
       --basedir=*) MY_BASEDIR_VERSION="$val" ;;
       --datadir=*|--data=*) DATADIR="$val" ;;
@@ -799,8 +801,6 @@
 
 while true
 do
-  rm -f $safe_mysql_unix_port "$pid_file"	# Some extra safety
-
   start_time=`date +%M%S`
 
   eval_log_error "$cmd"
@@ -874,6 +874,11 @@
     done
   fi
   log_notice "mysqld restarted"
+  if test -n "$CRASH_SCRIPT"
+  then
+    crash_script_output=`$CRASH_SCRIPT 2>&1`
+    log_error "$crash_script_output"
+  fi
 done
 
 log_notice "mysqld from pid file $pid_file ended"