Merge lp:~derks/mysql-mmm/angel-infinit-failures into lp:mysql-mmm

[Pascal Hofmann]

[BJ Dierkes (derks)]

Submitted as a proposed solution for LP #473446, and implements the changes from the patch I posted to that bug.

[Pascal Hofmann (pascalhofmann)]

I think a time condition should be added here. I don't want the angel to restart more than 10 times within 5 minutes, but I definitly want it to restart more than 10 times within hours/days/months.

review: Needs Fixing

[BJ Dierkes (derks)]

> I think a time condition should be added here. I don't want the angel to
> restart more than 10 times within 5 minutes, but I definitly want it to
> restart more than 10 times within hours/days/months.

I agree to that point, though any time there is a successful run the 'attempts' variable is reset to 0. I will look into wrapping the logic around a 'if < 5 minutes since the last time $attempts was reset' check.

[BJ Dierkes (derks)]

I have added the logic to only giveup if there have been 10 consecutive failures within 300 seconds (5 minutes). Please re-review the changes to lib/Common/Angel.pm from my temporary branch (at your convenience).

The result with these changes:

2009/11/17 12:23:18 FATAL Child exited with exitcode 99, restarting after 10 second sleep
2009/11/17 12:23:29 FATAL Listener: Can't create socket!
2009/11/17 12:23:29 FATAL Child exited with exitcode 99, restarting after 10 second sleep
2009/11/17 12:23:40 FATAL Listener: Can't create socket!
2009/11/17 12:23:40 FATAL Child exited with exitcode 99, restarting after 10 second sleep
2009/11/17 12:23:51 FATAL Listener: Can't create socket!
2009/11/17 12:23:51 FATAL Child exited with exitcode 99, restarting after 10 second sleep
2009/11/17 12:24:02 FATAL Listener: Can't create socket!
2009/11/17 12:24:02 FATAL Child exited with exitcode 99, restarting after 10 second sleep
2009/11/17 12:24:13 FATAL Listener: Can't create socket!
2009/11/17 12:24:13 FATAL Child exited with exitcode 99, restarting after 10 second sleep
2009/11/17 12:24:24 FATAL Listener: Can't create socket!
2009/11/17 12:24:24 FATAL Child exited with exitcode 99, restarting after 10 second sleep
2009/11/17 12:24:35 FATAL Listener: Can't create socket!
2009/11/17 12:24:35 FATAL Child exited with exitcode 99, restarting after 10 second sleep
2009/11/17 12:24:46 FATAL Listener: Can't create socket!
2009/11/17 12:24:47 FATAL Child exited with exitcode 99, restarting after 10 second sleep
2009/11/17 12:24:58 FATAL Listener: Can't create socket!
2009/11/17 12:24:58 FATAL Child exited with exitcode 99 and has failed more than 10 times consecutively in the last 5 minutes, not restarting

[Pascal Hofmann (pascalhofmann)]

review: Approve

=== modified file 'lib/Common/Angel.pm'
--- lib/Common/Angel.pm	2009-06-08 23:06:17 +0000
+++ lib/Common/Angel.pm	2010-02-03 09:59:10 +0000
@@ -10,12 +10,17 @@
 
 our $start_process;
 our $pid;
+our $attempts;
+our $starttime;
+
 
 sub Init($) { 
 	my $pidfile = shift;
 
 	
 	$MMM::Common::Angel::start_process	= 1;
+	$MMM::Common::Angel::attempts		= 0;
+	$MMM::Common::Angel::starttime		= time();
 	my $is_shutdown	= 0;
 
 	$pidfile->create() if (defined($pidfile));
@@ -25,6 +30,7 @@
 	local $SIG{QUIT}	= \&MMM::Common::Angel::SignalHandler;
 
 	do {
+		$MMM::Common::Angel::attempts++;
 
 		if ($MMM::Common::Angel::start_process) {
 			$MMM::Common::Angel::start_process = 0;
@@ -41,6 +47,9 @@
 
 		# Wait for child to exit
 		if (waitpid($MMM::Common::Angel::pid, 0) == -1) {
+			# child exited clean, reset attempts and starttime
+			$MMM::Common::Angel::attempts = 0;
+			$MMM::Common::Angel::starttime = time();
 			if ($ERRNO{ECHLD}) {
 				$is_shutdown = 1 unless ($MMM::Common::Angel::start_process);
 			}
@@ -52,8 +61,18 @@
 					$is_shutdown = 1;
 				}
 				else {
-					FATAL sprintf("Child exited with exitcode %s, restarting", WEXITSTATUS($?));
-					$MMM::Common::Angel::start_process	= 1;
+					my $now = time();
+					my $diff = $now - $MMM::Common::Angel::starttime;
+					if ($MMM::Common::Angel::attempts >= 10 && $diff < 300) {
+						FATAL sprintf("Child exited with exitcode %s and has failed more than 10 times consecutively in the last 5 minutes, not restarting", WEXITSTATUS($?));
+						$MMM::Common::Angel::start_process	= 0;
+						$is_shutdown = 1;
+					}
+					else {
+						FATAL sprintf("Child exited with exitcode %s, restarting after 10 second sleep", WEXITSTATUS($?));
+						sleep(10);
+						$MMM::Common::Angel::start_process	= 1;
+					}
 				}
 			}
 			if (WIFSIGNALED($CHILD_ERROR)) {