Merge lp:~mdcallag/maria/5.3fcntl into lp:maria/5.3

[Mark Callaghan]

This changes mysqld to use socket timeouts rather than non-blocking, setsockopt, blocking, setsockopt. It reduces fcntl calls significantly. Many Linux kernels use the big kernel lock for fcntl and all threads can get stuck on it on a big multi-core server with 500+ concurrent connections.

With this change I double peak QPS on sysbench readonly (100k -> 200k).

The change differs from my original change in the Facebook patch. The original change broke rpl.rpl_ssl. I excluded two lines while porting.

[Michael Widenius (monty)]

Hi!

>>>>> "Mark" == Mark Callaghan <email address hidden> writes:

Mark> Mark Callaghan has proposed merging lp:~mdcallag/maria/5.3fcntl into lp:maria.
Mark> Requested reviews:
Mark> Maria-captains (maria-captains)

Mark> For more details, see:
Mark> https://code.launchpad.net/~mdcallag/maria/5.3fcntl/+merge/51226

Mark> This changes mysqld to use socket timeouts rather than non-blocking, setsockopt, blocking, setsockopt. It reduces fcntl calls significantly. Many Linux kernels use the big kernel lock for fcntl and all threads can get stuck on it on a big multi-core server with 500+ concurrent connections.

Mark> With this change I double peak QPS on sysbench readonly (100k -> 200k).

Mark> The change differs from my original change in the Facebook patch. The original change broke rpl.rpl_ssl. I excluded two lines while porting.

I will look at this during this week. Thanks!

Regards,
Monty

[Michael Widenius (monty)]

Hi!

>>>>> "Mark" == Mark Callaghan <email address hidden> writes:

Mark> Mark Callaghan has proposed merging lp:~mdcallag/maria/5.3fcntl into lp:maria.
Mark> Requested reviews:
Mark> Maria-captains (maria-captains)

Mark> For more details, see:
Mark> https://code.launchpad.net/~mdcallag/maria/5.3fcntl/+merge/51226

Sorry for the delay; I have been very busy reviewing some other
big patches that needs to get into 5.3. (For the moment I don't do
anything else than reviews until all pending reviews are done)
This patch is on my TODO and hope to get to it soon.

At least, we will have time to talk about this in Lisbon.

Regards,
Monty

Unmerged revisions

2921. By Mark Callaghan on 2011-02-24

Port http://bazaar.launchpad.net/~mysqlatfacebook/mysqlatfacebook/5.1/revision/3540. When the
server is compiled with -DNO_ALARM this uses socket timeouts and avoids frequent fcntl calls.
Many Linux kernels use the big kernel lock during fcntl calls so this change can reduce
severe contention on fcntl and it doubles peak QPS for sysbench read-only from ~100k to ~200k.

This diff excludes 2 of the 7 lines changed in the original diff. I think those two lines were
incorrect and they broke the mtr test rpl.rpl_ssl

=== modified file 'sql/net_serv.cc'
--- sql/net_serv.cc	2011-01-03 22:55:41 +0000
+++ sql/net_serv.cc	2011-02-24 22:58:50 +0000
@@ -64,7 +64,7 @@
   can't normally do this the client should have a bigger max_allowed_packet.
 */
 
-#if defined(__WIN__) || !defined(MYSQL_SERVER)
+#if (defined(__WIN__) || !defined(MYSQL_SERVER)) && !defined(NO_ALARM)
   /* The following is because alarms doesn't work on windows. */
 #define NO_ALARM
 #endif
@@ -137,7 +137,7 @@
   if (vio != 0)					/* If real connection */
   {
     net->fd  = vio_fd(vio);			/* For perl DBI/DBD */
-#if defined(MYSQL_SERVER) && !defined(__WIN__)
+#if defined(MYSQL_SERVER) && !defined(__WIN__) && !defined(NO_ALARM)
     if (!(test_flags & TEST_BLOCKING))
     {
       my_bool old_mode;
@@ -630,7 +630,7 @@
     if ((long) (length= vio_write(net->vio,pos,(size_t) (end-pos))) <= 0)
     {
       my_bool interrupted = vio_should_retry(net->vio);
-#if !defined(__WIN__)
+#if !defined(NO_ALARM) && !defined(__WIN__)
       if ((interrupted || length == 0) && !thr_alarm_in_use(&alarmed))
       {
         if (!thr_alarm(&alarmed, net->write_timeout, &alarm_buff))
@@ -686,7 +686,7 @@
     pos+=length;
     update_statistics(thd_increment_bytes_sent(length));
   }
-#ifndef __WIN__
+#if !defined(NO_ALARM) && !defined(__WIN__)
  end:
 #endif
 #ifdef HAVE_COMPRESS
@@ -824,6 +824,7 @@
     thr_alarm(&alarmed,net->read_timeout,&alarm_buff);
 #else
   /* Read timeout is set in my_net_set_read_timeout */
+  DBUG_ASSERT(net_blocking);
 #endif /* NO_ALARM */
 
     pos = net->buff + net->where_b;		/* net->packet -4 */
@@ -838,7 +839,8 @@
 
 	  DBUG_PRINT("info",("vio_read returned %ld  errno: %d",
 			     (long) length, vio_errno(net->vio)));
-#if !defined(__WIN__) || defined(MYSQL_SERVER)
+
+#if !defined(NO_ALARM) && (!defined(__WIN__) || defined(MYSQL_SERVER))
 	  /*
 	    We got an error that there was no data on the socket. We now set up
 	    an alarm to not 'read forever', change the socket to non blocking