apache settings can yield frequent short outages

While investigating a brief self-resolved alert for one of our cloud mirror regions, I zeroed in on this charm's Apache settings as one possible source of problems.

All of the units in the region are logging frequent scoreboard errors: "AH00288: scoreboard is full, not at MaxRequestWorkers".

Apache is configured with 1 child process which is recycled every 10000 connections:

ubuntu@ip-10-6-64-180:~$ cat /etc/apache2/conf-available/000mpm-worker.conf
# [...]
    StartServers 1
    MinSpareThreads 1280
    MaxSpareThreads 2560
    ThreadLimit 2560
    ThreadsPerChild 2560
    ServerLimit 1
    MaxRequestWorkers 2560
    MaxConnectionsPerChild 10000
ubuntu@ip-10-6-64-180:~$ _

Generally you want MaxRequestWorkers == ServerLimit * ThreadsPerChild, which is the case here.

However, if the traffic is reasonably evenly distributed across all of the backends, they could easily hit their 10,000 request limits around the same time and all be trying to cycle out at the same time, which could lead to haproxy seeing them all down, which would cause these alerts and result in service being briefly unavailable.

For the environment I was looking at, the 10,000 connenction limit per child yields the equivalent of a graceful every 7-8 minutes.

And since there is only one child, which has to wait for pending requests to finish and logging to complete before it can be restarted, meanwhile not allowing empty scoreboard slots to be used for fresh requests, we can easily end up with the main apache process seeing a full scoreboard and yet not have reached MaxRequestWorkers.

The sole child will also cause a similar problem at logrotate time, since that happens at the same time on all units (in the whole world, even!).

I think at least the following needs to be done:

 - review whether MaxConnectionsPerChild is needed at all
 - configure Apache to have more than one child process