Merge lp:~hloeung/ubuntu-repository-cache/auto-apache2-mpm-tuning into lp:ubuntu-repository-cache

Proposed by Haw Loeung on 2020-10-28
Status: Merged
Approved by: Haw Loeung on 2020-11-02
Approved revision: 296
Merged at revision: 289
Proposed branch: lp:~hloeung/ubuntu-repository-cache/auto-apache2-mpm-tuning
Merge into: lp:ubuntu-repository-cache
Diff against target: 303 lines (+200/-18)
5 files modified
config.yaml (+15/-15)
lib/ubuntu_repository_cache/apache.py (+40/-0)
lib/ubuntu_repository_cache/tests/test_apache.py (+141/-0)
templates/apache2/mpm_worker.template (+3/-2)
tox.ini (+1/-1)
To merge this branch: bzr merge lp:~hloeung/ubuntu-repository-cache/auto-apache2-mpm-tuning
Reviewer Review Type Date Requested Status
Stuart Bishop 2020-10-28 Approve on 2020-11-02
Barry Price 2020-10-28 Approve on 2020-11-02
Review via email: mp+392920@code.launchpad.net

Commit message

Automatically calculate MPM worker tuning configs

Description of the change

Currently, these are the flavors/instance types generally used:

GCE - https://cloud.google.com/compute/docs/machine-types#n1_machine_types
n1-standard-16 = 16 vCPUs, 60GiB RAM

AWS - https://aws.amazon.com/ec2/instance-types/#Compute_Optimized
c5.xlarge = 4 vCPUs, 8GiB RAM
c3.4xlarge = 16 vCPUs, 30GiB RAM
c4.8xlarge = 36 vCPUs, 60GiB RAM

Azure - https://docs.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series
Standard_D2_v2 = 2 vCPUs, 7GiB RAM
Standard_D3_v2 = 4 vCPUs, 14GiB RAM

To post a comment you must log in.

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

296. By Haw Loeung on 2020-11-02

Fixed comments

Barry Price (barryprice) wrote :

LGTM +1

review: Approve
Stuart Bishop (stub) wrote :

This all looks great

review: Approve

Change successfully merged at revision 289

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'config.yaml'
2--- config.yaml 2020-09-30 04:32:54 +0000
3+++ config.yaml 2020-11-02 03:11:48 +0000
4@@ -111,37 +111,37 @@
5 description: Select the worker or prefork multi-processing module
6 apache2_mpm_startservers:
7 type: int
8- default: 2
9- description: Initial number of server processes to start
10+ default: 0
11+ description: Initial number of server processes to start (0 means auto calculate)
12 apache2_mpm_minsparethreads:
13 type: int
14- default: 100
15- description: Minimum number of worker threads which are kept spare
16+ default: 0
17+ description: Minimum number of worker threads which are kept spare (0 means auto calculate)
18 apache2_mpm_maxsparethreads:
19 type: int
20- default: 200
21- description: Maximum number of worker threads which are kept spare
22+ default: 0
23+ description: Maximum number of worker threads which are kept spare (0 means auto calculate)
24 apache2_mpm_threadlimit:
25 type: int
26- default: 64
27+ default: 0
28 description: |
29 Sets the upper limit on the configurable number of threads per
30- child process
31+ child process (0 means autocalculate)
32 apache2_mpm_threadsperchild:
33 type: int
34- default: 64
35- description: Constant number of worker threads in each server process
36+ default: 0
37+ description: Constant number of worker threads in each server process (0 means autocalculate)
38 apache2_mpm_serverlimit:
39 type: int
40- default: 256
41- description: Upper limit on configurable number of processes
42+ default: 0
43+ description: Upper limit on configurable number of processes (0 means autocalculate)
44 apache2_mpm_maxrequestworkers:
45 type: int
46- default: 16384
47- description: Maximum number of simultaneous client connections
48+ default: 0
49+ description: Maximum number of simultaneous client connections (0 means autocalculate)
50 apache2_mpm_maxconnectionsperchild:
51 type: int
52- default: 0
53+ default: 10000
54 description: Maximum number of requests a server process serves
55 logrotate_rotate:
56 type: string
57
58=== modified file 'lib/ubuntu_repository_cache/apache.py'
59--- lib/ubuntu_repository_cache/apache.py 2020-10-06 10:04:18 +0000
60+++ lib/ubuntu_repository_cache/apache.py 2020-11-02 03:11:48 +0000
61@@ -1,5 +1,6 @@
62 '''Functions related to apache use within this charm'''
63
64+import multiprocessing
65 import os
66 import subprocess
67 import json
68@@ -71,6 +72,44 @@
69 return host.service_running(SERVICE)
70
71
72+def tune_mpm_configs(config):
73+ cpu_count = multiprocessing.cpu_count()
74+
75+ if not config['startservers']:
76+ config['startservers'] = 1
77+
78+ if not config['serverlimit']:
79+ # Benchmarking shows single process multiple threads performs better
80+ # than multiple processes multiple threads.
81+ config['serverlimit'] = 1
82+
83+ if not config['threadsperchild']:
84+ # We could take into consideration how much memory the system
85+ # has but the ratio of vCPU to RAM between public clouds seems
86+ # good enough so this is not needed.
87+ config['threadsperchild'] = 256 * cpu_count
88+ # Cap at 8192
89+ if config['threadsperchild'] > 8192:
90+ config['threadsperchild'] = 8192
91+
92+ if not config['threadlimit']:
93+ # Hard limit should match per child limit.
94+ config['threadlimit'] = config['threadsperchild']
95+
96+ if not config['maxrequestworkers']:
97+ # Calculate max. workers based on serverlimit and threadsperchild.
98+ config['maxrequestworkers'] = config['serverlimit'] * config['threadsperchild']
99+
100+ if not config['minsparethreads']:
101+ # Use 50% of max. workers.
102+ config['minsparethreads'] = int(config['maxrequestworkers'] * 0.5)
103+
104+ if not config['maxsparethreads']:
105+ config['maxsparethreads'] = config['maxrequestworkers']
106+
107+ return config
108+
109+
110 def create_mpm_workerfile():
111 '''Create the multi-processing module (MPM) configuration'''
112
113@@ -85,6 +124,7 @@
114 'maxrequestworkers': config['apache2_mpm_maxrequestworkers'],
115 'maxconnectionsperchild': config['apache2_mpm_maxconnectionsperchild'],
116 }
117+ mpm_context = tune_mpm_configs(mpm_context)
118 conf_mpm_worker = '/etc/apache2/conf-available/000mpm-worker.conf'
119 templating.render('apache2/mpm_worker.template', conf_mpm_worker, mpm_context)
120 subprocess.check_call(['a2enconf', '000mpm-worker'])
121
122=== added file 'lib/ubuntu_repository_cache/tests/test_apache.py'
123--- lib/ubuntu_repository_cache/tests/test_apache.py 1970-01-01 00:00:00 +0000
124+++ lib/ubuntu_repository_cache/tests/test_apache.py 2020-11-02 03:11:48 +0000
125@@ -0,0 +1,141 @@
126+import os
127+import shutil
128+import tempfile
129+import unittest
130+from unittest import mock
131+
132+from .. import apache
133+
134+from charmhelpers.core import unitdata
135+
136+
137+class ApacheTestCase(unittest.TestCase):
138+ def setUp(self):
139+ self.maxDiff = None
140+ patcher = mock.patch('charmhelpers.core.hookenv.config')
141+ self.mock_config = patcher.start()
142+ self.addCleanup(patcher.stop)
143+
144+ self.tmpdir = tempfile.mkdtemp(prefix='charm-unittests-')
145+ self.addCleanup(shutil.rmtree, self.tmpdir)
146+ os.environ['UNIT_STATE_DB'] = os.path.join(self.tmpdir, '.unit-state.db')
147+ unitdata.kv().set('squid-cache-disk', 5000)
148+ unitdata.kv().set('squid-disk-caches', [('/srv', 2000)])
149+
150+ @mock.patch('multiprocessing.cpu_count')
151+ def test_tune_mpm_configs(self, cpu_count):
152+
153+ cpu_count.return_value = 4
154+ config = {
155+ 'startservers': 0,
156+ 'minsparethreads': 0,
157+ 'maxsparethreads': 0,
158+ 'threadlimit': 0,
159+ 'threadsperchild': 0,
160+ 'serverlimit': 0,
161+ 'maxrequestworkers': 0,
162+ 'maxconnectionsperchild': 10000,
163+ }
164+ expected = {
165+ 'startservers': 1,
166+ 'minsparethreads': 512,
167+ 'maxsparethreads': 1024,
168+ 'threadlimit': 1024,
169+ 'threadsperchild': 1024,
170+ 'serverlimit': 1,
171+ 'maxrequestworkers': 1024,
172+ 'maxconnectionsperchild': 10000,
173+ }
174+ self.assertEqual(apache.tune_mpm_configs(config), expected)
175+
176+ cpu_count.return_value = 40
177+ config = {
178+ 'startservers': 0,
179+ 'minsparethreads': 0,
180+ 'maxsparethreads': 0,
181+ 'threadlimit': 0,
182+ 'threadsperchild': 0,
183+ 'serverlimit': 0,
184+ 'maxrequestworkers': 0,
185+ 'maxconnectionsperchild': 10000,
186+ }
187+ expected = {
188+ 'startservers': 1,
189+ 'minsparethreads': 4096,
190+ 'maxsparethreads': 8192,
191+ 'threadlimit': 8192,
192+ 'threadsperchild': 8192,
193+ 'serverlimit': 1,
194+ 'maxrequestworkers': 8192,
195+ 'maxconnectionsperchild': 10000,
196+ }
197+ self.assertEqual(apache.tune_mpm_configs(config), expected)
198+
199+ cpu_count.return_value = 1
200+ config = {
201+ 'startservers': 0,
202+ 'minsparethreads': 0,
203+ 'maxsparethreads': 0,
204+ 'threadlimit': 0,
205+ 'threadsperchild': 0,
206+ 'serverlimit': 0,
207+ 'maxrequestworkers': 0,
208+ 'maxconnectionsperchild': 10000,
209+ }
210+ expected = {
211+ 'startservers': 1,
212+ 'minsparethreads': 128,
213+ 'maxsparethreads': 256,
214+ 'threadlimit': 256,
215+ 'threadsperchild': 256,
216+ 'serverlimit': 1,
217+ 'maxrequestworkers': 256,
218+ 'maxconnectionsperchild': 10000,
219+ }
220+ self.assertEqual(apache.tune_mpm_configs(config), expected)
221+
222+ cpu_count.return_value = 2
223+ config = {
224+ 'startservers': 0,
225+ 'minsparethreads': 0,
226+ 'maxsparethreads': 0,
227+ 'threadlimit': 0,
228+ 'threadsperchild': 0,
229+ 'serverlimit': 0,
230+ 'maxrequestworkers': 0,
231+ 'maxconnectionsperchild': 10000,
232+ }
233+ expected = {
234+ 'startservers': 1,
235+ 'minsparethreads': 256,
236+ 'maxsparethreads': 512,
237+ 'threadlimit': 512,
238+ 'threadsperchild': 512,
239+ 'serverlimit': 1,
240+ 'maxrequestworkers': 512,
241+ 'maxconnectionsperchild': 10000,
242+ }
243+ self.assertEqual(apache.tune_mpm_configs(config), expected)
244+
245+ cpu_count.return_value = 4
246+ config = {
247+ 'startservers': 5,
248+ 'minsparethreads': 10,
249+ 'maxsparethreads': 100,
250+ 'threadlimit': 32,
251+ 'threadsperchild': 32,
252+ 'serverlimit': 10,
253+ 'maxrequestworkers': 8192,
254+ 'maxconnectionsperchild': 5000,
255+ }
256+ expected = {
257+ 'startservers': 5,
258+ 'minsparethreads': 10,
259+ 'maxsparethreads': 100,
260+ 'threadlimit': 32,
261+ 'threadsperchild': 32,
262+ 'serverlimit': 10,
263+ 'maxrequestworkers': 8192,
264+ 'maxconnectionsperchild': 5000,
265+ }
266+ self.assertEqual(apache.tune_mpm_configs(config), expected)
267
268=== modified file 'templates/apache2/mpm_worker.template'
269--- templates/apache2/mpm_worker.template 2015-02-12 00:05:14 +0000
270+++ templates/apache2/mpm_worker.template 2020-11-02 03:11:48 +0000
271@@ -1,5 +1,5 @@
272 #
273-# JUJU WARNING: This file is managed by Juju, do NOT edit
274+# JUJU WARNING: This file is managed by Juju, do NOT edit
275 #
276
277 # worker MPM
278@@ -12,10 +12,11 @@
279 <IfModule mpm_worker_module>
280 StartServers {{ startservers }}
281 MinSpareThreads {{ minsparethreads }}
282- MaxSpareThreads {{ maxsparethreads }}
283+ MaxSpareThreads {{ maxsparethreads }}
284 ThreadLimit {{ threadlimit }}
285 ThreadsPerChild {{ threadsperchild }}
286 ServerLimit {{ serverlimit }}
287 MaxRequestWorkers {{ maxrequestworkers }}
288 MaxConnectionsPerChild {{ maxconnectionsperchild }}
289 </IfModule>
290+
291
292=== modified file 'tox.ini'
293--- tox.ini 2020-08-11 04:12:28 +0000
294+++ tox.ini 2020-11-02 03:11:48 +0000
295@@ -10,7 +10,7 @@
296 [testenv:unit]
297 commands =
298 pytest --ignore {toxinidir}/tests/functional \
299- {posargs:-v --cov=lib --cov=reactive --cov=actions --cov-report=term-missing --cov-branch}
300+ {posargs:-v --cov=lib --cov=reactive --cov-report=term-missing --cov-branch}
301 deps = -r{toxinidir}/tests/unit/requirements.txt
302 -r{toxinidir}/requirements.txt
303 setenv =

Subscribers

People subscribed via source and target branches