Merge lp:~camptocamp/anybox.recipe.openerp/20121204-add-daemon into lp:anybox.recipe.openerp

Proposed by Yannick Vaucher @ Camptocamp on 2012-12-04
Status: Needs review
Proposed branch: lp:~camptocamp/anybox.recipe.openerp/20121204-add-daemon
Merge into: lp:anybox.recipe.openerp
Diff against target: 437 lines (+319/-11)
7 files modified
anybox/recipe/openerp/base.py (+4/-1)
anybox/recipe/openerp/server.py (+124/-10)
anybox/recipe/openerp/templates/__init__.py (+1/-0)
anybox/recipe/openerp/templates/initd/__init__.py (+1/-0)
anybox/recipe/openerp/templates/initd/gunicorn_openerp_server.initd.in (+109/-0)
anybox/recipe/openerp/templates/initd/openerp_server.initd.in (+79/-0)
setup.py (+1/-0)
To merge this branch: bzr merge lp:~camptocamp/anybox.recipe.openerp/20121204-add-daemon
Reviewer Review Type Date Requested Status
Anybox 2012-12-04 Pending
Review via email: mp+137773@code.launchpad.net

Description of the change

Here is a merge proposal to insert the daemonization of openerp

It generates the following dirs and files:

in buildout dir:
- run (for pid files)
- init.d (for daemon launchers)
- log (for logs)

in init.d:
- openerp_server.initd
- gunicorn_openerp_server.initd

Those executable scripts are defined via templates in server.py of the recipe

New params in buildout config :
daemon.sys_user (used to launch daemon with a specific user if not set use current user)
daemon.current_instance (gives an instance name used to name pid file and daemon description)

Maybe this should be more optional or more generic ?
What is sure is that on our side in Camptocamp, we needs to be able to run openerp in daemon mode

To post a comment you must log in.
Georges Racinet (gracinet) wrote :

Just did a really quick read. That looks overall good, but I (we) will need a bit more time to review/merge that one.

About the use-case itself, at Anybox, we use supervisor to handle daemonization, console logs and startup. That explains the lack of such in the recipe. We have self-contained instances, with a supervisor part with the dedicated recipe as well as mutualizing servers, where a distribution-installed supervisor takes care of several buildout-built instances.

In the future, we may switch to circus (looks promising). Some other people may use other supervision processes, so I think we should maybe introduce a class of options for that purpose, in order to avoid confusion with too many different configuration files/scripts being generated.

Georges Racinet (gracinet) wrote :

How do you plan to handle the script (cron_worker_openerp) that handles ir.cron jobs ?

This is tricky, because that script needs a (list of) database names, and we already have use cases where limitation to thoses databases belonging to the system user won't be enough.

Init/update of the database through the recipe is not far behind that one. For the time being, we can live with a simple option to declare the database, even if that's a bit odd (bin/buildout would generate a rc script that can't work until the listed databases have been created).

We had long internal discussions about these issues, with no clear consensus. c2c's input would be appreciated.

Georges Racinet (gracinet) wrote :

Finally, this will require updates to README/CHANGES and, of course, new tests. About these, in case test_server.py isn't clear enough or appropriate, I can help.

Georges Racinet (gracinet) wrote :

Thank you, it's great to see new use cases and momentum.

We don't need to solve all the concerns expressed in the above comments for the contribution to be mergeable. Reaching a consistent state with room for further developments will be enough.

I won't have much more time for this today, but will check for new inputs at least tomorrow.

Hello,

My colleague Yannick will have a look on your remarks, though not sure he will have enough time today.

By the way, we may consider using a supervisor.
Is it actually a part of the buildout? I can see that the supervisor egg is in the recipes of the buildbot but do not find any configuration.

Why not to use collective.recipe.supervisor ?

Subsidiary question: Can we have more than one buildout deployed on 1 server, each one of them having their own self-contained supervisor ?

Christophe Combelles (ccomb) wrote :

Yes for a single instance we use collective.recipe.supervisor, that's perfect for quickly setting up a supervisor. For example we used it for a complex project with a libreoffice server (aeroo report), a django frontend with a cherrypy server and a Celery server for asynchronous jobs.
We use a single system-wide supervisor to manage multiple OpenERP instances from several buildouts.

Georges Racinet (gracinet) wrote :

> By the way, we may consider using a supervisor.
> Is it actually a part of the buildout? I can see that the supervisor egg is in
> the recipes of the buildbot but do not find any configuration.

That egg is indeed listed in [versions] of the buildbot/*.cfg, you are right. But it's a remnant from another configuration that had a supervisor part, it's not actually used.

Note that egg versions are shared among parts.

>
> Why not to use collective.recipe.supervisor ?

That's what we do as "local supervisors", sorry if I haven't been clear enough about that.

It wouldn't be the job of the openerp recipe to install supervisor.
A standalone, ready to go daemonized version with an example init script is interesting, especially if it needs some OpenERP specifics (like database declaration), meaning that it has a clear added value over an external approach, but there's a line to be drawn where the recipe responsibility should end, and that should be imho at the point where other recipes (supervisor) or the general OS environment (systemd/upstart/whatever) can interface themselves easily.

> Subsidiary question: Can we have more than one buildout deployed on 1 server,
> each one of them having their own self-contained supervisor ?

Yes, you probably can, if you take care to have them use separate control sockets, but I've not done this myself. I suppose that the supervisor recipe config defaults are sensible (in particular, Unix domain socket file should be local).

Then of course, you'll need an init/systemd/upstart to launch all these supervisors. The local supervisor is handy to multiplex all the needed subprocesses for that openerp installation (gunicorn, cron_worker, specific stuff) as one "service".

We prefer to use the supervisor provided by the distro with a process group per buildout, because rc script is provided out of the box. That implies that access to the web UI must be restricted to our staff (fine for us).

Thanks for all your explanations. I totally understand and agree.

I found an example a local supervisord setup using collective.recipe.supervisor
https://github.com/florentx/openobject-mirliton/blob/buildout/etc/buildout-base.cfg

This seems really good.

We'll do some tests and share the results. It would be nice then to have some documentation with an example of configuration. If our experiments are fine, we'll write it.

Georges Racinet (gracinet) wrote :

Hi,

we are on the verge of releasing version 1.4, mainly for the extraction-freeze features.

What are the results of your investigations ? Do you still need this ? Can it wait until, e.g, a point release or the next major release (1.5).

Regards,

Yes, we can wait on this. Finally we created a configuration using supervisor and though it isn't used in production yet we are moving to it for our next projects.

Unmerged revisions

250. By Yannick Vaucher @ Camptocamp on 2012-12-04

[ADD] daemon launcher for standalone and gunicorn mode

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'anybox/recipe/openerp/base.py'
2--- anybox/recipe/openerp/base.py 2012-11-20 10:25:02 +0000
3+++ anybox/recipe/openerp/base.py 2012-12-04 08:56:24 +0000
4@@ -57,6 +57,7 @@
5
6 self.downloads_dir = self.make_absolute(
7 self.b_options.get('openerp-downloads-directory', 'downloads'))
8+ self.daemon_dir = self.make_absolute('init.d')
9 self.version_wanted = None # from the buildout
10 self.version_detected = None # string from the openerp setup.py
11 self.parts = self.buildout['buildout']['parts-directory']
12@@ -81,9 +82,11 @@
13 self.openerp_installed = []
14
15 self.etc = self.make_absolute('etc')
16+ self.log_dir = self.make_absolute('log')
17+ self.run_dir = self.make_absolute('run')
18 self.bin_dir = self.buildout['buildout']['bin-directory']
19 self.config_path = join(self.etc, self.name + '.cfg')
20- for d in self.downloads_dir, self.etc:
21+ for d in self.downloads_dir, self.etc, self.daemon_dir, self.log_dir, self.run_dir:
22 if not os.path.exists(d):
23 logger.info('Created %s/ directory' % basename(d))
24 os.mkdir(d)
25
26=== modified file 'anybox/recipe/openerp/server.py'
27--- anybox/recipe/openerp/server.py 2012-11-29 17:10:20 +0000
28+++ anybox/recipe/openerp/server.py 2012-12-04 08:56:24 +0000
29@@ -1,14 +1,21 @@
30 # coding: utf-8
31 import os
32+import stat
33+import getpass
34 from os.path import join
35-import sys, logging
36+import sys
37+import logging
38 import subprocess
39 import zc.buildout
40+from mako.template import Template
41 from anybox.recipe.openerp import devtools
42 from anybox.recipe.openerp.base import BaseRecipe
43
44+from pkg_resources import resource_string
45+
46 logger = logging.getLogger(__name__)
47
48+
49 class ServerRecipe(BaseRecipe):
50 """Recipe for server install and config
51 """
52@@ -101,6 +108,112 @@
53
54 BaseRecipe.merge_requirements(self)
55
56+ def _create_daemon_launcher(self):
57+ """Generate daemon laucher in a init.d directory.
58+ """
59+ template = resource_string(__name__, join('templates', 'initd','openerp_server.initd.in'))
60+
61+ daemon_options = {}
62+ daemon_prefix = 'daemon.'
63+ daemon_options.update((k[len(daemon_prefix):], v)
64+ for k, v in self.options.items()
65+ if k.startswith(daemon_prefix))
66+
67+ system_user = daemon_options.get('sys_user')
68+ current_instance = daemon_options.get('current_instance')
69+ daemon_params = {
70+ 'path': '/sbin:/bin:/usr/sbin:/usr/bin',
71+ 'pypath': '/usr/bin/python',
72+ 'daemon': self.script_path,
73+ 'config': self.config_path,
74+ 'name': 'openerp-server',
75+ 'pid': join(self.buildout_dir, 'run', current_instance + '.pid'),
76+ 'desc': 'openerp-server-' + current_instance,
77+ 'logfile': join(self.log_dir, 'server.log'),
78+ 'user': system_user,
79+ }
80+
81+ template = template % daemon_params
82+
83+ daemon_file_name = join(self.daemon_dir, 'openerp_server.initd')
84+
85+ new_file = open(daemon_file_name, 'w')
86+ new_file.write(template)
87+ new_file.close()
88+
89+ os.chmod(daemon_file_name, stat.S_IRWXU)
90+ self.openerp_installed.append(daemon_file_name)
91+
92+ def _get_gunicorn_entry_point(self, gunicorn_options):
93+ # entry point in 7.0 has changed so we need it to be configurable
94+ gunicorn_entry_point = gunicorn_options.get('entry_point')
95+ if gunicorn_entry_point is None:
96+ if self.major_version >= (6, 2):
97+ # proxy vs direct now handled by an OpenERP server option
98+ gunicorn_entry_point = 'openerp:service.wsgi_server.application'
99+ else:
100+ gunicorn_entry_point = (
101+ 'openerp:wsgi.%s.application' % self.gunicorn_entry)
102+ return gunicorn_entry_point
103+
104+
105+ def _create_gunicorn_daemon_launcher(self):
106+ """Generate daemon laucher in a init.d directory with gunicorn support.
107+ """
108+ template = resource_string(__name__, join('templates', 'initd','gunicorn_openerp_server.initd.in'))
109+
110+ daemon_options = {}
111+ gunicorn_options = {}
112+ daemon_prefix = 'daemon.'
113+ gunicorn_prefix = 'gunicorn.'
114+
115+ daemon_options.update((k[len(daemon_prefix):], v)
116+ for k, v in self.options.items()
117+ if k.startswith(daemon_prefix))
118+
119+ system_user = daemon_options.get(
120+ 'sys_user',
121+ getpass.getuser())
122+ current_instance = daemon_options.get('current_instance')
123+
124+ gunicorn_options.update((k[len(gunicorn_prefix):], v)
125+ for k, v in self.options.items()
126+ if k.startswith(gunicorn_prefix))
127+ gunicorn_entry_point = self._get_gunicorn_entry_point(gunicorn_options)
128+
129+ database = self.options.get('options.db_name')
130+ version = self.version_detected
131+ daemon_params = {
132+ 'path': '/sbin:/bin:/usr/sbin:/usr/bin',
133+ 'pypath': '/usr/bin/python',
134+ 'daemon': join(self.bin_dir, 'gunicorn_openerp'),
135+ 'config': join(self.etc, 'gunicorn_openerp.conf.py'),
136+ 'app_module': gunicorn_entry_point,
137+ 'name': 'gunicorn-openerp-server',
138+ 'desc': 'gunicorn-openerp-server-' + current_instance,
139+ 'pid': join(self.buildout_dir, 'run', 'gunicorn_' + current_instance + '.pid'),
140+ 'logfile': join(self.log_dir, 'server.log'),
141+ 'user': system_user,
142+
143+ 'cron_pid': join(self.buildout_dir, 'run', 'cron_worker_' + current_instance + '.pid'),
144+ 'cron_daemon': join(self.bin_dir, 'cron_worker_openerp'),
145+ 'cron_daemon_opts': '--database=' + database,
146+ 'cron_config': self.config_path,
147+ 'version': version,
148+ }
149+
150+ template = Template(template).render(**daemon_params)
151+
152+ daemon_file_name = join(self.daemon_dir, 'gunicorn_openerp_server.initd')
153+
154+ new_file = open(daemon_file_name, 'w')
155+ new_file.write(template)
156+ new_file.close()
157+
158+ os.chmod(daemon_file_name, stat.S_IRWXU)
159+ self.openerp_installed.append(daemon_file_name)
160+
161+
162 def _create_default_config(self):
163 """Have OpenERP generate its default config file.
164 """
165@@ -135,11 +248,13 @@
166 if k.startswith(gunicorn_prefix))
167
168 f = open(join(self.etc, qualified_name + '.conf.py'), 'w')
169+ pidfile = join(self.run_dir, qualified_name + '.pid')
170+ gunicorn_options.update({'pidfile': pidfile})
171 conf = """'''Gunicorn configuration script.
172 Generated by buildout. Do NOT edit.'''
173 import openerp
174 bind = %(bind)r
175-pidfile = %(qualified_name)r + '.pid'
176+pidfile = %(pidfile)r
177 workers = %(workers)s
178
179 if openerp.release.major_version == '6.1':
180@@ -191,14 +306,7 @@
181 for k, v in self.options.items()
182 if k.startswith(gunicorn_prefix))
183
184- gunicorn_entry_point = gunicorn_options.get('entry_point')
185- if gunicorn_entry_point is None:
186- if self.major_version >= (6, 2):
187- # proxy vs direct now handled by an OpenERP server option
188- gunicorn_entry_point = 'openerp:service.wsgi_server.application'
189- else:
190- gunicorn_entry_point = (
191- 'openerp:wsgi.%s.application' % self.gunicorn_entry)
192+ gunicorn_entry_point = self._get_gunicorn_entry_point(gunicorn_options)
193
194 # gunicorn's main() does not take arguments, that's why we have
195 # to resort on hacking sys.argv
196@@ -345,6 +453,9 @@
197
198 self._install_main_startup_script()
199
200+ if os.name == 'posix':
201+ self._create_daemon_launcher()
202+
203 if self.with_openerp_command:
204 self._install_openerp_command(
205 self.options.get('openerp_command_name',
206@@ -363,6 +474,9 @@
207 'cron_worker_%s' % self.name)
208 self._install_cron_worker_startup_script(qualified_name)
209
210+ if os.name == 'posix':
211+ self._create_gunicorn_daemon_launcher()
212+
213 def _60_fix_root_path(self):
214 """Correction of root path for OpenERP 6.0 pure python install"""
215
216
217=== added directory 'anybox/recipe/openerp/templates'
218=== added file 'anybox/recipe/openerp/templates/__init__.py'
219--- anybox/recipe/openerp/templates/__init__.py 1970-01-01 00:00:00 +0000
220+++ anybox/recipe/openerp/templates/__init__.py 2012-12-04 08:56:24 +0000
221@@ -0,0 +1,1 @@
222+# This is a python package
223
224=== added directory 'anybox/recipe/openerp/templates/initd'
225=== added file 'anybox/recipe/openerp/templates/initd/__init__.py'
226--- anybox/recipe/openerp/templates/initd/__init__.py 1970-01-01 00:00:00 +0000
227+++ anybox/recipe/openerp/templates/initd/__init__.py 2012-12-04 08:56:24 +0000
228@@ -0,0 +1,1 @@
229+# This is a python package
230
231=== added file 'anybox/recipe/openerp/templates/initd/gunicorn_openerp_server.initd.in'
232--- anybox/recipe/openerp/templates/initd/gunicorn_openerp_server.initd.in 1970-01-01 00:00:00 +0000
233+++ anybox/recipe/openerp/templates/initd/gunicorn_openerp_server.initd.in 2012-12-04 08:56:24 +0000
234@@ -0,0 +1,109 @@
235+#!/bin/sh
236+
237+### BEGIN INIT INFO
238+# Provides: openerp-server
239+# Required-Start: $remote_fs $syslog
240+# Required-Stop: $remote_fs $syslog
241+# Should-Start: $network
242+# Should-Stop: $network
243+# Default-Start: 2 3 4 5
244+# Default-Stop: 0 1 6
245+# Short-Description: Enterprise Resource Management software
246+# Description: Open ERP is a complete ERP and CRM software.
247+### END INIT INFO
248+
249+PATH=${path}
250+PYTPATH=${pypath}
251+DAEMON=${daemon}
252+APP_MODULE=${app_module}
253+NAME=${name}
254+PID=${pid}
255+DESC=${desc}
256+CONFIG=${config}
257+LOGFILE=${logfile}
258+USER=${user}
259+
260+<%
261+use_cron_daemon = version.startswith('6.1')
262+%>
263+
264+%if use_cron_daemon:
265+CRON_PID=${cron_pid}
266+CRON_DAEMON=${cron_daemon}
267+CRON_DAEMON_OPTS="${cron_daemon_opts}"
268+CRON_CONFIG=${cron_config}
269+%endif
270+
271+test -x ${'$'}{DAEMON} || exit 0
272+
273+
274+%if use_cron_daemon:
275+test -x ${'$'}{CRON_DAEMON} || exit 0
276+%endif
277+
278+set -e
279+
280+do_start()
281+{
282+<%text>
283+ start-stop-daemon --start --quiet --pidfile ${PID} \
284+ --chuid ${USER} --background --make-pidfile \
285+ --exec ${DAEMON} -- ${APP_MODULE} --config=${CONFIG}
286+</%text>
287+%if use_cron_daemon:
288+<%text>
289+ start-stop-daemon --start --quiet --pidfile ${CRON_PID} \
290+ --chuid ${USER} --background --make-pidfile \
291+ --exec ${PYTPATH} \
292+ -- ${CRON_DAEMON} ${CRON_DAEMON_OPTS} --config=${CRON_CONFIG}
293+</%text>
294+%endif
295+}
296+
297+do_stop()
298+{
299+ start-stop-daemon --stop --quiet --pidfile ${'$'}{PID} --retry=TERM/3/KILL/2 --oknodo
300+%if use_cron_daemon:
301+ start-stop-daemon --stop --quiet --pidfile ${'$'}{CRON_PID} --retry=TERM/3/KILL/2 --oknodo
302+%endif
303+}
304+
305+<%text>
306+case "${1}" in
307+ start)
308+ echo -n "Starting ${DESC}: "
309+
310+ do_start
311+
312+ echo "${NAME}."
313+ ;;
314+
315+ stop)
316+ echo -n "Stopping ${DESC}: "
317+
318+ do_stop
319+
320+ echo "${NAME}."
321+ ;;
322+
323+ restart|force-reload)
324+ echo -n "Restarting ${DESC}: "
325+
326+ do_stop
327+
328+ sleep 1
329+
330+ do_start
331+
332+ echo "${NAME}."
333+ ;;
334+
335+ *)
336+ N=/etc/init.d/${NAME}
337+ echo "Usage: ${NAME} {start|stop|restart|force-reload}" >&2
338+ exit 1
339+ ;;
340+esac
341+
342+exit 0
343+</%text>
344
345=== added file 'anybox/recipe/openerp/templates/initd/openerp_server.initd.in'
346--- anybox/recipe/openerp/templates/initd/openerp_server.initd.in 1970-01-01 00:00:00 +0000
347+++ anybox/recipe/openerp/templates/initd/openerp_server.initd.in 2012-12-04 08:56:24 +0000
348@@ -0,0 +1,79 @@
349+#!/bin/sh
350+
351+### BEGIN INIT INFO
352+# Provides: openerp-server
353+# Required-Start: $remote_fs $syslog
354+# Required-Stop: $remote_fs $syslog
355+# Should-Start: $network
356+# Should-Stop: $network
357+# Default-Start: 2 3 4 5
358+# Default-Stop: 0 1 6
359+# Short-Description: Enterprise Resource Management software
360+# Description: Open ERP is a complete ERP and CRM software.
361+### END INIT INFO
362+
363+PATH=%(path)s
364+PYTPATH=%(pypath)s
365+DAEMON=%(daemon)s
366+NAME=%(name)s
367+PID=%(pid)s
368+DESC=%(desc)s
369+CONFIG=%(config)s
370+LOGFILE=%(logfile)s
371+USER=%(user)s
372+
373+test -x ${DAEMON} || exit 0
374+
375+set -e
376+
377+do_start()
378+{
379+ start-stop-daemon --start --quiet --pidfile ${PID} \
380+ --chuid ${USER} --background --make-pidfile \
381+ --exec ${PYTPATH} -- ${DAEMON} --config=${CONFIG} \
382+ --logfile=${LOGFILE}
383+}
384+
385+do_stop()
386+{
387+ start-stop-daemon --stop --quiet --pidfile ${PID} \
388+ --retry=TERM/10/KILL/5 --oknodo
389+}
390+
391+case "${1}" in
392+ start)
393+ echo -n "Starting ${DESC}: "
394+
395+ do_start
396+
397+ echo "${NAME}."
398+ ;;
399+
400+ stop)
401+ echo -n "Stopping ${DESC}: "
402+
403+ do_stop
404+
405+ echo "${NAME}."
406+ ;;
407+
408+ restart|force-reload)
409+ echo -n "Restarting ${DESC}: "
410+
411+ do_stop
412+
413+ sleep 1
414+
415+ do_start
416+
417+ echo "${NAME}."
418+ ;;
419+
420+ *)
421+ N=/etc/init.d/${NAME}
422+ echo "Usage: ${NAME} {start|stop|restart|force-reload}" >&2
423+ exit 1
424+ ;;
425+esac
426+
427+exit 0
428
429=== modified file 'setup.py'
430--- setup.py 2012-11-21 14:14:51 +0000
431+++ setup.py 2012-12-04 08:56:24 +0000
432@@ -37,6 +37,7 @@
433 'webclient = anybox.recipe.openerp.webclient:WebClientRecipe',
434 'gtkclient = anybox.recipe.openerp.gtkclient:GtkClientRecipe',
435 ]},
436+ package_data = { 'anybox.recipe.openerp': ['templates/initd/*.in'], },
437 )
438
439

Subscribers

People subscribed via source and target branches