Merge lp:~camptocamp/anybox.recipe.openerp/20121204-add-daemon into lp:anybox.recipe.openerp

[Yannick Vaucher @ Camptocamp]

Here is a merge proposal to insert the daemonization of openerp

It generates the following dirs and files:

in buildout dir:
- run (for pid files)
- init.d (for daemon launchers)
- log (for logs)

in init.d:
- openerp_server.initd
- gunicorn_openerp_server.initd

Those executable scripts are defined via templates in server.py of the recipe

New params in buildout config :
daemon.sys_user (used to launch daemon with a specific user if not set use current user)
daemon.current_instance (gives an instance name used to name pid file and daemon description)

Maybe this should be more optional or more generic ?
What is sure is that on our side in Camptocamp, we needs to be able to run openerp in daemon mode

[Alexandre Fayolle - camptocamp (alexandre-fayolle-c2c)]

LGTM

[Georges Racinet (gracinet)]

Just did a really quick read. That looks overall good, but I (we) will need a bit more time to review/merge that one.

About the use-case itself, at Anybox, we use supervisor to handle daemonization, console logs and startup. That explains the lack of such in the recipe. We have self-contained instances, with a supervisor part with the dedicated recipe as well as mutualizing servers, where a distribution-installed supervisor takes care of several buildout-built instances.

In the future, we may switch to circus (looks promising). Some other people may use other supervision processes, so I think we should maybe introduce a class of options for that purpose, in order to avoid confusion with too many different configuration files/scripts being generated.

[Georges Racinet (gracinet)]

How do you plan to handle the script (cron_worker_openerp) that handles ir.cron jobs ?

This is tricky, because that script needs a (list of) database names, and we already have use cases where limitation to thoses databases belonging to the system user won't be enough.

Init/update of the database through the recipe is not far behind that one. For the time being, we can live with a simple option to declare the database, even if that's a bit odd (bin/buildout would generate a rc script that can't work until the listed databases have been created).

We had long internal discussions about these issues, with no clear consensus. c2c's input would be appreciated.

[Georges Racinet (gracinet)]

Finally, this will require updates to README/CHANGES and, of course, new tests. About these, in case test_server.py isn't clear enough or appropriate, I can help.

[Georges Racinet (gracinet)]

Thank you, it's great to see new use cases and momentum.

We don't need to solve all the concerns expressed in the above comments for the contribution to be mergeable. Reaching a consistent state with room for further developments will be enough.

I won't have much more time for this today, but will check for new inputs at least tomorrow.

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

Hello,

My colleague Yannick will have a look on your remarks, though not sure he will have enough time today.

By the way, we may consider using a supervisor.
Is it actually a part of the buildout? I can see that the supervisor egg is in the recipes of the buildbot but do not find any configuration.

Why not to use collective.recipe.supervisor ?

Subsidiary question: Can we have more than one buildout deployed on 1 server, each one of them having their own self-contained supervisor ?

[Christophe Combelles (ccomb)]

Yes for a single instance we use collective.recipe.supervisor, that's perfect for quickly setting up a supervisor. For example we used it for a complex project with a libreoffice server (aeroo report), a django frontend with a cherrypy server and a Celery server for asynchronous jobs.
We use a single system-wide supervisor to manage multiple OpenERP instances from several buildouts.

[Georges Racinet (gracinet)]

> By the way, we may consider using a supervisor.
> Is it actually a part of the buildout? I can see that the supervisor egg is in
> the recipes of the buildbot but do not find any configuration.

That egg is indeed listed in [versions] of the buildbot/*.cfg, you are right. But it's a remnant from another configuration that had a supervisor part, it's not actually used.

Note that egg versions are shared among parts.

>
> Why not to use collective.recipe.supervisor ?

That's what we do as "local supervisors", sorry if I haven't been clear enough about that.

It wouldn't be the job of the openerp recipe to install supervisor.
A standalone, ready to go daemonized version with an example init script is interesting, especially if it needs some OpenERP specifics (like database declaration), meaning that it has a clear added value over an external approach, but there's a line to be drawn where the recipe responsibility should end, and that should be imho at the point where other recipes (supervisor) or the general OS environment (systemd/upstart/whatever) can interface themselves easily.

> Subsidiary question: Can we have more than one buildout deployed on 1 server,
> each one of them having their own self-contained supervisor ?

Yes, you probably can, if you take care to have them use separate control sockets, but I've not done this myself. I suppose that the supervisor recipe config defaults are sensible (in particular, Unix domain socket file should be local).

Then of course, you'll need an init/systemd/upstart to launch all these supervisors. The local supervisor is handy to multiplex all the needed subprocesses for that openerp installation (gunicorn, cron_worker, specific stuff) as one "service".

We prefer to use the supervisor provided by the distro with a process group per buildout, because rc script is provided out of the box. That implies that access to the web UI must be restricted to our staff (fine for us).

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

Thanks for all your explanations. I totally understand and agree.

I found an example a local supervisord setup using collective.recipe.supervisor
https://github.com/florentx/openobject-mirliton/blob/buildout/etc/buildout-base.cfg

This seems really good.

We'll do some tests and share the results. It would be nice then to have some documentation with an example of configuration. If our experiments are fine, we'll write it.

[Georges Racinet (gracinet)]

Hi,

we are on the verge of releasing version 1.4, mainly for the extraction-freeze features.

What are the results of your investigations ? Do you still need this ? Can it wait until, e.g, a point release or the next major release (1.5).

Regards,

[Yannick Vaucher @ Camptocamp (yvaucher-c2c)]

Yes, we can wait on this. Finally we created a configuration using supervisor and though it isn't used in production yet we are moving to it for our next projects.

Unmerged revisions

250. By Yannick Vaucher @ Camptocamp on 2012-12-04

[ADD] daemon launcher for standalone and gunicorn mode

=== modified file 'anybox/recipe/openerp/base.py'
--- anybox/recipe/openerp/base.py	2012-11-20 10:25:02 +0000
+++ anybox/recipe/openerp/base.py	2012-12-04 08:56:24 +0000
@@ -57,6 +57,7 @@
 
         self.downloads_dir = self.make_absolute(
             self.b_options.get('openerp-downloads-directory', 'downloads'))
+        self.daemon_dir = self.make_absolute('init.d')
         self.version_wanted = None  # from the buildout
         self.version_detected = None  # string from the openerp setup.py
         self.parts = self.buildout['buildout']['parts-directory']
@@ -81,9 +82,11 @@
         self.openerp_installed = []
 
         self.etc = self.make_absolute('etc')
+        self.log_dir = self.make_absolute('log')
+        self.run_dir = self.make_absolute('run')
         self.bin_dir = self.buildout['buildout']['bin-directory']
         self.config_path = join(self.etc, self.name + '.cfg')
-        for d in self.downloads_dir, self.etc:
+        for d in self.downloads_dir, self.etc, self.daemon_dir, self.log_dir, self.run_dir:
             if not os.path.exists(d):
                 logger.info('Created %s/ directory' % basename(d))
                 os.mkdir(d)

=== modified file 'anybox/recipe/openerp/server.py'
--- anybox/recipe/openerp/server.py	2012-11-29 17:10:20 +0000
+++ anybox/recipe/openerp/server.py	2012-12-04 08:56:24 +0000
@@ -1,14 +1,21 @@
 # coding: utf-8
 import os
+import stat
+import getpass
 from os.path import join
-import sys, logging
+import sys
+import logging
 import subprocess
 import zc.buildout
+from mako.template import Template
 from anybox.recipe.openerp import devtools
 from anybox.recipe.openerp.base import BaseRecipe
 
+from pkg_resources import resource_string
+
 logger = logging.getLogger(__name__)
 
+
 class ServerRecipe(BaseRecipe):
     """Recipe for server install and config
     """
@@ -101,6 +108,112 @@
 
         BaseRecipe.merge_requirements(self)
 
+    def _create_daemon_launcher(self):
+        """Generate daemon laucher in a init.d directory.
+        """
+        template = resource_string(__name__, join('templates', 'initd','openerp_server.initd.in'))
+
+        daemon_options = {}
+        daemon_prefix = 'daemon.'
+        daemon_options.update((k[len(daemon_prefix):], v)
+                                for k, v in self.options.items()
+                                if k.startswith(daemon_prefix))
+
+        system_user = daemon_options.get('sys_user')
+        current_instance = daemon_options.get('current_instance')
+        daemon_params = {
+                'path': '/sbin:/bin:/usr/sbin:/usr/bin',
+                'pypath': '/usr/bin/python',
+                'daemon': self.script_path,
+                'config': self.config_path,
+                'name': 'openerp-server',
+                'pid': join(self.buildout_dir, 'run', current_instance + '.pid'),
+                'desc': 'openerp-server-' + current_instance,
+                'logfile': join(self.log_dir, 'server.log'),
+                'user': system_user,
+                }
+
+        template = template % daemon_params
+
+        daemon_file_name = join(self.daemon_dir, 'openerp_server.initd')
+
+        new_file = open(daemon_file_name, 'w')
+        new_file.write(template)
+        new_file.close()
+
+        os.chmod(daemon_file_name, stat.S_IRWXU)
+        self.openerp_installed.append(daemon_file_name)
+
+    def _get_gunicorn_entry_point(self, gunicorn_options):
+        # entry point in 7.0 has changed so we need it to be configurable
+        gunicorn_entry_point = gunicorn_options.get('entry_point')
+        if gunicorn_entry_point is None:
+            if self.major_version >= (6, 2):
+                # proxy vs direct now handled by an OpenERP server option
+                gunicorn_entry_point = 'openerp:service.wsgi_server.application'
+            else:
+                gunicorn_entry_point = (
+                    'openerp:wsgi.%s.application' % self.gunicorn_entry)
+        return gunicorn_entry_point
+
+
+    def _create_gunicorn_daemon_launcher(self):
+        """Generate daemon laucher in a init.d directory with gunicorn support.
+        """
+        template = resource_string(__name__, join('templates', 'initd','gunicorn_openerp_server.initd.in'))
+
+        daemon_options = {}
+        gunicorn_options = {}
+        daemon_prefix = 'daemon.'
+        gunicorn_prefix = 'gunicorn.'
+
+        daemon_options.update((k[len(daemon_prefix):], v)
+                                for k, v in self.options.items()
+                                if k.startswith(daemon_prefix))
+
+        system_user = daemon_options.get(
+                'sys_user',
+                getpass.getuser())
+        current_instance = daemon_options.get('current_instance')
+
+        gunicorn_options.update((k[len(gunicorn_prefix):], v)
+                                for k, v in self.options.items()
+                                if k.startswith(gunicorn_prefix))
+        gunicorn_entry_point = self._get_gunicorn_entry_point(gunicorn_options)
+
+        database = self.options.get('options.db_name')
+        version = self.version_detected
+        daemon_params = {
+                'path': '/sbin:/bin:/usr/sbin:/usr/bin',
+                'pypath': '/usr/bin/python',
+                'daemon': join(self.bin_dir, 'gunicorn_openerp'),
+                'config': join(self.etc, 'gunicorn_openerp.conf.py'),
+                'app_module': gunicorn_entry_point,
+                'name': 'gunicorn-openerp-server',
+                'desc': 'gunicorn-openerp-server-' + current_instance,
+                'pid': join(self.buildout_dir, 'run', 'gunicorn_' + current_instance + '.pid'),
+                'logfile': join(self.log_dir, 'server.log'),
+                'user': system_user,
+
+                'cron_pid': join(self.buildout_dir, 'run', 'cron_worker_' + current_instance + '.pid'),
+                'cron_daemon': join(self.bin_dir, 'cron_worker_openerp'),
+                'cron_daemon_opts': '--database=' + database,
+                'cron_config': self.config_path,
+                'version': version,
+                }
+
+        template = Template(template).render(**daemon_params)
+
+        daemon_file_name = join(self.daemon_dir, 'gunicorn_openerp_server.initd')
+
+        new_file = open(daemon_file_name, 'w')
+        new_file.write(template)
+        new_file.close()
+
+        os.chmod(daemon_file_name, stat.S_IRWXU)
+        self.openerp_installed.append(daemon_file_name)
+
+
     def _create_default_config(self):
         """Have OpenERP generate its default config file.
         """
@@ -135,11 +248,13 @@
                                 if k.startswith(gunicorn_prefix))
 
         f = open(join(self.etc, qualified_name + '.conf.py'), 'w')
+        pidfile = join(self.run_dir, qualified_name + '.pid')
+        gunicorn_options.update({'pidfile': pidfile})
         conf = """'''Gunicorn configuration script.
 Generated by buildout. Do NOT edit.'''
 import openerp
 bind = %(bind)r
-pidfile = %(qualified_name)r + '.pid'
+pidfile = %(pidfile)r
 workers = %(workers)s
 
 if openerp.release.major_version == '6.1':
@@ -191,14 +306,7 @@
                                 for k, v in self.options.items()
                                 if k.startswith(gunicorn_prefix))
 
-        gunicorn_entry_point = gunicorn_options.get('entry_point')
-        if gunicorn_entry_point is None:
-            if self.major_version >= (6, 2):
-                # proxy vs direct now handled by an OpenERP server option
-                gunicorn_entry_point = 'openerp:service.wsgi_server.application'
-            else:
-                gunicorn_entry_point = (
-                    'openerp:wsgi.%s.application' % self.gunicorn_entry)
+        gunicorn_entry_point = self._get_gunicorn_entry_point(gunicorn_options)
 
         # gunicorn's main() does not take arguments, that's why we have
         # to resort on hacking sys.argv
@@ -345,6 +453,9 @@
 
         self._install_main_startup_script()
 
+        if os.name == 'posix':
+            self._create_daemon_launcher()
+
         if self.with_openerp_command:
             self._install_openerp_command(
                 self.options.get('openerp_command_name',
@@ -363,6 +474,9 @@
                                               'cron_worker_%s' % self.name)
             self._install_cron_worker_startup_script(qualified_name)
 
+            if os.name == 'posix':
+                self._create_gunicorn_daemon_launcher()
+
     def _60_fix_root_path(self):
         """Correction of root path for OpenERP 6.0 pure python install"""
 

=== added directory 'anybox/recipe/openerp/templates'
=== added file 'anybox/recipe/openerp/templates/__init__.py'
--- anybox/recipe/openerp/templates/__init__.py	1970-01-01 00:00:00 +0000
+++ anybox/recipe/openerp/templates/__init__.py	2012-12-04 08:56:24 +0000
@@ -0,0 +1,1 @@
+# This is a python package

=== added directory 'anybox/recipe/openerp/templates/initd'
=== added file 'anybox/recipe/openerp/templates/initd/__init__.py'
--- anybox/recipe/openerp/templates/initd/__init__.py	1970-01-01 00:00:00 +0000
+++ anybox/recipe/openerp/templates/initd/__init__.py	2012-12-04 08:56:24 +0000
@@ -0,0 +1,1 @@
+# This is a python package

=== added file 'anybox/recipe/openerp/templates/initd/gunicorn_openerp_server.initd.in'
--- anybox/recipe/openerp/templates/initd/gunicorn_openerp_server.initd.in	1970-01-01 00:00:00 +0000
+++ anybox/recipe/openerp/templates/initd/gunicorn_openerp_server.initd.in	2012-12-04 08:56:24 +0000
@@ -0,0 +1,109 @@
+#!/bin/sh
+
+### BEGIN INIT INFO
+# Provides:     openerp-server
+# Required-Start:   $remote_fs $syslog
+# Required-Stop:    $remote_fs $syslog
+# Should-Start:     $network
+# Should-Stop:      $network
+# Default-Start:    2 3 4 5
+# Default-Stop:     0 1 6
+# Short-Description:    Enterprise Resource Management software
+# Description:      Open ERP is a complete ERP and CRM software.
+### END INIT INFO
+
+PATH=${path}
+PYTPATH=${pypath}
+DAEMON=${daemon}
+APP_MODULE=${app_module}
+NAME=${name}
+PID=${pid}
+DESC=${desc}
+CONFIG=${config}
+LOGFILE=${logfile}
+USER=${user}
+
+<%
+use_cron_daemon = version.startswith('6.1')
+%>
+
+%if use_cron_daemon:
+CRON_PID=${cron_pid}
+CRON_DAEMON=${cron_daemon}
+CRON_DAEMON_OPTS="${cron_daemon_opts}"
+CRON_CONFIG=${cron_config}
+%endif
+
+test -x ${'$'}{DAEMON} || exit 0
+
+
+%if use_cron_daemon:
+test -x ${'$'}{CRON_DAEMON} || exit 0
+%endif
+
+set -e
+
+do_start()
+{
+<%text>
+    start-stop-daemon --start --quiet --pidfile ${PID} \
+        --chuid ${USER} --background --make-pidfile \
+        --exec ${DAEMON} -- ${APP_MODULE} --config=${CONFIG}
+</%text>
+%if use_cron_daemon:
+<%text>
+    start-stop-daemon --start --quiet --pidfile ${CRON_PID} \
+        --chuid ${USER} --background --make-pidfile \
+        --exec ${PYTPATH} \
+        -- ${CRON_DAEMON} ${CRON_DAEMON_OPTS} --config=${CRON_CONFIG}
+</%text>
+%endif
+}
+
+do_stop()
+{
+    start-stop-daemon --stop --quiet --pidfile ${'$'}{PID} --retry=TERM/3/KILL/2 --oknodo
+%if use_cron_daemon:
+    start-stop-daemon --stop --quiet --pidfile ${'$'}{CRON_PID} --retry=TERM/3/KILL/2 --oknodo
+%endif
+}
+
+<%text>
+case "${1}" in
+    start)
+        echo -n "Starting ${DESC}: "
+
+        do_start
+
+        echo "${NAME}."
+        ;;
+
+    stop)
+        echo -n "Stopping ${DESC}: "
+
+        do_stop
+
+        echo "${NAME}."
+        ;;
+
+    restart|force-reload)
+        echo -n "Restarting ${DESC}: "
+
+        do_stop
+
+        sleep 1
+
+        do_start
+
+        echo "${NAME}."
+        ;;
+
+    *)
+        N=/etc/init.d/${NAME}
+        echo "Usage: ${NAME} {start|stop|restart|force-reload}" >&2
+        exit 1
+        ;;
+esac
+
+exit 0
+</%text>

=== added file 'anybox/recipe/openerp/templates/initd/openerp_server.initd.in'
--- anybox/recipe/openerp/templates/initd/openerp_server.initd.in	1970-01-01 00:00:00 +0000
+++ anybox/recipe/openerp/templates/initd/openerp_server.initd.in	2012-12-04 08:56:24 +0000
@@ -0,0 +1,79 @@
+#!/bin/sh
+
+### BEGIN INIT INFO
+# Provides:     openerp-server
+# Required-Start:   $remote_fs $syslog
+# Required-Stop:    $remote_fs $syslog
+# Should-Start:     $network
+# Should-Stop:      $network
+# Default-Start:    2 3 4 5
+# Default-Stop:     0 1 6
+# Short-Description:    Enterprise Resource Management software
+# Description:      Open ERP is a complete ERP and CRM software.
+### END INIT INFO
+
+PATH=%(path)s
+PYTPATH=%(pypath)s
+DAEMON=%(daemon)s
+NAME=%(name)s
+PID=%(pid)s
+DESC=%(desc)s
+CONFIG=%(config)s
+LOGFILE=%(logfile)s
+USER=%(user)s
+
+test -x ${DAEMON} || exit 0
+
+set -e
+
+do_start()
+{
+    start-stop-daemon --start --quiet --pidfile ${PID} \
+        --chuid ${USER} --background --make-pidfile \
+        --exec ${PYTPATH} -- ${DAEMON} --config=${CONFIG} \
+        --logfile=${LOGFILE}
+}
+
+do_stop()
+{
+    start-stop-daemon --stop --quiet --pidfile ${PID} \
+        --retry=TERM/10/KILL/5 --oknodo
+}
+
+case "${1}" in
+    start)
+        echo -n "Starting ${DESC}: "
+
+        do_start
+
+        echo "${NAME}."
+        ;;
+
+    stop)
+        echo -n "Stopping ${DESC}: "
+
+        do_stop
+
+        echo "${NAME}."
+        ;;
+
+    restart|force-reload)
+        echo -n "Restarting ${DESC}: "
+
+        do_stop
+
+        sleep 1
+
+        do_start
+
+        echo "${NAME}."
+        ;;
+
+    *)
+        N=/etc/init.d/${NAME}
+        echo "Usage: ${NAME} {start|stop|restart|force-reload}" >&2
+        exit 1
+        ;;
+esac
+
+exit 0

=== modified file 'setup.py'
--- setup.py	2012-11-21 14:14:51 +0000
+++ setup.py	2012-12-04 08:56:24 +0000
@@ -37,6 +37,7 @@
         'webclient = anybox.recipe.openerp.webclient:WebClientRecipe',
         'gtkclient = anybox.recipe.openerp.gtkclient:GtkClientRecipe',
         ]},
+    package_data = { 'anybox.recipe.openerp': ['templates/initd/*.in'], },
     )