Merge lp:~jbaudoux/anybox.recipe.openerp/1.8.5_relative_paths_jailroot into lp:anybox.recipe.openerp

[Jacques-Etienne Baudoux]

Improvement to be compliant with buildout option "relative_paths = true"
It allows the creation of a buildout that can be moved to a jail root and run OpenERP in that jail root without any modifications to any of the files in the buildout.

This replace my first MP here:
https://code.launchpad.net/~jbaudoux/anybox.recipe.openerp/20130908_relative_paths/+merge/184473

and includes improvements from this MP
https://code.launchpad.net/~therp-nl/anybox.recipe.openerp/jbaudoux-relative_paths_explicit_jailroot_support/+merge/205099

as well as some rework of that latest MP to let it work on my current setup.

See the 2 previous MP for review history.

[Georges Racinet (gracinet)]

Hi, I'm at last in shape (well me and the recipe) to look at this request seriously. Sorry for being so slow.
First, could you please clarify some generalities for me ?

- I'm a bit confused by the jailroot terminology. Do you have something specific and hard in mind (such as a chroot or lxc container), or is this just about promising not to leave some directory, and therefore be admissible for a chroot ?

- I'm not sure to understand the use-case for the she-bang change. I'm used to control the she-bang by bootstrapping with the intended python. What's the difference here ? I got the "no modification of files" and find it a great perspective, but how does it play with the custom shebang (maybe a concrete example making the difference between building and running would help).

For the record, I've been playing with debian repackaging of buildouts lately, and I wonder if it would not precisely been served well by such features. Could it allow, e.g, to build on a CI worker in a certain directory and then run transparently in another ?

This could also be exactly what SlapOS (a distributed cloud system based on zc.buildout) would need

[Jacques-Etienne Baudoux (jbaudoux)]

Hello Georges,

In my case, odoo is build under a jailroot_dir directory:
/buildout.cfg
/jailroot_dir/odoo
/jailroot_dir/bin
/jailroot_dir/etc
/jailroot_dir/var
/jailroot_dir/dev

odoo is run inside jailroot_dir chroot. So paths in generated bin must be relative as well as path for etc/openerp.cfg

Regarding the python-scripts-executable for specify an alternative Python executable to be used in the she-bang of generated scripts, it is an addition of Terp that I'm not using. I only use the jailroot-dir option that implies relative-paths. This is enough for building the chroot environment.

[lp:~jbaudoux/anybox.recipe.openerp/1.8.5_relative_paths_jailroot]

435. By Jacques-Etienne Baudoux on 2015-05-21

[MOD] put base addons at the end of the addons path

Unmerged revisions

435. By Jacques-Etienne Baudoux on 2015-05-21

[MOD] put base addons at the end of the addons path

434. By Jacques-Etienne Baudoux on 2014-08-12

[RFR] Rewrite paths for running OpenERP in a jailroot: make jailroot-dir explicit in buildout.cfg, do not guess based on eggs directory; include addons path rewrite; rewrite calls to runtime.session

433. By Jacques-Etienne Baudoux on 2014-08-12

[MRG] lp:~anybox/anybox.recipe.openerp/1.8, revno 570

432. By Jacques-Etienne Baudoux on 2014-08-12

[FIX] Error in start_openerp script in calling _relativitiez

431. By Stefan Rijnhart (Opener) on 2014-06-02

[MRG] lp:~anybox/anybox.recipe.openerp/1.8, revno 534

430. By Stefan Rijnhart (Opener) on 2014-02-10

[FIX] Remove call to debugger

429. By Stefan Rijnhart (Opener) on 2014-02-06

[FIX] Terminology

428. By Stefan Rijnhart (Opener) on 2014-02-06

[IMP] Comment

427. By Stefan Rijnhart (Opener) on 2014-02-06

[FIX] Typo

426. By Stefan Rijnhart (Opener) on 2014-02-06

[RFR] Terminology

=== modified file 'anybox/recipe/openerp/base.py'
--- anybox/recipe/openerp/base.py	2014-11-30 17:44:41 +0000
+++ anybox/recipe/openerp/base.py	2015-05-21 14:27:54 +0000
@@ -45,6 +45,24 @@
     """Parse RFC 2822-formatted http header and return a time int."""
     rfc822.mktime_tz(rfc822.parsedate_tz(h))
 
+def _common_path(*path):
+    """Return the longest path prefix that is a prefix of all paths"""
+    return os.path.dirname(os.path.commonprefix(path))
+
+def _relative_path(common, path):
+    """Copied from easy_install"""
+    r = []
+    while 1:
+        dirname, basename = os.path.split(path)
+        r.append(basename)
+        if dirname == common:
+            break
+        if dirname == path:
+            raise AssertionError("dirname of %s is the same" % dirname)
+        path = dirname
+    r.reverse()
+    return os.path.join(*r)
+
 
 class MainSoftware(object):
     """Placeholder to represent the main software instead of an addon location.
@@ -152,8 +170,23 @@
         self.vcs_clear_locks = clear_locks == 'true'
         clear_retry = options.get('vcs-clear-retry', '').lower()
         self.clear_retry = clear_retry == 'true'
+        self.jailroot_dir = options.get('jailroot-dir')
+        self.python_scripts_executable = options.get(
+            'python-scripts-executable')
+
+        if self.jailroot_dir:
+            options['relative-paths'] = 'true'
 
         # same as in zc.recipe.eggs
+        relative_paths = options.get(
+            'relative-paths',
+            buildout['buildout'].get('relative-paths', 'false')
+            )
+        if relative_paths == 'true':
+            self._relative_paths = self.buildout_dir
+        else:
+            self._relative_paths = ''
+            assert relative_paths == 'false'
         self.extra_paths = [
             join(self.buildout_dir, p.strip())
             for p in option_splitlines(self.options.get('extra-paths'))
@@ -429,6 +462,21 @@
             return path
         return join(self.buildout_dir, path)
 
+    def make_jailroot_absolute(self, path):
+        """Make a path absolute to the jailroot."""
+        if self.jailroot_dir:
+            return join('/', self.make_relative(path))
+        return path
+
+    def make_relative(self, path):
+        """Make a path relative to the jailroot or buildout dir"""
+        if self._relative_paths:
+            root = self.jailroot_dir or self.buildout_dir
+            common = _common_path(path, root+'/')
+            relative = _relative_path(common, path)
+            return relative
+        return path
+
     def sandboxed_tar_extract(self, sandbox, tarfile, first=None):
         """Extract those members that are below the tarfile path 'sandbox'.
 
@@ -1340,7 +1388,7 @@
         else:
             base_addons = join(self.openerp_dir, 'openerp', 'addons')
         if os.path.exists(base_addons):
-            self.addons_paths.insert(0, base_addons)
+            self.addons_paths.append(base_addons)
 
         self.insert_odoo_git_addons(base_addons)
 
@@ -1349,6 +1397,11 @@
                 assert os.path.isdir(path), (
                     "Not a directory: %r (aborting)" % path)
 
+        addons_paths = list(self.addons_paths)
+        self.addons_paths = [
+            self.make_jailroot_absolute(addons_path)
+            for addons_path in addons_paths
+            ]
         self.options['options.addons_path'] = ','.join(self.addons_paths)
         if self.major_version <= (6, 0):
             self._60_fix_root_path()

=== modified file 'anybox/recipe/openerp/server.py'
--- anybox/recipe/openerp/server.py	2014-11-29 18:05:24 +0000
+++ anybox/recipe/openerp/server.py	2015-05-21 14:27:54 +0000
@@ -8,8 +8,12 @@
 import zc.buildout
 from zc.buildout import UserError
 from anybox.recipe.openerp import devtools
+<<<<<<< TREE
 from .base import BaseRecipe
 from .utils import option_splitlines, option_strip
+=======
+from anybox.recipe.openerp.base import BaseRecipe, _relative_path
+>>>>>>> MERGE-SOURCE
 
 logger = logging.getLogger(__name__)
 
@@ -64,6 +68,17 @@
             sw_modules = ('web', ) + sw_modules
         self.server_wide_modules = sw_modules
 
+        if self.python_scripts_executable:
+            # Monkeypatch the script headers to replace the python
+            # executable by the one configured by the user
+            new_header = '#!%s' % self.python_scripts_executable
+            zc.buildout.easy_install.script_template = (
+                zc.buildout.easy_install.script_template.replace(
+                    zc.buildout.easy_install.script_header, new_header))
+            zc.buildout.easy_install.py_script_template = (
+                zc.buildout.easy_install.py_script_template.replace(
+                    zc.buildout.easy_install.script_header, new_header))
+
     def apply_version_dependent_decisions(self):
         """Store some booleans depending on detected version.
 
@@ -294,17 +309,25 @@
             desc = self.openerp_scripts[name] = dict(entry=entry)
             return name, desc
 
+    def _relativitize(self, path):
+        """Inspired from easy_install"""
+        if self._relative_paths:
+            return "join(base, %r)" % self.make_relative(path)
+        return path
+
     def _register_main_startup_script(self, qualified_name):
         """Register main startup script, usually ``start_openerp`` for install.
         """
         desc = self._get_or_create_script('openerp_starter',
                                           name=qualified_name)[1]
 
-        arguments = '%r, %r, version=%r' % (self._get_server_command(),
-                                            self.config_path,
-                                            self.major_version)
+        arguments = '%s, %s, version=%r' % (
+            self._relativitize(self._get_server_command()),
+            self._relativitize(self.config_path),
+            self.major_version)
         if self.major_version >= (7, 3):
-            arguments += ', gevent_script_path=%r' % self.gevent_script_path
+            arguments += ', gevent_script_path=%s' % (
+                self._relativitize(self.gevent_script_path))
 
         if self.server_wide_modules:
             arguments += ', server_wide_modules=%r' % (
@@ -334,9 +357,9 @@
         """
         desc = self._get_or_create_script('openerp_tester',
                                           name=qualified_name)[1]
-        arguments = '%r, %r, version=%r, just_test=True' % (
-            self._get_server_command(),
-            self.config_path,
+        arguments = '%s, %s, version=%r, just_test=True' % (
+            self._relativitize(self._get_server_command()),
+            self._relativitize(self.config_path),
             self.major_version)
         if self.major_version >= (7, 3):
             arguments += ', gevent_script_path=%r' % self.gevent_script_path
@@ -364,9 +387,10 @@
         script_source_path = self.make_absolute(script[0])
         desc.update(
             entry='openerp_upgrader',
-            arguments='%r, %r, %r, %r' % (
-                script_source_path, script[1],
-                self.config_path, self.buildout_dir),
+            arguments='%s, %r, %s, %r' % (
+                self._relativitize(script_source_path), script[1],
+                self._relativitize(self.config_path),
+                self.jailroot_dir and '/' or self.buildout_dir),
         )
 
         if not os.path.exists(script_source_path):
@@ -403,7 +427,8 @@
         # to resort on hacking sys.argv
         desc['initialization'] = (
             "from sys import argv; argv[1:] = ['%s', '-c', '%s.conf.py']" % (
-                gunicorn_entry_point, join(self.etc, qualified_name)))
+                gunicorn_entry_point,
+                self._relativitize(join(self.etc, qualified_name))))
 
     def _register_openerp_command(self, qualified_name):
         """Register https://launchpad.net/openerp-command for install.
@@ -487,7 +512,9 @@
         desc = self._get_or_create_script('openerp_cron_worker',
                                           name=qualified_name)[1]
         desc.update(entry='openerp_cron_worker',
-                    arguments='%r, %r' % (script_src, self.config_path),
+                    arguments='%s, %s' % (
+                        self._relativitize(script_src),
+                        self._relativitize(self.config_path)),
                     initialization='',
                     )
 
@@ -502,8 +529,9 @@
         initialization = os.linesep.join((
             "",
             "from anybox.recipe.openerp.runtime.session import Session",
-            "session = Session(%r, %r)" % (self.config_path,
-                                           self.buildout_dir),
+            "session = Session(%s, %r)" % (
+                    self._relativitize(self.config_path),
+                    self.jailroot_dir and '/' or self.buildout_dir),
             "if len(sys.argv) <= 1:",
             "    print('To start the OpenERP working session, just do:')",
             "    print('    session.open(db=DATABASE_NAME)')",
@@ -526,8 +554,7 @@
             initialization=initialization,
             arguments=self.options.get('arguments', ''),
             extra_paths=self.extra_paths,
-            # TODO investigate these options:
-            # relative_paths=self._relative_paths,
+            relative_paths=self._relative_paths,
         )
 
     def _install_openerp_scripts(self):
@@ -544,8 +571,9 @@
         common_init = os.linesep.join((
             "",
             "from anybox.recipe.openerp.runtime.session import Session",
-            "session = Session(%r, %r)" % (self.config_path,
-                                           self.buildout_dir),
+            "session = Session(%s, %r)" % (
+                    self._relativitize(self.config_path),
+                    self.jailroot_dir and '/' or self.buildout_dir),
         ))
 
         for script_name, desc in self.openerp_scripts.items():
@@ -569,9 +597,8 @@
                 interpreter='',
                 initialization=initialization,
                 arguments=desc.get('arguments', ''),
-                # TODO investigate these options:
                 extra_paths=self.extra_paths,
-                # relative_paths=self._relative_paths,
+                relative_paths=self._relative_paths,
             )
             self.openerp_installed.append(join(self.bin_dir, script_name))
 

=== modified file 'doc/configuration.rst'
--- doc/configuration.rst	2014-11-12 03:33:02 +0000
+++ doc/configuration.rst	2015-05-21 14:27:54 +0000
@@ -408,6 +408,7 @@
 
 Note that tarball downloads get re-extracted afresh in any case.
 
+<<<<<<< TREE
 vcs-revert
 ----------
 
@@ -440,6 +441,45 @@
 
 .. note:: new in version 1.9.0
 
+=======
+:: _relocation_options:
+
+Options for buildout relocation and jail root
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. _relative-paths:
+
+relative-paths
+--------------
+
+If set to true, this option will set up a movable buildout. Paths in scripts
+will be made relative to the buildout directory so that it can be relocated
+on the file system.
+
+.. _jailroot-dir:
+
+jailroot-dir
+---------------------
+
+If you build a jail root environment, specify the absolute path of the jail.
+Note that it must be a sub-directory of the buildout dir as easy install cannot
+build outside of the buildout dir. The addons-paths in the OpenERP configuration
+file will be adapted to this path. This option implies :ref:`relative-paths`.
+For instance, if you deploy OpenERP under a ``myjail`` sub-directory::
+
+    jailroot-dir = /root/mybuildout-dir/myjail
+
+.. _python-scripts-executable:
+
+python-scripts-executable
+-------------------------
+
+Specify an alternative Python executable to be used in the she-bang of
+generated scripts. For use in a jail root, ``/usr/bin/env python`` should
+be a sane value, or you could hardcode the path to the Python executable
+in your jail root.
+
+>>>>>>> MERGE-SOURCE
 .. _openerp_options:
 
 OpenERP options