Merge lp:~gary/zc.buildout/betafix1 into lp:zc.buildout

[Gary Poster]

This change addresses two problems identified with the allowed_eggs_from_site_packages feature, as partially identified by bug 592524. I think the tests pretty clearly describe the problems they are addressing.

The first non-whitespace change in easy_install.py addresses the "allowed_eggs_from_site_packages_dependencies_bugfix" test. Hopefully the comment explains what is going on.

The remaining changes in easy_install.py address the "allowed_eggs_from_site_packages_bug_592524" test. The key line is "path = [p for p in path if p not in site_paths]" and, as with the other fix, it is explained with a comment.

[Gary Poster (gary)]

I'm not sure why the diff shows bootstrap as having been deleted.

betafix1$ bzr ls bootstrap/
bootstrap.py

Similarly, loggerhead shows the diff for the single revision that comprises the branch as not including a change to bootstrap:
http://bazaar.launchpad.net/~gary/zc.buildout/betafix1/revision/554

(Unfortunately, loggerhead is giving me an Internal Server Error when I try to go to the browse link on the branch page, http://bazaar.launchpad.net/~gary/zc.buildout/betafix1/files .)

[Francis J. Lacoste (flacoste)]

Allow good here also.

review: Approve

[Thomas Herve (therve)]

This branch seems ready to land?

[Gary Poster (gary)]

Thomas: Technically, yes. The problem is the fact that the buildout universe does not support betas well, and the last beta release for this branch caused a lot of problems. I have a plan to make the next release, and zc.buildout, better in this regard, but it involves more work.

Howvere, I can certainly make a local release easily, and now. If you would like me to move Launchpad to a new release so you all can use it too, I'm happy to: just let me know.

Gary

Unmerged revisions

554. By Gary Poster on 2010-06-15

Fix some problems with allowed_eggs_from_site_packages

553. By Gary Poster on 2010-06-15

get a baseline of zc.buildout with passing tests

=== removed file 'bootstrap/bootstrap.py'
--- bootstrap/bootstrap.py	2010-05-10 20:36:47 +0000
+++ bootstrap/bootstrap.py	1970-01-01 00:00:00 +0000
@@ -1,206 +0,0 @@
-##############################################################################
-#
-# Copyright (c) 2006 Zope Foundation and Contributors.
-# All Rights Reserved.
-#
-# This software is subject to the provisions of the Zope Public License,
-# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
-# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
-# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
-# FOR A PARTICULAR PURPOSE.
-#
-##############################################################################
-"""Bootstrap a buildout-based project
-
-Simply run this script in a directory containing a buildout.cfg.
-The script accepts buildout command-line options, so you can
-use the -c option to specify an alternate configuration file.
-
-$Id$
-"""
-
-import os, shutil, sys, tempfile, textwrap, urllib, urllib2
-from optparse import OptionParser
-
-if sys.platform == 'win32':
-    def quote(c):
-        if ' ' in c:
-            return '"%s"' % c # work around spawn lamosity on windows
-        else:
-            return c
-else:
-    quote = str
-
-# In order to be more robust in the face of system Pythons, we want to
-# run without site-packages loaded.  This is somewhat tricky, in
-# particular because Python 2.6's distutils imports site, so starting
-# with the -S flag is not sufficient.  However, we'll start with that:
-if 'site' in sys.modules:
-    # We will restart with python -S.
-    args = sys.argv[:]
-    args[0:0] = [sys.executable, '-S']
-    args = map(quote, args)
-    os.execv(sys.executable, args)
-# Now we are running with -S.  We'll get the clean sys.path, import site
-# because distutils will do it later, and then reset the path and clean
-# out any namespace packages from site-packages that might have been
-# loaded by .pth files.
-clean_path = sys.path[:]
-import site
-sys.path[:] = clean_path
-for k, v in sys.modules.items():
-    if (hasattr(v, '__path__') and
-        len(v.__path__)==1 and
-        not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
-        # This is a namespace package.  Remove it.
-        sys.modules.pop(k)
-
-is_jython = sys.platform.startswith('java')
-
-setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
-distribute_source = 'http://python-distribute.org/distribute_setup.py'
-
-# parsing arguments
-def normalize_to_url(option, opt_str, value, parser):
-    if value:
-        if '://' not in value: # It doesn't smell like a URL.
-            value = 'file://%s' % (
-                urllib.pathname2url(
-                    os.path.abspath(os.path.expanduser(value))),)
-        if opt_str == '--download-base' and not value.endswith('/'):
-            # Download base needs a trailing slash to make the world happy.
-            value += '/'
-    else:
-        value = None
-    name = opt_str[2:].replace('-', '_')
-    setattr(parser.values, name, value)
-
-usage = '''\
-[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
-
-Bootstraps a buildout-based project.
-
-Simply run this script in a directory containing a buildout.cfg, using the
-Python that you want bin/buildout to use.
-
-Note that by using --setup-source and --download-base to point to
-local resources, you can keep this script from going over the network.
-'''
-
-parser = OptionParser(usage=usage)
-parser.add_option("-v", "--version", dest="version",
-                          help="use a specific zc.buildout version")
-parser.add_option("-d", "--distribute",
-                   action="store_true", dest="use_distribute", default=False,
-                   help="Use Distribute rather than Setuptools.")
-parser.add_option("--setup-source", action="callback", dest="setup_source",
-                  callback=normalize_to_url, nargs=1, type="string",
-                  help=("Specify a URL or file location for the setup file. "
-                        "If you use Setuptools, this will default to " +
-                        setuptools_source + "; if you use Distribute, this "
-                        "will default to " + distribute_source +"."))
-parser.add_option("--download-base", action="callback", dest="download_base",
-                  callback=normalize_to_url, nargs=1, type="string",
-                  help=("Specify a URL or directory for downloading "
-                        "zc.buildout and either Setuptools or Distribute. "
-                        "Defaults to PyPI."))
-parser.add_option("--eggs",
-                  help=("Specify a directory for storing eggs.  Defaults to "
-                        "a temporary directory that is deleted when the "
-                        "bootstrap script completes."))
-parser.add_option("-c", None, action="store", dest="config_file",
-                   help=("Specify the path to the buildout configuration "
-                         "file to be used."))
-
-options, args = parser.parse_args()
-
-# if -c was provided, we push it back into args for buildout's main function
-if options.config_file is not None:
-    args += ['-c', options.config_file]
-
-if options.eggs:
-    eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
-else:
-    eggs_dir = tempfile.mkdtemp()
-
-if options.setup_source is None:
-    if options.use_distribute:
-        options.setup_source = distribute_source
-    else:
-        options.setup_source = setuptools_source
-
-args = args + ['bootstrap']
-
-
-try:
-    to_reload = False
-    import pkg_resources
-    to_reload = True
-    if not hasattr(pkg_resources, '_distribute'):
-        raise ImportError
-    import setuptools # A flag.  Sometimes pkg_resources is installed alone.
-except ImportError:
-    ez_code = urllib2.urlopen(
-        options.setup_source).read().replace('\r\n', '\n')
-    ez = {}
-    exec ez_code in ez
-    setup_args = dict(to_dir=eggs_dir, download_delay=0)
-    if options.download_base:
-        setup_args['download_base'] = options.download_base
-    if options.use_distribute:
-        setup_args['no_fake'] = True
-    ez['use_setuptools'](**setup_args)
-    if to_reload:
-        reload(pkg_resources)
-    else:
-        import pkg_resources
-    # This does not (always?) update the default working set.  We will
-    # do it.
-    for path in sys.path:
-        if path not in pkg_resources.working_set.entries:
-            pkg_resources.working_set.add_entry(path)
-
-cmd = [quote(sys.executable),
-       '-c',
-       quote('from setuptools.command.easy_install import main; main()'),
-       '-mqNxd',
-       quote(eggs_dir)]
-
-if options.download_base:
-    cmd.extend(['-f', quote(options.download_base)])
-
-requirement = 'zc.buildout'
-if options.version:
-    requirement = '=='.join((requirement, options.version))
-cmd.append(requirement)
-
-if options.use_distribute:
-    setup_requirement = 'distribute'
-else:
-    setup_requirement = 'setuptools'
-ws = pkg_resources.working_set
-env = dict(
-    os.environ,
-    PYTHONPATH=ws.find(
-        pkg_resources.Requirement.parse(setup_requirement)).location)
-
-if is_jython:
-    import subprocess
-    exitcode = subprocess.Popen(cmd, env=env).wait()
-else: # Windows prefers this, apparently; otherwise we would prefer subprocess
-    exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
-if exitcode != 0:
-    sys.stdout.flush()
-    sys.stderr.flush()
-    print ("An error occured when trying to install zc.buildout. "
-           "Look above this message for any errors that "
-           "were output by easy_install.")
-    sys.exit(exitcode)
-
-ws.add_entry(eggs_dir)
-ws.require(requirement)
-import zc.buildout.buildout
-zc.buildout.buildout.main(args)
-if not options.eggs: # clean up temporary egg directory
-    shutil.rmtree(eggs_dir)

=== modified file 'src/zc/buildout/easy_install.py'
--- src/zc/buildout/easy_install.py	2010-04-29 18:11:45 +0000
+++ src/zc/buildout/easy_install.py	2010-06-16 15:10:44 +0000
@@ -58,7 +58,6 @@
     import java.lang.System
     jython_os_name = (java.lang.System.getProperties()['os.name']).lower()
 
-
 setuptools_loc = pkg_resources.working_set.find(
     pkg_resources.Requirement.parse('setuptools')
     ).location
@@ -873,7 +872,13 @@
                         dist = best[req.key] = env.best_match(req, ws)
                     except pkg_resources.VersionConflict, err:
                         raise VersionConflict(err, ws)
-                    if dist is None:
+                    if dist is None or (
+                        dist.location in self._site_packages and not
+                        self.allow_site_package_egg(dist.project_name)):
+                        # If we didn't find a distribution in the
+                        # environment, or what we found is from site
+                        # packages and not allowed to be there, try
+                        # again.
                         if destination:
                             logger.debug('Getting required %r', str(req))
                         else:
@@ -1548,15 +1553,15 @@
     """
     path = _get_path(working_set, extra_paths)
     site_path = os.path.join(dest, 'site.py')
-    egg_path_string, preamble = _relative_path_and_setup(
-        site_path, path, relative_paths, indent_level=2, omit_os_import=True)
-    if preamble:
-        preamble = '\n'.join(
-            [(line and '    %s' % (line,) or line)
-             for line in preamble.split('\n')])
-    original_path_setup = ''
+    original_path_setup = preamble = ''
     if include_site_packages:
         stdlib, site_paths = _get_system_paths(executable)
+        # We want to make sure that paths from site-packages, such as those
+        # allowed by allowed_eggs_from_site_packages, always come last, or
+        # else site-packages paths may include packages that mask the eggs we
+        # really want.
+        path = [p for p in path if p not in site_paths]
+        # Now we set up the code we need.
         original_path_setup = original_path_snippet % (
             _format_paths((repr(p) for p in site_paths), 2),)
         distribution = working_set.find(
@@ -1570,10 +1575,17 @@
                     relative_paths)
             else:
                 location = repr(distribution.location)
-            preamble += namespace_include_site_packages_setup % (location,)
+            preamble = namespace_include_site_packages_setup % (location,)
             original_path_setup = (
                 addsitedir_namespace_originalpackages_snippet +
                 original_path_setup)
+    egg_path_string, relative_preamble = _relative_path_and_setup(
+        site_path, path, relative_paths, indent_level=2, omit_os_import=True)
+    if relative_preamble:
+        relative_preamble = '\n'.join(
+            [(line and '    %s' % (line,) or line)
+             for line in relative_preamble.split('\n')])
+        preamble = relative_preamble + preamble
     addsitepackages_marker = 'def addsitepackages('
     enableusersite_marker = 'ENABLE_USER_SITE = '
     successful_rewrite = False

=== modified file 'src/zc/buildout/tests.py'
--- src/zc/buildout/tests.py	1970-01-01 00:00:00 +0000
+++ src/zc/buildout/tests.py	2010-06-16 15:10:44 +0000
@@ -2255,6 +2255,75 @@
 
     """
 
+def allowed_eggs_from_site_packages_dependencies_bugfix():
+    """
+If you specify that a package with a dependency may come from site-packages,
+that doesn't mean that the dependency may come from site-packages.  This
+is a test for a bug fix to verify that this is true.
+
+    >>> py_path, site_packages_path = make_py()
+    >>> create_sample_sys_install(site_packages_path)
+    >>> interpreter_dir = tmpdir('interpreter')
+    >>> interpreter_parts_dir = os.path.join(
+    ...     interpreter_dir, 'parts', 'interpreter')
+    >>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin')
+    >>> mkdir(interpreter_bin_dir)
+    >>> mkdir(interpreter_dir, 'eggs')
+    >>> mkdir(interpreter_dir, 'parts')
+    >>> mkdir(interpreter_parts_dir)
+    >>> ws = zc.buildout.easy_install.install(
+    ...     ['demo'], join(interpreter_dir, 'eggs'), executable=py_path,
+    ...     links=[link_server], index=link_server+'index/',
+    ...     allowed_eggs_from_site_packages=['demo'])
+    >>> [dist.project_name for dist in ws]
+    ['demo', 'demoneeded']
+    >>> from pprint import pprint
+    >>> pprint([dist.location for dist in ws])
+    ['/executable_buildout/site-packages',
+     '/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
+
+    """
+
+def allowed_eggs_from_site_packages_bug_592524():
+    """
+When we use allowed_eggs_from_site_packages, we need to make sure that the
+site-packages paths are not inserted with the normal egg paths.  They already
+included at the end, and including them along with the normal egg paths will
+possibly mask subsequent egg paths.  This affects interpreters and scripts
+generated by sitepackage_safe_scripts.
+
+Our "py_path" has the "demoneeded" and "demo" packages available.
+
+    >>> py_path, site_packages_path = make_py()
+    >>> create_sample_sys_install(site_packages_path)
+    >>> interpreter_dir = tmpdir('interpreter')
+    >>> interpreter_parts_dir = os.path.join(
+    ...     interpreter_dir, 'parts', 'interpreter')
+    >>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin')
+    >>> mkdir(interpreter_bin_dir)
+    >>> mkdir(interpreter_dir, 'eggs')
+    >>> mkdir(interpreter_dir, 'parts')
+    >>> mkdir(interpreter_parts_dir)
+    >>> ws = zc.buildout.easy_install.install(
+    ...     ['demo', 'other'], join(interpreter_dir, 'eggs'), executable=py_path,
+    ...     links=[link_server], index=link_server+'index/',
+    ...     allowed_eggs_from_site_packages=['demo'])
+    >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
+    ...     interpreter_bin_dir, ws, py_path, interpreter_parts_dir,
+    ...     interpreter='py', include_site_packages=True)
+
+Now we will look at the paths in the site.py we generated.  Notice that the
+site-packages are at the end.  They were not before this bugfix.
+
+    >>> test = 'import pprint, sys; pprint.pprint(sys.path[-4:])'
+    >>> print call_py(join(interpreter_bin_dir, 'py'), test)
+    ['/interpreter/eggs/other-1.0-pyN.N.egg',
+     '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
+     '/executable_buildout/eggs/setuptools-0.0-pyN.N.egg',
+     '/executable_buildout/site-packages']
+    <BLANKLINE>
+    """
+
 def subprocesses_have_same_environment_by_default():
     """
 The scripts generated by sitepackage_safe_scripts set the PYTHONPATH so that,