Merge lp:~gary/zc.buildout/betafix3 into lp:zc.buildout

[Gary Poster]

This is the catch-all branch for three remaining small fixes.

1) I added MANIFEST.in files so that using non-svn branches/checkouts will produce workable distributions. See http://bazaar.launchpad.net/~gary/zc.buildout/betafix3/revision/556 .

2) I fixed bug 580563. I did not add a test. First, this is just a warning. Second, if at all possible, packages really should specify a dependency on setuptools, not distribute, since distribute can fulfill a setuptools dependency, but not the other way around. I didn't think it was worth the time. That said, I think it is doable (though distribute-related tests are sometimes tricky). See http://bazaar.launchpad.net/~gary/zc.buildout/betafix3/revision/557 .

3) I fixed bug 585188. Before this change buildout was more aggressive about processing .pth files than the standard site.py. This meant that it exposed a .pth problem that the Debian (or Ubuntu) packaging for repoze.what had with its plugins, when it was not visible in starting the system Python. I changed the zc.buildout-generated site.py to be more like the standard site.py. Tests show that the change is made, and that the script still works with the change. Tests do not show that the particular problem encountered with the repoze.what packaging is addressed: my brain rebelled when I tried to figure out a way to duplicate the nested .pth setup that would show the problem, but it would be fair to require me to try harder to do so. (See http://bazaar.launchpad.net/~gary/zc.buildout/betafix3/revision/558 and then http://bazaar.launchpad.net/~gary/zc.buildout/betafix3/revision/559 for the test adjustments.)

[Gary Poster (gary)]

Um, bootstrap.py is magically added back in this diff. I have no idea why--as noted in an earlier branch, it's been there all along. That part is spurious.

[Francis J. Lacoste (flacoste)]

Fair enough, I won't ask for more corner cases test to show that we cope around broken behavior.

review: Approve

Unmerged revisions

559. By Gary Poster on 2010-06-16

fix tests for changes

558. By Gary Poster on 2010-06-15

mimic standard site.py behavior for inclusion of .pth files

557. By Gary Poster on 2010-06-15

eliminate spurious warning if you are using distribute.

556. By Gary Poster on 2010-06-15

add files so releases can be made from a non-SVN checkout

555. By Gary Poster on 2010-06-15

fix virtualenv interaction by identfying broken virtualenv characteristic and reverting to previous behavior in that case.

554. By Gary Poster on 2010-06-15

Fix some problems with allowed_eggs_from_site_packages

553. By Gary Poster on 2010-06-15

get a baseline of zc.buildout with passing tests

=== added file 'MANIFEST.in'
--- MANIFEST.in	1970-01-01 00:00:00 +0000
+++ MANIFEST.in	2010-06-16 15:38:24 +0000
@@ -0,0 +1,3 @@
+include *.txt
+recursive-include src *.txt
+exclude MANIFEST.in buildout.cfg .bzrignore

=== added file 'bootstrap/bootstrap.py'
--- bootstrap/bootstrap.py	1970-01-01 00:00:00 +0000
+++ bootstrap/bootstrap.py	2010-06-16 15:38:24 +0000
@@ -0,0 +1,216 @@
+##############################################################################
+#
+# Copyright (c) 2006 Zope Foundation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Bootstrap a buildout-based project
+
+Simply run this script in a directory containing a buildout.cfg.
+The script accepts buildout command-line options, so you can
+use the -c option to specify an alternate configuration file.
+
+$Id$
+"""
+
+import os, shutil, sys, tempfile, textwrap, urllib, urllib2, subprocess
+from optparse import OptionParser
+
+if sys.platform == 'win32':
+    def quote(c):
+        if ' ' in c:
+            return '"%s"' % c # work around spawn lamosity on windows
+        else:
+            return c
+else:
+    quote = str
+
+# Detect https://bugs.launchpad.net/virtualenv/+bug/572545 .
+proc = subprocess.Popen(
+    [sys.executable, '-Sc', 'import ConfigParser'],
+    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+proc.communicate()
+has_broken_dash_S = bool(proc.returncode)
+
+# In order to be more robust in the face of system Pythons, we want to
+# run without site-packages loaded.  This is somewhat tricky, in
+# particular because Python 2.6's distutils imports site, so starting
+# with the -S flag is not sufficient.  However, we'll start with that:
+if not has_broken_dash_S and 'site' in sys.modules:
+    # We will restart with python -S.
+    args = sys.argv[:]
+    args[0:0] = [sys.executable, '-S']
+    args = map(quote, args)
+    os.execv(sys.executable, args)
+# Now we are running with -S.  We'll get the clean sys.path, import site
+# because distutils will do it later, and then reset the path and clean
+# out any namespace packages from site-packages that might have been
+# loaded by .pth files.
+clean_path = sys.path[:]
+import site
+sys.path[:] = clean_path
+for k, v in sys.modules.items():
+    if (hasattr(v, '__path__') and
+        len(v.__path__)==1 and
+        not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
+        # This is a namespace package.  Remove it.
+        sys.modules.pop(k)
+
+is_jython = sys.platform.startswith('java')
+
+setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
+distribute_source = 'http://python-distribute.org/distribute_setup.py'
+
+# parsing arguments
+def normalize_to_url(option, opt_str, value, parser):
+    if value:
+        if '://' not in value: # It doesn't smell like a URL.
+            value = 'file://%s' % (
+                urllib.pathname2url(
+                    os.path.abspath(os.path.expanduser(value))),)
+        if opt_str == '--download-base' and not value.endswith('/'):
+            # Download base needs a trailing slash to make the world happy.
+            value += '/'
+    else:
+        value = None
+    name = opt_str[2:].replace('-', '_')
+    setattr(parser.values, name, value)
+
+usage = '''\
+[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
+
+Bootstraps a buildout-based project.
+
+Simply run this script in a directory containing a buildout.cfg, using the
+Python that you want bin/buildout to use.
+
+Note that by using --setup-source and --download-base to point to
+local resources, you can keep this script from going over the network.
+'''
+
+parser = OptionParser(usage=usage)
+parser.add_option("-v", "--version", dest="version",
+                          help="use a specific zc.buildout version")
+parser.add_option("-d", "--distribute",
+                   action="store_true", dest="use_distribute", default=False,
+                   help="Use Distribute rather than Setuptools.")
+parser.add_option("--setup-source", action="callback", dest="setup_source",
+                  callback=normalize_to_url, nargs=1, type="string",
+                  help=("Specify a URL or file location for the setup file. "
+                        "If you use Setuptools, this will default to " +
+                        setuptools_source + "; if you use Distribute, this "
+                        "will default to " + distribute_source +"."))
+parser.add_option("--download-base", action="callback", dest="download_base",
+                  callback=normalize_to_url, nargs=1, type="string",
+                  help=("Specify a URL or directory for downloading "
+                        "zc.buildout and either Setuptools or Distribute. "
+                        "Defaults to PyPI."))
+parser.add_option("--eggs",
+                  help=("Specify a directory for storing eggs.  Defaults to "
+                        "a temporary directory that is deleted when the "
+                        "bootstrap script completes."))
+parser.add_option("-c", None, action="store", dest="config_file",
+                   help=("Specify the path to the buildout configuration "
+                         "file to be used."))
+
+options, args = parser.parse_args()
+
+# if -c was provided, we push it back into args for buildout's main function
+if options.config_file is not None:
+    args += ['-c', options.config_file]
+
+if options.eggs:
+    eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
+else:
+    eggs_dir = tempfile.mkdtemp()
+
+if options.setup_source is None:
+    if options.use_distribute:
+        options.setup_source = distribute_source
+    else:
+        options.setup_source = setuptools_source
+
+args = args + ['bootstrap']
+
+
+try:
+    to_reload = False
+    import pkg_resources
+    to_reload = True
+    if not hasattr(pkg_resources, '_distribute'):
+        raise ImportError
+    import setuptools # A flag.  Sometimes pkg_resources is installed alone.
+except ImportError:
+    ez_code = urllib2.urlopen(
+        options.setup_source).read().replace('\r\n', '\n')
+    ez = {}
+    exec ez_code in ez
+    setup_args = dict(to_dir=eggs_dir, download_delay=0)
+    if options.download_base:
+        setup_args['download_base'] = options.download_base
+    if options.use_distribute:
+        setup_args['no_fake'] = True
+    ez['use_setuptools'](**setup_args)
+    if to_reload:
+        reload(pkg_resources)
+    else:
+        import pkg_resources
+    # This does not (always?) update the default working set.  We will
+    # do it.
+    for path in sys.path:
+        if path not in pkg_resources.working_set.entries:
+            pkg_resources.working_set.add_entry(path)
+
+cmd = [quote(sys.executable),
+       '-c',
+       quote('from setuptools.command.easy_install import main; main()'),
+       '-mqNxd',
+       quote(eggs_dir)]
+
+if not has_broken_dash_S:
+    cmd.insert(1, '-S')
+
+if options.download_base:
+    cmd.extend(['-f', quote(options.download_base)])
+
+requirement = 'zc.buildout'
+if options.version:
+    requirement = '=='.join((requirement, options.version))
+cmd.append(requirement)
+
+if options.use_distribute:
+    setup_requirement = 'distribute'
+else:
+    setup_requirement = 'setuptools'
+ws = pkg_resources.working_set
+env = dict(
+    os.environ,
+    PYTHONPATH=ws.find(
+        pkg_resources.Requirement.parse(setup_requirement)).location)
+
+if is_jython:
+    import subprocess
+    exitcode = subprocess.Popen(cmd, env=env).wait()
+else: # Windows prefers this, apparently; otherwise we would prefer subprocess
+    exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
+if exitcode != 0:
+    sys.stdout.flush()
+    sys.stderr.flush()
+    print ("An error occured when trying to install zc.buildout. "
+           "Look above this message for any errors that "
+           "were output by easy_install.")
+    sys.exit(exitcode)
+
+ws.add_entry(eggs_dir)
+ws.require(requirement)
+import zc.buildout.buildout
+zc.buildout.buildout.main(args)
+if not options.eggs: # clean up temporary egg directory
+    shutil.rmtree(eggs_dir)

=== modified file 'src/zc/buildout/easy_install.py'
--- src/zc/buildout/easy_install.py	2010-06-16 15:38:24 +0000
+++ src/zc/buildout/easy_install.py	2010-06-16 15:38:24 +0000
@@ -818,7 +818,7 @@
     def _maybe_add_setuptools(self, ws, dist):
         if dist.has_metadata('namespace_packages.txt'):
             for r in dist.requires():
-                if r.project_name == 'setuptools':
+                if r.project_name in ('setuptools', 'distribute'):
                     break
             else:
                 # We have a namespace package but no requirement for setuptools
@@ -1673,7 +1673,8 @@
         %s
         ]
     for path in original_paths:
-        addsitedir(path, known_paths)'''
+        if path not in known_paths:
+            addsitedir(path, known_paths)'''
 
 addsitepackages_script = '''\
 def addsitepackages(known_paths):

=== modified file 'src/zc/buildout/easy_install.txt'
--- src/zc/buildout/easy_install.txt	1970-01-01 00:00:00 +0000
+++ src/zc/buildout/easy_install.txt	2010-06-16 15:38:24 +0000
@@ -1291,7 +1291,8 @@
             ...
             ]
         for path in original_paths:
-            addsitedir(path, known_paths)
+            if path not in known_paths:
+                addsitedir(path, known_paths)
         return known_paths
     <BLANKLINE>
     def original_addsitepackages(known_paths):...
@@ -1368,7 +1369,8 @@
             ...
             ]
         for path in original_paths:
-            addsitedir(path, known_paths)
+            if path not in known_paths:
+                addsitedir(path, known_paths)
         return known_paths
     <BLANKLINE>
     def original_addsitepackages(known_paths):...
@@ -1432,7 +1434,8 @@
             ...
             ]
         for path in original_paths:
-            addsitedir(path, known_paths)
+            if path not in known_paths:
+                addsitedir(path, known_paths)
         return known_paths
     <BLANKLINE>
     def original_addsitepackages(known_paths):...

=== added file 'z3c.recipe.scripts_/MANIFEST.in'
--- z3c.recipe.scripts_/MANIFEST.in	1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/MANIFEST.in	2010-06-16 15:38:24 +0000
@@ -0,0 +1,3 @@
+include *.txt
+recursive-include src *.txt
+exclude MANIFEST.in buildout.cfg .bzrignore

=== added file 'zc.recipe.egg_/MANIFEST.in'
--- zc.recipe.egg_/MANIFEST.in	1970-01-01 00:00:00 +0000
+++ zc.recipe.egg_/MANIFEST.in	2010-06-16 15:38:24 +0000
@@ -0,0 +1,3 @@
+include *.txt
+recursive-include src *.txt
+exclude MANIFEST.in buildout.cfg .bzrignore