Hi Sidnei. Looks good, but there's a practical problem in that it doesn't work for me on Karmic. Interesting that setuptools doesn't work for you on Lucid. More below. The Makefile fails for me on Karmic like this: bzr checkout lp:lazr-source-dependencies download-cache mkdir eggs if test ! -h src-py/lazrjs ; then \ ln -s ../src-js/lazrjs src-py/lazrjs;\ fi python -S bootstrap.py \ --distribute_setup-source=distribute_setup.py \ --download-base=download-cache/dist --eggs=eggs Downloading file:///home/gary/launchpad/lp-sourcedeps/newer-buildout/download-cache/dist/distribute-0.6.10.tar.gz Extracting in /tmp/tmpTmz0tV Now working in /tmp/tmpTmz0tV/distribute-0.6.10 Building a Distribute egg in /home/gary/launchpad/lp-sourcedeps/newer-buildout/eggs /home/gary/launchpad/lp-sourcedeps/newer-buildout/eggs/distribute-0.6.10-py2.6.egg Traceback (most recent call last): File "bootstrap.py", line 147, in PYTHONPATH=ws.find(pkg_resources.Requirement.parse('distribute')).location) AttributeError: 'NoneType' object has no attribute 'location' make: *** [bin/buildout] Error 1 This is a general problem with buildout and a system Python. I see we are using a hacked version of my old bootstrap.py. I've fixed this particular problem in my buildout branch for Jim. However, when I copy over my new bootstrap.py, I get a new error. gary@garybuntu:~/launchpad/lp-sourcedeps/newer-buildout$ make if test ! -h src-py/lazrjs ; then \ ln -s ../src-js/lazrjs src-py/lazrjs;\ fi python -S bootstrap.py \ --distribute \ --setup-source=distribute_setup.py \ --download-base=download-cache/dist --eggs=eggs Downloading file:///home/gary/launchpad/lp-sourcedeps/newer-buildout/download-cache/dist/distribute-0.6.10.tar.gz Extracting in /tmp/tmpBtweFh Now working in /tmp/tmpBtweFh/distribute-0.6.10 Building a Distribute egg in /home/gary/launchpad/lp-sourcedeps/newer-buildout/eggs /home/gary/launchpad/lp-sourcedeps/newer-buildout/eggs/distribute-0.6.10-py2.6.egg /home/gary/launchpad/lp-sourcedeps/newer-buildout/bootstrap.py:129: UserWarning: Module pkg_resources was already imported from /usr/lib/python2.6/dist-packages/pkg_resources.pyc, but /home/gary/launchpad/lp-sourcedeps/newer-buildout/eggs/distribute-0.6.10-py2.6.egg is being added to sys.path pkg_resources.working_set.add_entry(path) /home/gary/launchpad/lp-sourcedeps/newer-buildout/bootstrap.py:129: UserWarning: Module site was already imported from /usr/lib/python2.6/site.pyc, but /home/gary/launchpad/lp-sourcedeps/newer-buildout/eggs/distribute-0.6.10-py2.6.egg is being added to sys.path pkg_resources.working_set.add_entry(path) Traceback (most recent call last): File "bootstrap.py", line 179, in ws.require(requirement) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 626, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 524, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: setuptools make: *** [bin/buildout] Error 1 That's with the following diff. The bootstrap is just a straight copy from the bootstrap in lp:~gary/zc.buildout/python-support-8-support-subprocess (or svn.zope.org/repos/main/zc.buildout/branches/gary-8). The Makefile is a real change here in lazr-js though, to accommodate the spelling in the bootstrap.py. === modified file 'Makefile' --- Makefile 2010-03-16 00:48:52 +0000 +++ Makefile 2010-03-16 11:44:33 +0000 @@ -45,7 +45,8 @@ fi $(SHHH) $(PYTHON) -S bootstrap.py \ - --distribute_setup-source=distribute_setup.py \ + --distribute \ + --setup-source=distribute_setup.py \ --download-base=download-cache/dist --eggs=eggs $(PY): bin/buildout buildout.cfg setup.py === modified file 'bootstrap.py' --- bootstrap.py 2010-03-16 00:48:52 +0000 +++ bootstrap.py 2010-03-16 11:34:38 +0000 @@ -20,106 +20,113 @@ $Id$ """ -import os, re, shutil, sys, tempfile, textwrap, urllib, urllib2 - -# We have to manually parse our options rather than using one of the stdlib -# tools because we want to pass the ones we don't recognize along to -# zc.buildout.buildout.main. - -configuration = { - '--distribute_setup-source': 'http://python-distribute.org/distribute_setup.py', - '--version': '', - '--download-base': None, - '--eggs': None} - -helpstring = __doc__ + textwrap.dedent(''' - This script recognizes the following options itself. The first option it - encounters that is not one of these will cause the script to stop parsing - options and pass the rest on to buildout. Therefore, if you want to use - any of the following options *and* buildout command-line options like - -c, first use the following options, and then use the buildout options. - - Options: - --version=ZC_BUILDOUT_VERSION - Specify a version number of the zc.buildout to use - --distribute_setup-source=URL_OR_FILE - Specify a URL or file location for the distribute_setup file. - Defaults to - %(--distribute_setup-source)s - --download-base=URL_OR_DIRECTORY - Specify a URL or directory for downloading setuptools and - zc.buildout. Defaults to PyPI. - --eggs=DIRECTORY - Specify a directory for storing eggs. Defaults to a temporary - directory that is deleted when the bootstrap script completes. - - By using --distribute_setup-source and --download-base to point to local resources, - you can keep this script from going over the network. - ''' % configuration) -match_equals = re.compile(r'(%s)=(\S*)' % ('|'.join(configuration),)).match -args = sys.argv[1:] -if args == ['--help']: - print helpstring - sys.exit(0) - -# defaults -tmpeggs = None - -while args: - val = args[0] - if val in configuration: - del args[0] - if not args or args[0].startswith('-'): - print "ERROR: %s requires an argument." - print helpstring - sys.exit(1) - configuration[val] = args[0] +import os, shutil, sys, tempfile, textwrap, urllib, urllib2 +from optparse import OptionParser + +is_jython = sys.platform.startswith('java') + +setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py' +distribute_source = 'http://python-distribute.org/distribute_setup.py' + +# parsing arguments +def normalize_to_url(option, opt_str, value, parser): + if value: + if '://' not in value: # It doesn't smell like a URL. + value = 'file://%s' % ( + urllib.pathname2url( + os.path.abspath(os.path.expanduser(value))),) + if opt_str == '--download-base' and not value.endswith('/'): + # Download base needs a trailing slash to make the world happy. + value += '/' else: - match = match_equals(val) - if match and match.group(1) in configuration: - configuration[match.group(1)] = match.group(2) - else: - break - del args[0] - -for name in ('--distribute_setup-source', '--download-base'): - val = configuration[name] - if val is not None and '://' not in val: # we're being lazy. - configuration[name] = 'file://%s' % ( - urllib.pathname2url(os.path.abspath(os.path.expanduser(val))),) - -if (configuration['--download-base'] and - not configuration['--download-base'].endswith('/')): - # download base needs a trailing slash to make the world happy - configuration['--download-base'] += '/' - -if not configuration['--eggs']: - configuration['--eggs'] = tmpeggs = tempfile.mkdtemp() + value = None + name = opt_str[2:].replace('-', '_') + setattr(parser.values, name, value) + +usage = '''\ +[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options] + +Bootstraps a buildout-based project. + +Simply run this script in a directory containing a buildout.cfg, using the +Python that you want bin/buildout to use. + +Note that by using --setup-source and --download-base to point to +local resources, you can keep this script from going over the network. +''' + +parser = OptionParser(usage=usage) +parser.add_option("-v", "--version", dest="version", + help="use a specific zc.buildout version") +parser.add_option("-d", "--distribute", + action="store_true", dest="use_distribute", default=False, + help="Use Distribute rather than Setuptools.") +parser.add_option("--setup-source", action="callback", dest="setup_source", + callback=normalize_to_url, nargs=1, type="string", + help=("Specify a URL or file location for the setup file. " + "If you use Setuptools, this will default to " + + setuptools_source + "; if you use Distribute, this " + "will default to " + distribute_source +".")) +parser.add_option("--download-base", action="callback", dest="download_base", + callback=normalize_to_url, nargs=1, type="string", + help=("Specify a URL or directory for downloading " + "zc.buildout and either Setuptools or Distribute. " + "Defaults to PyPI.")) +parser.add_option("--eggs", + help=("Specify a directory for storing eggs. Defaults to " + "a temporary directory that is deleted when the " + "bootstrap script completes.")) +parser.add_option("-c", None, action="store", dest="config_file", + help=("Specify the path to the buildout configuration " + "file to be used.")) + +options, args = parser.parse_args() + +# if -c was provided, we push it back into args for buildout' main function +if options.config_file is not None: + args += ['-c', options.config_file] + +if options.eggs: + eggs_dir = os.path.abspath(os.path.expanduser(options.eggs)) else: - configuration['--eggs'] = os.path.abspath( - os.path.expanduser(configuration['--eggs'])) - -if configuration['--version']: - configuration['--version'] = '==' + configuration['--version'] - -to_reload = False + eggs_dir = tempfile.mkdtemp() + +if options.setup_source is None: + if options.use_distribute: + options.setup_source = distribute_source + else: + options.setup_source = setuptools_source + +args = args + ['bootstrap'] + + try: + to_reload = False import pkg_resources + to_reload = True if not hasattr(pkg_resources, '_distribute'): - to_reload = True raise ImportError + import setuptools # A flag. Sometimes pkg_resources is installed alone. except ImportError: + ez_code = urllib2.urlopen( + options.setup_source).read().replace('\r\n', '\n') ez = {} - exec urllib2.urlopen(configuration['--distribute_setup-source']).read() in ez - setuptools_args = dict(to_dir=configuration['--eggs'], download_delay=0, no_fake=True) - if configuration['--download-base']: - setuptools_args['download_base'] = configuration['--download-base'] - ez['use_setuptools'](**setuptools_args) - + exec ez_code in ez + setup_args = dict(to_dir=eggs_dir, download_delay=0) + if options.download_base: + setup_args['download_base'] = options.download_base + if options.use_distribute: + setup_args['no_fake'] = True + ez['use_setuptools'](**setup_args) if to_reload: reload(pkg_resources) else: import pkg_resources + # This does not (always?) update the default working set. We will + # do it. + for path in sys.path: + if path not in pkg_resources.working_set.entries: + pkg_resources.working_set.add_entry(path) if sys.platform == 'win32': def quote(c): @@ -130,32 +137,47 @@ else: def quote (c): return c + cmd = [quote(sys.executable), '-c', quote('from setuptools.command.easy_install import main; main()'), '-mqNxd', - quote(configuration['--eggs'])] - -if configuration['--download-base']: - cmd.extend(['-f', quote(configuration['--download-base'])]) - -cmd.append('zc.buildout' + configuration['--version']) - + quote(eggs_dir)] + +if options.download_base: + cmd.extend(['-f', quote(options.download_base)]) + +requirement = 'zc.buildout' +if options.version: + requirement = '=='.join((requirement, options.version)) +cmd.append(requirement) + +if options.use_distribute: + setup_requirement = 'distribute' +else: + setup_requirement = 'setuptools' ws = pkg_resources.working_set env = dict( os.environ, - PYTHONPATH=ws.find(pkg_resources.Requirement.parse('distribute')).location) + PYTHONPATH=ws.find( + pkg_resources.Requirement.parse(setup_requirement)).location) -exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env])) +if is_jython: + import subprocess + exitcode = subprocess.Popen(cmd, env=env).wait() +else: # Windows prefers this, apparently; otherwise we would prefer subprocess + exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env])) if exitcode != 0: - # we shouldn't need an error message because a failure - # should have generated a visible traceback in the subprocess. + sys.stdout.flush() + sys.stderr.flush() + print ("An error occured when trying to install zc.buildout. " + "Look above this message for any errors that " + "were output by easy_install.") sys.exit(exitcode) -ws.add_entry(configuration['--eggs']) -ws.require('zc.buildout' + configuration['--version']) +ws.add_entry(eggs_dir) +ws.require(requirement) import zc.buildout.buildout -args.append('bootstrap') zc.buildout.buildout.main(args) -if tmpeggs is not None: - shutil.rmtree(tmpeggs) +if not options.eggs: # clean up temporary egg directory + shutil.rmtree(eggs_dir) That's a new error I haven't seen before. It also seems to imply that that there's some nasty setuptools/distribute/system python interaction, because using a clean, non-system Python works fine for me. I notice that we are using -S in our Makefile. That no longer really works because in Python2.6 distutils imports site, so the effect of -S is removed. We can do nasty sys.module hacks, I suppose. For instance, if we add this to just after the imports of bootstrap.py, the build works on Karmic: clean_path = sys.path[:] import site sys.path[:] = clean_path for k, v in sys.modules.items(): if (hasattr(v, '__path__') and len(v.__path__)==1 and not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))): # This is a namespace package. Remove it. sys.modules.pop(k) That works if you run bootstrap with -S (as we do thanks to the Makefile, per above). A fully general solution would be to make the file check to see if site were in sys.modules, and restart with os.execv adding -S to the python invocation if it finds site around. I probably should do that for my buildout branch. I find this tricky in Windows, so help would be welcome--though there are working examples in buildout.py to crib off of. But anyway, I'd like this to work in Karmic before it is merged. I have put my changes in lp:~gary/lazr-js/newer-buildout if you want to look at them.