Merge lp:~gary/zc.buildout/python-support-8-support-subprocess into lp:zc.buildout

Proposed by Gary Poster on 2010-03-19
Status: Needs review
Proposed branch: lp:~gary/zc.buildout/python-support-8-support-subprocess
Merge into: lp:zc.buildout
Prerequisite: lp:~gary/zc.buildout/python-support-7
Diff against target: 644 lines (+324/-65) (has conflicts)
8 files modified
bootstrap/bootstrap.py (+33/-10)
buildout.cfg (+3/-3)
dev.py (+43/-3)
src/zc/buildout/buildout.py (+40/-19)
src/zc/buildout/easy_install.py (+26/-8)
src/zc/buildout/easy_install.txt (+11/-1)
src/zc/buildout/tests.py (+149/-5)
src/zc/buildout/update.txt (+19/-16)
Text conflict in CHANGES.txt
Text conflict in src/zc/buildout/buildout.py
Text conflict in src/zc/buildout/buildout.txt
Text conflict in src/zc/buildout/easy_install.py
Text conflict in src/zc/buildout/tests.py
To merge this branch: bzr merge lp:~gary/zc.buildout/python-support-8-support-subprocess
Reviewer Review Type Date Requested Status
Francis J. Lacoste (community) 2010-03-19 Approve on 2010-03-19
Review via email: mp+21732@code.launchpad.net

Description of the change

This branch does two things.

1) It makes it simple to start Python processes from scripts started with the new recipe: PYTHONPATH is set so that everything works by default.

2) It makes bootstrap.py (and dev.py) yet more robust in the face of system Pythons.

#1 affects #2.

To post a comment you must log in.
Francis J. Lacoste (flacoste) wrote :
Download full text (4.1 KiB)

> === modified file 'bootstrap/bootstrap.py'
> +# In order to be more robust in the face of system Pythons, we want to run
> +# with site-packages loaded. This is somewhat tricky, in particular because

With or without? It seems that you want to run with it, but remove any all
namespace from it? Anyway the comment could be clearer given the
non-obviousness of the code.

> +# Python 2.6's distutils imports site, so starting with the -S flag is not
> +# sufficient.
> +if 'site' in sys.modules:
> + # We will restart with python -S.
> + args = sys.argv[:]
> + args[0:0] = [sys.executable, '-S']
> + args = map(quote, args)
> + os.execv(sys.executable, args)
> +clean_path = sys.path[:]
> +import site
> +sys.path[:] = clean_path
> +for k, v in sys.modules.items():
> + if (hasattr(v, '__path__') and
> + len(v.__path__)==1 and
> + not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
> + # This is a namespace package. Remove it.
> + sys.modules.pop(k)
> +

> === modified file 'dev.py'
> +# In order to be more robust in the face of system Pythons, we want to run
> +# with site-packages loaded. This is somewhat tricky, in particular because
> +# Python 2.6's distutils imports site, so starting with the -S flag is not
> +# sufficient.

Since this is copy and paste from the other location, my other comment applies
here also.

> +if 'site' in sys.modules:
> + # We will restart with python -S.
> + args = sys.argv[:]
> + args[0:0] = [sys.executable, '-S']
> + args = map(quote, args)
> + os.execv(sys.executable, args)
> +clean_path = sys.path[:]
> +import site
> +sys.path[:] = clean_path
> +for k, v in sys.modules.items():
> + if (hasattr(v, '__path__') and
> + len(v.__path__)==1 and
> + not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
> + # This is a namespace package. Remove it.
> + sys.modules.pop(k)
> +
> is_jython = sys.platform.startswith('java')

> === modified file 'src/zc/buildout/buildout.py'

> - args.insert(0, zc.buildout.easy_install._safe_arg (sys.executable))
> -
> + args.insert(0, zc.buildout.easy_install._safe_arg(sys.executable))
> + env = os.environ.copy()
> + env['PYTHONPATH'] = partsdir
> if is_jython:
> - sys.exit(subprocess.Popen([sys.executable] + list(args)).wait())
> + sys.exit(
> + subprocess.Popen(
> + [sys.executable] + list(args), env=env).wait())
> else:
> - sys.exit(os.spawnv(os.P_WAIT, sys.executable, args))
> + sys.exit(os.spawnve(os.P_WAIT, sys.executable, args, env))
>

The intent here is to run the script with only partsdir in the path?

> === modified file 'src/zc/buildout/easy_install.py'
> --- src/zc/buildout/easy_install.py 2010-03-19 16:04:20 +0000
> +++ src/zc/buildout/easy_install.py 2010-03-19 16:04:20 +0000
> @@ -99,6 +99,7 @@
> "print repr([os.path.normpath(p) for p in sys.path if p])"])
> # Windows needs some (as yet to be determined) part of the real env.
> env = os.environ.copy()
> + env.pop('PYTHONPATH', None)

Care to explain w...

Read more...

review: Needs Information
564. By Gary Poster on 2010-03-19

add explanatory comments; extend test to show that scripts honor explicit PYTHONPATH

Gary Poster (gary) wrote :

Good call on all comments. In the new revision, I answered your questions in the new code comments, and added a test as you described.

(Yes, we want to run without site-packages loaded.)

Francis J. Lacoste (flacoste) wrote :

All good!

review: Approve

Unmerged revisions

564. By Gary Poster on 2010-03-19

add explanatory comments; extend test to show that scripts honor explicit PYTHONPATH

563. By Gary Poster on 2010-03-19

fix intermittent test failure in update.txt

562. By Gary Poster on 2010-03-19

fixes for bootstrap and a system Python; changes based on learning what would be necessary to be able to develop buildout with a system Python (zc.recipe.testing would also need to use sitepackage_safe_scripts)

561. By Gary Poster on 2010-03-11

set up PYTHONPATH for scripts too, so subprocesses are good to go by default.

560. By Gary Poster on 2010-02-24

merge from gary-6 <- gary-5

559. By Gary Poster on 2010-02-24

with these changes, I can build zc.buildout and run its tests successfully with my system Python. To make it fully robust, zc.recipe.test probably would need to use sitepackage_safe_scripts, but this works for now.

558. By Gary Poster on 2010-02-24

add test for recent fix for buildout

557. By Gary Poster on 2010-02-24

make the buildout script itself safe for a Python with site packages.

556. By Gary Poster on 2010-02-23

merge from gary-5 <- gary-4

555. By Gary Poster on 2010-02-23

merge from gary-5 <- gary-4

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'bootstrap/bootstrap.py'
--- bootstrap/bootstrap.py 2010-03-19 19:13:23 +0000
+++ bootstrap/bootstrap.py 2010-03-19 19:13:23 +0000
@@ -23,6 +23,39 @@
23import os, shutil, sys, tempfile, textwrap, urllib, urllib223import os, shutil, sys, tempfile, textwrap, urllib, urllib2
24from optparse import OptionParser24from optparse import OptionParser
2525
26if sys.platform == 'win32':
27 def quote(c):
28 if ' ' in c:
29 return '"%s"' % c # work around spawn lamosity on windows
30 else:
31 return c
32else:
33 quote = str
34
35# In order to be more robust in the face of system Pythons, we want to
36# run without site-packages loaded. This is somewhat tricky, in
37# particular because Python 2.6's distutils imports site, so starting
38# with the -S flag is not sufficient. However, we'll start with that:
39if 'site' in sys.modules:
40 # We will restart with python -S.
41 args = sys.argv[:]
42 args[0:0] = [sys.executable, '-S']
43 args = map(quote, args)
44 os.execv(sys.executable, args)
45# Now we are running with -S. We'll get the clean sys.path, import site
46# because distutils will do it later, and then reset the path and clean
47# out any namespace packages from site-packages that might have been
48# loaded by .pth files.
49clean_path = sys.path[:]
50import site
51sys.path[:] = clean_path
52for k, v in sys.modules.items():
53 if (hasattr(v, '__path__') and
54 len(v.__path__)==1 and
55 not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
56 # This is a namespace package. Remove it.
57 sys.modules.pop(k)
58
26is_jython = sys.platform.startswith('java')59is_jython = sys.platform.startswith('java')
2760
28setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'61setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
@@ -128,16 +161,6 @@
128 if path not in pkg_resources.working_set.entries:161 if path not in pkg_resources.working_set.entries:
129 pkg_resources.working_set.add_entry(path)162 pkg_resources.working_set.add_entry(path)
130163
131if sys.platform == 'win32':
132 def quote(c):
133 if ' ' in c:
134 return '"%s"' % c # work around spawn lamosity on windows
135 else:
136 return c
137else:
138 def quote (c):
139 return c
140
141cmd = [quote(sys.executable),164cmd = [quote(sys.executable),
142 '-c',165 '-c',
143 quote('from setuptools.command.easy_install import main; main()'),166 quote('from setuptools.command.easy_install import main; main()'),
144167
=== modified file 'buildout.cfg'
--- buildout.cfg 2010-03-19 19:13:23 +0000
+++ buildout.cfg 2010-03-19 19:13:23 +0000
@@ -3,14 +3,14 @@
3parts = test oltest py3parts = test oltest py
44
5[py]5[py]
6recipe = zc.recipe.egg6recipe = z3c.recipe.scripts
7eggs = zc.buildout7eggs = zc.buildout
8 zope.testing8 zope.testing
9interpreter = py9interpreter = py
1010
11[test]11[test]
12recipe = zc.recipe.testrunner12recipe = zc.recipe.testrunner
13eggs = 13eggs =
14 zc.buildout14 zc.buildout
15 zc.recipe.egg15 zc.recipe.egg
16 z3c.recipe.scripts16 z3c.recipe.scripts
@@ -18,7 +18,7 @@
18# Tests that can be run wo a network18# Tests that can be run wo a network
19[oltest]19[oltest]
20recipe = zc.recipe.testrunner20recipe = zc.recipe.testrunner
21eggs = 21eggs =
22 zc.buildout22 zc.buildout
23 zc.recipe.egg23 zc.recipe.egg
24 z3c.recipe.scripts24 z3c.recipe.scripts
2525
=== modified file 'dev.py'
--- dev.py 2010-03-19 19:13:23 +0000
+++ dev.py 2010-03-19 19:13:23 +0000
@@ -21,6 +21,39 @@
2121
22import os, shutil, sys, subprocess, urllib222import os, shutil, sys, subprocess, urllib2
2323
24if sys.platform == 'win32':
25 def quote(c):
26 if ' ' in c:
27 return '"%s"' % c # work around spawn lamosity on windows
28 else:
29 return c
30else:
31 quote = str
32
33# In order to be more robust in the face of system Pythons, we want to
34# run without site-packages loaded. This is somewhat tricky, in
35# particular because Python 2.6's distutils imports site, so starting
36# with the -S flag is not sufficient. However, we'll start with that:
37if 'site' in sys.modules:
38 # We will restart with python -S.
39 args = sys.argv[:]
40 args[0:0] = [sys.executable, '-S']
41 args = map(quote, args)
42 os.execv(sys.executable, args)
43# Now we are running with -S. We'll get the clean sys.path, import site
44# because distutils will do it later, and then reset the path and clean
45# out any namespace packages from site-packages that might have been
46# loaded by .pth files.
47clean_path = sys.path[:]
48import site
49sys.path[:] = clean_path
50for k, v in sys.modules.items():
51 if (hasattr(v, '__path__') and
52 len(v.__path__)==1 and
53 not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
54 # This is a namespace package. Remove it.
55 sys.modules.pop(k)
56
24is_jython = sys.platform.startswith('java')57is_jython = sys.platform.startswith('java')
2558
26for d in 'eggs', 'develop-eggs', 'bin':59for d in 'eggs', 'develop-eggs', 'bin':
@@ -49,14 +82,20 @@
49env['PYTHONPATH'] = os.path.dirname(pkg_resources.__file__)82env['PYTHONPATH'] = os.path.dirname(pkg_resources.__file__)
50subprocess.Popen(83subprocess.Popen(
51 [sys.executable] +84 [sys.executable] +
52 ['setup.py', '-q', 'develop', '-m', '-x', '-d', 'develop-eggs'],85 ['-S', 'setup.py', '-q', 'develop', '-m', '-x', '-d', 'develop-eggs'],
53 env=env).wait()86 env=env).wait()
5487
55pkg_resources.working_set.add_entry('src')88pkg_resources.working_set.add_entry('src')
5689
57import zc.buildout.easy_install90import zc.buildout.easy_install
58zc.buildout.easy_install.scripts(91if not os.path.exists('parts'):
59 ['zc.buildout'], pkg_resources.working_set , sys.executable, 'bin')92 os.mkdir('parts')
93partsdir = os.path.join('parts', 'buildout')
94if not os.path.exists(partsdir):
95 os.mkdir(partsdir)
96zc.buildout.easy_install.sitepackage_safe_scripts(
97 'bin', pkg_resources.working_set, sys.executable, partsdir,
98 reqs=['zc.buildout'])
6099
61bin_buildout = os.path.join('bin', 'buildout')100bin_buildout = os.path.join('bin', 'buildout')
62101
@@ -64,4 +103,5 @@
64 # Jython needs the script to be called twice via sys.executable103 # Jython needs the script to be called twice via sys.executable
65 assert subprocess.Popen([sys.executable] + [bin_buildout]).wait() == 0104 assert subprocess.Popen([sys.executable] + [bin_buildout]).wait() == 0
66105
106
67sys.exit(subprocess.Popen(bin_buildout).wait())107sys.exit(subprocess.Popen(bin_buildout).wait())
68108
=== modified file 'src/zc/buildout/buildout.py'
--- src/zc/buildout/buildout.py 2010-03-19 19:13:23 +0000
+++ src/zc/buildout/buildout.py 2010-03-19 19:13:23 +0000
@@ -375,7 +375,9 @@
375 if options.get('offline') == 'true':375 if options.get('offline') == 'true':
376 ws = zc.buildout.easy_install.working_set(376 ws = zc.buildout.easy_install.working_set(
377 distributions, options['executable'],377 distributions, options['executable'],
378 [options['develop-eggs-directory'], options['eggs-directory']]378 [options['develop-eggs-directory'],
379 options['eggs-directory']],
380 include_site_packages=False,
379 )381 )
380 else:382 else:
381 ws = zc.buildout.easy_install.install(383 ws = zc.buildout.easy_install.install(
@@ -385,7 +387,8 @@
385 executable=options['executable'],387 executable=options['executable'],
386 path=[options['develop-eggs-directory']],388 path=[options['develop-eggs-directory']],
387 newest=self.newest,389 newest=self.newest,
388 allow_hosts=self._allow_hosts390 allow_hosts=self._allow_hosts,
391 include_site_packages=False,
389 )392 )
390393
391 # Now copy buildout and setuptools eggs, and record destination eggs:394 # Now copy buildout and setuptools eggs, and record destination eggs:
@@ -851,16 +854,19 @@
851 if not self.newest:854 if not self.newest:
852 return855 return
853856
857 options = self['buildout']
858
854 ws = zc.buildout.easy_install.install(859 ws = zc.buildout.easy_install.install(
855 [860 [
856 (spec + ' ' + self['buildout'].get(spec+'-version', '')).strip()861 (spec + ' ' + options.get(spec+'-version', '')).strip()
857 for spec in ('zc.buildout', 'setuptools')862 for spec in ('zc.buildout', 'setuptools')
858 ],863 ],
859 self['buildout']['eggs-directory'],864 options['eggs-directory'],
860 links = self['buildout'].get('find-links', '').split(),865 links = options.get('find-links', '').split(),
861 index = self['buildout'].get('index'),866 index = options.get('index'),
862 path = [self['buildout']['develop-eggs-directory']],867 path = [options['develop-eggs-directory']],
863 allow_hosts = self._allow_hosts868 allow_hosts = self._allow_hosts,
869 include_site_packages=False
864 )870 )
865871
866 upgraded = []872 upgraded = []
@@ -876,7 +882,7 @@
876 __doing__ = 'Upgrading.'882 __doing__ = 'Upgrading.'
877883
878 should_run = realpath(884 should_run = realpath(
879 os.path.join(os.path.abspath(self['buildout']['bin-directory']),885 os.path.join(os.path.abspath(options['bin-directory']),
880 'buildout')886 'buildout')
881 )887 )
882 if sys.platform == 'win32':888 if sys.platform == 'win32':
@@ -908,21 +914,34 @@
908914
909 # the new dist is different, so we've upgraded.915 # the new dist is different, so we've upgraded.
910 # Update the scripts and return True916 # Update the scripts and return True
911 zc.buildout.easy_install.scripts(917 partsdir = os.path.join(options['parts-directory'], 'buildout')
912 ['zc.buildout'], ws, sys.executable,918 if os.path.exists(partsdir):
913 self['buildout']['bin-directory'],919 # This is primarily for unit tests, in which .py files change too
914 )920 # fast for Python to know to regenerate the .pyc/.pyo files.
921 shutil.rmtree(partsdir)
922 os.mkdir(partsdir)
923 zc.buildout.easy_install.sitepackage_safe_scripts(
924 options['bin-directory'], ws, sys.executable, partsdir,
925 reqs=['zc.buildout'])
915926
916 # Restart927 # Restart
917 args = map(zc.buildout.easy_install._safe_arg, sys.argv)928 args = map(zc.buildout.easy_install._safe_arg, sys.argv)
918 if not __debug__:929 if not __debug__:
919 args.insert(0, '-O')930 args.insert(0, '-O')
920 args.insert(0, zc.buildout.easy_install._safe_arg (sys.executable))931 args.insert(0, zc.buildout.easy_install._safe_arg(sys.executable))
921932 # We want to make sure that our new site.py is used for rerunning
933 # buildout, so we put the partsdir in PYTHONPATH for our restart.
934 # This overrides any set PYTHONPATH, but since we generally are
935 # trying to run with a completely "clean" python (only the standard
936 # library) then that should be fine.
937 env = os.environ.copy()
938 env['PYTHONPATH'] = partsdir
922 if is_jython:939 if is_jython:
923 sys.exit(subprocess.Popen([sys.executable] + list(args)).wait())940 sys.exit(
941 subprocess.Popen(
942 [sys.executable] + list(args), env=env).wait())
924 else:943 else:
925 sys.exit(os.spawnv(os.P_WAIT, sys.executable, args))944 sys.exit(os.spawnve(os.P_WAIT, sys.executable, args, env))
926945
927 def _load_extensions(self):946 def _load_extensions(self):
928 __doing__ = 'Loading extensions.'947 __doing__ = 'Loading extensions.'
@@ -943,7 +962,8 @@
943 working_set=pkg_resources.working_set,962 working_set=pkg_resources.working_set,
944 links = self['buildout'].get('find-links', '').split(),963 links = self['buildout'].get('find-links', '').split(),
945 index = self['buildout'].get('index'),964 index = self['buildout'].get('index'),
946 newest=self.newest, allow_hosts=self._allow_hosts)965 newest=self.newest, allow_hosts=self._allow_hosts,
966 include_site_packages=False)
947967
948 # Clear cache because extensions might now let us read pages we968 # Clear cache because extensions might now let us read pages we
949 # couldn't read before.969 # couldn't read before.
@@ -1060,7 +1080,8 @@
1060 path=path,1080 path=path,
1061 working_set=pkg_resources.working_set,1081 working_set=pkg_resources.working_set,
1062 newest=buildout.newest,1082 newest=buildout.newest,
1063 allow_hosts=buildout._allow_hosts1083 allow_hosts=buildout._allow_hosts,
1084 include_site_packages=False,
1064 )1085 )
10651086
1066 __doing__ = 'Loading %s recipe entry %s:%s.', group, spec, entry1087 __doing__ = 'Loading %s recipe entry %s:%s.', group, spec, entry
10671088
=== modified file 'src/zc/buildout/easy_install.py'
--- src/zc/buildout/easy_install.py 2010-03-19 19:13:23 +0000
+++ src/zc/buildout/easy_install.py 2010-03-19 19:13:23 +0000
@@ -99,6 +99,10 @@
99 "print repr([os.path.normpath(p) for p in sys.path if p])"])99 "print repr([os.path.normpath(p) for p in sys.path if p])"])
100 # Windows needs some (as yet to be determined) part of the real env.100 # Windows needs some (as yet to be determined) part of the real env.
101 env = os.environ.copy()101 env = os.environ.copy()
102 # We need to make sure that PYTHONPATH, which will often be set
103 # to include a custom buildout-generated site.py, is not set, or
104 # else we will not get an accurate sys.path for the executable.
105 env.pop('PYTHONPATH', None)
102 env.update(kwargs)106 env.update(kwargs)
103 _proc = subprocess.Popen(107 _proc = subprocess.Popen(
104 cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)108 cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
@@ -337,9 +341,6 @@
337 self._site_packages))341 self._site_packages))
338 if self._include_site_packages:342 if self._include_site_packages:
339 path.extend(self._site_packages)343 path.extend(self._site_packages)
340 # else we could try to still include the buildout_and_setuptools_path
341 # if the elements are not in site_packages, but we're not bothering
342 # with this optimization for now, in the name of code simplicity.
343 if dest is not None and dest not in path:344 if dest is not None and dest not in path:
344 path.insert(0, dest)345 path.insert(0, dest)
345 self._path = path346 self._path = path
@@ -1209,9 +1210,9 @@
1209 generated.append(_generate_site(1210 generated.append(_generate_site(
1210 site_py_dest, working_set, executable, extra_paths,1211 site_py_dest, working_set, executable, extra_paths,
1211 include_site_packages, relative_paths))1212 include_site_packages, relative_paths))
1212 script_initialization = (1213 script_initialization = _script_initialization_template % dict(
1213 '\nimport site # imports custom buildout-generated site.py\n%s' % (1214 site_py_dest=site_py_dest,
1214 script_initialization,))1215 script_initialization=script_initialization)
1215 if not script_initialization.endswith('\n'):1216 if not script_initialization.endswith('\n'):
1216 script_initialization += '\n'1217 script_initialization += '\n'
1217 generated.extend(_generate_scripts(1218 generated.extend(_generate_scripts(
@@ -1222,6 +1223,15 @@
1222 interpreter, dest, executable, site_py_dest, relative_paths))1223 interpreter, dest, executable, site_py_dest, relative_paths))
1223 return generated1224 return generated
12241225
1226_script_initialization_template = '''
1227import site # imports custom buildout-generated site.py
1228import os
1229path = %(site_py_dest)r
1230if os.environ.get('PYTHONPATH'):
1231 path = os.pathsep.join([path, os.environ['PYTHONPATH']])
1232os.environ['PYTHONPATH'] = path
1233%(script_initialization)s'''
1234
1225# Utilities for the script generation functions.1235# Utilities for the script generation functions.
12261236
1227# These are shared by both ``scripts`` and ``sitepackage_safe_scripts``1237# These are shared by both ``scripts`` and ``sitepackage_safe_scripts``
@@ -1504,8 +1514,14 @@
1504 "fp, path, desc = imp.find_module(%r); "1514 "fp, path, desc = imp.find_module(%r); "
1505 "fp.close; "1515 "fp.close; "
1506 "print path" % (name,)]1516 "print path" % (name,)]
1517 env = os.environ.copy()
1518 # We need to make sure that PYTHONPATH, which will often be set to
1519 # include a custom buildout-generated site.py, is not set, or else
1520 # we will not get an accurate value for the "real" site.py and
1521 # sitecustomize.py.
1522 env.pop('PYTHONPATH', None)
1507 _proc = subprocess.Popen(1523 _proc = subprocess.Popen(
1508 cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)1524 cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
1509 stdout, stderr = _proc.communicate();1525 stdout, stderr = _proc.communicate();
1510 if _proc.returncode:1526 if _proc.returncode:
1511 logger.info(1527 logger.info(
@@ -1601,7 +1617,9 @@
1601 site.close()1617 site.close()
1602 real_site.close()1618 real_site.close()
1603 if not successful_rewrite:1619 if not successful_rewrite:
1604 raise RuntimeError('Buildout did not successfully rewrite site.py')1620 raise RuntimeError(
1621 'Buildout did not successfully rewrite %s to %s' %
1622 (real_site_path, site_path))
1605 return site_path1623 return site_path
16061624
1607namespace_include_site_packages_setup = '''1625namespace_include_site_packages_setup = '''
16081626
=== modified file 'src/zc/buildout/easy_install.txt'
--- src/zc/buildout/easy_install.txt 2010-03-19 19:13:23 +0000
+++ src/zc/buildout/easy_install.txt 2010-03-19 19:13:23 +0000
@@ -1499,6 +1499,11 @@
1499 <BLANKLINE>1499 <BLANKLINE>
1500 <BLANKLINE>1500 <BLANKLINE>
1501 import site # imports custom buildout-generated site.py1501 import site # imports custom buildout-generated site.py
1502 import os
1503 path = '/interpreter/parts/interpreter'
1504 if os.environ.get('PYTHONPATH'):
1505 path = os.pathsep.join([path, os.environ['PYTHONPATH']])
1506 os.environ['PYTHONPATH'] = path
1502 <BLANKLINE>1507 <BLANKLINE>
1503 import eggrecipedemo1508 import eggrecipedemo
1504 <BLANKLINE>1509 <BLANKLINE>
@@ -1528,7 +1533,7 @@
1528 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(1533 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
1529 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,1534 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
1530 ... reqs=['demo'], script_arguments='1, 2',1535 ... reqs=['demo'], script_arguments='1, 2',
1531 ... script_initialization='import os\nos.chdir("foo")')1536 ... script_initialization='import os\nos.chdir("foo")')
15321537
1533 >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE1538 >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE
1534 #!/usr/local/bin/python2.4 -S1539 #!/usr/local/bin/python2.4 -S
@@ -1539,6 +1544,11 @@
1539 <BLANKLINE>1544 <BLANKLINE>
1540 import site # imports custom buildout-generated site.py1545 import site # imports custom buildout-generated site.py
1541 import os1546 import os
1547 path = '/interpreter/parts/interpreter'
1548 if os.environ.get('PYTHONPATH'):
1549 path = os.pathsep.join([path, os.environ['PYTHONPATH']])
1550 os.environ['PYTHONPATH'] = path
1551 import os
1542 os.chdir("foo")1552 os.chdir("foo")
1543 <BLANKLINE>1553 <BLANKLINE>
1544 import eggrecipedemo1554 import eggrecipedemo
15451555
=== modified file 'src/zc/buildout/tests.py'
--- src/zc/buildout/tests.py 2010-03-19 19:13:23 +0000
+++ src/zc/buildout/tests.py 2010-03-19 19:13:23 +0000
@@ -2254,8 +2254,95 @@
22542254
2255 """2255 """
22562256
2257def subprocesses_have_same_environment_by_default():
2258 """
2259The scripts generated by sitepackage_safe_scripts set the PYTHONPATH so that,
2260if the environment is maintained (the default behavior), subprocesses get
2261the same Python packages.
2262
2263First, we set up a script and an interpreter.
2264
2265 >>> interpreter_dir = tmpdir('interpreter')
2266 >>> interpreter_parts_dir = os.path.join(
2267 ... interpreter_dir, 'parts', 'interpreter')
2268 >>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin')
2269 >>> mkdir(interpreter_bin_dir)
2270 >>> mkdir(interpreter_dir, 'eggs')
2271 >>> mkdir(interpreter_dir, 'parts')
2272 >>> mkdir(interpreter_parts_dir)
2273 >>> ws = zc.buildout.easy_install.install(
2274 ... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
2275 ... index=link_server+'index/')
2276 >>> test = (
2277 ... "import subprocess, sys; subprocess.call("
2278 ... "[sys.executable, '-c', "
2279 ... "'import eggrecipedemo; print eggrecipedemo.x'])")
2280 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
2281 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
2282 ... reqs=['demo'], interpreter='py',
2283 ... script_initialization=test + '; sys.exit(0)')
2284
2285This works for the script.
2286
2287 >>> print system(join(interpreter_bin_dir, 'demo'))
2288 3
2289 <BLANKLINE>
2290
2291This also works for the generated interpreter.
2292
2293 >>> print call_py(join(interpreter_bin_dir, 'py'), test)
2294 3
2295 <BLANKLINE>
2296
2297If you have a PYTHONPATH in your environment, it will be honored, after
2298the buildout-generated path.
2299
2300 >>> original_pythonpath = os.environ.get('PYTHONPATH')
2301 >>> os.environ['PYTHONPATH'] = 'foo'
2302 >>> test = (
2303 ... "import subprocess, sys; subprocess.call("
2304 ... "[sys.executable, '-c', "
2305 ... "'import sys, pprint; pprint.pprint(sys.path)'])")
2306 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
2307 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
2308 ... reqs=['demo'], interpreter='py',
2309 ... script_initialization=test + '; sys.exit(0)')
2310
2311This works for the script. As you can see, /sample_buildout/foo is included
2312right after the "parts" directory that contains site.py and sitecustomize.py.
2313You can also see, actually more easily than in the other example, that we
2314have the desired eggs available.
2315
2316 >>> print system(join(interpreter_bin_dir, 'demo')), # doctest: +ELLIPSIS
2317 ['',
2318 '/interpreter/parts/interpreter',
2319 '/sample-buildout/foo',
2320 ...
2321 '/interpreter/eggs/demo-0.3-pyN.N.egg',
2322 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
2323
2324This also works for the generated interpreter, with identical results.
2325
2326 >>> print call_py(join(interpreter_bin_dir, 'py'), test),
2327 ... # doctest: +ELLIPSIS
2328 ['',
2329 '/interpreter/parts/interpreter',
2330 '/sample-buildout/foo',
2331 ...
2332 '/interpreter/eggs/demo-0.3-pyN.N.egg',
2333 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
2334
2335 >>> # Cleanup
2336 >>> if original_pythonpath:
2337 ... os.environ['PYTHONPATH'] = original_pythonpath
2338 ... else:
2339 ... del os.environ['PYTHONPATH']
2340 ...
2341
2342 """
2343
2257def bootstrap_makes_buildout_that_works_with_system_python():2344def bootstrap_makes_buildout_that_works_with_system_python():
2258 """2345 r"""
2259In order to work smoothly with a system Python, bootstrapping creates2346In order to work smoothly with a system Python, bootstrapping creates
2260the buildout script with2347the buildout script with
2261zc.buildout.easy_install.sitepackage_safe_scripts. If it did not, a2348zc.buildout.easy_install.sitepackage_safe_scripts. If it did not, a
@@ -2274,15 +2361,11 @@
2274 >>> write(sample_buildout, 'recipes', 'dummy.py',2361 >>> write(sample_buildout, 'recipes', 'dummy.py',
2275 ... '''2362 ... '''
2276 ... import logging, os, zc.buildout2363 ... import logging, os, zc.buildout
2277 ...
2278 ... class Dummy:2364 ... class Dummy:
2279 ...
2280 ... def __init__(self, buildout, name, options):2365 ... def __init__(self, buildout, name, options):
2281 ... pass2366 ... pass
2282 ...
2283 ... def install(self):2367 ... def install(self):
2284 ... return ()2368 ... return ()
2285 ...
2286 ... def update(self):2369 ... def update(self):
2287 ... pass2370 ... pass
2288 ... ''')2371 ... ''')
@@ -2340,6 +2423,67 @@
2340 Installing dummy.2423 Installing dummy.
2341 <BLANKLINE>2424 <BLANKLINE>
23422425
2426Here's the same story with a namespace package, which has some additional
2427complications behind the scenes. First, a recipe, in the "tellmy" namespace.
2428
2429 >>> mkdir(sample_buildout, 'ns')
2430 >>> mkdir(sample_buildout, 'ns', 'tellmy')
2431 >>> write(sample_buildout, 'ns', 'tellmy', '__init__.py',
2432 ... "__import__('pkg_resources').declare_namespace(__name__)\n")
2433 >>> mkdir(sample_buildout, 'ns', 'tellmy', 'recipes')
2434 >>> write(sample_buildout, 'ns', 'tellmy', 'recipes', '__init__.py', ' ')
2435 >>> write(sample_buildout, 'ns', 'tellmy', 'recipes', 'dummy.py',
2436 ... '''
2437 ... import logging, os, zc.buildout
2438 ... class Dummy:
2439 ... def __init__(self, buildout, name, options):
2440 ... pass
2441 ... def install(self):
2442 ... return ()
2443 ... def update(self):
2444 ... pass
2445 ... ''')
2446 >>> write(sample_buildout, 'ns', 'setup.py',
2447 ... '''
2448 ... from setuptools import setup
2449 ... setup(
2450 ... name="tellmy.recipes",
2451 ... packages=['tellmy', 'tellmy.recipes'],
2452 ... install_requires=['setuptools'],
2453 ... namespace_packages=['tellmy'],
2454 ... entry_points = {'zc.buildout':
2455 ... ['dummy = tellmy.recipes.dummy:Dummy']},
2456 ... )
2457 ... ''')
2458
2459Now, a buildout that uses it.
2460
2461 >>> create_sample_namespace_eggs(sample_eggs, site_packages_path)
2462 >>> rmdir('develop-eggs')
2463 >>> from zc.buildout.testing import make_buildout
2464 >>> make_buildout(executable=py_path)
2465 >>> write(sample_buildout, 'buildout.cfg',
2466 ... '''
2467 ... [buildout]
2468 ... develop = ns
2469 ... recipes
2470 ... parts = dummy
2471 ... find-links = %(link_server)s
2472 ... executable = %(py_path)s
2473 ...
2474 ... [dummy]
2475 ... recipe = tellmy.recipes:dummy
2476 ... ''' % globals())
2477
2478Now we actually run the buildout.
2479
2480 >>> print system(buildout)
2481 Develop: '/sample-buildout/ns'
2482 Develop: '/sample-buildout/recipes'
2483 Uninstalling dummy.
2484 Installing dummy.
2485 <BLANKLINE>
2486
2343 """2487 """
23442488
2345if sys.version_info > (2, 4):2489if sys.version_info > (2, 4):
23462490
=== modified file 'src/zc/buildout/update.txt'
--- src/zc/buildout/update.txt 2010-03-19 19:13:23 +0000
+++ src/zc/buildout/update.txt 2010-03-19 19:13:23 +0000
@@ -78,22 +78,26 @@
78 zc.buildout 99.9978 zc.buildout 99.99
79 setuptools 99.9979 setuptools 99.99
8080
81Our buildout script has been updated to use the new eggs:81Our buildout script's site.py has been updated to use the new eggs:
8282
83 >>> cat(sample_buildout, 'bin', 'buildout')83 >>> cat(sample_buildout, 'parts', 'buildout', 'site.py')
84 ... # doctest: +NORMALIZE_WHITESPACE84 ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
85 #!/usr/local/bin/python2.485 "...
86 <BLANKLINE>86 def addsitepackages(known_paths):
87 import sys87 """Add site packages, as determined by zc.buildout.
88 sys.path[0:0] = [88 <BLANKLINE>
89 '/sample-buildout/eggs/zc.buildout-99.99-py2.4.egg',89 See original_addsitepackages, below, for the original version."""
90 '/sample-buildout/eggs/setuptools-99.99-py2.4.egg',90 buildout_paths = [
91 ]91 '/sample-buildout/eggs/zc.buildout-99.99-pyN.N.egg',
92 <BLANKLINE>92 '/sample-buildout/eggs/setuptools-99.99-pyN.N.egg'
93 import zc.buildout.buildout93 ]
94 <BLANKLINE>94 for path in buildout_paths:
95 if __name__ == '__main__':95 sitedir, sitedircase = makepath(path)
96 zc.buildout.buildout.main()96 if not sitedircase in known_paths and os.path.exists(sitedir):
97 sys.path.append(sitedir)
98 known_paths.add(sitedircase)
99 return known_paths
100 ...
97101
98Now, let's recreate the sample buildout. If we specify constraints on102Now, let's recreate the sample buildout. If we specify constraints on
99the versions of zc.buildout and setuptools to use, running the103the versions of zc.buildout and setuptools to use, running the
@@ -120,7 +124,6 @@
120 zc.buildout version 1.0.0,124 zc.buildout version 1.0.0,
121 setuptools version 0.6;125 setuptools version 0.6;
122 restarting.126 restarting.
123 Generated script '/sample-buildout/bin/buildout'.
124 Develop: '/sample-buildout/showversions'127 Develop: '/sample-buildout/showversions'
125 Updating show-versions.128 Updating show-versions.
126 zc.buildout 1.0.0129 zc.buildout 1.0.0

Subscribers

People subscribed via source and target branches

to all changes: