Merge lp:~gary/zc.buildout/python-support-4 into lp:zc.buildout

Proposed by Gary Poster on 2010-02-17
Status: Needs review
Proposed branch: lp:~gary/zc.buildout/python-support-4
Merge into: lp:zc.buildout
Prerequisite: lp:~gary/zc.buildout/python-support-3-options
Diff against target: 3435 lines (+2506/-195)
26 files modified
CHANGES.txt (+32/-0)
README.txt (+9/-1)
buildout.cfg (+3/-1)
setup.py (+1/-1)
src/zc/buildout/bootstrap.txt (+3/-3)
src/zc/buildout/easy_install.py (+523/-107)
src/zc/buildout/easy_install.txt (+587/-23)
src/zc/buildout/testing.py (+86/-22)
src/zc/buildout/tests.py (+285/-0)
src/zc/buildout/testselectingpython.py (+28/-1)
src/zc/buildout/update.txt (+1/-0)
z3c.recipe.scripts_/CHANGES.txt (+7/-0)
z3c.recipe.scripts_/README.txt (+10/-0)
z3c.recipe.scripts_/setup.py (+76/-0)
z3c.recipe.scripts_/src/z3c/__init__.py (+1/-0)
z3c.recipe.scripts_/src/z3c/recipe/__init__.py (+1/-0)
z3c.recipe.scripts_/src/z3c/recipe/scripts/README.txt (+402/-0)
z3c.recipe.scripts_/src/z3c/recipe/scripts/__init__.py (+1/-0)
z3c.recipe.scripts_/src/z3c/recipe/scripts/scripts.py (+101/-0)
z3c.recipe.scripts_/src/z3c/recipe/scripts/tests.py (+293/-0)
zc.recipe.egg_/setup.py (+2/-2)
zc.recipe.egg_/src/zc/recipe/egg/README.txt (+4/-1)
zc.recipe.egg_/src/zc/recipe/egg/api.txt (+1/-0)
zc.recipe.egg_/src/zc/recipe/egg/custom.txt (+5/-0)
zc.recipe.egg_/src/zc/recipe/egg/egg.py (+42/-28)
zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt (+2/-5)
To merge this branch: bzr merge lp:~gary/zc.buildout/python-support-4
Reviewer Review Type Date Requested Status
Francis J. Lacoste (community) 2010-02-17 Approve on 2010-02-18
Review via email: mp+19547@code.launchpad.net
To post a comment you must log in.
Gary Poster (gary) wrote :
Download full text (3.5 KiB)

This is a completely reworked approach to using zc.buildout with a system Python (or any Python with a populated site packages directory), deprecating my previous attempt.

The basic approach revolves around writing our own site.py and sitecustomize.py that are responsible for setting up paths and any other environment changes desired. Then if the path to these rewritten files are included (via PYTHONPATH, or with -S followed by sys.path manipulation and ``import site``) you have the environment set up, using the standard Python mechanism.

Similarly to my past work, the point that got complicated was namespace packages. The solution shown here is straightforward, as you can see in the easy_install.txt discussion.

However, if you try to use a system Python, you expose its packages using ``add-site-packages=true``, and you use eggs with namespace packages, it will incur a startup cost for generated scripts that is quadratically related to the number of eggs added by zc.buildout, because of code in pkg_resources. (We also have that same behavior now in Launchpad.) While I did a number of experiments to try and bypass the inefficiencies of the pkg_resources code, I discarded them, because Launchpad is using pkg_resources and expects it to be fully set up. Other code may be the same. The right solution is to fix pkg_resources, in Setuptools or Distribute, at which point the approach I have in this branch should still work, and be much faster.

I did punt a bit in the Windows version of the new interpreter script: I could not get the standard Windows path hacks to work for sys.exec*. I used subprocess instead for Windows, and commented this section heavily.

Obviously, this is a huge branch. I did not see a good way of subdividing it beyond what I already did (~gary/zc.buildout/python-support-1-cleanup, ~gary/zc.buildout/python-support-2-bootstrap, and ~gary/zc.buildout/python-support-3-options). I did consider the following subdivision:

- First, you could look at the changes to easy_install.py and easy_install.txt. This is the heart of the changes. It refactors the existing script generation into bits I could reuse more easily, and then makes a new version script generation function (``generated_scripts``) that works as I describe above.

- Second, you could look at zc.recipe.egg and z3c.recipe.scripts. zc.recipe.egg is refactored so I could reuse bits more easily. z3c.recipe.scripts is a brand new recipe that uses generate_scripts, as described above.

- Finally, you could look at the new tests and test changes in zc.buildout. They use zc.recipe.egg and z3c.recipe.scripts to test that system installs can work as desired--essentially showing that all of the earlier work does in fact produce a system that does what we want.

I didn't feel that separating them made a lot of sense, because you could not test whether generate_scripts actually did what we wanted until we had z3c.recipe.scripts also available. The changes are all interrelated.

Something to note is that zc.recipe.egg does not have any changed behavior. Also, it is not safe to use with a system Python. If you want that behavior for scripts and interpreters, use z3c.rec...

Read more...

Francis J. Lacoste (flacoste) wrote :
Download full text (13.2 KiB)

Hi Gary,

Wow, this was a big one. I have a bunch of comments, mostly about
clarifications and typos.

But good to merge otherwise as far as Launchpad is concerned.

Cheers

> === modified file 'CHANGES.txt'
> --- CHANGES.txt 2010-02-17 22:17:12 +0000
> +++ CHANGES.txt 2010-02-17 22:17:12 +0000
> @@ -6,6 +6,25 @@
>
> New Features:
>
> +- Buildout can be safely used with a system Python, as long as you use the
> + new z3c.recipe.scripts recipe to generate scripts and interpreters, rather
> + than zc.recipe.egg (which is still a fully supported, and simpler, way of
> + generating scripts and interpreters if you are using a "clean" Python).

Really, I tohugh zc.recipe.egg would also do the right thing. Can you explain
the differences to me?

> +
> + A hopefully slight limitation: in no cases are distributions in your
> + site-packages used to satisfy buildout dependencies. The
> + site-packages can be used in addition to the dependencies specified in
> + your buildout, and buildout dependencies can override code in your
> + site-packages, but even if your Python's site-packages has the same
> + exact version as specified in your buildout configuration, buildout
> + will still use its own copy.

I assume dependencies related to zc.buildout itself like distribute,
and setuptools. Not general dependencies managed by buildout, right?

>
> +- Installing a namespace package using a Python that already has a package
> + in the same namespace (e.g., in the Python's site-packages) failed in
> + some cases.
> +

I assume you mean this is fixed. (IOW, that is in a fixed bugs section?)

> +- Another variation of this error showed itself when at least two
> + dependencies were in a shared location like site-packages, and the
> + first one met the "versions" setting. The first dependency would be
> + added, but subsequent dependencies from the same location (e.g.,
> + site-packages) would use the version of the package found in the
> + shared location, ignoring the version setting.
> +

I assume the same thing here.

> === modified file 'README.txt'
> --- README.txt 2010-02-17 22:17:12 +0000
> +++ README.txt 2010-02-17 22:17:12 +0000
> @@ -37,6 +37,11 @@
> dependencies. It installs their console-script entry points with
> the needed eggs included in their paths.
>
> +`z3c.recipe.scripts <http://pypi.python.org/pypi/z3c.recipe.scripts>`_
> + This scripts recipe builds interpreter scripts and entry point scripts
> + based on eggs. These scripts have more features and flexibility than the
> + ones offered by zc.recipe.egg.
> +

Again, might be worthwhile to go into more details of when one should be used
instead of the other.

=== modified file 'src/zc/buildout/easy_install.py'
--- src/zc/buildout/easy_install.py 2010-02-17 22:17:12 +0000
+++ src/zc/buildout/easy_install.py 2010-02-17 22:17:12 +0000
@@ -137,9 +137,50 @@
 else:
     _safe_arg = str

-_easy_install_cmd = _safe_arg(
- 'from setuptools.command.easy_install import main; main()'
- )
+# The following string is used to run easy_install in
+# Installer._call_easy_install. It is started with python -S (that is,
+# don't import site at start). That flag, a...

review: Approve
550. By Gary Poster on 2010-02-19

merge from gary-3

551. By Gary Poster on 2010-02-22

switch ``generate_scripts`` to ``sitepackage_safe_scripts`` per review

552. By Gary Poster on 2010-02-22

add missing reset_interpreter calls

553. By Gary Poster on 2010-02-22

change the way that we identify namespace packages and better comment the code, per review.

554. By Gary Poster on 2010-02-22

doc fixes per review.

555. By Gary Poster on 2010-02-23

try again to describe the change.

556. By Gary Poster on 2010-02-23

fix docstring

Gary Poster (gary) wrote :
Download full text (20.7 KiB)

Thank you, Francis!

Francis and I talked about the review. On the basis of his comments and the discussion, I did the revisions listed above as well as two further branches: https://code.edge.launchpad.net/~gary/zc.buildout/python-support-5-initial-egg-control and https://code.edge.launchpad.net/~gary/zc.buildout/python-support-6-egg-control .

I'll respond to his comments point by point now.

> === modified file 'CHANGES.txt'
> --- CHANGES.txt 2010-02-17 22:17:12 +0000
> +++ CHANGES.txt 2010-02-17 22:17:12 +0000
> @@ -6,6 +6,25 @@
>
> New Features:
>
> +- Buildout can be safely used with a system Python, as long as you use the
> + new z3c.recipe.scripts recipe to generate scripts and interpreters, rather
> + than zc.recipe.egg (which is still a fully supported, and simpler, way of
> + generating scripts and interpreters if you are using a "clean" Python).
>
> Really, I tohugh zc.recipe.egg would also do the right thing. Can you explain
> the differences to me?

We discussed this. He meant that he wanted the CHANGES document to more clearly describe the differences. I tried this:

- Buildout can be safely used with a system Python (or any Python with code
  in site-packages), as long as you use the new z3c.recipe.scripts
  recipe to generate scripts and interpreters, rather than zc.recipe.egg.

  zc.recipe.egg is still a fully supported, and simpler, way of
  generating scripts and interpreters if you are using a "clean" Python,
  without code installed in site-packages. It keeps its previous behavior in
  order to provide backwards compatibility.

>
> +
> + A hopefully slight limitation: in no cases are distributions in your
> + site-packages used to satisfy buildout dependencies. The
> + site-packages can be used in addition to the dependencies specified in
> + your buildout, and buildout dependencies can override code in your
> + site-packages, but even if your Python's site-packages has the same
> + exact version as specified in your buildout configuration, buildout
> + will still use its own copy.
>
> I assume dependencies related to zc.buildout itself like distribute,
> and setuptools. Not general dependencies managed by buildout, right?

My comment was wrong--and the actual situation turned out to be a big problem, leading to the two subsequent branches I mentioned above.

The actual situation was that dependencies would be obtained from site-packages--even if you said you didn't want to use site-packages. This is completely broken for the use case of wanting to use a system Python without site-packages.

I fixed this in the later branches.

The later branches mean that one can choose to get general dependencies from site-packages, or not.

> +- Installing a namespace package using a Python that already has a package
> + in the same namespace (e.g., in the Python's site-packages) failed in
> + some cases.
> +
>
> I assume you mean this is fixed. (IOW, that is in a fixed bugs section?)

Right.

> +- Another variation of this error showed itself when at least two
> + dependencies were in a shared location like site-packages, and the
> + first one met the "versions" setting. The first dependency would be
> + ad...

Unmerged revisions

556. By Gary Poster on 2010-02-23

fix docstring

555. By Gary Poster on 2010-02-23

try again to describe the change.

554. By Gary Poster on 2010-02-22

doc fixes per review.

553. By Gary Poster on 2010-02-22

change the way that we identify namespace packages and better comment the code, per review.

552. By Gary Poster on 2010-02-22

add missing reset_interpreter calls

551. By Gary Poster on 2010-02-22

switch ``generate_scripts`` to ``sitepackage_safe_scripts`` per review

550. By Gary Poster on 2010-02-19

merge from gary-3

549. By Gary Poster on 2010-02-17

propagate merge from gary-3 <- gary-2 <- gary-1 <- trunk

548. By gary on 2010-02-13

fix some tests on other Python versions

547. By gary on 2010-02-12

revert attempt to skip some of the pkg_resources dance: it caused me trouble.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'CHANGES.txt'
--- CHANGES.txt 2010-02-23 20:41:17 +0000
+++ CHANGES.txt 2010-02-23 20:41:17 +0000
@@ -6,6 +6,27 @@
66
7New Features:7New Features:
88
9- Buildout can be safely used with a system Python (or any Python with code
10 in site-packages), as long as you use the new z3c.recipe.scripts
11 recipe to generate scripts and interpreters, rather than zc.recipe.egg.
12
13 zc.recipe.egg is still a fully supported, and simpler, way of
14 generating scripts and interpreters if you are using a "clean" Python,
15 without code installed in site-packages. It keeps its previous behavior in
16 order to provide backwards compatibility.
17
18 (Note that this branch is incomplete in its implementation of this feature:
19 if eggs are in installed in site-packages but you do not want to use
20 site-packages, the eggs will drag in site-packages even if you try to
21 exclude it. This is addressed in subsequent branches in the series of
22 which this one is a part.)
23
24- Added new function, ``zc.buildout.easy_install.sitepackage_safe_scripts``,
25 to generate scripts and interpreter. It produces a full-featured
26 interpreter (all command-line options supported) and the ability to
27 safely let scripts include site packages, such as with a system
28 Python. The ``z3c.recipe.scripts`` recipe uses this new function.
29
9- Improve bootstrap.30- Improve bootstrap.
1031
11 * New options let you specify where to find ez_setup.py and where to find32 * New options let you specify where to find ez_setup.py and where to find
@@ -23,6 +44,17 @@
23 This means, among other things, that ``bin/buildout -vv`` and44 This means, among other things, that ``bin/buildout -vv`` and
24 ``bin/buildout annotate`` correctly list more of the options.45 ``bin/buildout annotate`` correctly list more of the options.
2546
47- Installing a namespace package using a Python that already has a package
48 in the same namespace (e.g., in the Python's site-packages) failed in
49 some cases.
50
51- Another variation of this error showed itself when at least two
52 dependencies were in a shared location like site-packages, and the
53 first one met the "versions" setting. The first dependency would be
54 added, but subsequent dependencies from the same location (e.g.,
55 site-packages) would use the version of the package found in the
56 shared location, ignoring the version setting.
57
261.4.3 (2009-12-10)581.4.3 (2009-12-10)
27==================59==================
2860
2961
=== modified file 'README.txt'
--- README.txt 2010-02-23 20:41:17 +0000
+++ README.txt 2010-02-23 20:41:17 +0000
@@ -35,7 +35,15 @@
35`zc.recipe.egg <http://pypi.python.org/pypi/zc.recipe.egg>`_35`zc.recipe.egg <http://pypi.python.org/pypi/zc.recipe.egg>`_
36 The egg recipe installes one or more eggs, with their36 The egg recipe installes one or more eggs, with their
37 dependencies. It installs their console-script entry points with37 dependencies. It installs their console-script entry points with
38 the needed eggs included in their paths.38 the needed eggs included in their paths. It is suitable for use with
39 a "clean" Python: one without packages installed in site-packages.
40
41`z3c.recipe.scripts <http://pypi.python.org/pypi/z3c.recipe.scripts>`_
42 Like zc.recipe.egg, this recipe builds interpreter scripts and entry
43 point scripts based on eggs. It can be used with a Python that has
44 packages installed in site-packages, such as a system Python. The
45 interpreter also has more features than the one offered by
46 zc.recipe.egg.
3947
40`zc.recipe.testrunner <http://pypi.python.org/pypi/zc.recipe.testrunner>`_48`zc.recipe.testrunner <http://pypi.python.org/pypi/zc.recipe.testrunner>`_
41 The testrunner egg creates a test runner script for one or49 The testrunner egg creates a test runner script for one or
4250
=== modified file 'buildout.cfg'
--- buildout.cfg 2009-10-23 08:25:24 +0000
+++ buildout.cfg 2010-02-23 20:41:17 +0000
@@ -1,5 +1,5 @@
1[buildout]1[buildout]
2develop = zc.recipe.egg_ .2develop = zc.recipe.egg_ z3c.recipe.scripts_ .
3parts = test oltest py3parts = test oltest py
44
5[py]5[py]
@@ -13,6 +13,7 @@
13eggs = 13eggs =
14 zc.buildout14 zc.buildout
15 zc.recipe.egg15 zc.recipe.egg
16 z3c.recipe.scripts
1617
17# Tests that can be run wo a network18# Tests that can be run wo a network
18[oltest]19[oltest]
@@ -20,6 +21,7 @@
20eggs = 21eggs =
21 zc.buildout22 zc.buildout
22 zc.recipe.egg23 zc.recipe.egg
24 z3c.recipe.scripts
23defaults =25defaults =
24 [26 [
25 '-t',27 '-t',
2628
=== modified file 'setup.py'
--- setup.py 2009-12-10 16:19:55 +0000
+++ setup.py 2010-02-23 20:41:17 +0000
@@ -12,7 +12,7 @@
12#12#
13##############################################################################13##############################################################################
14name = "zc.buildout"14name = "zc.buildout"
15version = "1.4.4dev"15version = "1.5.0dev"
1616
17import os17import os
18from setuptools import setup18from setuptools import setup
1919
=== modified file 'src/zc/buildout/bootstrap.txt'
--- src/zc/buildout/bootstrap.txt 2010-02-23 20:41:17 +0000
+++ src/zc/buildout/bootstrap.txt 2010-02-23 20:41:17 +0000
@@ -232,8 +232,8 @@
232 >>> print system(232 >>> print system(
233 ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+233 ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
234 ... 'bootstrap.py --help'),234 ... 'bootstrap.py --help'),
235 ... # doctest: +ELLIPSIS235 ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
236 usage: [DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]236 Usage: [DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
237 <BLANKLINE>237 <BLANKLINE>
238 Bootstraps a buildout-based project.238 Bootstraps a buildout-based project.
239 <BLANKLINE>239 <BLANKLINE>
@@ -244,7 +244,7 @@
244 local resources, you can keep this script from going over the network.244 local resources, you can keep this script from going over the network.
245 <BLANKLINE>245 <BLANKLINE>
246 <BLANKLINE>246 <BLANKLINE>
247 options:247 Options:
248 -h, --help show this help message and exit248 -h, --help show this help message and exit
249 -v VERSION, --version=VERSION249 -v VERSION, --version=VERSION
250 use a specific zc.buildout version250 use a specific zc.buildout version
251251
=== modified file 'src/zc/buildout/easy_install.py'
--- src/zc/buildout/easy_install.py 2010-02-23 20:41:17 +0000
+++ src/zc/buildout/easy_install.py 2010-02-23 20:41:17 +0000
@@ -60,12 +60,13 @@
60 pkg_resources.Requirement.parse('setuptools')60 pkg_resources.Requirement.parse('setuptools')
61 ).location61 ).location
6262
63# Include buildout and setuptools eggs in paths63# Include buildout and setuptools eggs in paths. We prevent dupes just to
64buildout_and_setuptools_path = [64# keep from duplicating any log messages about them.
65 setuptools_loc,65buildout_loc = pkg_resources.working_set.find(
66 pkg_resources.working_set.find(66 pkg_resources.Requirement.parse('zc.buildout')).location
67 pkg_resources.Requirement.parse('zc.buildout')).location,67buildout_and_setuptools_path = [setuptools_loc]
68 ]68if os.path.normpath(setuptools_loc) != os.path.normpath(buildout_loc):
69 buildout_and_setuptools_path.append(buildout_loc)
6970
7071
71class IncompatibleVersionError(zc.buildout.UserError):72class IncompatibleVersionError(zc.buildout.UserError):
@@ -137,9 +138,73 @@
137else:138else:
138 _safe_arg = str139 _safe_arg = str
139140
140_easy_install_cmd = _safe_arg(141# The following string is used to run easy_install in
141 'from setuptools.command.easy_install import main; main()'142# Installer._call_easy_install. It is started with python -S (that is,
142 )143# don't import site at start). That flag, and all of the code in this
144# snippet above the last two lines, exist to work around a relatively rare
145# problem. If
146#
147# - your buildout configuration is trying to install a package that is within
148# a namespace package, and
149#
150# - you use a Python that has a different version of this package
151# installed in in its site-packages using
152# --single-version-externally-managed (that is, using the mechanism
153# sometimes used by system packagers:
154# http://peak.telecommunity.com/DevCenter/setuptools#install-command ), and
155#
156# - the new package tries to do sys.path tricks in the setup.py to get a
157# __version__,
158#
159# then the older package will be loaded first, making the setup version
160# the wrong number. While very arguably packages simply shouldn't do
161# the sys.path tricks, some do, and we don't want buildout to fall over
162# when they do.
163#
164# The namespace packages installed in site-packages with
165# --single-version-externally-managed use a mechanism that cause them to
166# be processed when site.py is imported (see
167# http://mail.python.org/pipermail/distutils-sig/2009-May/011730.html
168# for another description of the problem). Simply starting Python with
169# -S addresses the problem in Python 2.4 and 2.5, but Python 2.6's
170# distutils imports a value from the site module, so we unfortunately
171# have to do more drastic surgery in the _easy_install_cmd code below.
172#
173# Here's an example of the .pth files created by setuptools when using that
174# flag:
175#
176# import sys,new,os;
177# p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('<NAMESPACE>',));
178# ie = os.path.exists(os.path.join(p,'__init__.py'));
179# m = not ie and sys.modules.setdefault('<NAMESPACE>',new.module('<NAMESPACE>'));
180# mp = (m or []) and m.__dict__.setdefault('__path__',[]);
181# (p not in mp) and mp.append(p)
182#
183# The code, below, then, runs under -S, indicating that site.py should
184# not be loaded initially. It gets the initial sys.path under these
185# circumstances, and then imports site (because Python 2.6's distutils
186# will want it, as mentioned above). It then reinstates the old sys.path
187# value. Then it removes namespace packages (created by the setuptools
188# code above) from sys.modules. It identifies namespace packages by
189# iterating over every loaded module. It first looks if there is a
190# __path__, so it is a package; and then it sees if that __path__ does
191# not have an __init__.py. (Note that PEP 382,
192# http://www.python.org/dev/peps/pep-0382, makes it possible to have a
193# namespace package that has an __init__.py, but also should make it
194# unnecessary for site.py to preprocess these packages, so it should be
195# fine, as far as can be guessed as of this writing.) Finally, it
196# imports easy_install and runs it.
197
198_easy_install_cmd = _safe_arg('''\
199import sys,os;\
200p = sys.path[:];\
201import site;\
202sys.path[:] = p;\
203[sys.modules.pop(k) for k, v in sys.modules.items()\
204 if hasattr(v, '__path__') and len(v.__path__)==1 and\
205 not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))];\
206from setuptools.command.easy_install import main;\
207main()''')
143208
144209
145class Installer:210class Installer:
@@ -301,7 +366,7 @@
301 try:366 try:
302 path = setuptools_loc367 path = setuptools_loc
303368
304 args = ('-c', _easy_install_cmd, '-mUNxd', _safe_arg(tmp))369 args = ('-Sc', _easy_install_cmd, '-mUNxd', _safe_arg(tmp))
305 if self._always_unzip:370 if self._always_unzip:
306 args += ('-Z', )371 args += ('-Z', )
307 level = logger.getEffectiveLevel()372 level = logger.getEffectiveLevel()
@@ -904,6 +969,9 @@
904def working_set(specs, executable, path):969def working_set(specs, executable, path):
905 return install(specs, None, executable=executable, path=path)970 return install(specs, None, executable=executable, path=path)
906971
972############################################################################
973# Script generation functions
974
907def scripts(reqs, working_set, executable, dest,975def scripts(reqs, working_set, executable, dest,
908 scripts=None,976 scripts=None,
909 extra_paths=(),977 extra_paths=(),
@@ -912,20 +980,86 @@
912 initialization='',980 initialization='',
913 relative_paths=False,981 relative_paths=False,
914 ):982 ):
915983 """Generate scripts and/or an interpreter.
984
985 See sitepackage_safe_scripts for a version that can be used with a Python
986 that has code installed in site-packages. It has more options and a
987 different approach.
988 """
989 path = _get_path(working_set, extra_paths)
990 if initialization:
991 initialization = '\n'+initialization+'\n'
992 generated = _generate_scripts(
993 reqs, working_set, dest, path, scripts, relative_paths,
994 initialization, executable, arguments)
995 if interpreter:
996 sname = os.path.join(dest, interpreter)
997 spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
998 generated.extend(
999 _pyscript(spath, sname, executable, rpsetup))
1000 return generated
1001
1002def sitepackage_safe_scripts(
1003 dest, working_set, executable, site_py_dest,
1004 reqs=(), scripts=None, interpreter=None, extra_paths=(),
1005 initialization='', add_site_packages=False, exec_sitecustomize=False,
1006 relative_paths=False, script_arguments='', script_initialization=''):
1007 """Generate scripts and/or an interpreter from a system Python.
1008
1009 This accomplishes the same job as the ``scripts`` function, above,
1010 but it does so in an alternative way that allows safely including
1011 Python site packages, if desired, and choosing to execute the Python's
1012 sitecustomize.
1013 """
1014 generated = []
1015 generated.append(_generate_sitecustomize(
1016 site_py_dest, executable, initialization, exec_sitecustomize))
1017 generated.append(_generate_site(
1018 site_py_dest, working_set, executable, extra_paths,
1019 add_site_packages, relative_paths))
1020 script_initialization = (
1021 '\nimport site # imports custom buildout-generated site.py\n%s' % (
1022 script_initialization,))
1023 if not script_initialization.endswith('\n'):
1024 script_initialization += '\n'
1025 generated.extend(_generate_scripts(
1026 reqs, working_set, dest, [site_py_dest], scripts, relative_paths,
1027 script_initialization, executable, script_arguments, block_site=True))
1028 if interpreter:
1029 generated.extend(_generate_interpreter(
1030 interpreter, dest, executable, site_py_dest, relative_paths))
1031 return generated
1032
1033# Utilities for the script generation functions.
1034
1035# These are shared by both ``scripts`` and ``sitepackage_safe_scripts``
1036
1037def _get_path(working_set, extra_paths=()):
1038 """Given working set and extra paths, return a normalized path list."""
916 path = [dist.location for dist in working_set]1039 path = [dist.location for dist in working_set]
917 path.extend(extra_paths)1040 path.extend(extra_paths)
918 path = map(realpath, path)1041 return map(realpath, path)
9191042
920 generated = []1043def _generate_scripts(reqs, working_set, dest, path, scripts, relative_paths,
9211044 initialization, executable, arguments,
1045 block_site=False):
1046 """Generate scripts for the given requirements.
1047
1048 - reqs is an iterable of string requirements or entry points.
1049 - The requirements must be findable in the given working_set.
1050 - The dest is the directory in which the scripts should be created.
1051 - The path is a list of paths that should be added to sys.path.
1052 - The scripts is an optional dictionary. If included, the keys should be
1053 the names of the scripts that should be created, as identified in their
1054 entry points; and the values should be the name the script should
1055 actually be created with.
1056 - relative_paths, if given, should be the path that is the root of the
1057 buildout (the common path that should be the root of what is relative).
1058 """
922 if isinstance(reqs, str):1059 if isinstance(reqs, str):
923 raise TypeError('Expected iterable of requirements or entry points,'1060 raise TypeError('Expected iterable of requirements or entry points,'
924 ' got string.')1061 ' got string.')
9251062 generated = []
926 if initialization:
927 initialization = '\n'+initialization+'\n'
928
929 entry_points = []1063 entry_points = []
930 for req in reqs:1064 for req in reqs:
931 if isinstance(req, str):1065 if isinstance(req, str):
@@ -939,7 +1073,6 @@
939 )1073 )
940 else:1074 else:
941 entry_points.append(req)1075 entry_points.append(req)
942
943 for name, module_name, attrs in entry_points:1076 for name, module_name, attrs in entry_points:
944 if scripts is not None:1077 if scripts is not None:
945 sname = scripts.get(name)1078 sname = scripts.get(name)
@@ -947,40 +1080,51 @@
947 continue1080 continue
948 else:1081 else:
949 sname = name1082 sname = name
950
951 sname = os.path.join(dest, sname)1083 sname = os.path.join(dest, sname)
952 spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)1084 spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
953
954 generated.extend(1085 generated.extend(
955 _script(module_name, attrs, spath, sname, executable, arguments,1086 _script(sname, executable, rpsetup, spath, initialization,
956 initialization, rpsetup)1087 module_name, attrs, arguments, block_site=block_site))
957 )
958
959 if interpreter:
960 sname = os.path.join(dest, interpreter)
961 spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
962 generated.extend(_pyscript(spath, sname, executable, rpsetup))
963
964 return generated1088 return generated
9651089
966def _relative_path_and_setup(sname, path, relative_paths):1090def _relative_path_and_setup(sname, path,
1091 relative_paths=False, indent_level=1,
1092 omit_os_import=False):
1093 """Return a string of code of paths and of setup if appropriate.
1094
1095 - sname is the full path to the script name to be created.
1096 - path is the list of paths to be added to sys.path.
1097 - relative_paths, if given, should be the path that is the root of the
1098 buildout (the common path that should be the root of what is relative).
1099 - indent_level is the number of four-space indents that the path should
1100 insert before each element of the path.
1101 """
967 if relative_paths:1102 if relative_paths:
968 relative_paths = os.path.normcase(relative_paths)1103 relative_paths = os.path.normcase(relative_paths)
969 sname = os.path.normcase(os.path.abspath(sname))1104 sname = os.path.normcase(os.path.abspath(sname))
970 spath = ',\n '.join(1105 spath = _format_paths(
971 [_relativitize(os.path.normcase(path_item), sname, relative_paths)1106 [_relativitize(os.path.normcase(path_item), sname, relative_paths)
972 for path_item in path]1107 for path_item in path], indent_level=indent_level)
973 )
974 rpsetup = relative_paths_setup1108 rpsetup = relative_paths_setup
1109 if not omit_os_import:
1110 rpsetup = '\n\nimport os\n' + rpsetup
975 for i in range(_relative_depth(relative_paths, sname)):1111 for i in range(_relative_depth(relative_paths, sname)):
976 rpsetup += "base = os.path.dirname(base)\n"1112 rpsetup += "\nbase = os.path.dirname(base)"
977 else:1113 else:
978 spath = repr(path)[1:-1].replace(', ', ',\n ')1114 spath = _format_paths((repr(p) for p in path),
1115 indent_level=indent_level)
979 rpsetup = ''1116 rpsetup = ''
980 return spath, rpsetup1117 return spath, rpsetup
9811118
982
983def _relative_depth(common, path):1119def _relative_depth(common, path):
1120 """Return number of dirs separating ``path`` from ancestor, ``common``.
1121
1122 For instance, if path is /foo/bar/baz/bing, and common is /foo, this will
1123 return 2--in UNIX, the number of ".." to get from bing's directory
1124 to foo.
1125
1126 This is a helper for _relative_path_and_setup.
1127 """
984 n = 01128 n = 0
985 while 1:1129 while 1:
986 dirname = os.path.dirname(path)1130 dirname = os.path.dirname(path)
@@ -993,6 +1137,11 @@
993 return n1137 return n
9941138
995def _relative_path(common, path):1139def _relative_path(common, path):
1140 """Return the relative path from ``common`` to ``path``.
1141
1142 This is a helper for _relativitize, which is a helper to
1143 _relative_path_and_setup.
1144 """
996 r = []1145 r = []
997 while 1:1146 while 1:
998 dirname, basename = os.path.split(path)1147 dirname, basename = os.path.split(path)
@@ -1006,6 +1155,11 @@
1006 return os.path.join(*r)1155 return os.path.join(*r)
10071156
1008def _relativitize(path, script, relative_paths):1157def _relativitize(path, script, relative_paths):
1158 """Return a code string for the given path.
1159
1160 Path is relative to the base path ``relative_paths``if the common prefix
1161 between ``path`` and ``script`` starts with ``relative_paths``.
1162 """
1009 if path == script:1163 if path == script:
1010 raise AssertionError("path == script")1164 raise AssertionError("path == script")
1011 common = os.path.dirname(os.path.commonprefix([path, script]))1165 common = os.path.dirname(os.path.commonprefix([path, script]))
@@ -1016,66 +1170,82 @@
1016 else:1170 else:
1017 return repr(path)1171 return repr(path)
10181172
1019
1020relative_paths_setup = """1173relative_paths_setup = """
1021import os
1022
1023join = os.path.join1174join = os.path.join
1024base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))1175base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))"""
1025"""1176
10261177def _write_script(full_name, contents, logged_type):
1027def _script(module_name, attrs, path, dest, executable, arguments,1178 """Write contents of script in full_name, logging the action.
1028 initialization, rsetup):1179
1180 The only tricky bit in this function is that it supports Windows by
1181 creating exe files using a pkg_resources helper.
1182 """
1029 generated = []1183 generated = []
1030 script = dest1184 script_name = full_name
1031 if is_win32:1185 if is_win32:
1032 dest += '-script.py'1186 script_name += '-script.py'
10331187 # Generate exe file and give the script a magic name.
1034 contents = script_template % dict(1188 exe = full_name + '.exe'
1035 python = _safe_arg(executable),
1036 path = path,
1037 module_name = module_name,
1038 attrs = attrs,
1039 arguments = arguments,
1040 initialization = initialization,
1041 relative_paths_setup = rsetup,
1042 )
1043 changed = not (os.path.exists(dest) and open(dest).read() == contents)
1044
1045 if is_win32:
1046 # generate exe file and give the script a magic name:
1047 exe = script+'.exe'
1048 new_data = pkg_resources.resource_string('setuptools', 'cli.exe')1189 new_data = pkg_resources.resource_string('setuptools', 'cli.exe')
1049 if not os.path.exists(exe) or (open(exe, 'rb').read() != new_data):1190 if not os.path.exists(exe) or (open(exe, 'rb').read() != new_data):
1050 # Only write it if it's different.1191 # Only write it if it's different.
1051 open(exe, 'wb').write(new_data)1192 open(exe, 'wb').write(new_data)
1052 generated.append(exe)1193 generated.append(exe)
10531194 changed = not (os.path.exists(script_name) and
1195 open(script_name).read() == contents)
1054 if changed:1196 if changed:
1055 open(dest, 'w').write(contents)1197 open(script_name, 'w').write(contents)
1056 logger.info("Generated script %r.", script)
1057
1058 try:1198 try:
1059 os.chmod(dest, 0755)1199 os.chmod(script_name, 0755)
1060 except (AttributeError, os.error):1200 except (AttributeError, os.error):
1061 pass1201 pass
10621202 logger.info("Generated %s %r.", logged_type, full_name)
1063 generated.append(dest)1203 generated.append(script_name)
1064 return generated1204 return generated
10651205
1206def _format_paths(paths, indent_level=1):
1207 """Format paths for inclusion in a script."""
1208 separator = ',\n' + indent_level * ' '
1209 return separator.join(paths)
1210
1211def _script(dest, executable, relative_paths_setup, path, initialization,
1212 module_name, attrs, arguments, block_site=False):
1213 if block_site:
1214 dash_S = ' -S'
1215 else:
1216 dash_S = ''
1217 contents = script_template % dict(
1218 python=_safe_arg(executable),
1219 dash_S=dash_S,
1220 path=path,
1221 module_name=module_name,
1222 attrs=attrs,
1223 arguments=arguments,
1224 initialization=initialization,
1225 relative_paths_setup=relative_paths_setup,
1226 )
1227 return _write_script(dest, contents, 'script')
1228
1066if is_jython and jython_os_name == 'linux':1229if is_jython and jython_os_name == 'linux':
1067 script_header = '#!/usr/bin/env %(python)s'1230 script_header = '#!/usr/bin/env %(python)s%(dash_S)s'
1068else:1231else:
1069 script_header = '#!%(python)s'1232 script_header = '#!%(python)s%(dash_S)s'
10701233
1234sys_path_template = '''\
1235import sys
1236sys.path[0:0] = [
1237 %s,
1238 ]
1239'''
10711240
1072script_template = script_header + '''\1241script_template = script_header + '''\
1073
1074%(relative_paths_setup)s1242%(relative_paths_setup)s
1243
1075import sys1244import sys
1076sys.path[0:0] = [1245sys.path[0:0] = [
1077 %(path)s,1246 %(path)s,
1078 ]1247 ]
1248
1079%(initialization)s1249%(initialization)s
1080import %(module_name)s1250import %(module_name)s
10811251
@@ -1083,47 +1253,25 @@
1083 %(module_name)s.%(attrs)s(%(arguments)s)1253 %(module_name)s.%(attrs)s(%(arguments)s)
1084'''1254'''
10851255
1256# These are used only by the older ``scripts`` function.
10861257
1087def _pyscript(path, dest, executable, rsetup):1258def _pyscript(path, dest, executable, rsetup):
1088 generated = []
1089 script = dest
1090 if is_win32:
1091 dest += '-script.py'
1092
1093 contents = py_script_template % dict(1259 contents = py_script_template % dict(
1094 python = _safe_arg(executable),1260 python=_safe_arg(executable),
1095 path = path,1261 dash_S='',
1096 relative_paths_setup = rsetup,1262 path=path,
1263 relative_paths_setup=rsetup,
1097 )1264 )
1098 changed = not (os.path.exists(dest) and open(dest).read() == contents)1265 return _write_script(dest, contents, 'interpreter')
1099
1100 if is_win32:
1101 # generate exe file and give the script a magic name:
1102 exe = script + '.exe'
1103 open(exe, 'wb').write(
1104 pkg_resources.resource_string('setuptools', 'cli.exe')
1105 )
1106 generated.append(exe)
1107
1108 if changed:
1109 open(dest, 'w').write(contents)
1110 try:
1111 os.chmod(dest,0755)
1112 except (AttributeError, os.error):
1113 pass
1114 logger.info("Generated interpreter %r.", script)
1115
1116 generated.append(dest)
1117 return generated
11181266
1119py_script_template = script_header + '''\1267py_script_template = script_header + '''\
1120
1121%(relative_paths_setup)s1268%(relative_paths_setup)s
1269
1122import sys1270import sys
11231271
1124sys.path[0:0] = [1272sys.path[0:0] = [
1125 %(path)s,1273 %(path)s,
1126 ]1274 ]
11271275
1128_interactive = True1276_interactive = True
1129if len(sys.argv) > 1:1277if len(sys.argv) > 1:
@@ -1151,6 +1299,274 @@
1151 __import__("code").interact(banner="", local=globals())1299 __import__("code").interact(banner="", local=globals())
1152'''1300'''
11531301
1302# These are used only by the newer ``sitepackage_safe_scripts`` function.
1303
1304def _get_system_paths(executable):
1305 """Return lists of standard lib and site paths for executable.
1306 """
1307 # We want to get a list of the site packages, which is not easy.
1308 # The canonical way to do this is to use
1309 # distutils.sysconfig.get_python_lib(), but that only returns a
1310 # single path, which does not reflect reality for many system
1311 # Pythons, which have multiple additions. Instead, we start Python
1312 # with -S, which does not import site.py and set up the extra paths
1313 # like site-packages or (Ubuntu/Debian) dist-packages and
1314 # python-support. We then compare that sys.path with the normal one
1315 # (minus user packages if this is Python 2.6, because we don't
1316 # support those (yet?). The set of the normal one minus the set of
1317 # the ones in ``python -S`` is the set of packages that are
1318 # effectively site-packages.
1319 #
1320 # The given executable might not be the current executable, so it is
1321 # appropriate to do another subprocess to figure out what the
1322 # additional site-package paths are. Moreover, even if this
1323 # executable *is* the current executable, this code might be run in
1324 # the context of code that has manipulated the sys.path--for
1325 # instance, to add local zc.buildout or setuptools eggs.
1326 def get_sys_path(*args, **kwargs):
1327 cmd = [executable]
1328 cmd.extend(args)
1329 cmd.extend([
1330 "-c", "import sys, os;"
1331 "print repr([os.path.normpath(p) for p in sys.path if p])"])
1332 # Windows needs some (as yet to be determined) part of the real env.
1333 env = os.environ.copy()
1334 env.update(kwargs)
1335 _proc = subprocess.Popen(
1336 cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
1337 stdout, stderr = _proc.communicate();
1338 if _proc.returncode:
1339 raise RuntimeError(
1340 'error trying to get system packages:\n%s' % (stderr,))
1341 res = eval(stdout.strip())
1342 try:
1343 res.remove('.')
1344 except ValueError:
1345 pass
1346 return res
1347 stdlib = get_sys_path('-S') # stdlib only
1348 no_user_paths = get_sys_path(PYTHONNOUSERSITE='x')
1349 site_paths = [p for p in no_user_paths if p not in stdlib]
1350 return (stdlib, site_paths)
1351
1352def _get_module_file(executable, name):
1353 """Return a module's file path.
1354
1355 - executable is a path to the desired Python executable.
1356 - name is the name of the (pure, not C) Python module.
1357 """
1358 cmd = [executable, "-c",
1359 "import imp; "
1360 "fp, path, desc = imp.find_module(%r); "
1361 "fp.close; "
1362 "print path" % (name,)]
1363 _proc = subprocess.Popen(
1364 cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
1365 stdout, stderr = _proc.communicate();
1366 if _proc.returncode:
1367 logger.info(
1368 'Could not find file for module %s:\n%s', name, stderr)
1369 return None
1370 # else: ...
1371 res = stdout.strip()
1372 if res.endswith('.pyc') or res.endswith('.pyo'):
1373 raise RuntimeError('Cannot find uncompiled version of %s' % (name,))
1374 if not os.path.exists(res):
1375 raise RuntimeError(
1376 'File does not exist for module %s:\n%s' % (name, res))
1377 return res
1378
1379def _generate_sitecustomize(dest, executable, initialization='',
1380 exec_sitecustomize=False):
1381 """Write a sitecustomize file with optional custom initialization.
1382
1383 The created script will execute the underlying Python's
1384 sitecustomize if exec_sitecustomize is True.
1385 """
1386 sitecustomize_path = os.path.join(dest, 'sitecustomize.py')
1387 sitecustomize = open(sitecustomize_path, 'w')
1388 if initialization:
1389 sitecustomize.write(initialization + '\n')
1390 if exec_sitecustomize:
1391 real_sitecustomize_path = _get_module_file(
1392 executable, 'sitecustomize')
1393 if real_sitecustomize_path:
1394 real_sitecustomize = open(real_sitecustomize_path, 'r')
1395 sitecustomize.write(
1396 '\n# The following is from\n# %s\n' %
1397 (real_sitecustomize_path,))
1398 sitecustomize.write(real_sitecustomize.read())
1399 real_sitecustomize.close()
1400 sitecustomize.close()
1401 return sitecustomize_path
1402
1403def _generate_site(dest, working_set, executable, extra_paths=(),
1404 add_site_packages=False, relative_paths=False):
1405 """Write a site.py file with eggs from working_set.
1406
1407 extra_paths will be added to the path. If add_site_packages is True,
1408 paths from the underlying Python will be added.
1409 """
1410 path = _get_path(working_set, extra_paths)
1411 site_path = os.path.join(dest, 'site.py')
1412 egg_path_string, preamble = _relative_path_and_setup(
1413 site_path, path, relative_paths, indent_level=2, omit_os_import=True)
1414 if preamble:
1415 preamble = '\n'.join(
1416 [(line and ' %s' % (line,) or line)
1417 for line in preamble.split('\n')])
1418 original_path_setup = ''
1419 if add_site_packages:
1420 stdlib, site_paths = _get_system_paths(executable)
1421 original_path_setup = original_path_snippet % (
1422 _format_paths((repr(p) for p in site_paths), 2),)
1423 distribution = working_set.find(
1424 pkg_resources.Requirement.parse('setuptools'))
1425 if distribution is not None:
1426 # We need to worry about namespace packages.
1427 if relative_paths:
1428 location = _relativitize(
1429 distribution.location,
1430 os.path.normcase(os.path.abspath(site_path)),
1431 relative_paths)
1432 else:
1433 location = repr(distribution.location)
1434 preamble += namespace_add_site_packages_setup % (location,)
1435 original_path_setup = (
1436 addsitedir_namespace_originalpackages_snippet +
1437 original_path_setup)
1438 addsitepackages_marker = 'def addsitepackages('
1439 enableusersite_marker = 'ENABLE_USER_SITE = '
1440 successful_rewrite = False
1441 real_site_path = _get_module_file(executable, 'site')
1442 real_site = open(real_site_path, 'r')
1443 site = open(site_path, 'w')
1444 try:
1445 for line in real_site.readlines():
1446 if line.startswith(enableusersite_marker):
1447 site.write(enableusersite_marker)
1448 site.write('False # buildout does not support user sites.\n')
1449 elif line.startswith(addsitepackages_marker):
1450 site.write(addsitepackages_script % (
1451 preamble, egg_path_string, original_path_setup))
1452 site.write(line[len(addsitepackages_marker):])
1453 successful_rewrite = True
1454 else:
1455 site.write(line)
1456 finally:
1457 site.close()
1458 real_site.close()
1459 if not successful_rewrite:
1460 raise RuntimeError('Buildout did not successfully rewrite site.py')
1461 return site_path
1462
1463namespace_add_site_packages_setup = '''
1464 setuptools_path = %s
1465 sys.path.append(setuptools_path)
1466 known_paths.add(os.path.normcase(setuptools_path))
1467 import pkg_resources'''
1468
1469addsitedir_namespace_originalpackages_snippet = '''
1470 pkg_resources.working_set.add_entry(sitedir)'''
1471
1472original_path_snippet = '''
1473 original_paths = [
1474 %s
1475 ]
1476 for path in original_paths:
1477 addsitedir(path, known_paths)'''
1478
1479addsitepackages_script = '''\
1480def addsitepackages(known_paths):
1481 """Add site packages, as determined by zc.buildout.
1482
1483 See original_addsitepackages, below, for the original version."""%s
1484 buildout_paths = [
1485 %s
1486 ]
1487 for path in buildout_paths:
1488 sitedir, sitedircase = makepath(path)
1489 if not sitedircase in known_paths and os.path.exists(sitedir):
1490 sys.path.append(sitedir)
1491 known_paths.add(sitedircase)%s
1492 return known_paths
1493
1494def original_addsitepackages('''
1495
1496def _generate_interpreter(name, dest, executable, site_py_dest,
1497 relative_paths=False):
1498 """Write an interpreter script, using the site.py approach."""
1499 full_name = os.path.join(dest, name)
1500 site_py_dest_string, rpsetup = _relative_path_and_setup(
1501 full_name, [site_py_dest], relative_paths, omit_os_import=True)
1502 if rpsetup:
1503 rpsetup += "\n"
1504 if sys.platform == 'win32':
1505 windows_import = '\nimport subprocess'
1506 # os.exec* is a mess on Windows, particularly if the path
1507 # to the executable has spaces and the Python is using MSVCRT.
1508 # The standard fix is to surround the executable's path with quotes,
1509 # but that has been unreliable in testing.
1510 #
1511 # Here's a demonstration of the problem. Given a Python
1512 # compiled with a MSVCRT-based compiler, such as the free Visual
1513 # C++ 2008 Express Edition, and an executable path with spaces
1514 # in it such as the below, we see the following.
1515 #
1516 # >>> import os
1517 # >>> p0 = 'C:\\Documents and Settings\\Administrator\\My Documents\\Downloads\\Python-2.6.4\\PCbuild\\python.exe'
1518 # >>> os.path.exists(p0)
1519 # True
1520 # >>> os.execv(p0, [])
1521 # Traceback (most recent call last):
1522 # File "<stdin>", line 1, in <module>
1523 # OSError: [Errno 22] Invalid argument
1524 #
1525 # That seems like a standard problem. The standard solution is
1526 # to quote the path (see, for instance
1527 # http://bugs.python.org/issue436259). However, this solution,
1528 # and other variations, fail:
1529 #
1530 # >>> p1 = '"C:\\Documents and Settings\\Administrator\\My Documents\\Downloads\\Python-2.6.4\\PCbuild\\python.exe"'
1531 # >>> os.execv(p1, [])
1532 # Traceback (most recent call last):
1533 # File "<stdin>", line 1, in <module>
1534 # OSError: [Errno 22] Invalid argument
1535 #
1536 # We simply use subprocess instead, since it handles everything
1537 # nicely, and the transparency of exec* (that is, not running,
1538 # perhaps unexpectedly, in a subprocess) is arguably not a
1539 # necessity, at least for many use cases.
1540 execute = 'subprocess.call(argv, env=environ)'
1541 else:
1542 windows_import = ''
1543 execute = 'os.execve(sys.executable, argv, environ)'
1544 contents = interpreter_template % dict(
1545 python=_safe_arg(executable),
1546 dash_S=' -S',
1547 site_dest=site_py_dest_string,
1548 relative_paths_setup=rpsetup,
1549 windows_import=windows_import,
1550 execute=execute,
1551 )
1552 return _write_script(full_name, contents, 'interpreter')
1553
1554interpreter_template = script_header + '''
1555import os
1556import sys%(windows_import)s
1557%(relative_paths_setup)s
1558argv = [sys.executable] + sys.argv[1:]
1559environ = os.environ.copy()
1560path = %(site_dest)s
1561if environ.get('PYTHONPATH'):
1562 path = os.pathsep.join([path, environ['PYTHONPATH']])
1563environ['PYTHONPATH'] = path
1564%(execute)s
1565'''
1566
1567# End of script generation code.
1568############################################################################
1569
1154runsetup_template = """1570runsetup_template = """
1155import sys1571import sys
1156sys.path.insert(0, %(setupdir)r)1572sys.path.insert(0, %(setupdir)r)
11571573
=== modified file 'src/zc/buildout/easy_install.txt'
--- src/zc/buildout/easy_install.txt 2009-11-11 21:21:11 +0000
+++ src/zc/buildout/easy_install.txt 2010-02-23 20:41:17 +0000
@@ -521,25 +521,38 @@
521Script generation521Script generation
522-----------------522-----------------
523523
524The easy_install module provides support for creating scripts from524The easy_install module provides support for creating scripts from eggs.
525eggs. It provides a function similar to setuptools except that it525It provides two competing functions. One, ``scripts``, is a
526provides facilities for baking a script's path into the script. This526well-established approach to generating reliable scripts with a "clean"
527has two advantages:527Python--e.g., one that does not have any packages in its site-packages.
528The other, ``sitepackage_safe_scripts``, is newer, a bit trickier, and is
529designed to work with a Python that has code in its site-packages, such
530as a system Python.
531
532Both are similar to setuptools except that they provides facilities for
533baking a script's path into the script. This has two advantages:
528534
529- The eggs to be used by a script are not chosen at run time, making535- The eggs to be used by a script are not chosen at run time, making
530 startup faster and, more importantly, deterministic.536 startup faster and, more importantly, deterministic.
531537
532- The script doesn't have to import pkg_resources because the logic538- The script doesn't have to import pkg_resources because the logic that
533 that pkg_resources would execute at run time is executed at539 pkg_resources would execute at run time is executed at script-creation
534 script-creation time.540 time. (There is an exception in ``sitepackage_safe_scripts`` if you
535541 want to have your Python's site packages available, as discussed
536The scripts method can be used to generate scripts. Let's create a542 below, but even in that case pkg_resources is only partially
537destination directory for it to place them in:543 activated, which can be a significant time savings.)
538544
539 >>> import tempfile545
546The ``scripts`` function
547~~~~~~~~~~~~~~~~~~~~~~~~
548
549The ``scripts`` function is the first way to generate scripts that we'll
550examine. It is the earlier approach that the package offered. Let's
551create a destination directory for it to place them in:
552
540 >>> bin = tmpdir('bin')553 >>> bin = tmpdir('bin')
541554
542Now, we'll use the scripts method to generate scripts in this directory555Now, we'll use the scripts function to generate scripts in this directory
543from the demo egg:556from the demo egg:
544557
545 >>> import sys558 >>> import sys
@@ -736,8 +749,8 @@
736 >>> print system(os.path.join(bin, 'run')),749 >>> print system(os.path.join(bin, 'run')),
737 3 1750 3 1
738751
739Including extra paths in scripts752The ``scripts`` function: Including extra paths in scripts
740--------------------------------753~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
741754
742We can pass a keyword argument, extra paths, to cause additional paths755We can pass a keyword argument, extra paths, to cause additional paths
743to be included in the a generated script:756to be included in the a generated script:
@@ -762,8 +775,8 @@
762 if __name__ == '__main__':775 if __name__ == '__main__':
763 eggrecipedemo.main()776 eggrecipedemo.main()
764777
765Providing script arguments778The ``scripts`` function: Providing script arguments
766--------------------------779~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
767780
768An "argument" keyword argument can be used to pass arguments to an781An "argument" keyword argument can be used to pass arguments to an
769entry point. The value passed is a source string to be placed between the782entry point. The value passed is a source string to be placed between the
@@ -786,8 +799,8 @@
786 if __name__ == '__main__':799 if __name__ == '__main__':
787 eggrecipedemo.main(1, 2)800 eggrecipedemo.main(1, 2)
788801
789Passing initialization code802The ``scripts`` function: Passing initialization code
790---------------------------803~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
791804
792You can also pass script initialization code:805You can also pass script initialization code:
793806
@@ -812,8 +825,8 @@
812 if __name__ == '__main__':825 if __name__ == '__main__':
813 eggrecipedemo.main(1, 2)826 eggrecipedemo.main(1, 2)
814827
815Relative paths828The ``scripts`` function: Relative paths
816--------------829~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
817830
818Sometimes, you want to be able to move a buildout directory around and831Sometimes, you want to be able to move a buildout directory around and
819have scripts still work without having to rebuild them. We can832have scripts still work without having to rebuild them. We can
@@ -836,7 +849,7 @@
836 ... interpreter='py',849 ... interpreter='py',
837 ... relative_paths=bo)850 ... relative_paths=bo)
838851
839 >>> cat(bo, 'bin', 'run')852 >>> cat(bo, 'bin', 'run') # doctest: +NORMALIZE_WHITESPACE
840 #!/usr/local/bin/python2.4853 #!/usr/local/bin/python2.4
841 <BLANKLINE>854 <BLANKLINE>
842 import os855 import os
@@ -868,7 +881,7 @@
868881
869We specified an interpreter and its paths are adjusted too:882We specified an interpreter and its paths are adjusted too:
870883
871 >>> cat(bo, 'bin', 'py')884 >>> cat(bo, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE
872 #!/usr/local/bin/python2.4885 #!/usr/local/bin/python2.4
873 <BLANKLINE>886 <BLANKLINE>
874 import os887 import os
@@ -911,6 +924,557 @@
911 del _interactive924 del _interactive
912 __import__("code").interact(banner="", local=globals())925 __import__("code").interact(banner="", local=globals())
913926
927The ``sitepackage_safe_scripts`` function
928~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
929
930The newer function for creating scripts is ``sitepackage_safe_scripts``.
931 It has the same basic functionality as the ``scripts`` function: it can
932create scripts to run arbitrary entry points, and to run a Python
933interpreter. The following are the differences from a user's
934perspective.
935
936- It can be used safely with a Python that has packages installed itself,
937 such as a system-installed Python.
938
939- In contrast to the interpreter generated by the ``scripts`` method, which
940 supports only a small subset of the usual Python executable's options,
941 the interpreter generated by ``sitepackage_safe_scripts`` supports all
942 of them. This makes it possible to use as full Python replacement for
943 scripts that need the distributions specified in your buildout.
944
945- Both the interpreter and the entry point scripts allow you to include the
946 site packages, and/or the sitecustomize, of the Python executable, if
947 desired.
948
949It works by creating site.py and sitecustomize.py files that set up the
950desired paths and initialization. These must be placed within an otherwise
951empty directory. Typically this is in a recipe's parts directory.
952
953Here's the simplest example, building an interpreter script.
954
955 >>> interpreter_dir = tmpdir('interpreter')
956 >>> interpreter_parts_dir = os.path.join(
957 ... interpreter_dir, 'parts', 'interpreter')
958 >>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin')
959 >>> mkdir(interpreter_bin_dir)
960 >>> mkdir(interpreter_dir, 'eggs')
961 >>> mkdir(interpreter_dir, 'parts')
962 >>> mkdir(interpreter_parts_dir)
963
964 >>> ws = zc.buildout.easy_install.install(
965 ... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
966 ... index=link_server+'index/')
967 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
968 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
969 ... interpreter='py')
970
971Depending on whether the machine being used is running Windows or not, this
972produces either three or four files. In both cases, we have site.py and
973sitecustomize.py generated in the parts/interpreter directory. For Windows,
974we have py.exe and py-script.py; for other operating systems, we have py.
975
976 >>> sitecustomize_path = os.path.join(
977 ... interpreter_parts_dir, 'sitecustomize.py')
978 >>> site_path = os.path.join(interpreter_parts_dir, 'site.py')
979 >>> interpreter_path = os.path.join(interpreter_bin_dir, 'py')
980 >>> if sys.platform == 'win32':
981 ... py_path = os.path.join(interpreter_bin_dir, 'py-script.py')
982 ... expected = [sitecustomize_path,
983 ... site_path,
984 ... os.path.join(interpreter_bin_dir, 'py.exe'),
985 ... py_path]
986 ... else:
987 ... py_path = interpreter_path
988 ... expected = [sitecustomize_path, site_path, py_path]
989 ...
990 >>> assert generated == expected, repr((generated, expected))
991
992We didn't ask for any initialization, and we didn't ask to use the underlying
993sitecustomization, so sitecustomize.py is empty.
994
995 >>> cat(sitecustomize_path)
996
997The interpreter script is simple. It puts the directory with the
998site.py and sitecustomize.py on the PYTHONPATH and (re)starts Python.
999
1000 >>> cat(py_path)
1001 #!/usr/bin/python -S
1002 import os
1003 import sys
1004 <BLANKLINE>
1005 argv = [sys.executable] + sys.argv[1:]
1006 environ = os.environ.copy()
1007 path = '/interpreter/parts/interpreter'
1008 if environ.get('PYTHONPATH'):
1009 path = os.pathsep.join([path, environ['PYTHONPATH']])
1010 environ['PYTHONPATH'] = path
1011 os.execve(sys.executable, argv, environ)
1012
1013The site.py file is a modified version of the underlying Python's site.py.
1014The most important modification is that it has a different version of the
1015addsitepackages function. It sets up the Python path, similarly to the
1016behavior of the function it replaces. The following shows the part that
1017buildout inserts, in the simplest case.
1018
1019 >>> sys.stdout.write('#\n'); cat(site_path)
1020 ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
1021 #...
1022 def addsitepackages(known_paths):
1023 """Add site packages, as determined by zc.buildout.
1024 <BLANKLINE>
1025 See original_addsitepackages, below, for the original version."""
1026 buildout_paths = [
1027 '/interpreter/eggs/demo-0.3-pyN.N.egg',
1028 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
1029 ]
1030 for path in buildout_paths:
1031 sitedir, sitedircase = makepath(path)
1032 if not sitedircase in known_paths and os.path.exists(sitedir):
1033 sys.path.append(sitedir)
1034 known_paths.add(sitedircase)
1035 return known_paths
1036 <BLANKLINE>
1037 def original_addsitepackages(known_paths):...
1038
1039Here are some examples of the interpreter in use.
1040
1041 >>> print call_py(interpreter_path, "print 16+26")
1042 42
1043 <BLANKLINE>
1044 >>> res = call_py(interpreter_path, "import sys; print sys.path")
1045 >>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
1046 ['',
1047 '/interpreter/parts/interpreter',
1048 ...,
1049 '/interpreter/eggs/demo-0.3-pyN.N.egg',
1050 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
1051 <BLANKLINE>
1052 >>> clean_paths = eval(res.strip()) # This is used later for comparison.
1053
1054If you provide initialization, it goes in sitecustomize.py.
1055
1056 >>> def reset_interpreter():
1057 ... # This is necessary because, in our tests, the timestamps of the
1058 ... # .pyc files are not outdated when we want them to be.
1059 ... rmdir(interpreter_bin_dir)
1060 ... mkdir(interpreter_bin_dir)
1061 ... rmdir(interpreter_parts_dir)
1062 ... mkdir(interpreter_parts_dir)
1063 ...
1064 >>> reset_interpreter()
1065
1066 >>> initialization_string = """\
1067 ... import os
1068 ... os.environ['FOO'] = 'bar baz bing shazam'"""
1069 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
1070 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
1071 ... interpreter='py', initialization=initialization_string)
1072 >>> cat(sitecustomize_path)
1073 import os
1074 os.environ['FOO'] = 'bar baz bing shazam'
1075 >>> print call_py(interpreter_path, "import os; print os.environ['FOO']")
1076 bar baz bing shazam
1077 <BLANKLINE>
1078
1079If you use relative paths, this affects the interpreter and site.py. (This is
1080again the UNIX version; the Windows version uses subprocess instead of
1081os.execve.)
1082
1083 >>> reset_interpreter()
1084 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
1085 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
1086 ... interpreter='py', relative_paths=interpreter_dir)
1087 >>> cat(py_path)
1088 #!/usr/bin/python -S
1089 import os
1090 import sys
1091 <BLANKLINE>
1092 join = os.path.join
1093 base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
1094 base = os.path.dirname(base)
1095 <BLANKLINE>
1096 argv = [sys.executable] + sys.argv[1:]
1097 environ = os.environ.copy()
1098 path = join(base, 'parts/interpreter')
1099 if environ.get('PYTHONPATH'):
1100 path = os.pathsep.join([path, environ['PYTHONPATH']])
1101 environ['PYTHONPATH'] = path
1102 os.execve(sys.executable, argv, environ)
1103
1104For site.py, we again show only the pertinent parts. Notice that the egg
1105paths join a base to a path, as with the use of this argument in the
1106``scripts`` function.
1107
1108 >>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS
1109 #...
1110 def addsitepackages(known_paths):
1111 """Add site packages, as determined by zc.buildout.
1112 <BLANKLINE>
1113 See original_addsitepackages, below, for the original version."""
1114 join = os.path.join
1115 base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
1116 base = os.path.dirname(base)
1117 base = os.path.dirname(base)
1118 buildout_paths = [
1119 join(base, 'eggs/demo-0.3-pyN.N.egg'),
1120 join(base, 'eggs/demoneeded-1.1-pyN.N.egg')
1121 ]...
1122
1123The paths resolve in practice as you would expect.
1124
1125 >>> print call_py(interpreter_path,
1126 ... "import sys, pprint; pprint.pprint(sys.path)")
1127 ... # doctest: +ELLIPSIS
1128 ['',
1129 '/interpreter/parts/interpreter',
1130 ...,
1131 '/interpreter/eggs/demo-0.3-pyN.N.egg',
1132 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
1133 <BLANKLINE>
1134
1135The ``extra_paths`` argument affects the path in site.py. Notice that
1136/interpreter/other is added after the eggs.
1137
1138 >>> reset_interpreter()
1139 >>> mkdir(interpreter_dir, 'other')
1140 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
1141 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
1142 ... interpreter='py', extra_paths=[join(interpreter_dir, 'other')])
1143 >>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS
1144 #...
1145 def addsitepackages(known_paths):
1146 """Add site packages, as determined by zc.buildout.
1147 <BLANKLINE>
1148 See original_addsitepackages, below, for the original version."""
1149 buildout_paths = [
1150 '/interpreter/eggs/demo-0.3-pyN.N.egg',
1151 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
1152 '/interpreter/other'
1153 ]...
1154
1155 >>> print call_py(interpreter_path,
1156 ... "import sys, pprint; pprint.pprint(sys.path)")
1157 ... # doctest: +ELLIPSIS
1158 ['',
1159 '/interpreter/parts/interpreter',
1160 ...,
1161 '/interpreter/eggs/demo-0.3-pyN.N.egg',
1162 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
1163 '/interpreter/other']
1164 <BLANKLINE>
1165
1166The ``sitepackage_safe_scripts`` function: using site-packages
1167~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1168
1169The ``sitepackage_safe_scripts`` function supports including site
1170packages. This has some advantages and some serious dangers.
1171
1172A typical reason to include site-packages is that it is easier to
1173install one or more dependencies in your Python than it is with
1174buildout. Some packages, such as lxml or Python PostgreSQL integration,
1175have dependencies that can be much easier to build and/or install using
1176other mechanisms, such as your operating system's package manager. By
1177installing some core packages into your Python's site-packages, this can
1178significantly simplify some application installations.
1179
1180However, doing this has a significant danger. One of the primary goals
1181of buildout is to provide repeatability. Some packages (one of the
1182better known Python openid packages, for instance) change their behavior
1183depending on what packages are available. If Python curl bindings are
1184available, these may be preferred by the library. If a certain XML
1185package is installed, it may be preferred by the library. These hidden
1186choices may cause small or large behavior differences. The fact that
1187they can be rarely encountered can actually make it worse: you forget
1188that this might be a problem, and debugging the differences can be
1189difficult. If you allow site-packages to be included in your buildout,
1190and the Python you use is not managed precisely by your application (for
1191instance, it is a system Python), you open yourself up to these
1192possibilities. Don't be unaware of the dangers.
1193
1194That explained, let's see how it works. If you don't use namespace packages,
1195this is very straightforward.
1196
1197 >>> reset_interpreter()
1198 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
1199 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
1200 ... interpreter='py', add_site_packages=True)
1201 >>> sys.stdout.write('#\n'); cat(site_path)
1202 ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
1203 #...
1204 def addsitepackages(known_paths):
1205 """Add site packages, as determined by zc.buildout.
1206 <BLANKLINE>
1207 See original_addsitepackages, below, for the original version."""
1208 buildout_paths = [
1209 '/interpreter/eggs/demo-0.3-pyN.N.egg',
1210 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
1211 ]
1212 for path in buildout_paths:
1213 sitedir, sitedircase = makepath(path)
1214 if not sitedircase in known_paths and os.path.exists(sitedir):
1215 sys.path.append(sitedir)
1216 known_paths.add(sitedircase)
1217 original_paths = [
1218 ...
1219 ]
1220 for path in original_paths:
1221 addsitedir(path, known_paths)
1222 return known_paths
1223 <BLANKLINE>
1224 def original_addsitepackages(known_paths):...
1225
1226It simply adds the original paths using addsitedir after the code to add the
1227buildout paths.
1228
1229Here's an example of the new script in use. Other documents and tests in
1230this package give the feature a more thorough workout, but this should
1231give you an idea of the feature.
1232
1233 >>> res = call_py(interpreter_path, "import sys; print sys.path")
1234 >>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
1235 ['',
1236 '/interpreter/parts/interpreter',
1237 ...,
1238 '/interpreter/eggs/demo-0.3-py2.4.egg',
1239 '/interpreter/eggs/demoneeded-1.1-py2.4.egg',
1240 ...]
1241 <BLANKLINE>
1242
1243The clean_paths gathered earlier is a subset of this full list of paths.
1244
1245 >>> full_paths = eval(res.strip())
1246 >>> len(clean_paths) < len(full_paths)
1247 True
1248 >>> set(os.path.normpath(p) for p in clean_paths).issubset(
1249 ... os.path.normpath(p) for p in full_paths)
1250 True
1251
1252Unfortunately, because of how setuptools namespace packages are implemented
1253differently for operating system packages (debs or rpms) as opposed to
1254standard setuptools installation, there's a slightly trickier dance if you
1255use them. To show this we'll needs some extra eggs that use namespaces.
1256We'll use the ``tellmy.fortune`` package, which we'll need to make an initial
1257call to another text fixture to create.
1258
1259 >>> from zc.buildout.tests import create_sample_namespace_eggs
1260 >>> namespace_eggs = tmpdir('namespace_eggs')
1261 >>> create_sample_namespace_eggs(namespace_eggs)
1262
1263 >>> reset_interpreter()
1264 >>> ws = zc.buildout.easy_install.install(
1265 ... ['demo', 'tellmy.fortune'], join(interpreter_dir, 'eggs'),
1266 ... links=[link_server, namespace_eggs], index=link_server+'index/')
1267 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
1268 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
1269 ... interpreter='py', add_site_packages=True)
1270 >>> sys.stdout.write('#\n'); cat(site_path)
1271 ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
1272 #...
1273 def addsitepackages(known_paths):
1274 """Add site packages, as determined by zc.buildout.
1275 <BLANKLINE>
1276 See original_addsitepackages, below, for the original version."""
1277 setuptools_path = '...setuptools...'
1278 sys.path.append(setuptools_path)
1279 known_paths.add(os.path.normcase(setuptools_path))
1280 import pkg_resources
1281 buildout_paths = [
1282 '/interpreter/eggs/demo-0.3-pyN.N.egg',
1283 '/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg',
1284 '...setuptools...',
1285 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
1286 ]
1287 for path in buildout_paths:
1288 sitedir, sitedircase = makepath(path)
1289 if not sitedircase in known_paths and os.path.exists(sitedir):
1290 sys.path.append(sitedir)
1291 known_paths.add(sitedircase)
1292 pkg_resources.working_set.add_entry(sitedir)
1293 original_paths = [
1294 ...
1295 ]
1296 for path in original_paths:
1297 addsitedir(path, known_paths)
1298 return known_paths
1299 <BLANKLINE>
1300 def original_addsitepackages(known_paths):...
1301
1302 >>> print call_py(interpreter_path, "import sys; print sys.path")
1303 ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
1304 ['',
1305 '/interpreter/parts/interpreter',
1306 ...,
1307 '...setuptools...',
1308 '/interpreter/eggs/demo-0.3-pyN.N.egg',
1309 '/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg',
1310 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
1311 ...]
1312
1313As you can see, the script now first imports pkg_resources. Then we
1314need to process egg files specially to look for namespace packages there
1315*before* we process process lines in .pth files that use the "import"
1316feature--lines that might be part of the setuptools namespace package
1317implementation for system packages, as mentioned above, and that must
1318come after processing egg namespaces.
1319
1320The most complex that this function gets is if you use namespace packages,
1321include site-packages, and use relative paths. For completeness, we'll look
1322at that result.
1323
1324 >>> reset_interpreter()
1325 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
1326 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
1327 ... interpreter='py', add_site_packages=True,
1328 ... relative_paths=interpreter_dir)
1329 >>> sys.stdout.write('#\n'); cat(site_path)
1330 ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
1331 #...
1332 def addsitepackages(known_paths):
1333 """Add site packages, as determined by zc.buildout.
1334 <BLANKLINE>
1335 See original_addsitepackages, below, for the original version."""
1336 join = os.path.join
1337 base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
1338 base = os.path.dirname(base)
1339 base = os.path.dirname(base)
1340 setuptools_path = '...setuptools...'
1341 sys.path.append(setuptools_path)
1342 known_paths.add(os.path.normcase(setuptools_path))
1343 import pkg_resources
1344 buildout_paths = [
1345 join(base, 'eggs/demo-0.3-pyN.N.egg'),
1346 join(base, 'eggs/tellmy.fortune-1.0-pyN.N.egg'),
1347 '...setuptools...',
1348 join(base, 'eggs/demoneeded-1.1-pyN.N.egg')
1349 ]
1350 for path in buildout_paths:
1351 sitedir, sitedircase = makepath(path)
1352 if not sitedircase in known_paths and os.path.exists(sitedir):
1353 sys.path.append(sitedir)
1354 known_paths.add(sitedircase)
1355 pkg_resources.working_set.add_entry(sitedir)
1356 original_paths = [
1357 ...
1358 ]
1359 for path in original_paths:
1360 addsitedir(path, known_paths)
1361 return known_paths
1362 <BLANKLINE>
1363 def original_addsitepackages(known_paths):...
1364
1365 >>> print call_py(interpreter_path, "import sys; print sys.path")
1366 ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
1367 ['',
1368 '/interpreter/parts/interpreter',
1369 ...,
1370 '...setuptools...',
1371 '/interpreter/eggs/demo-0.3-pyN.N.egg',
1372 '/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg',
1373 '/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
1374 ...]
1375
1376The ``exec_sitecustomize`` argument does the same thing for the
1377sitecustomize module--it allows you to include the code from the
1378sitecustomize module in the underlying Python if you set the argument to
1379True. The z3c.recipe.scripts package sets up the full environment necessary
1380to demonstrate this piece.
1381
1382The ``sitepackage_safe_scripts`` function: writing scripts for entry points
1383~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1384
1385All of the examples so far for this function have been creating
1386interpreters. The function can also write scripts for entry
1387points. They are almost identical to the scripts that we saw for the
1388``scripts`` function except that they ``import site`` after setting the
1389sys.path to include our custom site.py and sitecustomize.py files. These
1390files then initialize the Python environment as we have already seen. Let's
1391see a simple example.
1392
1393 >>> reset_interpreter()
1394 >>> ws = zc.buildout.easy_install.install(
1395 ... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
1396 ... index=link_server+'index/')
1397 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
1398 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
1399 ... reqs=['demo'])
1400
1401As before, in Windows, 2 files are generated for each script. A script
1402file, ending in '-script.py', and an exe file that allows the script
1403to be invoked directly without having to specify the Python
1404interpreter and without having to provide a '.py' suffix. This is in addition
1405to the site.py and sitecustomize.py files that are generated as with our
1406interpreter examples above.
1407
1408 >>> if sys.platform == 'win32':
1409 ... demo_path = os.path.join(interpreter_bin_dir, 'demo-script.py')
1410 ... expected = [sitecustomize_path,
1411 ... site_path,
1412 ... os.path.join(interpreter_bin_dir, 'demo.exe'),
1413 ... demo_path]
1414 ... else:
1415 ... demo_path = os.path.join(interpreter_bin_dir, 'demo')
1416 ... expected = [sitecustomize_path, site_path, demo_path]
1417 ...
1418 >>> assert generated == expected, repr((generated, expected))
1419
1420The demo script runs the entry point defined in the demo egg:
1421
1422 >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE
1423 #!/usr/local/bin/python2.4 -S
1424 <BLANKLINE>
1425 import sys
1426 sys.path[0:0] = [
1427 '/interpreter/parts/interpreter',
1428 ]
1429 <BLANKLINE>
1430 <BLANKLINE>
1431 import site # imports custom buildout-generated site.py
1432 <BLANKLINE>
1433 import eggrecipedemo
1434 <BLANKLINE>
1435 if __name__ == '__main__':
1436 eggrecipedemo.main()
1437
1438 >>> demo_call = join(interpreter_bin_dir, 'demo')
1439 >>> if sys.platform == 'win32':
1440 ... demo_call = '"%s"' % demo_call
1441 >>> print system(demo_call)
1442 3 1
1443 <BLANKLINE>
1444
1445There are a few differences from the ``scripts`` function. First, the
1446``reqs`` argument (an iterable of string requirements or entry point
1447tuples) is a keyword argument here. We see that in the example above.
1448Second, the ``arguments`` argument is now named ``script_arguments`` to
1449try and clarify that it does not affect interpreters. While the
1450``initialization`` argument continues to affect both the interpreters
1451and the entry point scripts, if you have initialization that is only
1452pertinent to the entry point scripts, you can use the
1453``script_initialization`` argument.
1454
1455Let's see ``script_arguments`` and ``script_initialization`` in action.
1456
1457 >>> reset_interpreter()
1458 >>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
1459 ... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
1460 ... reqs=['demo'], script_arguments='1, 2',
1461 ... script_initialization='import os\nos.chdir("foo")')
1462
1463 >>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE
1464 #!/usr/local/bin/python2.4 -S
1465 import sys
1466 sys.path[0:0] = [
1467 '/interpreter/parts/interpreter',
1468 ]
1469 <BLANKLINE>
1470 import site # imports custom buildout-generated site.py
1471 import os
1472 os.chdir("foo")
1473 <BLANKLINE>
1474 import eggrecipedemo
1475 <BLANKLINE>
1476 if __name__ == '__main__':
1477 eggrecipedemo.main(1, 2)
9141478
915Handling custom build options for extensions provided in source distributions1479Handling custom build options for extensions provided in source distributions
916-----------------------------------------------------------------------------1480-----------------------------------------------------------------------------
9171481
=== modified file 'src/zc/buildout/testing.py'
--- src/zc/buildout/testing.py 2010-02-23 20:41:17 +0000
+++ src/zc/buildout/testing.py 2010-02-23 20:41:17 +0000
@@ -28,6 +28,7 @@
28import subprocess28import subprocess
29import sys29import sys
30import tempfile30import tempfile
31import textwrap
31import threading32import threading
32import time33import time
33import urllib234import urllib2
@@ -105,6 +106,16 @@
105 e.close()106 e.close()
106 return result107 return result
107108
109def call_py(interpreter, cmd, flags=None):
110 if sys.platform == 'win32':
111 args = ['"%s"' % arg for arg in (interpreter, flags, cmd) if arg]
112 args.insert(-1, '"-c"')
113 return system('"%s"' % ' '.join(args))
114 else:
115 cmd = repr(cmd)
116 return system(
117 ' '.join(arg for arg in (interpreter, flags, '-c', cmd) if arg))
118
108def get(url):119def get(url):
109 return urllib2.urlopen(url).read()120 return urllib2.urlopen(url).read()
110121
@@ -116,7 +127,11 @@
116 args = [zc.buildout.easy_install._safe_arg(arg)127 args = [zc.buildout.easy_install._safe_arg(arg)
117 for arg in args]128 for arg in args]
118 args.insert(0, '-q')129 args.insert(0, '-q')
119 args.append(dict(os.environ, PYTHONPATH=setuptools_location))130 env = dict(os.environ)
131 if executable == sys.executable:
132 env['PYTHONPATH'] = setuptools_location
133 # else pass an executable that has setuptools! See testselectingpython.py.
134 args.append(env)
120135
121 here = os.getcwd()136 here = os.getcwd()
122 try:137 try:
@@ -135,6 +150,11 @@
135def bdist_egg(setup, executable, dest):150def bdist_egg(setup, executable, dest):
136 _runsetup(setup, executable, 'bdist_egg', '-d', dest)151 _runsetup(setup, executable, 'bdist_egg', '-d', dest)
137152
153def sys_install(setup, dest):
154 _runsetup(setup, sys.executable, 'install', '--install-purelib', dest,
155 '--record', os.path.join(dest, '__added_files__'),
156 '--single-version-externally-managed')
157
138def find_python(version):158def find_python(version):
139 e = os.environ.get('PYTHON%s' % version)159 e = os.environ.get('PYTHON%s' % version)
140 if e is not None:160 if e is not None:
@@ -202,6 +222,24 @@
202 time.sleep(0.01)222 time.sleep(0.01)
203 raise ValueError('Timed out waiting for: '+label)223 raise ValueError('Timed out waiting for: '+label)
204224
225def make_buildout():
226 # Create a basic buildout.cfg to avoid a warning from buildout:
227 open('buildout.cfg', 'w').write(
228 "[buildout]\nparts =\n"
229 )
230 # Use the buildout bootstrap command to create a buildout
231 zc.buildout.buildout.Buildout(
232 'buildout.cfg',
233 [('buildout', 'log-level', 'WARNING'),
234 # trick bootstrap into putting the buildout develop egg
235 # in the eggs dir.
236 ('buildout', 'develop-eggs-directory', 'eggs'),
237 ]
238 ).bootstrap([])
239 # Create the develop-eggs dir, which didn't get created the usual
240 # way due to the trick above:
241 os.mkdir('develop-eggs')
242
205def buildoutSetUp(test):243def buildoutSetUp(test):
206244
207 test.globs['__tear_downs'] = __tear_downs = []245 test.globs['__tear_downs'] = __tear_downs = []
@@ -255,27 +293,7 @@
255 sample = tmpdir('sample-buildout')293 sample = tmpdir('sample-buildout')
256294
257 os.chdir(sample)295 os.chdir(sample)
258296 make_buildout()
259 # Create a basic buildout.cfg to avoid a warning from buildout:
260 open('buildout.cfg', 'w').write(
261 "[buildout]\nparts =\n"
262 )
263
264 # Use the buildout bootstrap command to create a buildout
265 zc.buildout.buildout.Buildout(
266 'buildout.cfg',
267 [('buildout', 'log-level', 'WARNING'),
268 # trick bootstrap into putting the buildout develop egg
269 # in the eggs dir.
270 ('buildout', 'develop-eggs-directory', 'eggs'),
271 ]
272 ).bootstrap([])
273
274
275
276 # Create the develop-eggs dir, which didn't get created the usual
277 # way due to the trick above:
278 os.mkdir('develop-eggs')
279297
280 def start_server(path):298 def start_server(path):
281 port, thread = _start_server(path, name=path)299 port, thread = _start_server(path, name=path)
@@ -283,6 +301,50 @@
283 register_teardown(lambda: stop_server(url, thread))301 register_teardown(lambda: stop_server(url, thread))
284 return url302 return url
285303
304 def make_py(initialization=''):
305 """Returns paths to new executable and to its site-packages.
306 """
307 buildout = tmpdir('executable_buildout')
308 site_packages_dir = os.path.join(buildout, 'site-packages')
309 mkdir(site_packages_dir)
310 old_wd = os.getcwd()
311 os.chdir(buildout)
312 make_buildout()
313 # Normally we don't process .pth files in extra-paths. We want to
314 # in this case so that we can test with setuptools system installs
315 # (--single-version-externally-managed), which use .pth files.
316 initialization = (
317 ('import sys\n'
318 'import site\n'
319 'known_paths = set(sys.path)\n'
320 'site_packages_dir = %r\n'
321 'site.addsitedir(site_packages_dir, known_paths)\n'
322 ) % (site_packages_dir,)) + initialization
323 initialization = '\n'.join(
324 ' ' + line for line in initialization.split('\n'))
325 install_develop(
326 'zc.recipe.egg', os.path.join(buildout, 'develop-eggs'))
327 install_develop(
328 'z3c.recipe.scripts', os.path.join(buildout, 'develop-eggs'))
329 write('buildout.cfg', textwrap.dedent('''\
330 [buildout]
331 parts = py
332
333 [py]
334 recipe = z3c.recipe.scripts
335 interpreter = py
336 initialization =
337 %(initialization)s
338 extra-paths = %(site-packages)s
339 eggs = setuptools
340 ''') % {
341 'initialization': initialization,
342 'site-packages': site_packages_dir})
343 system(os.path.join(buildout, 'bin', 'buildout'))
344 os.chdir(old_wd)
345 return (
346 os.path.join(buildout, 'bin', 'py'), site_packages_dir)
347
286 test.globs.update(dict(348 test.globs.update(dict(
287 sample_buildout = sample,349 sample_buildout = sample,
288 ls = ls,350 ls = ls,
@@ -293,6 +355,7 @@
293 tmpdir = tmpdir,355 tmpdir = tmpdir,
294 write = write,356 write = write,
295 system = system,357 system = system,
358 call_py = call_py,
296 get = get,359 get = get,
297 cd = (lambda *path: os.chdir(os.path.join(*path))),360 cd = (lambda *path: os.chdir(os.path.join(*path))),
298 join = os.path.join,361 join = os.path.join,
@@ -301,6 +364,7 @@
301 start_server = start_server,364 start_server = start_server,
302 buildout = os.path.join(sample, 'bin', 'buildout'),365 buildout = os.path.join(sample, 'bin', 'buildout'),
303 wait_until = wait_until,366 wait_until = wait_until,
367 make_py = make_py
304 ))368 ))
305369
306 zc.buildout.easy_install.prefer_final(prefer_final)370 zc.buildout.easy_install.prefer_final(prefer_final)
307371
=== modified file 'src/zc/buildout/tests.py'
--- src/zc/buildout/tests.py 2010-02-23 20:41:17 +0000
+++ src/zc/buildout/tests.py 2010-02-23 20:41:17 +0000
@@ -53,6 +53,7 @@
5353
54 >>> ls('develop-eggs')54 >>> ls('develop-eggs')
55 - foo.egg-link55 - foo.egg-link
56 - z3c.recipe.scripts.egg-link
56 - zc.recipe.egg.egg-link57 - zc.recipe.egg.egg-link
5758
58 """59 """
@@ -84,6 +85,7 @@
8485
85 >>> ls('develop-eggs')86 >>> ls('develop-eggs')
86 - foo.egg-link87 - foo.egg-link
88 - z3c.recipe.scripts.egg-link
87 - zc.recipe.egg.egg-link89 - zc.recipe.egg.egg-link
8890
89 >>> print system(join('bin', 'buildout')+' -vvv'), # doctest: +ELLIPSIS91 >>> print system(join('bin', 'buildout')+' -vvv'), # doctest: +ELLIPSIS
@@ -668,6 +670,7 @@
668670
669 >>> ls('develop-eggs')671 >>> ls('develop-eggs')
670 - foox.egg-link672 - foox.egg-link
673 - z3c.recipe.scripts.egg-link
671 - zc.recipe.egg.egg-link674 - zc.recipe.egg.egg-link
672675
673Create another:676Create another:
@@ -692,6 +695,7 @@
692 >>> ls('develop-eggs')695 >>> ls('develop-eggs')
693 - foox.egg-link696 - foox.egg-link
694 - fooy.egg-link697 - fooy.egg-link
698 - z3c.recipe.scripts.egg-link
695 - zc.recipe.egg.egg-link699 - zc.recipe.egg.egg-link
696700
697Remove one:701Remove one:
@@ -709,6 +713,7 @@
709713
710 >>> ls('develop-eggs')714 >>> ls('develop-eggs')
711 - fooy.egg-link715 - fooy.egg-link
716 - z3c.recipe.scripts.egg-link
712 - zc.recipe.egg.egg-link717 - zc.recipe.egg.egg-link
713718
714Remove the other:719Remove the other:
@@ -723,6 +728,7 @@
723All gone728All gone
724729
725 >>> ls('develop-eggs')730 >>> ls('develop-eggs')
731 - z3c.recipe.scripts.egg-link
726 - zc.recipe.egg.egg-link732 - zc.recipe.egg.egg-link
727 '''733 '''
728734
@@ -797,6 +803,7 @@
797 ... + join(sample_buildout, 'eggs'))803 ... + join(sample_buildout, 'eggs'))
798804
799 >>> ls('develop-eggs')805 >>> ls('develop-eggs')
806 - z3c.recipe.scripts.egg-link
800 - zc.recipe.egg.egg-link807 - zc.recipe.egg.egg-link
801808
802 >>> ls('eggs') # doctest: +ELLIPSIS809 >>> ls('eggs') # doctest: +ELLIPSIS
@@ -1769,6 +1776,235 @@
1769 1 21776 1 2
1770 """1777 """
17711778
1779def versions_section_ignored_for_dependency_in_favor_of_site_packages():
1780 r"""
1781This is a test for a bugfix.
1782
1783The error showed itself when at least two dependencies were in a shared
1784location like site-packages, and the first one met the "versions" setting. The
1785first dependency would be added, but subsequent dependencies from the same
1786location (e.g., site-packages) would use the version of the package found in
1787the shared location, ignoring the version setting.
1788
1789We begin with a Python that has demoneeded version 1.1 installed and a
1790demo version 0.3, all in a site-packages-like shared directory. We need
1791to create this. ``eggrecipedemo.main()`` shows the number after the dot
1792(that is, ``X`` in ``1.X``), for the demo package and the demoneeded
1793package, so this demonstrates that our Python does in fact have demo
1794version 0.3 and demoneeded version 1.1.
1795
1796 >>> py_path = make_py_with_system_install(make_py, sample_eggs)
1797 >>> print call_py(
1798 ... py_path,
1799 ... "import tellmy.version; print tellmy.version.__version__"),
1800 1.1
1801
1802Now here's a setup that would expose the bug, using the
1803zc.buildout.easy_install API.
1804
1805 >>> example_dest = tmpdir('example_dest')
1806 >>> workingset = zc.buildout.easy_install.install(
1807 ... ['tellmy.version'], example_dest, links=[sample_eggs],
1808 ... executable=py_path,
1809 ... index=None,
1810 ... versions={'tellmy.version': '1.0'})
1811 >>> for dist in workingset:
1812 ... res = str(dist)
1813 ... if res.startswith('tellmy.version'):
1814 ... print res
1815 ... break
1816 tellmy.version 1.0
1817
1818Before the bugfix, the desired tellmy.version distribution would have
1819been blocked the one in site-packages.
1820"""
1821
1822def handle_namespace_package_in_both_site_packages_and_buildout_eggs():
1823 r"""
1824If you have the same namespace package in both site-packages and in
1825buildout, we need to be very careful that faux-Python-executables and
1826scripts generated by easy_install.sitepackage_safe_scripts correctly
1827combine the two. We show this with the local recipe that uses the
1828function, z3c.recipe.scripts.
1829
1830To demonstrate this, we will create three packages: tellmy.version 1.0,
1831tellmy.version 1.1, and tellmy.fortune 1.0. tellmy.version 1.1 is installed.
1832
1833 >>> py_path = make_py_with_system_install(make_py, sample_eggs)
1834 >>> print call_py(
1835 ... py_path,
1836 ... "import tellmy.version; print tellmy.version.__version__")
1837 1.1
1838 <BLANKLINE>
1839
1840Now we will create a buildout that creates a script and a faux-Python script.
1841We want to see that both can successfully import the specified versions of
1842tellmy.version and tellmy.fortune.
1843
1844 >>> write('buildout.cfg',
1845 ... '''
1846 ... [buildout]
1847 ... parts = eggs
1848 ... find-links = %(link_server)s
1849 ...
1850 ... [primed_python]
1851 ... executable = %(py_path)s
1852 ...
1853 ... [eggs]
1854 ... recipe = z3c.recipe.scripts
1855 ... python = primed_python
1856 ... interpreter = py
1857 ... add-site-packages = true
1858 ... eggs = tellmy.version == 1.0
1859 ... tellmy.fortune == 1.0
1860 ... demo
1861 ... script-initialization =
1862 ... import tellmy.version
1863 ... print tellmy.version.__version__
1864 ... import tellmy.fortune
1865 ... print tellmy.fortune.__version__
1866 ... ''' % globals())
1867
1868 >>> print system(buildout)
1869 Installing eggs.
1870 Getting distribution for 'tellmy.version==1.0'.
1871 Got tellmy.version 1.0.
1872 Getting distribution for 'tellmy.fortune==1.0'.
1873 Got tellmy.fortune 1.0.
1874 Getting distribution for 'demo'.
1875 Got demo 0.4c1.
1876 Getting distribution for 'demoneeded'.
1877 Got demoneeded 1.2c1.
1878 Generated script '/sample-buildout/bin/demo'.
1879 Generated interpreter '/sample-buildout/bin/py'.
1880 <BLANKLINE>
1881
1882Finally, we are ready for the actual test. Prior to the bug fix that
1883this tests, the results of both calls below was the following::
1884
1885 1.1
1886 Traceback (most recent call last):
1887 ...
1888 ImportError: No module named fortune
1889 <BLANKLINE>
1890
1891In other words, we got the site-packages version of tellmy.version, and
1892we could not import tellmy.fortune at all. The following are the correct
1893results for the interpreter and for the script.
1894
1895 >>> print call_py(
1896 ... join('bin', 'py'),
1897 ... "import tellmy.version; " +
1898 ... "print tellmy.version.__version__; " +
1899 ... "import tellmy.fortune; " +
1900 ... "print tellmy.fortune.__version__") # doctest: +ELLIPSIS
1901 1.0
1902 1.0...
1903
1904 >>> print system(join('bin', 'demo'))
1905 1.0
1906 1.0
1907 4 2
1908 <BLANKLINE>
1909 """
1910
1911def handle_sys_path_version_hack():
1912 r"""
1913This is a test for a bugfix.
1914
1915If you use a Python that has a different version of one of your
1916dependencies, and the new package tries to do sys.path tricks in the
1917setup.py to get a __version__, and it uses namespace packages, the older
1918package will be loaded first, making the setup version the wrong number.
1919While very arguably packages simply shouldn't do this, some do, and we
1920don't want buildout to fall over when they do.
1921
1922To demonstrate this, we will need to create a distribution that has one of
1923these unpleasant tricks, and a Python that has an older version installed.
1924
1925 >>> py_path, site_packages_path = make_py()
1926 >>> for version in ('1.0', '1.1'):
1927 ... tmp = tempfile.mkdtemp()
1928 ... try:
1929 ... write(tmp, 'README.txt', '')
1930 ... mkdir(tmp, 'src')
1931 ... mkdir(tmp, 'src', 'tellmy')
1932 ... write(tmp, 'src', 'tellmy', '__init__.py',
1933 ... "__import__("
1934 ... "'pkg_resources').declare_namespace(__name__)\n")
1935 ... mkdir(tmp, 'src', 'tellmy', 'version')
1936 ... write(tmp, 'src', 'tellmy', 'version',
1937 ... '__init__.py', '__version__=%r\n' % version)
1938 ... write(
1939 ... tmp, 'setup.py',
1940 ... "from setuptools import setup\n"
1941 ... "import sys\n"
1942 ... "sys.path.insert(0, 'src')\n"
1943 ... "from tellmy.version import __version__\n"
1944 ... "setup(\n"
1945 ... " name='tellmy.version',\n"
1946 ... " package_dir = {'': 'src'},\n"
1947 ... " packages = ['tellmy', 'tellmy.version'],\n"
1948 ... " install_requires = ['setuptools'],\n"
1949 ... " namespace_packages=['tellmy'],\n"
1950 ... " zip_safe=True, version=__version__,\n"
1951 ... " author='bob', url='bob', author_email='bob')\n"
1952 ... )
1953 ... zc.buildout.testing.sdist(tmp, sample_eggs)
1954 ... if version == '1.0':
1955 ... # We install the 1.0 version in site packages the way a
1956 ... # system packaging system (debs, rpms) would do it.
1957 ... zc.buildout.testing.sys_install(tmp, site_packages_path)
1958 ... finally:
1959 ... shutil.rmtree(tmp)
1960 >>> print call_py(
1961 ... py_path,
1962 ... "import tellmy.version; print tellmy.version.__version__")
1963 1.0
1964 <BLANKLINE>
1965 >>> write('buildout.cfg',
1966 ... '''
1967 ... [buildout]
1968 ... parts = eggs
1969 ... find-links = %(sample_eggs)s
1970 ...
1971 ... [primed_python]
1972 ... executable = %(py_path)s
1973 ...
1974 ... [eggs]
1975 ... recipe = zc.recipe.egg:eggs
1976 ... python = primed_python
1977 ... eggs = tellmy.version == 1.1
1978 ... ''' % globals())
1979
1980Before the bugfix, running this buildout would generate this error:
1981
1982 Installing eggs.
1983 Getting distribution for 'tellmy.version==1.1'.
1984 Installing tellmy.version 1.1
1985 Caused installation of a distribution:
1986 tellmy.version 1.0
1987 with a different version.
1988 Got None.
1989 While:
1990 Installing eggs.
1991 Error: There is a version conflict.
1992 We already have: tellmy.version 1.0
1993 <BLANKLINE>
1994
1995You can see the copiously commented fix for this in easy_install.py (see
1996zc.buildout.easy_install.Installer._call_easy_install and particularly
1997the comment leading up to zc.buildout.easy_install._easy_install_cmd).
1998Now the install works correctly, as seen here.
1999
2000 >>> print system(buildout)
2001 Installing eggs.
2002 Getting distribution for 'tellmy.version==1.1'.
2003 Got tellmy.version 1.1.
2004 <BLANKLINE>
2005
2006 """
2007
1772if sys.version_info > (2, 4):2008if sys.version_info > (2, 4):
1773 def test_exit_codes():2009 def test_exit_codes():
1774 """2010 """
@@ -2367,6 +2603,7 @@
23672603
2368 >>> ls('develop-eggs')2604 >>> ls('develop-eggs')
2369 - foo.egg-link2605 - foo.egg-link
2606 - z3c.recipe.scripts.egg-link
2370 - zc.recipe.egg.egg-link2607 - zc.recipe.egg.egg-link
23712608
2372 """2609 """
@@ -2654,6 +2891,47 @@
26542891
2655######################################################################2892######################################################################
26562893
2894def make_py_with_system_install(make_py, sample_eggs):
2895 py_path, site_packages_path = make_py()
2896 create_sample_namespace_eggs(sample_eggs, site_packages_path)
2897 return py_path
2898
2899def create_sample_namespace_eggs(dest, site_packages_path=None):
2900 from zc.buildout.testing import write, mkdir
2901 for pkg, version in (('version', '1.0'), ('version', '1.1'),
2902 ('fortune', '1.0')):
2903 tmp = tempfile.mkdtemp()
2904 try:
2905 write(tmp, 'README.txt', '')
2906 mkdir(tmp, 'src')
2907 mkdir(tmp, 'src', 'tellmy')
2908 write(tmp, 'src', 'tellmy', '__init__.py',
2909 "__import__("
2910 "'pkg_resources').declare_namespace(__name__)\n")
2911 mkdir(tmp, 'src', 'tellmy', pkg)
2912 write(tmp, 'src', 'tellmy', pkg,
2913 '__init__.py', '__version__=%r\n' % version)
2914 write(
2915 tmp, 'setup.py',
2916 "from setuptools import setup\n"
2917 "setup(\n"
2918 " name='tellmy.%(pkg)s',\n"
2919 " package_dir = {'': 'src'},\n"
2920 " packages = ['tellmy', 'tellmy.%(pkg)s'],\n"
2921 " install_requires = ['setuptools'],\n"
2922 " namespace_packages=['tellmy'],\n"
2923 " zip_safe=True, version=%(version)r,\n"
2924 " author='bob', url='bob', author_email='bob')\n"
2925 % locals()
2926 )
2927 zc.buildout.testing.sdist(tmp, dest)
2928 if (site_packages_path and pkg == 'version' and version == '1.1'):
2929 # We install the 1.1 version in site packages the way a
2930 # system packaging system (debs, rpms) would do it.
2931 zc.buildout.testing.sys_install(tmp, site_packages_path)
2932 finally:
2933 shutil.rmtree(tmp)
2934
2657def create_sample_eggs(test, executable=sys.executable):2935def create_sample_eggs(test, executable=sys.executable):
2658 write = test.globs['write']2936 write = test.globs['write']
2659 dest = test.globs['sample_eggs']2937 dest = test.globs['sample_eggs']
@@ -2776,6 +3054,7 @@
2776 test.globs['sample_eggs'])3054 test.globs['sample_eggs'])
2777 test.globs['update_extdemo'] = lambda : add_source_dist(test, 1.5)3055 test.globs['update_extdemo'] = lambda : add_source_dist(test, 1.5)
2778 zc.buildout.testing.install_develop('zc.recipe.egg', test)3056 zc.buildout.testing.install_develop('zc.recipe.egg', test)
3057 zc.buildout.testing.install_develop('z3c.recipe.scripts', test)
27793058
2780egg_parse = re.compile('([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d).egg$'3059egg_parse = re.compile('([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d).egg$'
2781 ).match3060 ).match
@@ -2934,6 +3213,10 @@
2934 (re.compile('[-d] setuptools-\S+[.]egg'), 'setuptools.egg'),3213 (re.compile('[-d] setuptools-\S+[.]egg'), 'setuptools.egg'),
2935 (re.compile(r'\\[\\]?'), '/'),3214 (re.compile(r'\\[\\]?'), '/'),
2936 (re.compile(r'\#!\S+\bpython\S*'), '#!/usr/bin/python'),3215 (re.compile(r'\#!\S+\bpython\S*'), '#!/usr/bin/python'),
3216 # Normalize generate_script's Windows interpreter to UNIX:
3217 (re.compile(r'\nimport subprocess\n'), '\n'),
3218 (re.compile('subprocess\\.call\\(argv, env=environ\\)'),
3219 'os.execve(sys.executable, argv, environ)'),
2937 ]+(sys.version_info < (2, 5) and [3220 ]+(sys.version_info < (2, 5) and [
2938 (re.compile('.*No module named runpy.*', re.S), ''),3221 (re.compile('.*No module named runpy.*', re.S), ''),
2939 (re.compile('.*usage: pdb.py scriptfile .*', re.S), ''),3222 (re.compile('.*usage: pdb.py scriptfile .*', re.S), ''),
@@ -3043,6 +3326,8 @@
3043 zc.buildout.testing.normalize_script,3326 zc.buildout.testing.normalize_script,
3044 normalize_bang,3327 normalize_bang,
3045 (re.compile('Downloading.*setuptools.*egg\n'), ''),3328 (re.compile('Downloading.*setuptools.*egg\n'), ''),
3329 (re.compile('options:'), 'Options:'),
3330 (re.compile('usage:'), 'Usage:'),
3046 ]),3331 ]),
3047 ))3332 ))
30483333
30493334
=== modified file 'src/zc/buildout/testselectingpython.py'
--- src/zc/buildout/testselectingpython.py 2009-11-27 13:20:19 +0000
+++ src/zc/buildout/testselectingpython.py 2010-02-23 20:41:17 +0000
@@ -11,7 +11,7 @@
11# FOR A PARTICULAR PURPOSE.11# FOR A PARTICULAR PURPOSE.
12#12#
13##############################################################################13##############################################################################
14import os, re, sys, unittest14import os, re, subprocess, sys, textwrap, unittest
15from zope.testing import doctest, renormalizing15from zope.testing import doctest, renormalizing
16import zc.buildout.tests16import zc.buildout.tests
17import zc.buildout.testing17import zc.buildout.testing
@@ -42,6 +42,33 @@
4242
43def multi_python(test):43def multi_python(test):
44 other_executable = zc.buildout.testing.find_python(other_version)44 other_executable = zc.buildout.testing.find_python(other_version)
45 command = textwrap.dedent('''\
46 try:
47 import setuptools
48 except ImportError:
49 import sys
50 sys.exit(1)
51 ''')
52 if subprocess.call([other_executable, '-c', command],
53 env=os.environ):
54 # the other executable does not have setuptools. Get setuptools.
55 # We will do this using the same tools we are testing, for better or
56 # worse. Alternatively, we could try using bootstrap.
57 executable_dir = test.globs['tmpdir']('executable_dir')
58 executable_parts = os.path.join(executable_dir, 'parts')
59 test.globs['mkdir'](executable_parts)
60 ws = zc.buildout.easy_install.install(
61 ['setuptools'], executable_dir,
62 index='http://www.python.org/pypi/',
63 always_unzip=True, executable=other_executable)
64 zc.buildout.easy_install.sitepackage_safe_scripts(
65 executable_dir, ws, other_executable, executable_parts,
66 reqs=['setuptools'], interpreter='py')
67 original_executable = other_executable
68 other_executable = os.path.join(executable_dir, 'py')
69 assert not subprocess.call(
70 [other_executable, '-c', command], env=os.environ), (
71 'test set up failed')
45 sample_eggs = test.globs['tmpdir']('sample_eggs')72 sample_eggs = test.globs['tmpdir']('sample_eggs')
46 os.mkdir(os.path.join(sample_eggs, 'index'))73 os.mkdir(os.path.join(sample_eggs, 'index'))
47 test.globs['sample_eggs'] = sample_eggs74 test.globs['sample_eggs'] = sample_eggs
4875
=== modified file 'src/zc/buildout/update.txt'
--- src/zc/buildout/update.txt 2009-11-06 22:33:23 +0000
+++ src/zc/buildout/update.txt 2010-02-23 20:41:17 +0000
@@ -81,6 +81,7 @@
81Our buildout script has been updated to use the new eggs:81Our buildout script has been updated to use the new eggs:
8282
83 >>> cat(sample_buildout, 'bin', 'buildout')83 >>> cat(sample_buildout, 'bin', 'buildout')
84 ... # doctest: +NORMALIZE_WHITESPACE
84 #!/usr/local/bin/python2.485 #!/usr/local/bin/python2.4
85 <BLANKLINE>86 <BLANKLINE>
86 import sys87 import sys
8788
=== added directory 'z3c.recipe.scripts_'
=== added file 'z3c.recipe.scripts_/CHANGES.txt'
--- z3c.recipe.scripts_/CHANGES.txt 1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/CHANGES.txt 2010-02-23 20:41:17 +0000
@@ -0,0 +1,7 @@
1Change History
2**************
3
41.0.0
5=====
6
7Initial public version.
08
=== added file 'z3c.recipe.scripts_/README.txt'
--- z3c.recipe.scripts_/README.txt 1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/README.txt 2010-02-23 20:41:17 +0000
@@ -0,0 +1,10 @@
1********************************
2Buildout Script Recipe
3********************************
4
5.. contents::
6
7The script recipe installs eggs into a buildout eggs directory, exactly
8like zc.recipe.egg, and then generates scripts in a buildout bin
9directory with egg paths baked into them.
10
011
=== added file 'z3c.recipe.scripts_/setup.py'
--- z3c.recipe.scripts_/setup.py 1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/setup.py 2010-02-23 20:41:17 +0000
@@ -0,0 +1,76 @@
1##############################################################################
2#
3# Copyright (c) 2007 Zope Corporation and Contributors.
4# All Rights Reserved.
5#
6# This software is subject to the provisions of the Zope Public License,
7# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
8# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
9# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
10# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
11# FOR A PARTICULAR PURPOSE.
12#
13##############################################################################
14"""Setup for z3c.recipe.scripts package
15
16$Id: setup.py 106736 2009-12-18 02:33:08Z gary $
17"""
18
19version = '1.0.0dev'
20
21import os
22from setuptools import setup, find_packages
23
24def read(*rnames):
25 return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
26
27name = "z3c.recipe.scripts"
28setup(
29 name = name,
30 version = version,
31 author = "Gary Poster",
32 author_email = "gary.poster@canonical.com",
33 description = "Recipe for installing Python scripts",
34 long_description = (
35 read('README.txt')
36 + '\n' +
37 read('CHANGES.txt')
38 + '\n' +
39 'Detailed Documentation\n'
40 '**********************\n'
41 + '\n' +
42 read('src', 'z3c', 'recipe', 'scripts', 'README.txt')
43 + '\n' +
44 'Download\n'
45 '*********\n'
46 ),
47 keywords = "development build",
48 classifiers = [
49 'Development Status :: 5 - Production/Stable',
50 'Framework :: Buildout',
51 'Intended Audience :: Developers',
52 'License :: OSI Approved :: Zope Public License',
53 'Topic :: Software Development :: Build Tools',
54 'Topic :: Software Development :: Libraries :: Python Modules',
55 ],
56 url='http://cheeseshop.python.org/pypi/z3c.recipe.scripts',
57 license = "ZPL 2.1",
58
59 packages = find_packages('src'),
60 package_dir = {'':'src'},
61 namespace_packages = ['z3c', 'z3c.recipe'],
62 install_requires = [
63 'zc.buildout >=1.5.0dev',
64 'zc.recipe.egg >=1.2.3dev',
65 'setuptools'],
66 tests_require = ['zope.testing'],
67 test_suite = name+'.tests.test_suite',
68 entry_points = {'zc.buildout': ['default = %s:Scripts' % name,
69 'script = %s:Scripts' % name,
70 'scripts = %s:Scripts' % name,
71 'interpreter = %s:Interpreter' % name,
72 ]
73 },
74 include_package_data = True,
75 zip_safe=False,
76 )
077
=== added directory 'z3c.recipe.scripts_/src'
=== added directory 'z3c.recipe.scripts_/src/z3c'
=== added file 'z3c.recipe.scripts_/src/z3c/__init__.py'
--- z3c.recipe.scripts_/src/z3c/__init__.py 1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/src/z3c/__init__.py 2010-02-23 20:41:17 +0000
@@ -0,0 +1,1 @@
1__import__('pkg_resources').declare_namespace(__name__)
02
=== added directory 'z3c.recipe.scripts_/src/z3c/recipe'
=== added file 'z3c.recipe.scripts_/src/z3c/recipe/__init__.py'
--- z3c.recipe.scripts_/src/z3c/recipe/__init__.py 1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/src/z3c/recipe/__init__.py 2010-02-23 20:41:17 +0000
@@ -0,0 +1,1 @@
1__import__('pkg_resources').declare_namespace(__name__)
02
=== added directory 'z3c.recipe.scripts_/src/z3c/recipe/scripts'
=== added file 'z3c.recipe.scripts_/src/z3c/recipe/scripts/README.txt'
--- z3c.recipe.scripts_/src/z3c/recipe/scripts/README.txt 1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/src/z3c/recipe/scripts/README.txt 2010-02-23 20:41:17 +0000
@@ -0,0 +1,402 @@
1Script and interpreter generation
2=================================
3
4This recipe is very similar to zc.recipe.egg, and if you are familiar with its
5options, you will be able to use this one easily.
6
7The script and interpreter generation in this recipe are improved from
8those provided by zc.recipe.egg in two basic ways.
9
10- The interpreter generated by the script supports all interpreter
11 options, as opposed to the subset provided by zc.recipe.egg.
12
13- Both scripts and interpreters from this recipe can optionally choose
14 to include site-packages, and even sitecustomize.
15
16The recipe takes several options. First, here's the list of the options
17that overlap from the standard zc.recipe.eggs scripts recipe. After
18this, we'll list the new options and describe them.
19
20* eggs
21* find-links
22* index
23* python
24* extra-paths
25* entry-points
26* scripts
27* dependent-scripts
28* interpreter
29* arguments
30* initialization
31* relative-paths
32
33In addition to these, the recipe offers these new options. They are
34introduced here, and described more in depth below.
35
36add-site-packages
37 You can choose to have the site-packages of the underlying Python
38 available to your script or interpreter, in addition to the packages
39 from your eggs. See the section on this option for motivations and
40 warnings.
41
42extends
43 You can extend another section using this value. It is intended to be
44 used by extending a section that uses this package's scripts recipe.
45 In this manner, you can avoid repeating yourself.
46
47exec-sitecustomize
48 Normally the Python's real sitecustomize module is not processed.
49 If you want it to be processed, set this value to 'true'. This will
50 be honored irrespective of the setting for add-site-packages.
51
52script-initialization
53 The standard initialization code affects both an interpreter and scripts.
54 The code in script-initialization is used only for the generated scripts.
55
56Finally, the "interpreter" entry point ignores ``script-initialization``,
57``scripts``, and ``arguments``, and provides yet another additional option.
58
59name
60 While, by default, the interpreter recipe takes the name of the
61 section to be the desired interpreter name, you can specify the
62 interpreter name here instead.
63
64Script generation
65-----------------
66
67Generating a basic script looks virtually identical to using zc.recipe.egg.
68
69(Note that the find-links and index values are typically not needed; they
70are included to help make this document run as a test successfully.)
71
72 >>> write(sample_buildout, 'buildout.cfg',
73 ... """
74 ... [buildout]
75 ... parts = demo
76 ...
77 ... [demo]
78 ... recipe = z3c.recipe.scripts
79 ... eggs = demo<0.3
80 ... find-links = %(server)s
81 ... index = %(server)s/index
82 ... """ % dict(server=link_server))
83
84 >>> print system(buildout),
85 Installing demo.
86 Getting distribution for 'demo<0.3'.
87 Got demo 0.2.
88 Getting distribution for 'demoneeded'.
89 Got demoneeded 1.2c1.
90 Generated script '/sample-buildout/bin/demo'.
91
92 >>> print system(join(sample_buildout, 'bin', 'demo')),
93 2 2
94
95Interpreter generation
96----------------------
97
98As with zc.recipe.egg, you can generate an interpreter with the default
99script recipe shown above by supplying the "interpreter" option.
100This example will create both an entry point script and an interpreter.
101
102 >>> write(sample_buildout, 'buildout.cfg',
103 ... """
104 ... [buildout]
105 ... parts = demo
106 ...
107 ... [demo]
108 ... recipe = z3c.recipe.scripts
109 ... eggs = demo<0.3
110 ... find-links = %(server)s
111 ... index = %(server)s/index
112 ... interpreter = py
113 ... """ % dict(server=link_server))
114
115 >>> print system(buildout),
116 Uninstalling demo.
117 Installing demo.
118 Generated script '/sample-buildout/bin/demo'.
119 Generated interpreter '/sample-buildout/bin/py'.
120
121You can also generate an interpreter alone with the ``interpreter`` recipe.
122
123 >>> write(sample_buildout, 'buildout.cfg',
124 ... """
125 ... [buildout]
126 ... parts = py
127 ...
128 ... [py]
129 ... recipe = z3c.recipe.scripts:interpreter
130 ... eggs = demo<0.3
131 ... find-links = %(server)s
132 ... index = %(server)s/index
133 ... """ % dict(server=link_server))
134
135 >>> print system(buildout),
136 Uninstalling demo.
137 Installing py.
138 Generated interpreter '/sample-buildout/bin/py'.
139
140In both cases, the bin/py script works by restarting Python after
141specifying a special path in PYTHONPATH. This example shows the UNIX version;
142the Windows version actually uses subprocess instead.
143
144 >>> cat(sample_buildout, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE
145 #!/usr/bin/python2.4 -S
146 <BLANKLINE>
147 import os
148 import sys
149 <BLANKLINE>
150 argv = [sys.executable] + sys.argv[1:]
151 environ = os.environ.copy()
152 path = '/sample-buildout/parts/py'
153 if environ.get('PYTHONPATH'):
154 path = os.pathsep.join([path, environ['PYTHONPATH']])
155 environ['PYTHONPATH'] = path
156 os.execve(sys.executable, argv, environ)
157
158The path is a directory that contains two files: our own site.py and
159sitecustomize.py. The site.py is modified from the underlying Python's
160site.py, and is responsible for setting up our paths. The
161sitecustomize.py is responsible for running the initialization code
162provided.
163
164 >>> ls(sample_buildout, 'parts', 'py')
165 - site.py
166 - sitecustomize.py
167
168Here's an example of using the generated interpreter.
169
170 >>> print system(join(sample_buildout, 'bin', 'py') +
171 ... ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
172 ['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
173 '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
174 <BLANKLINE>
175
176Including site-packages and sitecustomize
177-----------------------------------------
178
179As introduced above, this recipe supports including site packages. This has
180some advantages and some serious dangers.
181
182A typical reason to include site-packages is that it is easier to
183install one or more dependencies in your Python than it is with
184buildout. Some packages, such as lxml or Python PostgreSQL integration,
185have dependencies that can be much easier to build and/or install using
186other mechanisms, such as your operating system's package manager. By
187installing some core packages into your Python's site-packages, this can
188significantly simplify some application installations.
189
190However, doing this has a significant danger. One of the primary goals
191of buildout is to provide repeatability. Some packages (one of the
192better known Python openid packages, for instance) change their behavior
193depending on what packages are available. If Python curl bindings are
194available, these may be preferred by the library. If a certain XML
195package is installed, it may be preferred by the library. These hidden
196choices may cause small or large behavior differences. The fact that
197they can be rarely encountered can actually make it worse: you forget
198that this might be a problem, and debugging the differences can be
199difficult. If you allow site-packages to be included in your buildout,
200and the Python you use is not managed precisely by your application (for
201instance, it is a system Python), you open yourself up to these
202possibilities. Don't be unaware of the dangers.
203
204To show off these features, we need to use buildout with a Python
205executable with some extra paths to show ``add-site-packages``; and one
206guaranteed to have a sitecustomize module to show
207``exec-sitecustomize``. We'll make one using a test fixture called
208``make_py``. The os.environ change below will go into the sitecustomize,
209and the site_packages_path will be in the Python's path.
210
211 >>> py_path, site_packages_path = make_py(initialization='''\
212 ... import os
213 ... os.environ['zc.buildout'] = 'foo bar baz shazam'
214 ... ''')
215 >>> print site_packages_path
216 /executable_buildout/site-packages
217
218Now let's take a look at add-site-packages.
219
220 >>> write(sample_buildout, 'buildout.cfg',
221 ... """
222 ... [buildout]
223 ... parts = py
224 ... executable = %(py_path)s
225 ...
226 ... [py]
227 ... recipe = z3c.recipe.scripts:interpreter
228 ... add-site-packages = true
229 ... eggs = demo<0.3
230 ... find-links = %(server)s
231 ... index = %(server)s/index
232 ... """ % dict(server=link_server, py_path=py_path))
233
234 >>> print system(buildout),
235 Uninstalling py.
236 Installing py.
237 Generated interpreter '/sample-buildout/bin/py'.
238
239 >>> print system(join(sample_buildout, 'bin', 'py') +
240 ... ''' -c "import sys, pprint; pprint.pprint(sys.path)"''')
241 ... # doctest: +ELLIPSIS
242 ['',
243 '/sample-buildout/parts/py',
244 ...,
245 '/sample-buildout/eggs/demo-0.2-pyN.N.egg',
246 '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg',
247 '/executable_buildout/eggs/setuptools-X-pyN.N.egg',
248 '/executable_buildout/site-packages']
249 <BLANKLINE>
250
251Next we will use the exec-sitecustomize option. It simply copies
252Python's underlying sitecustomize module, if it exists, to the local
253version. The os.environ change shown above in the make_py call will go
254into the sitecustomize.
255
256 >>> write(sample_buildout, 'buildout.cfg',
257 ... """
258 ... [buildout]
259 ... parts = py
260 ... executable = %(py_path)s
261 ...
262 ... [py]
263 ... recipe = z3c.recipe.scripts:interpreter
264 ... exec-sitecustomize = true
265 ... eggs = demo<0.3
266 ... find-links = %(server)s
267 ... index = %(server)s/index
268 ... """ % dict(server=link_server, py_path=py_path))
269
270 >>> print system(buildout),
271 Uninstalling py.
272 Installing py.
273 Generated interpreter '/sample-buildout/bin/py'.
274
275 >>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
276 ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
277 <BLANKLINE>
278 # The following is from
279 # /executable_buildout/parts/py/sitecustomize.py
280 ...
281 import os
282 os.environ['zc.buildout'] = 'foo bar baz shazam'
283
284 >>> print system(join(sample_buildout, 'bin', 'py') +
285 ... ''' -c "import os; print os.environ['zc.buildout']"''')
286 foo bar baz shazam
287 <BLANKLINE>
288
289Options
290-------
291
292We'll focus now on the options that are different than zc.recipe.egg.
293
294Let's look at the ``extends`` option first.
295
296 >>> write(sample_buildout, 'buildout.cfg',
297 ... """
298 ... [buildout]
299 ... parts = demo python
300 ...
301 ... [demo]
302 ... recipe = z3c.recipe.scripts
303 ... eggs = demo<0.3
304 ... find-links = %(server)s
305 ... index = %(server)s/index
306 ...
307 ... [python]
308 ... recipe = z3c.recipe.scripts:interpreter
309 ... extends = demo
310 ... initialization =
311 ... import os
312 ... os.environ['zc.buildout'] = 'foo bar baz shazam'
313 ... """ % dict(server=link_server))
314
315That makes it easier to specify some initialization for the interpreter
316that is different than a script, while duplicating other configuration.
317
318Now let's put it in action.
319
320 >>> print system(buildout),
321 Uninstalling py.
322 Installing demo.
323 Generated script '/sample-buildout/bin/demo'.
324 Installing python.
325 Generated interpreter '/sample-buildout/bin/python'.
326
327 >>> print system(join(sample_buildout, 'bin', 'python') +
328 ... ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
329 ['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
330 '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
331 <BLANKLINE>
332 >>> print system(join(sample_buildout, 'bin', 'python') +
333 ... ''' -c "import os; print os.environ['zc.buildout']"'''),
334 foo bar baz shazam
335
336Note that the parts/py directory has been cleaned up, and parts/python has
337been created.
338
339 >>> ls(sample_buildout, 'parts')
340 d demo
341 d python
342
343If you want to have initialization that only affects scripts, not the
344interpreter, you can use script-initialization. Here's a demonstration.
345
346 >>> write(sample_buildout, 'buildout.cfg',
347 ... """
348 ... [buildout]
349 ... parts = demo
350 ...
351 ... [demo]
352 ... recipe = z3c.recipe.scripts
353 ... eggs = demo<0.3
354 ... find-links = %(server)s
355 ... index = %(server)s/index
356 ... interpreter = py
357 ... script-initialization =
358 ... print "Hi from the script"
359 ... """ % dict(server=link_server))
360
361 >>> print system(buildout),
362 Uninstalling python.
363 Uninstalling demo.
364 Installing demo.
365 Generated script '/sample-buildout/bin/demo'.
366 Generated interpreter '/sample-buildout/bin/py'.
367
368 >>> print system(join(sample_buildout, 'bin', 'py') +
369 ... ''' -c "print 'Hi from the interpreter'"'''),
370 Hi from the interpreter
371
372 >>> print system(join(sample_buildout, 'bin', 'demo')),
373 Hi from the script
374 2 2
375
376The last new option is ``name``. This simply changes the name of the
377interpreter, so that you are not forced to use the name of the section.
378
379 >>> write(sample_buildout, 'buildout.cfg',
380 ... """
381 ... [buildout]
382 ... parts = interpreter
383 ...
384 ... [interpreter]
385 ... name = python2
386 ... recipe = z3c.recipe.scripts:interpreter
387 ... eggs = demo<0.3
388 ... find-links = %(server)s
389 ... index = %(server)s/index
390 ... """ % dict(server=link_server))
391
392 >>> print system(buildout),
393 Uninstalling demo.
394 Installing interpreter.
395 Generated interpreter '/sample-buildout/bin/python2'.
396
397 >>> print system(join(sample_buildout, 'bin', 'python2') +
398 ... ' -c "print 42"')
399 42
400 <BLANKLINE>
401
402The other options all identical to zc.recipe.egg.
0403
=== added file 'z3c.recipe.scripts_/src/z3c/recipe/scripts/__init__.py'
--- z3c.recipe.scripts_/src/z3c/recipe/scripts/__init__.py 1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/src/z3c/recipe/scripts/__init__.py 2010-02-23 20:41:17 +0000
@@ -0,0 +1,1 @@
1from z3c.recipe.scripts.scripts import Scripts, Interpreter
02
=== added file 'z3c.recipe.scripts_/src/z3c/recipe/scripts/scripts.py'
--- z3c.recipe.scripts_/src/z3c/recipe/scripts/scripts.py 1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/src/z3c/recipe/scripts/scripts.py 2010-02-23 20:41:17 +0000
@@ -0,0 +1,101 @@
1##############################################################################
2#
3# Copyright (c) 2009-2010 Zope Corporation and Contributors.
4# All Rights Reserved.
5#
6# This software is subject to the provisions of the Zope Public License,
7# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
8# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
9# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
10# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
11# FOR A PARTICULAR PURPOSE.
12#
13##############################################################################
14"""Install scripts from eggs.
15"""
16import os
17import zc.buildout
18import zc.buildout.easy_install
19from zc.recipe.egg.egg import ScriptBase
20
21
22class Base(ScriptBase):
23
24 def __init__(self, buildout, name, options):
25 if 'extends' in options:
26 options.update(buildout[options['extends']])
27 super(Base, self).__init__(buildout, name, options)
28 self.default_eggs = '' # Disables feature from zc.recipe.egg.
29 b_options = buildout['buildout']
30 options['parts-directory'] = os.path.join(
31 b_options['parts-directory'], self.name)
32
33 value = options.setdefault(
34 'add-site-packages',
35 b_options.get('add-site-packages', 'false'))
36 if value not in ('true', 'false'):
37 raise zc.buildout.UserError(
38 "Invalid value for add-site-packages option: %s" %
39 (value,))
40 self.add_site_packages = (value == 'true')
41
42 value = options.setdefault(
43 'exec-sitecustomize',
44 b_options.get('exec-sitecustomize', 'false'))
45 if value not in ('true', 'false'):
46 raise zc.buildout.UserError(
47 "Invalid value for exec-sitecustomize option: %s" %
48 (value,))
49 self.exec_sitecustomize = (value == 'true')
50
51
52class Interpreter(Base):
53
54 def __init__(self, buildout, name, options):
55 super(Interpreter, self).__init__(buildout, name, options)
56
57 options.setdefault('name', name)
58
59 def install(self):
60 reqs, ws = self.working_set()
61 options = self.options
62 generated = []
63 if not os.path.exists(options['parts-directory']):
64 os.mkdir(options['parts-directory'])
65 generated.append(options['parts-directory'])
66 generated.extend(zc.buildout.easy_install.sitepackage_safe_scripts(
67 options['bin-directory'], ws, options['executable'],
68 options['parts-directory'],
69 interpreter=options['name'],
70 extra_paths=self.extra_paths,
71 initialization=options.get('initialization', ''),
72 add_site_packages=self.add_site_packages,
73 exec_sitecustomize=self.exec_sitecustomize,
74 relative_paths=self._relative_paths,
75 ))
76 return generated
77
78 update = install
79
80
81class Scripts(Base):
82
83 def _install(self, reqs, ws, scripts):
84 options = self.options
85 generated = []
86 if not os.path.exists(options['parts-directory']):
87 os.mkdir(options['parts-directory'])
88 generated.append(options['parts-directory'])
89 generated.extend(zc.buildout.easy_install.sitepackage_safe_scripts(
90 options['bin-directory'], ws, options['executable'],
91 options['parts-directory'], reqs=reqs, scripts=scripts,
92 interpreter=options.get('interpreter'),
93 extra_paths=self.extra_paths,
94 initialization=options.get('initialization', ''),
95 add_site_packages=self.add_site_packages,
96 exec_sitecustomize=self.exec_sitecustomize,
97 relative_paths=self._relative_paths,
98 script_arguments=options.get('arguments', ''),
99 script_initialization=options.get('script-initialization', '')
100 ))
101 return generated
0102
=== added file 'z3c.recipe.scripts_/src/z3c/recipe/scripts/tests.py'
--- z3c.recipe.scripts_/src/z3c/recipe/scripts/tests.py 1970-01-01 00:00:00 +0000
+++ z3c.recipe.scripts_/src/z3c/recipe/scripts/tests.py 2010-02-23 20:41:17 +0000
@@ -0,0 +1,293 @@
1##############################################################################
2#
3# Copyright (c) 2006 Zope Corporation and Contributors.
4# All Rights Reserved.
5#
6# This software is subject to the provisions of the Zope Public License,
7# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
8# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
9# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
10# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
11# FOR A PARTICULAR PURPOSE.
12#
13##############################################################################
14
15import os, re, shutil, sys
16import zc.buildout.tests
17import zc.buildout.testselectingpython
18import zc.buildout.testing
19
20import unittest
21from zope.testing import doctest, renormalizing
22
23# We do not explicitly test the recipe support for the ``eggs``,
24# ``find-links``, and ``index`` options because they are used for most or
25# all of the examples. The README tests ``extends``,
26# ``include-site-customization`` and ``name``. That leaves ``python``,
27# ``extra-paths``, ``initialization``, ``relative-paths``, and
28# ``add-site-packages``.
29
30def supports_python_option():
31 """
32This simply shows that the ``python`` option can specify another section to
33find the ``executable``. (The ``python`` option defaults to looking in the
34``buildout`` section.) We do this by creating a custom Python that will have
35some initialization that we can look for.
36
37 >>> py_path, site_packages_path = make_py(initialization='''
38 ... import os
39 ... os.environ['zc.buildout'] = 'foo bar baz shazam'
40 ... ''')
41
42 >>> write(sample_buildout, 'buildout.cfg',
43 ... '''
44 ... [buildout]
45 ... parts = py
46 ...
47 ... [custom_python]
48 ... executable = %(py_path)s
49 ...
50 ... [py]
51 ... recipe = z3c.recipe.scripts:interpreter
52 ... exec-sitecustomize = true
53 ... eggs = demo<0.3
54 ... find-links = %(server)s
55 ... index = %(server)s/index
56 ... python = custom_python
57 ... ''' % dict(server=link_server, py_path=py_path))
58
59 >>> print system(buildout),
60 Installing py.
61 Getting distribution for 'demo<0.3'.
62 Got demo 0.2.
63 Getting distribution for 'demoneeded'.
64 Got demoneeded 1.2c1.
65 Generated interpreter '/sample-buildout/bin/py'.
66
67 >>> print system(join(sample_buildout, 'bin', 'py') +
68 ... ''' -c "import os; print os.environ['zc.buildout']"'''),
69 foo bar baz shazam
70"""
71
72def interpreter_recipe_supports_extra_paths_option():
73 """
74This shows that specifying extra-paths will affect sys.path.
75
76This recipe will not add paths that do not exist, so we create them.
77
78 >>> mkdir(sample_buildout, 'foo')
79 >>> mkdir(sample_buildout, 'foo', 'bar')
80 >>> mkdir(sample_buildout, 'spam')
81
82 >>> write(sample_buildout, 'buildout.cfg',
83 ... '''
84 ... [buildout]
85 ... parts = py
86 ...
87 ... [py]
88 ... recipe = z3c.recipe.scripts:interpreter
89 ... find-links = %(server)s
90 ... index = %(server)s/index
91 ... extra-paths =
92 ... ${buildout:directory}/foo/bar
93 ... ${buildout:directory}/spam
94 ... ''' % dict(server=link_server))
95
96 >>> print system(buildout),
97 Installing py.
98 Generated interpreter '/sample-buildout/bin/py'.
99 >>> print system(join(sample_buildout, 'bin', 'py') +
100 ... ''' -c "import sys;print 'path' + ' '.join(sys.path)"''')
101 ... # doctest:+ELLIPSIS
102 path.../foo/bar /sample-buildout/spam...
103
104"""
105
106def interpreter_recipe_supports_initialization_option():
107 """
108This simply shows that the ``initialization`` option can specify code to
109run on initialization.
110
111 >>> write(sample_buildout, 'buildout.cfg',
112 ... '''
113 ... [buildout]
114 ... parts = py
115 ...
116 ... [py]
117 ... recipe = z3c.recipe.scripts:interpreter
118 ... initialization =
119 ... import os
120 ... os.environ['zc.buildout'] = 'foo bar baz shazam'
121 ... eggs = demo<0.3
122 ... find-links = %(server)s
123 ... index = %(server)s/index
124 ... ''' % dict(server=link_server))
125
126 >>> print system(buildout),
127 Installing py.
128 Getting distribution for 'demo<0.3'.
129 Got demo 0.2.
130 Getting distribution for 'demoneeded'.
131 Got demoneeded 1.2c1.
132 Generated interpreter '/sample-buildout/bin/py'.
133
134 >>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
135 ... # doctest: +NORMALIZE_WHITESPACE
136 <BLANKLINE>
137 import os
138 os.environ['zc.buildout'] = 'foo bar baz shazam'
139 >>> print system(join(sample_buildout, 'bin', 'py') +
140 ... ''' -c "import os; print os.environ['zc.buildout']"'''),
141 foo bar baz shazam
142
143This also works with the exec-sitecustomize option, processing local
144initialization, and then the Python's initialization. We show this with a
145custom Python.
146
147 >>> py_path, site_packages_path = make_py(initialization='''
148 ... import os
149 ... os.environ['zc.buildout'] = 'foo bar baz shazam'
150 ... ''')
151
152 >>> write(sample_buildout, 'buildout.cfg',
153 ... '''
154 ... [buildout]
155 ... parts = py
156 ...
157 ... [custom_python]
158 ... executable = %(py_path)s
159 ...
160 ... [py]
161 ... recipe = z3c.recipe.scripts:interpreter
162 ... initialization =
163 ... import os
164 ... os.environ['zc.recipe.egg'] = 'baLOOba'
165 ... exec-sitecustomize = true
166 ... eggs = demo<0.3
167 ... find-links = %(server)s
168 ... index = %(server)s/index
169 ... python = custom_python
170 ... ''' % dict(server=link_server, py_path=py_path))
171
172 >>> print system(buildout),
173 Uninstalling py.
174 Installing py.
175 Generated interpreter '/sample-buildout/bin/py'.
176
177 >>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
178 ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
179 <BLANKLINE>
180 import os
181 os.environ['zc.recipe.egg'] = 'baLOOba'
182 <BLANKLINE>
183 # The following is from
184 # /executable_buildout/parts/py/sitecustomize.py
185 ...
186 import os
187 os.environ['zc.buildout'] = 'foo bar baz shazam'
188
189 >>> print system(join(sample_buildout, 'bin', 'py') + ' -c ' +
190 ... '''"import os; print os.environ['zc.recipe.egg']"'''),
191 baLOOba
192 >>> print system(join(sample_buildout, 'bin', 'py') +
193 ... ''' -c "import os; print os.environ['zc.buildout']"'''),
194 foo bar baz shazam
195
196"""
197
198def interpreter_recipe_supports_relative_paths_option():
199 """
200This shows that the relative-paths option affects the code for inserting
201paths into sys.path.
202
203 >>> write(sample_buildout, 'buildout.cfg',
204 ... '''
205 ... [buildout]
206 ... parts = py
207 ...
208 ... [py]
209 ... recipe = z3c.recipe.scripts:interpreter
210 ... find-links = %(server)s
211 ... index = %(server)s/index
212 ... relative-paths = true
213 ... extra-paths =
214 ... /foo/bar
215 ... ${buildout:directory}/spam
216 ... ''' % dict(server=link_server))
217
218 >>> print system(buildout),
219 Installing py.
220 Generated interpreter '/sample-buildout/bin/py'.
221
222Let's look at the site.py that was generated:
223
224 >>> import sys
225 >>> sys.stdout.write('#'); cat(sample_buildout, 'parts', 'py', 'site.py')
226 ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
227 #...
228 def addsitepackages(known_paths):
229 "..."
230 join = os.path.join
231 base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
232 base = os.path.dirname(base)
233 base = os.path.dirname(base)
234 buildout_paths = [
235 '/foo/bar',
236 join(base, 'spam')
237 ]...
238
239
240"""
241
242def setUp(test):
243 zc.buildout.tests.easy_install_SetUp(test)
244 zc.buildout.testing.install_develop('zc.recipe.egg', test)
245 zc.buildout.testing.install_develop('z3c.recipe.scripts', test)
246
247def setUpSelecting(test):
248 zc.buildout.testselectingpython.setup(test)
249 zc.buildout.testing.install_develop('zc.recipe.egg', test)
250 zc.buildout.testing.install_develop('z3c.recipe.scripts', test)
251
252def test_suite():
253 suite = unittest.TestSuite((
254 doctest.DocFileSuite(
255 'README.txt',
256 setUp=setUp, tearDown=zc.buildout.testing.buildoutTearDown,
257 checker=renormalizing.RENormalizing([
258 zc.buildout.testing.normalize_path,
259 zc.buildout.testing.normalize_endings,
260 zc.buildout.testing.normalize_script,
261 zc.buildout.testing.normalize_egg_py,
262 zc.buildout.tests.normalize_bang,
263 (re.compile(r'zc.buildout(-\S+)?[.]egg(-link)?'),
264 'zc.buildout.egg'),
265 (re.compile('[-d] setuptools-[^-]+-'), 'setuptools-X-'),
266 (re.compile(r'setuptools-[\w.]+-py'), 'setuptools-X-py'),
267 (re.compile(r'eggs\\\\demo'), 'eggs/demo'),
268 (re.compile(r'[a-zA-Z]:\\\\foo\\\\bar'), '/foo/bar'),
269 (re.compile(r'\#!\S+\bpython\S*'), '#!/usr/bin/python'),
270 # Normalize generate_script's Windows interpreter to UNIX:
271 (re.compile(r'\nimport subprocess\n'), '\n'),
272 (re.compile('subprocess\\.call\\(argv, env=environ\\)'),
273 'os.execve(sys.executable, argv, environ)'),
274 ])
275 ),
276 doctest.DocTestSuite(
277 setUp=setUp,
278 tearDown=zc.buildout.testing.buildoutTearDown,
279 checker=renormalizing.RENormalizing([
280 zc.buildout.testing.normalize_path,
281 zc.buildout.testing.normalize_endings,
282 zc.buildout.testing.normalize_egg_py,
283 (re.compile(r'[a-zA-Z]:\\\\foo\\\\bar'), '/foo/bar'),
284 ]),
285 ),
286
287 ))
288
289 return suite
290
291if __name__ == '__main__':
292 unittest.main(defaultTest='test_suite')
293
0294
=== modified file 'zc.recipe.egg_/setup.py'
--- zc.recipe.egg_/setup.py 2009-08-06 13:49:47 +0000
+++ zc.recipe.egg_/setup.py 2010-02-23 20:41:17 +0000
@@ -16,7 +16,7 @@
16$Id$16$Id$
17"""17"""
1818
19version = '0'19version = '1.2.3dev'
2020
21import os21import os
22from setuptools import setup, find_packages22from setuptools import setup, find_packages
@@ -66,7 +66,7 @@
66 package_dir = {'':'src'},66 package_dir = {'':'src'},
67 namespace_packages = ['zc', 'zc.recipe'],67 namespace_packages = ['zc', 'zc.recipe'],
68 install_requires = [68 install_requires = [
69 'zc.buildout >=1.2.0',69 'zc.buildout >=1.5.0dev',
70 'setuptools'],70 'setuptools'],
71 tests_require = ['zope.testing'],71 tests_require = ['zope.testing'],
72 test_suite = name+'.tests.test_suite',72 test_suite = name+'.tests.test_suite',
7373
=== modified file 'zc.recipe.egg_/src/zc/recipe/egg/README.txt'
--- zc.recipe.egg_/src/zc/recipe/egg/README.txt 2010-02-23 20:41:17 +0000
+++ zc.recipe.egg_/src/zc/recipe/egg/README.txt 2010-02-23 20:41:17 +0000
@@ -154,6 +154,8 @@
154interpreter154interpreter
155 The name of a script to generate that allows access to a Python155 The name of a script to generate that allows access to a Python
156 interpreter that has the path set based on the eggs installed.156 interpreter that has the path set based on the eggs installed.
157 (See the ``z3c.recipe.scripts`` recipe for a more full-featured
158 interpreter.)
157159
158extra-paths160extra-paths
159 Extra paths to include in a generated script.161 Extra paths to include in a generated script.
@@ -577,7 +579,7 @@
577 - demo579 - demo
578 - other580 - other
579581
580 >>> cat(sample_buildout, 'bin', 'other')582 >>> cat(sample_buildout, 'bin', 'other') # doctest: +NORMALIZE_WHITESPACE
581 #!/usr/local/bin/python2.4583 #!/usr/local/bin/python2.4
582 <BLANKLINE>584 <BLANKLINE>
583 import sys585 import sys
@@ -640,3 +642,4 @@
640 Uninstalling bigdemo.642 Uninstalling bigdemo.
641 Installing demo.643 Installing demo.
642 Generated script '/sample-buildout/bin/foo'.644 Generated script '/sample-buildout/bin/foo'.
645
643646
=== modified file 'zc.recipe.egg_/src/zc/recipe/egg/api.txt'
--- zc.recipe.egg_/src/zc/recipe/egg/api.txt 2010-02-23 20:41:17 +0000
+++ zc.recipe.egg_/src/zc/recipe/egg/api.txt 2010-02-23 20:41:17 +0000
@@ -117,6 +117,7 @@
117 extras = other117 extras = other
118 find-links = http://localhost:27071/118 find-links = http://localhost:27071/
119 index = http://localhost:27071/index119 index = http://localhost:27071/index
120 python = buildout
120 recipe = sample121 recipe = sample
121122
122If we use the extra-paths option:123If we use the extra-paths option:
123124
=== modified file 'zc.recipe.egg_/src/zc/recipe/egg/custom.txt'
--- zc.recipe.egg_/src/zc/recipe/egg/custom.txt 2010-02-23 20:41:17 +0000
+++ zc.recipe.egg_/src/zc/recipe/egg/custom.txt 2010-02-23 20:41:17 +0000
@@ -150,6 +150,7 @@
150150
151 >>> ls(sample_buildout, 'develop-eggs')151 >>> ls(sample_buildout, 'develop-eggs')
152 d extdemo-1.4-py2.4-unix-i686.egg152 d extdemo-1.4-py2.4-unix-i686.egg
153 - z3c.recipe.scripts.egg-link
153 - zc.recipe.egg.egg-link154 - zc.recipe.egg.egg-link
154155
155Note that no scripts or dependencies are installed. To install156Note that no scripts or dependencies are installed. To install
@@ -231,6 +232,7 @@
231 >>> ls(sample_buildout, 'develop-eggs')232 >>> ls(sample_buildout, 'develop-eggs')
232 - demo.egg-link233 - demo.egg-link
233 d extdemo-1.4-py2.4-unix-i686.egg234 d extdemo-1.4-py2.4-unix-i686.egg
235 - z3c.recipe.scripts.egg-link
234 - zc.recipe.egg.egg-link236 - zc.recipe.egg.egg-link
235237
236But if we run the buildout in the default on-line and newest modes, we238But if we run the buildout in the default on-line and newest modes, we
@@ -248,6 +250,7 @@
248 - demo.egg-link250 - demo.egg-link
249 d extdemo-1.4-py2.4-linux-i686.egg251 d extdemo-1.4-py2.4-linux-i686.egg
250 d extdemo-1.5-py2.4-linux-i686.egg252 d extdemo-1.5-py2.4-linux-i686.egg
253 - z3c.recipe.scripts.egg-link
251 - zc.recipe.egg.egg-link254 - zc.recipe.egg.egg-link
252255
253Controlling the version used256Controlling the version used
@@ -287,6 +290,7 @@
287 >>> ls(sample_buildout, 'develop-eggs')290 >>> ls(sample_buildout, 'develop-eggs')
288 - demo.egg-link291 - demo.egg-link
289 d extdemo-1.4-py2.4-linux-i686.egg292 d extdemo-1.4-py2.4-linux-i686.egg
293 - z3c.recipe.scripts.egg-link
290 - zc.recipe.egg.egg-link294 - zc.recipe.egg.egg-link
291295
292296
@@ -553,6 +557,7 @@
553 >>> ls('develop-eggs')557 >>> ls('develop-eggs')
554 - demo.egg-link558 - demo.egg-link
555 - extdemo.egg-link559 - extdemo.egg-link
560 - z3c.recipe.scripts.egg-link
556 - zc.recipe.egg.egg-link561 - zc.recipe.egg.egg-link
557562
558and the extdemo now has a built extension:563and the extdemo now has a built extension:
559564
=== modified file 'zc.recipe.egg_/src/zc/recipe/egg/egg.py'
--- zc.recipe.egg_/src/zc/recipe/egg/egg.py 2010-02-23 20:41:17 +0000
+++ zc.recipe.egg_/src/zc/recipe/egg/egg.py 2010-02-23 20:41:17 +0000
@@ -19,11 +19,12 @@
19import logging, os, re, zipfile19import logging, os, re, zipfile
20import zc.buildout.easy_install20import zc.buildout.easy_install
2121
22
22class Eggs(object):23class Eggs(object):
2324
24 def __init__(self, buildout, name, options):25 def __init__(self, buildout, name, options):
25 self.buildout = buildout26 self.buildout = buildout
26 self.name = name27 self.name = self.default_eggs = name
27 self.options = options28 self.options = options
28 b_options = buildout['buildout']29 b_options = buildout['buildout']
29 links = options.get('find-links', b_options['find-links'])30 links = options.get('find-links', b_options['find-links'])
@@ -52,7 +53,7 @@
52 # verify that this is None, 'true' or 'false'53 # verify that this is None, 'true' or 'false'
53 get_bool(options, 'unzip')54 get_bool(options, 'unzip')
5455
55 python = options.get('python', b_options['python'])56 python = options.setdefault('python', b_options['python'])
56 options['executable'] = buildout[python]['executable']57 options['executable'] = buildout[python]['executable']
5758
58 def working_set(self, extra=()):59 def working_set(self, extra=()):
@@ -65,15 +66,16 @@
6566
66 distributions = [67 distributions = [
67 r.strip()68 r.strip()
68 for r in options.get('eggs', self.name).split('\n')69 for r in options.get('eggs', self.default_eggs).split('\n')
69 if r.strip()]70 if r.strip()]
70 orig_distributions = distributions[:]71 orig_distributions = distributions[:]
71 distributions.extend(extra)72 distributions.extend(extra)
7273
73 if self.buildout['buildout'].get('offline') == 'true':74 if b_options.get('offline') == 'true':
74 ws = zc.buildout.easy_install.working_set(75 ws = zc.buildout.easy_install.working_set(
75 distributions, options['executable'],76 distributions, options['executable'],
76 [options['develop-eggs-directory'], options['eggs-directory']]77 [options['develop-eggs-directory'],
78 options['eggs-directory']],
77 )79 )
78 else:80 else:
79 kw = {}81 kw = {}
@@ -85,7 +87,7 @@
85 index=self.index,87 index=self.index,
86 executable=options['executable'],88 executable=options['executable'],
87 path=[options['develop-eggs-directory']],89 path=[options['develop-eggs-directory']],
88 newest=self.buildout['buildout'].get('newest') == 'true',90 newest=b_options.get('newest') == 'true',
89 allow_hosts=self.allow_hosts,91 allow_hosts=self.allow_hosts,
90 **kw)92 **kw)
9193
@@ -97,16 +99,19 @@
9799
98 update = install100 update = install
99101
100class Scripts(Eggs):102
103class ScriptBase(Eggs):
101104
102 def __init__(self, buildout, name, options):105 def __init__(self, buildout, name, options):
103 super(Scripts, self).__init__(buildout, name, options)106 super(ScriptBase, self).__init__(buildout, name, options)
104107
105 options['bin-directory'] = buildout['buildout']['bin-directory']108 b_options = buildout['buildout']
109
110 options['bin-directory'] = b_options['bin-directory']
106 options['_b'] = options['bin-directory'] # backward compat.111 options['_b'] = options['bin-directory'] # backward compat.
107112
108 self.extra_paths = [113 self.extra_paths = [
109 os.path.join(buildout['buildout']['directory'], p.strip())114 os.path.join(b_options['directory'], p.strip())
110 for p in options.get('extra-paths', '').split('\n')115 for p in options.get('extra-paths', '').split('\n')
111 if p.strip()116 if p.strip()
112 ]117 ]
@@ -115,11 +120,9 @@
115120
116121
117 relative_paths = options.get(122 relative_paths = options.get(
118 'relative-paths',123 'relative-paths', b_options.get('relative-paths', 'false'))
119 buildout['buildout'].get('relative-paths', 'false')
120 )
121 if relative_paths == 'true':124 if relative_paths == 'true':
122 options['buildout-directory'] = buildout['buildout']['directory']125 options['buildout-directory'] = b_options['directory']
123 self._relative_paths = options['buildout-directory']126 self._relative_paths = options['buildout-directory']
124 else:127 else:
125 self._relative_paths = ''128 self._relative_paths = ''
@@ -128,12 +131,13 @@
128 parse_entry_point = re.compile(131 parse_entry_point = re.compile(
129 '([^=]+)=(\w+(?:[.]\w+)*):(\w+(?:[.]\w+)*)$'132 '([^=]+)=(\w+(?:[.]\w+)*):(\w+(?:[.]\w+)*)$'
130 ).match133 ).match
134
131 def install(self):135 def install(self):
132 reqs, ws = self.working_set()136 reqs, ws = self.working_set()
133 options = self.options137 options = self.options
134138
135 scripts = options.get('scripts')139 scripts = options.get('scripts')
136 if scripts or scripts is None:140 if scripts or scripts is None or options.get('interpreter'):
137 if scripts is not None:141 if scripts is not None:
138 scripts = scripts.split()142 scripts = scripts.split()
139 scripts = dict([143 scripts = dict([
@@ -157,22 +161,32 @@
157 name = dist.project_name161 name = dist.project_name
158 if name != 'setuptools' and name not in reqs:162 if name != 'setuptools' and name not in reqs:
159 reqs.append(name)163 reqs.append(name)
160164 return self._install(reqs, ws, scripts)
161 return zc.buildout.easy_install.scripts(
162 reqs, ws, options['executable'],
163 options['bin-directory'],
164 scripts=scripts,
165 extra_paths=self.extra_paths,
166 interpreter=options.get('interpreter'),
167 initialization=options.get('initialization', ''),
168 arguments=options.get('arguments', ''),
169 relative_paths=self._relative_paths,
170 )
171
172 return ()165 return ()
173166
174 update = install167 update = install
175168
169 def _install(self, reqs, ws, scripts):
170 # Subclasses implement this.
171 raise NotImplementedError()
172
173
174class Scripts(ScriptBase):
175
176 def _install(self, reqs, ws, scripts):
177 options = self.options
178 return zc.buildout.easy_install.scripts(
179 reqs, ws, options['executable'],
180 options['bin-directory'],
181 scripts=scripts,
182 extra_paths=self.extra_paths,
183 interpreter=options.get('interpreter'),
184 initialization=options.get('initialization', ''),
185 arguments=options.get('arguments', ''),
186 relative_paths=self._relative_paths
187 )
188
189
176def get_bool(options, name, default=False):190def get_bool(options, name, default=False):
177 value = options.get(name)191 value = options.get(name)
178 if not value:192 if not value:
179193
=== modified file 'zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt'
--- zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt 2010-02-23 20:41:17 +0000
+++ zc.recipe.egg_/src/zc/recipe/egg/selecting-python.txt 2010-02-23 20:41:17 +0000
@@ -35,7 +35,7 @@
35 ... index = http://www.python.org/pypi/35 ... index = http://www.python.org/pypi/
36 ...36 ...
37 ... [python2.4]37 ... [python2.4]
38 ... executable = %(python23)s38 ... executable = %(python24)s
39 ...39 ...
40 ... [demo]40 ... [demo]
41 ... recipe = zc.recipe.egg41 ... recipe = zc.recipe.egg
@@ -43,7 +43,7 @@
43 ... find-links = %(server)s43 ... find-links = %(server)s
44 ... python = python2.444 ... python = python2.4
45 ... interpreter = py-demo45 ... interpreter = py-demo
46 ... """ % dict(server=link_server, python23=other_executable))46 ... """ % dict(server=link_server, python24=other_executable))
4747
48Now, if we run the buildout:48Now, if we run the buildout:
4949
@@ -55,8 +55,6 @@
55 Getting distribution for 'demo<0.3'.55 Getting distribution for 'demo<0.3'.
56 Got demo 0.2.56 Got demo 0.2.
57 Getting distribution for 'demoneeded'.57 Getting distribution for 'demoneeded'.
58 Getting distribution for 'setuptools'.
59 Got setuptools 0.6.
60 Got demoneeded 1.2c1.58 Got demoneeded 1.2c1.
61 Generated script '/sample-buildout/bin/demo'.59 Generated script '/sample-buildout/bin/demo'.
62 Generated interpreter '/sample-buildout/bin/py-demo'.60 Generated interpreter '/sample-buildout/bin/py-demo'.
@@ -66,7 +64,6 @@
66 >>> ls(sample_buildout, 'eggs')64 >>> ls(sample_buildout, 'eggs')
67 - demo-0.2-py2.4.egg65 - demo-0.2-py2.4.egg
68 - demoneeded-1.2c1-py2.4.egg66 - demoneeded-1.2c1-py2.4.egg
69 d setuptools-0.6-py2.4.egg
70 d setuptools-0.6-py2.5.egg67 d setuptools-0.6-py2.5.egg
71 - zc.buildout-1.0-py2.5.egg68 - zc.buildout-1.0-py2.5.egg
7269

Subscribers

People subscribed via source and target branches

to all changes: