Merge lp:~gary/zc.buildout/betafix2 into lp:zc.buildout
- betafix2
- Merge into trunk
Status: | Needs review |
---|---|
Proposed branch: | lp:~gary/zc.buildout/betafix2 |
Merge into: | lp:zc.buildout |
Prerequisite: | lp:~gary/zc.buildout/betafix1 |
Diff against target: |
634 lines (+366/-34) 6 files modified
dev.py (+14/-6) src/zc/buildout/buildout.py (+33/-9) src/zc/buildout/easy_install.py (+55/-10) src/zc/buildout/testing.py (+1/-1) src/zc/buildout/tests.py (+23/-8) src/zc/buildout/virtualenv.txt (+240/-0) |
To merge this branch: | bzr merge lp:~gary/zc.buildout/betafix2 |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Francis J. Lacoste (community) | Approve | ||
Review via email: mp+27733@code.launchpad.net |
Commit message
Description of the change
This is the set of changes to make buildout play nicely with virtualenv again.
The core problem with the virtualenv interaction is that the virtualenv executable's -S behavior is broken. Rather than detecting virtualenv, I try to detect that brokenness.
The basic goal with these changes was to make virtualenv users have the "classic" behavior with as little hindrance as possible. As shown in the tests, everything now is as transparent as I could make it. The only time you even get a complaint is when you get a recipe that tries to use the new features: it ignores the request for the new features and warns you. If you want to make the warning go away, you can use -s (new) or one or more -q options (the existing mechanism to request a "quiet" build).
It would be legitimate to request more tests, for instance of the restart behavior within zc.buildout. I have not yet done so because I felt that the tests I did add were reasonably comprehensive and representative, even though they were not exhaustive.
A reasonable approach to the review might be to look at the virtualenv.txt test first, and then go out from there.
Unmerged revisions
- 555. By Gary Poster
-
fix virtualenv interaction by identfying broken virtualenv characteristic and reverting to previous behavior in that case.
- 554. By Gary Poster
-
Fix some problems with allowed_
eggs_from_ site_packages - 553. By Gary Poster
-
get a baseline of zc.buildout with passing tests
Preview Diff
1 | === modified file 'dev.py' |
2 | --- dev.py 1970-01-01 00:00:00 +0000 |
3 | +++ dev.py 2010-06-16 15:24:27 +0000 |
4 | @@ -19,7 +19,7 @@ |
5 | $Id$ |
6 | """ |
7 | |
8 | -import os, shutil, sys, subprocess, urllib2 |
9 | +import os, shutil, sys, subprocess, urllib2, subprocess |
10 | from optparse import OptionParser |
11 | |
12 | if sys.platform == 'win32': |
13 | @@ -31,11 +31,15 @@ |
14 | else: |
15 | quote = str |
16 | |
17 | +# Detect https://bugs.launchpad.net/virtualenv/+bug/572545 . |
18 | +has_broken_dash_S = subprocess.call( |
19 | + [sys.executable, '-Sc', 'import ConfigParser']) |
20 | + |
21 | # In order to be more robust in the face of system Pythons, we want to |
22 | # run without site-packages loaded. This is somewhat tricky, in |
23 | # particular because Python 2.6's distutils imports site, so starting |
24 | # with the -S flag is not sufficient. However, we'll start with that: |
25 | -if 'site' in sys.modules: |
26 | +if not has_broken_dash_S and 'site' in sys.modules: |
27 | # We will restart with python -S. |
28 | args = sys.argv[:] |
29 | args[0:0] = [sys.executable, '-S'] |
30 | @@ -117,10 +121,14 @@ |
31 | |
32 | env = os.environ.copy() # Windows needs yet-to-be-determined values from this. |
33 | env['PYTHONPATH'] = os.path.dirname(pkg_resources.__file__) |
34 | -subprocess.Popen( |
35 | - [sys.executable] + |
36 | - ['setup.py', '-q', 'develop', '-m', '-x', '-d', 'develop-eggs'], |
37 | - env=env).wait() |
38 | + |
39 | +cmd = [quote(sys.executable), |
40 | + 'setup.py', '-q', 'develop', '-m', '-x', '-d', 'develop-eggs'] |
41 | + |
42 | +if not has_broken_dash_S: |
43 | + cmd.insert(1, '-S') |
44 | + |
45 | +subprocess.Popen(cmd, env=env).wait() |
46 | |
47 | pkg_resources.working_set.add_entry('src') |
48 | |
49 | |
50 | === modified file 'src/zc/buildout/buildout.py' |
51 | --- src/zc/buildout/buildout.py 2010-04-29 18:11:45 +0000 |
52 | +++ src/zc/buildout/buildout.py 2010-06-16 15:24:27 +0000 |
53 | @@ -34,6 +34,7 @@ |
54 | import sys |
55 | import tempfile |
56 | import UserDict |
57 | +import warnings |
58 | import zc.buildout |
59 | import zc.buildout.download |
60 | import zc.buildout.easy_install |
61 | @@ -51,6 +52,9 @@ |
62 | if is_jython: |
63 | import subprocess |
64 | |
65 | +_sys_executable_has_broken_dash_S = ( |
66 | + zc.buildout.easy_install._has_broken_dash_S(sys.executable)) |
67 | + |
68 | class MissingOption(zc.buildout.UserError, KeyError): |
69 | """A required option was missing. |
70 | """ |
71 | @@ -359,7 +363,7 @@ |
72 | distributions, options['executable'], |
73 | [options['develop-eggs-directory'], |
74 | options['eggs-directory']], |
75 | - include_site_packages=False, |
76 | + include_site_packages=_sys_executable_has_broken_dash_S, |
77 | ) |
78 | else: |
79 | ws = zc.buildout.easy_install.install( |
80 | @@ -370,7 +374,7 @@ |
81 | path=[options['develop-eggs-directory']], |
82 | newest=self.newest, |
83 | allow_hosts=self._allow_hosts, |
84 | - include_site_packages=False, |
85 | + include_site_packages=_sys_executable_has_broken_dash_S, |
86 | ) |
87 | |
88 | # Now copy buildout and setuptools eggs, and record destination eggs: |
89 | @@ -408,7 +412,8 @@ |
90 | relative_paths = '' |
91 | zc.buildout.easy_install.sitepackage_safe_scripts( |
92 | options['bin-directory'], ws, options['executable'], partsdir, |
93 | - reqs=['zc.buildout'], relative_paths=relative_paths) |
94 | + reqs=['zc.buildout'], relative_paths=relative_paths, |
95 | + include_site_packages=_sys_executable_has_broken_dash_S) |
96 | |
97 | init = bootstrap |
98 | |
99 | @@ -854,7 +859,7 @@ |
100 | index = options.get('index'), |
101 | path = [options['develop-eggs-directory']], |
102 | allow_hosts = self._allow_hosts, |
103 | - include_site_packages=False |
104 | + include_site_packages=_sys_executable_has_broken_dash_S |
105 | ) |
106 | |
107 | upgraded = [] |
108 | @@ -910,7 +915,8 @@ |
109 | os.mkdir(partsdir) |
110 | zc.buildout.easy_install.sitepackage_safe_scripts( |
111 | options['bin-directory'], ws, sys.executable, partsdir, |
112 | - reqs=['zc.buildout']) |
113 | + reqs=['zc.buildout'], |
114 | + include_site_packages=_sys_executable_has_broken_dash_S) |
115 | |
116 | # Restart |
117 | args = map(zc.buildout.easy_install._safe_arg, sys.argv) |
118 | @@ -951,7 +957,7 @@ |
119 | links = self['buildout'].get('find-links', '').split(), |
120 | index = self['buildout'].get('index'), |
121 | newest=self.newest, allow_hosts=self._allow_hosts, |
122 | - include_site_packages=False) |
123 | + include_site_packages=_sys_executable_has_broken_dash_S) |
124 | |
125 | # Clear cache because extensions might now let us read pages we |
126 | # couldn't read before. |
127 | @@ -1069,8 +1075,7 @@ |
128 | working_set=pkg_resources.working_set, |
129 | newest=buildout.newest, |
130 | allow_hosts=buildout._allow_hosts, |
131 | - include_site_packages=False, |
132 | - ) |
133 | + include_site_packages=_sys_executable_has_broken_dash_S) |
134 | |
135 | __doing__ = 'Loading %s recipe entry %s:%s.', group, spec, entry |
136 | return pkg_resources.load_entry_point( |
137 | @@ -1577,6 +1582,11 @@ |
138 | will be started. This is especially useful for debuging recipe |
139 | problems. |
140 | |
141 | + -s |
142 | + |
143 | + Squelch warnings about using an executable with a broken -S |
144 | + implementation. |
145 | + |
146 | Assignments are of the form: section:option=value and are used to |
147 | provide configuration options that override those given in the |
148 | configuration file. For example, to run the buildout in offline mode, |
149 | @@ -1642,11 +1652,12 @@ |
150 | windows_restart = False |
151 | user_defaults = True |
152 | debug = False |
153 | + ignore_broken_dash_s = False |
154 | while args: |
155 | if args[0][0] == '-': |
156 | op = orig_op = args.pop(0) |
157 | op = op[1:] |
158 | - while op and op[0] in 'vqhWUoOnNDA': |
159 | + while op and op[0] in 'vqhWUoOnNDAs': |
160 | if op[0] == 'v': |
161 | verbosity += 10 |
162 | elif op[0] == 'q': |
163 | @@ -1665,6 +1676,8 @@ |
164 | options.append(('buildout', 'newest', 'false')) |
165 | elif op[0] == 'D': |
166 | debug = True |
167 | + elif op[0] == 's': |
168 | + ignore_broken_dash_s = True |
169 | else: |
170 | _help() |
171 | op = op[1:] |
172 | @@ -1708,6 +1721,17 @@ |
173 | # The rest should be commands, so we'll stop here |
174 | break |
175 | |
176 | + if verbosity < 0 or ignore_broken_dash_s: |
177 | + broken_dash_S_filter_action = 'ignore' |
178 | + elif verbosity == 0: # This is the default. |
179 | + broken_dash_S_filter_action = 'once' |
180 | + else: |
181 | + broken_dash_S_filter_action = 'default' |
182 | + warnings.filterwarnings( |
183 | + broken_dash_S_filter_action, |
184 | + re.escape( |
185 | + zc.buildout.easy_install.BROKEN_DASH_S_WARNING), |
186 | + UserWarning) |
187 | if verbosity: |
188 | options.append(('buildout', 'verbosity', str(verbosity))) |
189 | |
190 | |
191 | === modified file 'src/zc/buildout/easy_install.py' |
192 | --- src/zc/buildout/easy_install.py 2010-06-16 15:24:27 +0000 |
193 | +++ src/zc/buildout/easy_install.py 2010-06-16 15:24:27 +0000 |
194 | @@ -33,6 +33,7 @@ |
195 | import subprocess |
196 | import sys |
197 | import tempfile |
198 | +import warnings |
199 | import zc.buildout |
200 | import zipimport |
201 | |
202 | @@ -54,6 +55,18 @@ |
203 | is_distribute = ( |
204 | pkg_resources.Requirement.parse('setuptools').key=='distribute') |
205 | |
206 | +BROKEN_DASH_S_WARNING = ( |
207 | + 'Buildout has been asked to exclude or limit site-packages so that ' |
208 | + 'builds can be repeatable when using a system Python. However, ' |
209 | + 'the chosen Python executable has a broken implementation of -S (see ' |
210 | + 'https://bugs.launchpad.net/virtualenv/+bug/572545 for an example ' |
211 | + "problem) and this breaks buildout's ability to isolate site-packages. " |
212 | + "If the executable already has a clean site-packages (e.g., " |
213 | + "using virtualenv's ``--no-site-packages`` option) you may be getting " |
214 | + 'equivalent repeatability. To silence this warning, use the -s argument ' |
215 | + 'to the buildout script. Alternatively, use a Python executable with a ' |
216 | + 'working -S (such as a standard Python binary).') |
217 | + |
218 | if is_jython: |
219 | import java.lang.System |
220 | jython_os_name = (java.lang.System.getProperties()['os.name']).lower() |
221 | @@ -70,6 +83,14 @@ |
222 | if os.path.normpath(setuptools_loc) != os.path.normpath(buildout_loc): |
223 | buildout_and_setuptools_path.append(buildout_loc) |
224 | |
225 | +def _has_broken_dash_S(executable): |
226 | + """Detect https://bugs.launchpad.net/virtualenv/+bug/572545 .""" |
227 | + proc = subprocess.Popen( |
228 | + [executable, '-Sc', 'import ConfigParser'], |
229 | + stdout=subprocess.PIPE, stderr=subprocess.PIPE) |
230 | + proc.communicate() |
231 | + return bool(proc.returncode) |
232 | + |
233 | def _get_system_paths(executable): |
234 | """Return lists of standard lib and site paths for executable. |
235 | """ |
236 | @@ -209,10 +230,10 @@ |
237 | _safe_arg = str |
238 | |
239 | # The following string is used to run easy_install in |
240 | -# Installer._call_easy_install. It is started with python -S (that is, |
241 | -# don't import site at start). That flag, and all of the code in this |
242 | -# snippet above the last two lines, exist to work around a relatively rare |
243 | -# problem. If |
244 | +# Installer._call_easy_install. It is usually started with python -S |
245 | +# (that is, don't import site at start). That flag, and all of the code |
246 | +# in this snippet above the last two lines, exist to work around a |
247 | +# relatively rare problem. If |
248 | # |
249 | # - your buildout configuration is trying to install a package that is within |
250 | # a namespace package, and |
251 | @@ -264,17 +285,16 @@ |
252 | # unnecessary for site.py to preprocess these packages, so it should be |
253 | # fine, as far as can be guessed as of this writing.) Finally, it |
254 | # imports easy_install and runs it. |
255 | - |
256 | -_easy_install_cmd = _safe_arg('''\ |
257 | +_easy_install_preface = '''\ |
258 | import sys,os;\ |
259 | p = sys.path[:];\ |
260 | import site;\ |
261 | sys.path[:] = p;\ |
262 | [sys.modules.pop(k) for k, v in sys.modules.items()\ |
263 | if hasattr(v, '__path__') and len(v.__path__)==1 and\ |
264 | - not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))];\ |
265 | -from setuptools.command.easy_install import main;\ |
266 | -main()''') |
267 | + not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))];''' |
268 | +_easy_install_cmd = ( |
269 | + 'from setuptools.command.easy_install import main;main()') |
270 | |
271 | |
272 | class Installer: |
273 | @@ -321,6 +341,7 @@ |
274 | |
275 | self._index_url = index |
276 | self._executable = executable |
277 | + self._has_broken_dash_S = _has_broken_dash_S(self._executable) |
278 | if always_unzip is not None: |
279 | self._always_unzip = always_unzip |
280 | path = (path and path[:] or []) |
281 | @@ -329,6 +350,17 @@ |
282 | if allowed_eggs_from_site_packages is not None: |
283 | self._allowed_eggs_from_site_packages = tuple( |
284 | allowed_eggs_from_site_packages) |
285 | + if self._has_broken_dash_S: |
286 | + if (not self._include_site_packages or |
287 | + self._allowed_eggs_from_site_packages != ('*',)): |
288 | + # We can't do this if the executable has a broken -S. |
289 | + warnings.warn(BROKEN_DASH_S_WARNING) |
290 | + self._include_site_packages = True |
291 | + self._allowed_eggs_from_site_packages = ('*',) |
292 | + self._easy_install_cmd = _easy_install_preface + _easy_install_cmd |
293 | + else: |
294 | + self._easy_install_cmd = _easy_install_cmd |
295 | + self._easy_install_cmd = _safe_arg(self._easy_install_cmd) |
296 | stdlib, self._site_packages = _get_system_paths(executable) |
297 | version_info = _get_version_info(executable) |
298 | if version_info == sys.version_info: |
299 | @@ -487,7 +519,9 @@ |
300 | try: |
301 | path = setuptools_loc |
302 | |
303 | - args = ('-Sc', _easy_install_cmd, '-mUNxd', _safe_arg(tmp)) |
304 | + args = ('-c', self._easy_install_cmd, '-mUNxd', _safe_arg(tmp)) |
305 | + if not self._has_broken_dash_S: |
306 | + args = ('-S',) + args |
307 | if self._always_unzip: |
308 | args += ('-Z', ) |
309 | level = logger.getEffectiveLevel() |
310 | @@ -1176,6 +1210,11 @@ |
311 | _pyscript(spath, sname, executable, rpsetup)) |
312 | return generated |
313 | |
314 | +# We need to give an alternate name to the ``scripts`` function so that it |
315 | +# can be referenced within sitepackage_safe_scripts, which uses ``scripts`` |
316 | +# as an argument name. |
317 | +_original_scripts_function = scripts |
318 | + |
319 | def sitepackage_safe_scripts( |
320 | dest, working_set, executable, site_py_dest, |
321 | reqs=(), scripts=None, interpreter=None, extra_paths=(), |
322 | @@ -1188,6 +1227,12 @@ |
323 | Python site packages, if desired, and choosing to execute the Python's |
324 | sitecustomize. |
325 | """ |
326 | + if _has_broken_dash_S(executable): |
327 | + if not include_site_packages: |
328 | + warnings.warn(BROKEN_DASH_S_WARNING) |
329 | + return _original_scripts_function( |
330 | + reqs, working_set, executable, dest, scripts, extra_paths, |
331 | + script_arguments, interpreter, initialization, relative_paths) |
332 | generated = [] |
333 | generated.append(_generate_sitecustomize( |
334 | site_py_dest, executable, initialization, exec_sitecustomize)) |
335 | |
336 | === modified file 'src/zc/buildout/testing.py' |
337 | --- src/zc/buildout/testing.py 1970-01-01 00:00:00 +0000 |
338 | +++ src/zc/buildout/testing.py 2010-06-16 15:24:27 +0000 |
339 | @@ -596,7 +596,7 @@ |
340 | sep = re.escape(os.path.sep) |
341 | normalize_path = ( |
342 | re.compile( |
343 | - r'''[^'" \t\n\r]+%(sep)s_[Tt][Ee][Ss][Tt]_%(sep)s([^"' \t\n\r]+)''' |
344 | + r'''[^'" \t\n\r!]+%(sep)s_[Tt][Ee][Ss][Tt]_%(sep)s([^"' \t\n\r]+)''' |
345 | % dict(sep=sep)), |
346 | _normalize_path, |
347 | ) |
348 | |
349 | === modified file 'src/zc/buildout/tests.py' |
350 | --- src/zc/buildout/tests.py 2010-06-16 15:24:27 +0000 |
351 | +++ src/zc/buildout/tests.py 2010-06-16 15:24:27 +0000 |
352 | @@ -3923,14 +3923,29 @@ |
353 | setUp=easy_install_SetUp, |
354 | tearDown=zc.buildout.testing.buildoutTearDown, |
355 | checker=renormalizing.RENormalizing([ |
356 | - zc.buildout.testing.normalize_path, |
357 | - zc.buildout.testing.normalize_endings, |
358 | - zc.buildout.testing.normalize_script, |
359 | - normalize_bang, |
360 | - (re.compile('Downloading.*setuptools.*egg\n'), ''), |
361 | - (re.compile('options:'), 'Options:'), |
362 | - (re.compile('usage:'), 'Usage:'), |
363 | - ]), |
364 | + zc.buildout.testing.normalize_path, |
365 | + zc.buildout.testing.normalize_endings, |
366 | + zc.buildout.testing.normalize_script, |
367 | + normalize_bang, |
368 | + (re.compile('Downloading.*setuptools.*egg\n'), ''), |
369 | + (re.compile('options:'), 'Options:'), |
370 | + (re.compile('usage:'), 'Usage:'), |
371 | + ]), |
372 | + )) |
373 | + test_suite.append(doctest.DocFileSuite( |
374 | + 'virtualenv.txt', |
375 | + setUp=easy_install_SetUp, |
376 | + tearDown=zc.buildout.testing.buildoutTearDown, |
377 | + checker=renormalizing.RENormalizing([ |
378 | + zc.buildout.testing.normalize_path, |
379 | + zc.buildout.testing.normalize_endings, |
380 | + zc.buildout.testing.normalize_script, |
381 | + zc.buildout.testing.normalize_egg_py, |
382 | + (re.compile('(setuptools|distribute)-\S+-'), |
383 | + 'setuptools.egg'), |
384 | + (re.compile('zc.buildout-\S+-'), |
385 | + 'zc.buildout.egg'), |
386 | + ]), |
387 | )) |
388 | |
389 | return unittest.TestSuite(test_suite) |
390 | |
391 | === added file 'src/zc/buildout/virtualenv.txt' |
392 | --- src/zc/buildout/virtualenv.txt 1970-01-01 00:00:00 +0000 |
393 | +++ src/zc/buildout/virtualenv.txt 2010-06-16 15:24:27 +0000 |
394 | @@ -0,0 +1,240 @@ |
395 | +Version 1.5.0 of buildout (and higher) provides the ability to use |
396 | +buildout directly with a system Python if you use z3c.recipe.scripts or |
397 | +other isolation-aware recipes that use the sitepackage_safe_scripts function. |
398 | + |
399 | +Some people use virtualenv to provide similar functionality. |
400 | +Unfortunately, a problem with the virtualenv executable as of this |
401 | +writing means that -S will not work properly with it (see |
402 | +https://bugs.launchpad.net/virtualenv/+bug/572545). This breaks |
403 | +buildout's approach to providing isolation. |
404 | + |
405 | +Because of this, if buildout detects an executable with a broken -S |
406 | +option, it will revert to its pre-1.5.0 behavior. If buildout has been |
407 | +asked to provide isolation, it will warn the user that isolation will |
408 | +not be provided by buildout, but proceed. This should give full |
409 | +backwards compatibility to virtualenv users. |
410 | + |
411 | +The only minor annoyance in the future may be recipes that explicitly |
412 | +use the new buildout functionality to provide isolation: as described |
413 | +above, the builds will proceed, but users will receive warnings that |
414 | +buildout is not providing isolation itself. The warnings themselves can |
415 | +be squelched when running bin/buildout with the ``-s`` option or with a |
416 | +lower verbosity than usual (e.g., one or more ``-q`` options). |
417 | + |
418 | +For tests, then, we can examine several things. We'll focus on four. |
419 | + |
420 | +- Running bootstrap with an executable broken in this way will not try to do |
421 | + any -S tricks. |
422 | + |
423 | +- Running sitepackage_safe_scripts with a virtualenv will create an |
424 | + old-style script. This will affect the bin/buildout script that is |
425 | + created, for instance. If the sitepackage_safe_scripts function is asked |
426 | + to provide isolation under these circumstances, it will warn that isolation |
427 | + will not be available, but still create the desired script. |
428 | + |
429 | +- Using the easy_install Installer or install or build functions and trying |
430 | + to request isolation will generate a warning and then the isolation request |
431 | + will be ignored as it proceeds. |
432 | + |
433 | +- Passing -s (or -q) to the bin/buildout script will squelch warnings. |
434 | + |
435 | +Testing these involves first creating a Python that exhibits the same |
436 | +behavior as the problematic one we care about from virtualenv. Let's do that |
437 | +first. |
438 | + |
439 | + >>> import os, sys |
440 | + >>> py_path, site_packages_path = make_py() |
441 | + >>> py_file = open(py_path) |
442 | + >>> py_lines = py_file.readlines() |
443 | + >>> py_file.close() |
444 | + >>> py_file = open(py_path, 'w') |
445 | + >>> extra = '''\ |
446 | + ... new_argv = argv[:1] |
447 | + ... for ix, val in enumerate(argv[1:]): |
448 | + ... if val.startswith('--'): |
449 | + ... new_argv.append(val) |
450 | + ... if val.startswith('-') and len(val) > 1: |
451 | + ... if 'S' in val: |
452 | + ... val = val.replace('S', '') |
453 | + ... environ['BROKEN_DASH_S'] = 'Y' |
454 | + ... if val != '-': |
455 | + ... new_argv.append(val) |
456 | + ... if 'c' in val: |
457 | + ... new_argv.extend(argv[ix+2:]) |
458 | + ... break |
459 | + ... else: |
460 | + ... new_argv.extend(argv[ix+1:]) |
461 | + ... argv = new_argv |
462 | + ... ''' |
463 | + >>> for line in py_lines: |
464 | + ... py_file.write(line) |
465 | + ... if line.startswith('environ = os.environ.copy()'): |
466 | + ... py_file.write(extra) |
467 | + ... print 'Rewritten.' |
468 | + ... |
469 | + Rewritten. |
470 | + >>> py_file.close() |
471 | + >>> sitecustomize_path = join(os.path.dirname(site_packages_path), |
472 | + ... 'parts', 'py', 'sitecustomize.py') |
473 | + >>> sitecustomize_file = open(sitecustomize_path, 'a') |
474 | + >>> sitecustomize_file.write(''' |
475 | + ... import os, sys |
476 | + ... sys.executable = %r |
477 | + ... if 'BROKEN_DASH_S' in os.environ: |
478 | + ... class ImportHook: |
479 | + ... @staticmethod |
480 | + ... def find_module(fullname, path=None): |
481 | + ... if fullname == 'ConfigParser': |
482 | + ... raise ImportError() |
483 | + ... |
484 | + ... sys.meta_path.append(ImportHook) |
485 | + ... sys.modules.pop('site', None) # Keeps site out of sys.modules. |
486 | + ... # This will be a close-enough approximation of site not being |
487 | + ... # loaded for our tests--it lets us provoke the right errors when |
488 | + ... # the fixes are absent, and works well enough when the fixes are |
489 | + ... # present. |
490 | + ... ''' % (py_path,)) |
491 | + >>> sitecustomize_file.close() |
492 | + >>> print call_py( |
493 | + ... py_path, |
494 | + ... "import ConfigParser") |
495 | + <BLANKLINE> |
496 | + >>> print 'X'; print call_py( |
497 | + ... py_path, |
498 | + ... "import ConfigParser", |
499 | + ... '-S') # doctest: +ELLIPSIS |
500 | + X...Traceback (most recent call last): |
501 | + ... |
502 | + ImportError: No module named ConfigParser |
503 | + <BLANKLINE> |
504 | + >>> from zc.buildout.easy_install import _has_broken_dash_S |
505 | + >>> _has_broken_dash_S(py_path) |
506 | + True |
507 | + |
508 | +Well, that was ugly, but it seems to have done the trick. The |
509 | +executable represented by py_path has the same problematic |
510 | +characteristic as the virtualenv one: -S results in a Python that does |
511 | +not allow the import of some packages from the standard library. We'll |
512 | +test with this. |
513 | + |
514 | +First, let's try running bootstrap. |
515 | + |
516 | + >>> from os.path import dirname, join |
517 | + >>> import zc.buildout |
518 | + >>> bootstrap_py = join( |
519 | + ... dirname( |
520 | + ... dirname( |
521 | + ... dirname( |
522 | + ... dirname(zc.buildout.__file__) |
523 | + ... ) |
524 | + ... ) |
525 | + ... ), |
526 | + ... 'bootstrap', 'bootstrap.py') |
527 | + >>> broken_S_buildout = tmpdir('broken_S') |
528 | + >>> os.chdir(broken_S_buildout) |
529 | + >>> write('buildout.cfg', |
530 | + ... ''' |
531 | + ... [buildout] |
532 | + ... parts = |
533 | + ... ''') |
534 | + >>> write('bootstrap.py', open(bootstrap_py).read()) |
535 | + >>> print 'X'; print system( |
536 | + ... zc.buildout.easy_install._safe_arg(py_path)+' '+ |
537 | + ... 'bootstrap.py'); print 'X' # doctest: +ELLIPSIS |
538 | + X... |
539 | + Generated script '/broken_S/bin/buildout'. |
540 | + ... |
541 | + |
542 | +If bootstrap didn't look out for a broken -S, that would have failed. Moreover, |
543 | +take a look at bin/buildout: |
544 | + |
545 | + >>> cat('bin', 'buildout') |
546 | + #!/executable_buildout/bin/py |
547 | + <BLANKLINE> |
548 | + import sys |
549 | + sys.path[0:0] = [ |
550 | + '/broken_S/eggs/setuptools-0.0-pyN.N.egg', |
551 | + '/broken_S/eggs/zc.buildout-0.0-pyN.N.egg', |
552 | + ] |
553 | + <BLANKLINE> |
554 | + import zc.buildout.buildout |
555 | + <BLANKLINE> |
556 | + if __name__ == '__main__': |
557 | + zc.buildout.buildout.main() |
558 | + |
559 | +That's the old-style buildout script: no changes for users with this issue. |
560 | + |
561 | +Of course, they don't get the new features either, presumably because |
562 | +they don't need or want them. This means that if they use a recipe that |
563 | +tries to use a new feature, the behavior needs to degrade gracefully. |
564 | + |
565 | +Here's an example. We'll switch to another buildout in which it is easier to |
566 | +use local dev versions of zc.buildout and z3c.recipe.scripts. |
567 | + |
568 | + >>> os.chdir(dirname(dirname(buildout))) |
569 | + >>> write('buildout.cfg', |
570 | + ... ''' |
571 | + ... [buildout] |
572 | + ... parts = eggs |
573 | + ... find-links = %(link_server)s |
574 | + ... |
575 | + ... [primed_python] |
576 | + ... executable = %(py_path)s |
577 | + ... |
578 | + ... [eggs] |
579 | + ... recipe = z3c.recipe.scripts |
580 | + ... python = primed_python |
581 | + ... interpreter = py |
582 | + ... eggs = demo |
583 | + ... ''' % globals()) |
584 | + |
585 | + >>> print system(buildout) # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS |
586 | + Installing eggs. |
587 | + Getting distribution for 'demo'. |
588 | + Got demo 0.4c1. |
589 | + Getting distribution for 'demoneeded'. |
590 | + Got demoneeded 1.2c1. |
591 | + Generated script '/sample-buildout/bin/demo'. |
592 | + Generated interpreter '/sample-buildout/bin/py'. |
593 | + ...UserWarning: Buildout has been asked to exclude or limit site-packages |
594 | + so that builds can be repeatable when using a system Python. However, |
595 | + the chosen Python executable has a broken implementation of -S (see |
596 | + https://bugs.launchpad.net/virtualenv/+bug/572545 for an example |
597 | + problem) and this breaks buildout's ability to isolate site-packages. |
598 | + If the executable already has a clean site-packages (e.g., using |
599 | + virtualenv's ``--no-site-packages`` option) you may be getting |
600 | + equivalent repeatability. To silence this warning, use the -s argument |
601 | + to the buildout script. Alternatively, use a Python executable with a |
602 | + working -S (such as a standard Python binary). |
603 | + warnings.warn(BROKEN_DASH_S_WARNING) |
604 | + <BLANKLINE> |
605 | + |
606 | +So, it did what we asked as best it could, but gave a big warning. If |
607 | +you don't want those warnings for those particular recipes that use the |
608 | +new features, you can use the "-s" option to squelch the warnings. |
609 | + |
610 | + >>> print system(buildout + ' -s') |
611 | + Updating eggs. |
612 | + <BLANKLINE> |
613 | + |
614 | +A lower verbosity (one or more -q options) also quiets the warning. |
615 | + |
616 | + >>> print system(buildout + ' -q') |
617 | + <BLANKLINE> |
618 | + |
619 | +Notice that, as we saw before with bin/buildout, the generated scripts |
620 | +are old-style, because the new-style feature gracefully degrades to the |
621 | +previous implementation when it encounters an executable with a broken |
622 | +dash-S. |
623 | + |
624 | + >>> print 'X'; cat('bin', 'py') # doctest: +ELLIPSIS |
625 | + X... |
626 | + <BLANKLINE> |
627 | + import sys |
628 | + <BLANKLINE> |
629 | + sys.path[0:0] = [ |
630 | + '/sample-buildout/eggs/demo-0.4c1-pyN.N.egg', |
631 | + '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg', |
632 | + ] |
633 | + ... |
634 | + |
> +Well, that was ugly, but it seems to have done the trick. The
> +executable represented by py_path has the same problematic
Indeed!
But other than that, all is good. Test is very readable and I think is enough.