Merge lp:~gary/z3c.recipe.filetemplate/relative-paths into lp:z3c.recipe.filetemplate
- relative-paths
- Merge into svn
Status: | Needs review |
---|---|
Proposed branch: | lp:~gary/z3c.recipe.filetemplate/relative-paths |
Merge into: | lp:z3c.recipe.filetemplate |
Prerequisite: | lp:~gary/z3c.recipe.filetemplate/cleanup |
Diff against target: |
1409 lines (+954/-192) 7 files modified
.bzrignore (+7/-0) CHANGES.txt (+37/-4) MANIFEST.in (+3/-0) setup.py (+1/-1) z3c/recipe/filetemplate/README.txt (+531/-123) z3c/recipe/filetemplate/__init__.py (+300/-56) z3c/recipe/filetemplate/tests.txt (+75/-8) |
To merge this branch: | bzr merge lp:~gary/z3c.recipe.filetemplate/relative-paths |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Francis J. Lacoste (community) | Approve | ||
Review via email: mp+23701@code.launchpad.net |
Commit message
Description of the change
This branch adds support for the buildout relative-paths option to z3c.recipe.
The approach chosen adds two features to the recipe: ``path extensions`` and ``filters``. With these features, and some magic variables, I was able to provide a workable solution to the problem of relative paths.
As noted in the MP fields, this builds on the "cleanup" branch.
Francis J. Lacoste (flacoste) wrote : | # |
I have a couple of minor comments and questions. Should be good to go.
> === modified file 'z3c/recipe/
> + >>> cat(sample_
> + #!/bin/sh
> + Z3C_RECIPE_
> + readlink -f "$0" 2>/dev/null || \
> + realpath "$0" 2>/dev/null || \
> + type -P "$0" 2>/dev/null`
> + Z3C_RECIPE_
> + Z3C_RECIPE_
> + cat "$Z3C_RECIPE_
> +
The multiple mutation of Z3C_RECIPE_
Especially, the two identical Z3C_RECIPE_
I assume that's the way to get to the parent directory of the script. Would be
clearer to use multiple variables, at the detriment of more namespace
polluting. At the very least a comment would help the reader. Your call.
> === modified file 'z3c/recipe/
> + def _get(self, section, option, start):
> + if section is None:
> + section = self.recipe.name # This sets up error messages properly.
> + if section == self.recipe.name:
> + factory = self.recipe.
> + if factory is not None:
> + try:
> + return factory(self, start, option)
> + except:
> + # Argh. Would like to raise wrapped exception.
> + colno, lineno = self.get_
> + msg = ('Dynamic option %r in line %d, col %d of %s '
> + 'crashed.') % (option, lineno, colno, self.source)
> + self.recipe.
> + raise
> + # else...
> + options = self.recipe.options
>
Shouldn't you catch and re-raise SystemExit and KeyboardInterrupt? Or was this
fixed in Python 2.5?
> - def substitute(self, recipe, seen):
> + def substitute(self):
> def convert(mo):
> + start = mo.start()
> # Check the most common path first.
> - option = mo.group(
> + option = mo.group('option')
> if option is not None:
> - val = self._get(
> - mo.start(
> + section = mo.group('section')
> + val = self._get(section, option, start)
> + path_extension = mo.group(
> + filters = mo.group('filters')
> + if path_extension is not None:
> + val = os.path.join(val, *path_extension
> + if filters is not None:
> + for filter in filters.
> + filter = filter.strip()
> + if filter not in self.recipe.
> + colno, lineno = self.get_
> + raise ValueError(
> + '...
Francis J. Lacoste (flacoste) wrote : | # |
On April 19, 2010, Gary Poster wrote:
> For reference, this is a patch to Launchpad that takes advantage of this
> new feature.
>
> http://
>
> After this patch and lp~gary/
> absolute paths based on the build are only found in Launchpad in three
> locations:
>
> - in scripts generated by z3c.recipe.tag (would be easy to fix)
We don't care about that one in deployment.
> - in scripts generated by z3c.recipe.i18n (would be easy to fix)
Nor about this one.
> - in code generated by our Mailman integration (would be hard to fix, but
> after we move to Python 2.6 Barry Warsaw may be willing and able to help
> us move to the new Mailman code base).
That one means that we'll probably still need to run make clean on the mailman
machine at least?
--
Francis J. Lacoste
<email address hidden>
- 22. By Gary Poster
-
add comments to generated code; only log a problem in filters and dynamic options if they are not stopped because of SystemExit and KeyboardInterrupt.
Gary Poster (gary) wrote : | # |
On Apr 19, 2010, at 4:28 PM, Francis J. Lacoste wrote:
> Review: Approve
> I have a couple of minor comments and questions. Should be good to go.
>
>> === modified file 'z3c/recipe/
>
>> + >>> cat(sample_
>> + #!/bin/sh
>> + Z3C_RECIPE_
>> + readlink -f "$0" 2>/dev/null || \
>> + realpath "$0" 2>/dev/null || \
>> + type -P "$0" 2>/dev/null`
>> + Z3C_RECIPE_
>> + Z3C_RECIPE_
>> + cat "$Z3C_RECIPE_
>> +
>
> The multiple mutation of Z3C_RECIPE_
> Especially, the two identical Z3C_RECIPE_
> I assume that's the way to get to the parent directory of the script. Would be
> clearer to use multiple variables, at the detriment of more namespace
> polluting. At the very least a comment would help the reader. Your call.
Done, with variable name changes and comments, for both shell and Python versions.
>
>> === modified file 'z3c/recipe/
>
>> + def _get(self, section, option, start):
>> + if section is None:
>> + section = self.recipe.name # This sets up error messages properly.
>> + if section == self.recipe.name:
>> + factory = self.recipe.
>> + if factory is not None:
>> + try:
>> + return factory(self, start, option)
>> + except:
>> + # Argh. Would like to raise wrapped exception.
>> + colno, lineno = self.get_
>> + msg = ('Dynamic option %r in line %d, col %d of %s '
>> + 'crashed.') % (option, lineno, colno, self.source)
>> + self.recipe.
>> + raise
>> + # else...
>> + options = self.recipe.options
>>
>
> Shouldn't you catch and re-raise SystemExit and KeyboardInterrupt? Or was this
> fixed in Python 2.5?
Changed.
>> - def substitute(self, recipe, seen):
>> + def substitute(self):
>> def convert(mo):
>> + start = mo.start()
>> # Check the most common path first.
>> - option = mo.group(
>> + option = mo.group('option')
>> if option is not None:
>> - val = self._get(
>> - mo.start(
>> + section = mo.group('section')
>> + val = self._get(section, option, start)
>> + path_extension = mo.group(
>> + filters = mo.group('filters')
>> + if path_extension is not None:
>> + val = os.path.join(val, *path_extension
>> + if filters is not None:
>> + for filter in filters.
>> + filte...
Unmerged revisions
- 22. By Gary Poster
-
add comments to generated code; only log a problem in filters and dynamic options if they are not stopped because of SystemExit and KeyboardInterrupt.
- 21. By Gary Poster
-
remove comment that turned out not to be true.
- 20. By Gary Poster
-
tweak based on usage
- 19. By Gary Poster
-
add support for relative paths
- 18. By Gary Poster
-
re-commit the pertinent work from lp:~gary/z3c.recipe.filetemplate/support-system-python
Preview Diff
1 | === added file '.bzrignore' |
2 | --- .bzrignore 1970-01-01 00:00:00 +0000 |
3 | +++ .bzrignore 2010-04-20 19:34:27 +0000 |
4 | @@ -0,0 +1,7 @@ |
5 | +.installed.cfg |
6 | +bin |
7 | +develop-eggs |
8 | +eggs |
9 | +parts |
10 | +z3c.recipe.filetemplate.egg-info |
11 | +dist |
12 | |
13 | === modified file 'CHANGES.txt' |
14 | --- CHANGES.txt 2010-04-20 19:34:27 +0000 |
15 | +++ CHANGES.txt 2010-04-20 19:34:27 +0000 |
16 | @@ -9,10 +9,35 @@ |
17 | Features |
18 | -------- |
19 | |
20 | -- Support escaping "${...}" with "$${...}" in templates. This is particularly |
21 | - useful for *NIX shell scripts. |
22 | - |
23 | -- Support the relative-paths buildout option. XXX describe design |
24 | +- Enable cross-platform paths by allowing an extended syntax for path |
25 | + suffixes. Example: If ``${buildout:directory}`` resolves to |
26 | + ``/sample_buildout`` on a POSIX system and ``C:\sample_buildout`` in |
27 | + Windows, ``${buildout:directory/foo.txt}`` will resolve to |
28 | + ``/sample_buildout/foo.txt`` and ``C:\sample_buildout\foo.txt``, |
29 | + respectively. |
30 | + |
31 | +- Add filters via a pipe syntax, reminiscent of UNIX pipes or Django template |
32 | + filters. Simple example: if ``${name}`` resolves to ``harry`` then |
33 | + ``${name|upper}`` resolves to ``HARRY``. Simple string filters are |
34 | + upper, lower, title, and capitalize, just like the Python string |
35 | + methods. Also see the next bullet. |
36 | + |
37 | +- Added support for the buildout relative-paths option. Shell scripts should |
38 | + include ``${shell-relative-path-setup}`` before commands with |
39 | + buildout-generated paths are executed. Python scripts should use |
40 | + ``${python-relative-path-setup}`` similarly. ``${os-paths}`` (shell), |
41 | + ``${space-paths}`` (shell), and ``${string-paths}`` (Python) will have |
42 | + relative paths if the buildout relative-paths option is used. To convert |
43 | + individual absolute paths to relative paths, use the ``path-repr`` filter |
44 | + in Python scripts and the ``shell-path`` filter in shell scripts. Path |
45 | + suffixes can be combined with these filters, so, if buildout's |
46 | + relative-paths option is true, ``${buildout:directory/foo.txt|path-repr}`` |
47 | + will produce a buildout-relative, platform appropriate path to |
48 | + foo.txt. Note that for shell scripts, Windows is not supported at |
49 | + this time. |
50 | + |
51 | +- Support escaping ``${...}`` with ``$${...}`` in templates. This is |
52 | + particularly useful for *NIX shell scripts. |
53 | |
54 | ----- |
55 | Fixes |
56 | @@ -20,6 +45,14 @@ |
57 | |
58 | - Make tests less susceptible to timing errors. |
59 | |
60 | +------- |
61 | +Changes |
62 | +------- |
63 | + |
64 | +- ``${os-paths}`` and ``${space-paths}`` no longer filter out .zip paths. |
65 | + |
66 | +- The entries in ``${string-paths}`` now are separated by newlines. Each |
67 | + entry is indented to the level of the initial placement of the marker. |
68 | |
69 | 2.0.3 (2009-07-02) |
70 | ================== |
71 | |
72 | === added file 'MANIFEST.in' |
73 | --- MANIFEST.in 1970-01-01 00:00:00 +0000 |
74 | +++ MANIFEST.in 2010-04-20 19:34:27 +0000 |
75 | @@ -0,0 +1,3 @@ |
76 | +include *.txt |
77 | +recursive-include z3c *.txt |
78 | +exclude MANIFEST.in buildout.cfg .bzrignore |
79 | |
80 | === modified file 'setup.py' |
81 | --- setup.py 2009-05-04 17:31:38 +0000 |
82 | +++ setup.py 2010-04-20 19:34:27 +0000 |
83 | @@ -19,7 +19,7 @@ |
84 | return open(os.path.join(os.path.dirname(__file__), *rnames)).read() |
85 | |
86 | setup(name='z3c.recipe.filetemplate', |
87 | - version = '2.1dev', |
88 | + version = '2.1', |
89 | license='ZPL 2.1', |
90 | url='http://pypi.python.org/pypi/z3c.recipe.filetemplate', |
91 | description="zc.buildout recipe for creating files from file templates", |
92 | |
93 | === modified file 'z3c/recipe/filetemplate/README.txt' |
94 | --- z3c/recipe/filetemplate/README.txt 2010-04-20 19:34:27 +0000 |
95 | +++ z3c/recipe/filetemplate/README.txt 2010-04-20 19:34:27 +0000 |
96 | @@ -224,6 +224,12 @@ |
97 | Also note that, if you use a source directory and your ``files`` specify a |
98 | directory, the directory must match precisely. |
99 | |
100 | + >>> # Clean up for later test. |
101 | + >>> import shutil |
102 | + >>> shutil.rmtree(os.path.join(sample_buildout, 'template', 'etc')) |
103 | + >>> os.remove(os.path.join( |
104 | + ... sample_buildout, 'template', 'bin', 'helloworld.sh.in')) |
105 | + |
106 | ============== |
107 | Advanced Usage |
108 | ============== |
109 | @@ -235,7 +241,7 @@ |
110 | standard buildout syntax, but used in the template. Notice |
111 | ``${buildout:parts}`` in the template below. |
112 | |
113 | - >>> write(sample_buildout, 'helloworld.txt.in', |
114 | + >>> update_file(sample_buildout, 'helloworld.txt.in', |
115 | ... """ |
116 | ... Hello ${world}. I used these parts: ${buildout:parts}. |
117 | ... """) |
118 | @@ -257,11 +263,108 @@ |
119 | >>> cat(sample_buildout, 'helloworld.txt') |
120 | Hello Philipp. I used these parts: message. |
121 | |
122 | -Sharing variables |
123 | +Path Extensions |
124 | +=============== |
125 | + |
126 | +Substitutions can have path suffixes using the POSIX "/" path separator. |
127 | +The template will convert these to the proper path separator for the current |
128 | +OS. They also then are part of the value passed to filters, the feature |
129 | +described next. Notice ``${buildout:directory/foo/bar.txt}`` in the template |
130 | +below. |
131 | + |
132 | + >>> update_file(sample_buildout, 'helloworld.txt.in', |
133 | + ... """ |
134 | + ... Here's foo/bar.txt in the buildout: |
135 | + ... ${buildout:directory/foo/bar.txt} |
136 | + ... """) |
137 | + |
138 | + >>> print system(buildout) |
139 | + Uninstalling message. |
140 | + Installing message. |
141 | + |
142 | + >>> cat(sample_buildout, 'helloworld.txt') # doctest: +ELLIPSIS |
143 | + Here's foo/bar.txt in the buildout: |
144 | + /.../sample-buildout/foo/bar.txt |
145 | + |
146 | +Filters |
147 | +======= |
148 | + |
149 | +You can use pipes within a substitution to filter the original value. This |
150 | +recipe provides several filters for you to use. The syntax is reminiscent of |
151 | +(and inspired by) POSIX pipes and Django template filters. For example, |
152 | +if world = Philipp, ``HELLO ${world|upper}!`` would result in ``HELLO |
153 | +PHILIPP!``. |
154 | + |
155 | +A few simple Python string methods are exposed as filters right now: |
156 | + |
157 | +- capitalize: First letter in string is capitalized. |
158 | +- lower: All letters in string are lowercase. |
159 | +- title: First letter of each word in string is capitalized. |
160 | +- upper: All letters in string are uppercase. |
161 | + |
162 | +Other filters are important for handling paths if buildout's relative-paths |
163 | +option is true. See `Working with Paths`_ for more details. |
164 | + |
165 | +- path-repr: Converts the path to a Python expression for the path. If |
166 | + buildout's relative-paths option is false, this will simply be a repr |
167 | + of the absolute path. If relative-paths is true, this will be a |
168 | + function call to convert a buildout-relative path to an absolute path; |
169 | + it requires that ``${python-relative-path-setup}`` be included earlier |
170 | + in the template. |
171 | + |
172 | +- shell-path: Converts the path to a shell expression for the path. Only |
173 | + POSIX is supported at this time. If buildout's relative-paths option |
174 | + is false, this will simply be the absolute path. If relative-paths is |
175 | + true, this will be an expression to convert a buildout-relative path |
176 | + to an absolute path; it requires that ``${shell-relative-path-setup}`` |
177 | + be included earlier in the template. |
178 | + |
179 | +Combining the three advanced features described so far, then, if the |
180 | +buildout relative-paths option were false, we were in a POSIX system, and |
181 | +the sample buildout were in the root of the system, the template |
182 | +expression ``${buildout:bin-directory/data/initial.csv|path-repr}`` |
183 | +would result in ``'/sample-buildout/bin/data/initial.csv'``. |
184 | + |
185 | +Here's a real, working example of the string method filters. We'll have |
186 | +examples of the path filters in the `Working with Paths`_ section. |
187 | + |
188 | + >>> update_file(sample_buildout, 'helloworld.txt.in', |
189 | + ... """ |
190 | + ... HELLO ${world|upper}! |
191 | + ... hello ${world|lower}. |
192 | + ... ${name|title} and the Chocolate Factory |
193 | + ... ${sentence|capitalize} |
194 | + ... """) |
195 | + |
196 | + >>> write(sample_buildout, 'buildout.cfg', |
197 | + ... """ |
198 | + ... [buildout] |
199 | + ... parts = message |
200 | + ... |
201 | + ... [message] |
202 | + ... recipe = z3c.recipe.filetemplate |
203 | + ... files = helloworld.txt |
204 | + ... world = Philipp |
205 | + ... name = willy wonka |
206 | + ... sentence = that is a good book. |
207 | + ... """) |
208 | + |
209 | + >>> print system(buildout) |
210 | + Uninstalling message. |
211 | + Installing message. |
212 | + |
213 | + >>> cat(sample_buildout, 'helloworld.txt') # doctest: +ELLIPSIS |
214 | + HELLO PHILIPP! |
215 | + hello philipp. |
216 | + Willy Wonka and the Chocolate Factory |
217 | + That is a good book. |
218 | + |
219 | +Sharing Variables |
220 | ================= |
221 | |
222 | -The recipe allows extending one or more sections, to decrease repetition, using |
223 | -the ``extends`` option. For instance, consider the following buildout. |
224 | +The recipe allows extending one or more sections, to decrease |
225 | +repetition, using the ``extends`` option. For instance, consider the |
226 | +following buildout. |
227 | |
228 | >>> write(sample_buildout, 'buildout.cfg', |
229 | ... """ |
230 | @@ -284,7 +387,7 @@ |
231 | section, and overwritten locally. A template of |
232 | ``${mygreeting}, ${myaudience}!``... |
233 | |
234 | - >>> write(sample_buildout, 'helloworld.txt.in', |
235 | + >>> update_file(sample_buildout, 'helloworld.txt.in', |
236 | ... """ |
237 | ... ${mygreeting}, ${myaudience}! |
238 | ... """) |
239 | @@ -298,114 +401,15 @@ |
240 | >>> cat(sample_buildout, 'helloworld.txt') |
241 | Hi, everybody! |
242 | |
243 | -Specifying paths |
244 | -================ |
245 | - |
246 | -You can specify eggs and extra-paths in the recipe. If you do, three |
247 | -predefined options will be available in the recipe's options for the template. |
248 | -If "paths" are the non-zip paths, and "all_paths" are all paths, then the |
249 | -options would be defined roughly as given here: |
250 | - |
251 | -``os-paths`` |
252 | - ``(os.pathsep).join(paths)`` |
253 | - |
254 | -``string-paths`` |
255 | - ``', '.join(repr(p) for p in all_paths)`` |
256 | - |
257 | -``space-paths`` |
258 | - ``' '.join(paths)`` |
259 | - |
260 | -For instance, consider this example. |
261 | - |
262 | - >>> write(sample_buildout, 'buildout.cfg', |
263 | - ... """ |
264 | - ... [buildout] |
265 | - ... parts = message |
266 | - ... |
267 | - ... [message] |
268 | - ... recipe = z3c.recipe.filetemplate |
269 | - ... files = helloworld.txt |
270 | - ... eggs = demo<0.3 |
271 | - ... |
272 | - ... find-links = %(server)s |
273 | - ... index = %(server)s/index |
274 | - ... """ % dict(server=link_server)) |
275 | - |
276 | - |
277 | - >>> write(sample_buildout, 'helloworld.txt.in', |
278 | - ... """ |
279 | - ... Hello! Here are the paths for the ${eggs} eggs. |
280 | - ... OS paths: |
281 | - ... ${os-paths} |
282 | - ... --- |
283 | - ... String paths: |
284 | - ... ${string-paths} |
285 | - ... --- |
286 | - ... Space paths: |
287 | - ... ${space-paths} |
288 | - ... """) |
289 | - |
290 | - >>> print system(buildout) |
291 | - Getting distribution for 'demo<0.3'. |
292 | - Got demo 0.2. |
293 | - Getting distribution for 'demoneeded'. |
294 | - Got demoneeded 1.2c1. |
295 | - Uninstalling message. |
296 | - Installing message. |
297 | - |
298 | - >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS |
299 | - Hello! Here are the paths for the demo<0.3 eggs. |
300 | - OS paths: |
301 | - .../eggs/demo-0.2...egg:.../eggs/demoneeded-1.2c1...egg |
302 | - --- |
303 | - String paths: |
304 | - '.../eggs/demo-0.2...egg', '.../eggs/demoneeded-1.2c1...egg' |
305 | - --- |
306 | - Space paths: |
307 | - .../eggs/demo-0.2...egg .../eggs/demoneeded-1.2c1...egg |
308 | - |
309 | -You can specify extra-paths as well, which will go at the end of the egg paths. |
310 | - |
311 | - >>> write(sample_buildout, 'buildout.cfg', |
312 | - ... """ |
313 | - ... [buildout] |
314 | - ... parts = message |
315 | - ... |
316 | - ... [message] |
317 | - ... recipe = z3c.recipe.filetemplate |
318 | - ... files = helloworld.txt |
319 | - ... eggs = demo<0.3 |
320 | - ... extra-paths = ${buildout:directory}/foo |
321 | - ... |
322 | - ... find-links = %(server)s |
323 | - ... index = %(server)s/index |
324 | - ... """ % dict(server=link_server)) |
325 | - |
326 | - >>> print system(buildout) |
327 | - Uninstalling message. |
328 | - Installing message. |
329 | - |
330 | - >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS |
331 | - Hello! Here are the paths for the demo<0.3 eggs. |
332 | - OS paths: |
333 | - ...demo...:...demoneeded...:.../sample-buildout/foo |
334 | - --- |
335 | - String paths: |
336 | - '...demo...', '...demoneeded...', '.../sample-buildout/foo' |
337 | - --- |
338 | - Space paths: |
339 | - ...demo... ...demoneeded... .../sample-buildout/foo |
340 | - |
341 | Defining options in Python |
342 | ========================== |
343 | |
344 | You can specify that certain variables should be interpreted as Python using |
345 | ``interpreted-options``. This takes zero or more lines. Each line should |
346 | -specify an option. It can define immediately (see ``duplicate-os-paths``, |
347 | -``foo-paths``, and ``silly-range`` in the example below) or point to an option |
348 | -to be interepreted, which can be useful if you want to define a |
349 | -multi-line expression (see ``first-interpreted-option`` and |
350 | -``message-reversed-is-egassem``). |
351 | +specify an option. It can define immediately (see ``silly-range`` in |
352 | +the example below) or point to an option to be interepreted, which can |
353 | +be useful if you want to define a multi-line expression (see |
354 | +``first-interpreted-option`` and ``message-reversed-is-egassem``). |
355 | |
356 | >>> write(sample_buildout, 'buildout.cfg', |
357 | ... """ |
358 | @@ -415,28 +419,20 @@ |
359 | ... [message] |
360 | ... recipe = z3c.recipe.filetemplate |
361 | ... files = helloworld.txt |
362 | - ... eggs = demo<0.3 |
363 | - ... interpreted-options = duplicate-os-paths=(os.pathsep).join(paths) |
364 | - ... foo-paths='FOO'.join(all_paths) |
365 | - ... silly-range = repr(range(5)) |
366 | + ... interpreted-options = silly-range = repr(range(5)) |
367 | ... first-interpreted-option |
368 | ... message-reversed-is-egassem |
369 | ... first-interpreted-option = |
370 | - ... options['interpreted-options'].split()[0].strip() |
371 | + ... options['interpreted-options'].splitlines()[0].strip() |
372 | ... message-reversed-is-egassem= |
373 | ... ''.join( |
374 | ... reversed( |
375 | ... buildout['buildout']['parts'])) |
376 | ... not-interpreted=hello world |
377 | - ... |
378 | - ... find-links = %(server)s |
379 | - ... index = %(server)s/index |
380 | - ... """ % dict(server=link_server)) |
381 | + ... """) |
382 | |
383 | - >>> write(sample_buildout, 'helloworld.txt.in', """\ |
384 | + >>> update_file(sample_buildout, 'helloworld.txt.in', """\ |
385 | ... ${not-interpreted}! |
386 | - ... duplicate-os-paths: ${duplicate-os-paths} |
387 | - ... foo-paths: ${foo-paths} |
388 | ... silly-range: ${silly-range} |
389 | ... first-interpreted-option: ${first-interpreted-option} |
390 | ... message-reversed-is-egassem: ${message-reversed-is-egassem} |
391 | @@ -448,9 +444,421 @@ |
392 | |
393 | >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS |
394 | hello world! |
395 | - duplicate-os-paths: ...demo-0.2...egg:...demoneeded-1.2c1...egg |
396 | - foo-paths: ...demo-0.2...eggFOO...demoneeded-1.2c1...egg |
397 | silly-range: [0, 1, 2, 3, 4] |
398 | - first-interpreted-option: duplicate-os-paths=(os.pathsep).join(paths) |
399 | + first-interpreted-option: silly-range = repr(range(5)) |
400 | message-reversed-is-egassem: egassem |
401 | |
402 | +Working with Paths |
403 | +================== |
404 | + |
405 | +We've already mentioned how to handle buildout's relative-paths option |
406 | +in the discussion of filters. This section has some concrete examples |
407 | +and discussion of that. It also introduces how to get a set of paths |
408 | +from specifying dependencies. |
409 | + |
410 | +Here are concrete examples of the path-repr and shell-path filters. |
411 | +We'll show results when relative-paths is true and when it is false. |
412 | + |
413 | +------------------------------ |
414 | +Demonstration of ``path-repr`` |
415 | +------------------------------ |
416 | + |
417 | +Let's say we want to make a custom Python script in the bin directory. |
418 | +It will print some information from a file in a ``data`` directory |
419 | +within the buildout root. Here's the template. |
420 | + |
421 | + >>> write(sample_buildout, 'template', 'bin', 'dosomething.py.in', '''\ |
422 | + ... #!${buildout:executable} |
423 | + ... ${python-relative-path-setup} |
424 | + ... f = open(${buildout:directory/data/info.csv|path-repr}) |
425 | + ... print f.read() |
426 | + ... ''') |
427 | + >>> os.chmod( |
428 | + ... os.path.join( |
429 | + ... sample_buildout, 'template', 'bin', 'dosomething.py.in'), |
430 | + ... 0711) |
431 | + |
432 | +If we evaluate that template with relative-paths set to false, the results |
433 | +shouldn't be too surprising. |
434 | + |
435 | + >>> write(sample_buildout, 'buildout.cfg', |
436 | + ... """ |
437 | + ... [buildout] |
438 | + ... parts = message |
439 | + ... |
440 | + ... [message] |
441 | + ... recipe = z3c.recipe.filetemplate |
442 | + ... source-directory = template |
443 | + ... """) |
444 | + |
445 | + >>> print system(buildout) |
446 | + Uninstalling message. |
447 | + Installing message. |
448 | + |
449 | + >>> cat(sample_buildout, 'bin', 'dosomething.py') # doctest: +ELLIPSIS |
450 | + #!... |
451 | + <BLANKLINE> |
452 | + f = open('/.../sample-buildout/data/info.csv') |
453 | + print f.read() |
454 | + |
455 | +``${python-relative-path-setup}`` evaluated to an empty string. The path |
456 | +is absolute and quoted. |
457 | + |
458 | +If we evaluate it with relative-paths set to true, the results are much... |
459 | +bigger. |
460 | + |
461 | + >>> write(sample_buildout, 'buildout.cfg', |
462 | + ... """ |
463 | + ... [buildout] |
464 | + ... parts = message |
465 | + ... relative-paths = true |
466 | + ... |
467 | + ... [message] |
468 | + ... recipe = z3c.recipe.filetemplate |
469 | + ... source-directory = template |
470 | + ... """) |
471 | + |
472 | + >>> print system(buildout) |
473 | + Uninstalling message. |
474 | + Installing message. |
475 | + |
476 | + >>> cat(sample_buildout, 'bin', 'dosomething.py') # doctest: +ELLIPSIS |
477 | + #!... |
478 | + import os, imp |
479 | + # Get path to this file. |
480 | + if __name__ == '__main__': |
481 | + _z3c_recipe_filetemplate_filename = __file__ |
482 | + else: |
483 | + # If this is an imported module, we want the location of the .py |
484 | + # file, not the .pyc, because the .py file may have been symlinked. |
485 | + _z3c_recipe_filetemplate_filename = imp.find_module(__name__)[1] |
486 | + # Get the full, non-symbolic-link directory for this file. |
487 | + _z3c_recipe_filetemplate_base = os.path.dirname( |
488 | + os.path.abspath(os.path.realpath(_z3c_recipe_filetemplate_filename))) |
489 | + # Ascend to buildout root. |
490 | + _z3c_recipe_filetemplate_base = os.path.dirname( |
491 | + _z3c_recipe_filetemplate_base) |
492 | + def _z3c_recipe_filetemplate_path_repr(path): |
493 | + "Return absolute version of buildout-relative path." |
494 | + return os.path.join(_z3c_recipe_filetemplate_base, path) |
495 | + <BLANKLINE> |
496 | + f = open(_z3c_recipe_filetemplate_path_repr('data/info.csv')) |
497 | + print f.read() |
498 | + |
499 | +That's quite a bit of code. You might wonder why we don't just use '..' for |
500 | +parent directories. The reason is that we want our scripts to be usable |
501 | +from any place on the filesystem. If we used '..' to construct paths |
502 | +relative to the generated file, then the paths would only work from |
503 | +certain directories. |
504 | + |
505 | +So that's how path-repr works. It can really come in handy if you want |
506 | +to support relative paths in buildout. Now let's look at the shell-path |
507 | +filter. |
508 | + |
509 | +------------------------------- |
510 | +Demonstration of ``shell-path`` |
511 | +------------------------------- |
512 | + |
513 | +Maybe you want to write some shell scripts. The shell-path filter will help |
514 | +you support buildout relative-paths fairly painlessly. |
515 | + |
516 | +Right now, only POSIX is supported with the shell-path filter, as mentioned |
517 | +before. |
518 | + |
519 | +Usage is very similar to the ``path-repr`` filter. You need to include |
520 | +``${shell-relative-path-setup}`` before you use it, just as you include |
521 | +``${python-relative-path-setup}`` before using ``path-repr``. |
522 | + |
523 | +Let's say we want to make a custom shell script in the bin directory. |
524 | +It will print some information from a file in a ``data`` directory |
525 | +within the buildout root. Here's the template. |
526 | + |
527 | + >>> write(sample_buildout, 'template', 'bin', 'dosomething.sh.in', '''\ |
528 | + ... #!/bin/sh |
529 | + ... ${shell-relative-path-setup} |
530 | + ... cat ${buildout:directory/data/info.csv|shell-path} |
531 | + ... ''') |
532 | + >>> os.chmod( |
533 | + ... os.path.join( |
534 | + ... sample_buildout, 'template', 'bin', 'dosomething.sh.in'), |
535 | + ... 0711) |
536 | + |
537 | +If relative-paths is set to false (the default), the results are simple. |
538 | + |
539 | + >>> write(sample_buildout, 'buildout.cfg', |
540 | + ... """ |
541 | + ... [buildout] |
542 | + ... parts = message |
543 | + ... |
544 | + ... [message] |
545 | + ... recipe = z3c.recipe.filetemplate |
546 | + ... source-directory = template |
547 | + ... """) |
548 | + |
549 | + >>> print system(buildout) |
550 | + Uninstalling message. |
551 | + Installing message. |
552 | + |
553 | + >>> cat(sample_buildout, 'bin', 'dosomething.sh') # doctest: +ELLIPSIS |
554 | + #!/bin/sh |
555 | + <BLANKLINE> |
556 | + cat /.../sample-buildout/data/info.csv |
557 | + |
558 | +``${shell-relative-path-setup}`` evaluated to an empty string. The path |
559 | +is absolute. |
560 | + |
561 | +Now let's look at the larger code when relative-paths is set to true. |
562 | + |
563 | + >>> write(sample_buildout, 'buildout.cfg', |
564 | + ... """ |
565 | + ... [buildout] |
566 | + ... parts = message |
567 | + ... relative-paths = true |
568 | + ... |
569 | + ... [message] |
570 | + ... recipe = z3c.recipe.filetemplate |
571 | + ... source-directory = template |
572 | + ... """) |
573 | + |
574 | + >>> print system(buildout) |
575 | + Uninstalling message. |
576 | + Installing message. |
577 | + |
578 | + >>> cat(sample_buildout, 'bin', 'dosomething.sh') # doctest: +ELLIPSIS |
579 | + #!/bin/sh |
580 | + # Get full, non-symbolic-link path to this file. |
581 | + Z3C_RECIPE_FILETEMPLATE_FILENAME=`\ |
582 | + readlink -f "$0" 2>/dev/null || \ |
583 | + realpath "$0" 2>/dev/null || \ |
584 | + type -P "$0" 2>/dev/null` |
585 | + # Get directory of file. |
586 | + Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_FILENAME}` |
587 | + # Ascend to buildout root. |
588 | + Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_BASE}` |
589 | + <BLANKLINE> |
590 | + cat "$Z3C_RECIPE_FILETEMPLATE_BASE"/data/info.csv |
591 | + |
592 | +As with the Python code, we don't just use '..' for |
593 | +parent directories because we want our scripts to be usable |
594 | +from any place on the filesystem. |
595 | + |
596 | +---------------------------------- |
597 | +Getting Arbitrary Dependency Paths |
598 | +---------------------------------- |
599 | + |
600 | +You can specify ``eggs`` and ``extra-paths`` in the recipe. The |
601 | +mechanism is the same as the one provided by the zc.recipe.egg, so |
602 | +pertinent options such as find-links and index are available. |
603 | + |
604 | +If you do, the paths for the dependencies will be calculated. They will |
605 | +be available as a list in the namespace of the interpreted options as |
606 | +``paths``. Also, three predefined options will be available in the |
607 | +recipe's options for the template. |
608 | + |
609 | +If ``paths`` are the paths, ``shell_path`` is the ``shell-path`` filter, and |
610 | +``path_repr`` is the ``path-repr`` filter, then the pre-defined options |
611 | +would be defined roughly as given here: |
612 | + |
613 | +``os-paths`` (for shell scripts) |
614 | + ``(os.pathsep).join(shell_path(path) for path in paths)`` |
615 | + |
616 | +``string-paths`` (for Python scripts) |
617 | + ``',\n '.join(path_repr(path) for path in paths)`` |
618 | + |
619 | +``space-paths`` (for shell scripts) |
620 | + ``' '.join(shell_path(path) for path in paths)`` |
621 | + |
622 | +Therefore, if you want to support the relative-paths option, you should |
623 | +include ``${shell-relative-path-setup}`` (for ``os-paths`` and |
624 | +``space-paths``) or ``${python-relative-path-setup}`` (for ``string-paths``) |
625 | +as appropriate at the top of your template. |
626 | + |
627 | +Let's consider a simple example. |
628 | + |
629 | + >>> write(sample_buildout, 'buildout.cfg', |
630 | + ... """ |
631 | + ... [buildout] |
632 | + ... parts = message |
633 | + ... |
634 | + ... [message] |
635 | + ... recipe = z3c.recipe.filetemplate |
636 | + ... files = helloworld.txt |
637 | + ... eggs = demo<0.3 |
638 | + ... |
639 | + ... find-links = %(server)s |
640 | + ... index = %(server)s/index |
641 | + ... """ % dict(server=link_server)) |
642 | + |
643 | +The relative-paths option is false, the default. |
644 | + |
645 | + >>> write(sample_buildout, 'helloworld.txt.in', |
646 | + ... """ |
647 | + ... Hello! Here are the paths for the ${eggs} eggs. |
648 | + ... OS paths: |
649 | + ... ${os-paths} |
650 | + ... --- |
651 | + ... String paths: |
652 | + ... ${string-paths} |
653 | + ... --- |
654 | + ... Space paths: |
655 | + ... ${space-paths} |
656 | + ... """) |
657 | + |
658 | + >>> print system(buildout) |
659 | + Getting distribution for 'demo<0.3'. |
660 | + Got demo 0.2. |
661 | + Getting distribution for 'demoneeded'. |
662 | + Got demoneeded 1.2c1. |
663 | + Uninstalling message. |
664 | + Installing message. |
665 | + |
666 | + >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS |
667 | + Hello! Here are the paths for the demo<0.3 eggs. |
668 | + OS paths: |
669 | + /.../eggs/demo-0.2...egg:/.../eggs/demoneeded-1.2c1...egg |
670 | + --- |
671 | + String paths: |
672 | + '/.../eggs/demo-0.2...egg', |
673 | + '/.../eggs/demoneeded-1.2c1...egg' |
674 | + --- |
675 | + Space paths: |
676 | + /.../eggs/demo-0.2...egg /.../eggs/demoneeded-1.2c1...egg |
677 | + |
678 | +You can specify extra-paths as well, which will go at the end of the egg |
679 | +paths. |
680 | + |
681 | + >>> write(sample_buildout, 'buildout.cfg', |
682 | + ... """ |
683 | + ... [buildout] |
684 | + ... parts = message |
685 | + ... |
686 | + ... [message] |
687 | + ... recipe = z3c.recipe.filetemplate |
688 | + ... files = helloworld.txt |
689 | + ... eggs = demo<0.3 |
690 | + ... extra-paths = ${buildout:directory}/foo |
691 | + ... |
692 | + ... find-links = %(server)s |
693 | + ... index = %(server)s/index |
694 | + ... """ % dict(server=link_server)) |
695 | + |
696 | + >>> print system(buildout) |
697 | + Uninstalling message. |
698 | + Installing message. |
699 | + |
700 | + >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS |
701 | + Hello! Here are the paths for the demo<0.3 eggs. |
702 | + OS paths: |
703 | + /...demo...:/...demoneeded...:/.../sample-buildout/foo |
704 | + --- |
705 | + String paths: |
706 | + '/...demo...', |
707 | + '/...demoneeded...', |
708 | + '/.../sample-buildout/foo' |
709 | + --- |
710 | + Space paths: |
711 | + /...demo... /...demoneeded... .../sample-buildout/foo |
712 | + |
713 | +To emphasize the effect of the relative-paths option, let's see what it looks |
714 | +like when we set relative-paths to True. |
715 | + |
716 | + >>> write(sample_buildout, 'buildout.cfg', |
717 | + ... """ |
718 | + ... [buildout] |
719 | + ... parts = message |
720 | + ... relative-paths = true |
721 | + ... |
722 | + ... [message] |
723 | + ... recipe = z3c.recipe.filetemplate |
724 | + ... files = helloworld.txt |
725 | + ... eggs = demo<0.3 |
726 | + ... extra-paths = ${buildout:directory}/foo |
727 | + ... |
728 | + ... find-links = %(server)s |
729 | + ... index = %(server)s/index |
730 | + ... """ % dict(server=link_server)) |
731 | + |
732 | + >>> print system(buildout) |
733 | + Uninstalling message. |
734 | + Installing message. |
735 | + |
736 | + >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS |
737 | + Hello! Here are the paths for the demo<0.3 eggs. |
738 | + OS paths: |
739 | + "$Z3C_RECIPE_FILETEMPLATE_BASE"/eggs/demo-0.2-py...egg:"$Z3C_RECIPE_FILETEMPLATE_BASE"/eggs/demoneeded-1.2c1-py...egg:"$Z3C_RECIPE_FILETEMPLATE_BASE"/foo |
740 | + --- |
741 | + String paths: |
742 | + _z3c_recipe_filetemplate_path_repr('eggs/demo-0.2-py...egg'), |
743 | + _z3c_recipe_filetemplate_path_repr('eggs/demoneeded-1.2c1-py...egg'), |
744 | + _z3c_recipe_filetemplate_path_repr('foo') |
745 | + --- |
746 | + Space paths: |
747 | + "$Z3C_RECIPE_FILETEMPLATE_BASE"/eggs/demo-0.2-py...egg "$Z3C_RECIPE_FILETEMPLATE_BASE"/eggs/demoneeded-1.2c1-py...egg "$Z3C_RECIPE_FILETEMPLATE_BASE"/foo |
748 | + |
749 | + |
750 | +Remember, your script won't really work unless you include |
751 | +``${shell-relative-path-setup}`` (for ``os-paths`` and ``space-paths``) |
752 | +or ``${python-relative-path-setup}`` (for ``string-paths``) as |
753 | +appropriate at the top of your template. |
754 | + |
755 | +Getting Dependency Paths from ``zc.recipe.egg`` |
756 | +----------------------------------------------- |
757 | + |
758 | +You can get the ``eggs`` and ``extra-paths`` from another section using |
759 | +zc.recipe.egg by using the ``extends`` option from the `Sharing Variables`_ |
760 | +section above. Then you can use the template options described above to |
761 | +build your paths in your templates. |
762 | + |
763 | +Getting Dependency Paths from ``z3c.recipe.scripts`` |
764 | +---------------------------------------------------- |
765 | + |
766 | +If, like the Launchpad project, you are using Gary Poster's unreleased |
767 | +package ``z3c.recipe.scripts`` to generate your scripts, and you want to |
768 | +have your scripts use the same Python environment as generated by that |
769 | +recipe, you can just use the path-repr and shell-path filters with standard |
770 | +buildout directories. Here is an example buildout.cfg. |
771 | + |
772 | +:: |
773 | + |
774 | + [buildout] |
775 | + parts = scripts message |
776 | + relative-paths = true |
777 | + |
778 | + [scripts] |
779 | + recipe = z3c.recipe.scripts |
780 | + eggs = demo<0.3 |
781 | + |
782 | + [message] |
783 | + recipe = z3c.recipe.filetemplate |
784 | + files = helloworld.py |
785 | + |
786 | +Then the template to use this would want to simply put |
787 | +``${scripts:parts-directory|path-repr}`` at the beginning of Python's path. |
788 | + |
789 | +You can do this for subprocesses with PYTHONPATH. |
790 | + |
791 | + ${python-relative-path-setup} |
792 | + import os |
793 | + import subprocess |
794 | + env = os.environ.copy() |
795 | + env['PYTHONPATH'] = ${scripts:parts-directory|path-repr} |
796 | + subprocess.call('myscript', env=env) |
797 | + |
798 | +That's it. |
799 | + |
800 | +Similarly, here's an approach to making a script that will have the |
801 | +right environment. You want to put the parts directory of the |
802 | +z3c.recipe.scripts section in the sys.path before site.py is loaded. |
803 | +This is usually handled by z3c.recipe.scripts itself, but sometimes you |
804 | +may want to write Python scripts in your template for some reason. |
805 | + |
806 | + #!/usr/bin/env python -S |
807 | + ${python-relative-path-setup} |
808 | + import sys |
809 | + sys.path.insert(0, ${scripts:parts-directory|path-repr}) |
810 | + import site |
811 | + # do stuff... |
812 | + |
813 | +If you do this for many scripts, put this entire snippet in an option in the |
814 | +recipe and use this snippet as a single substitution in the top of your |
815 | +scripts. |
816 | |
817 | === modified file 'z3c/recipe/filetemplate/__init__.py' |
818 | --- z3c/recipe/filetemplate/__init__.py 2010-04-20 19:34:27 +0000 |
819 | +++ z3c/recipe/filetemplate/__init__.py 2010-04-20 19:34:27 +0000 |
820 | @@ -29,10 +29,15 @@ |
821 | |
822 | class FileTemplate(object): |
823 | |
824 | + filters = {} |
825 | + dynamic_options = {} |
826 | + |
827 | def __init__(self, buildout, name, options): |
828 | self.buildout = buildout |
829 | self.name = name |
830 | self.options = options |
831 | + self.buildout_root = zc.buildout.easy_install.realpath( |
832 | + buildout['buildout']['directory']) |
833 | self.logger=logging.getLogger(self.name) |
834 | # get defaults from extended sections |
835 | defaults = {} |
836 | @@ -42,34 +47,33 @@ |
837 | defaults.update(self.buildout[section_name]) |
838 | for key, value in defaults.items(): |
839 | self.options.setdefault(key, value) |
840 | + relative_paths = self.options.setdefault( |
841 | + 'relative-paths', |
842 | + buildout['buildout'].get('relative-paths', 'false') |
843 | + ) |
844 | + if relative_paths not in ('true', 'false'): |
845 | + self._user_error( |
846 | + 'The relative-paths option must have the value of ' |
847 | + 'true or false.') |
848 | + self.relative_paths = relative_paths = (relative_paths == 'true') |
849 | + self.paths = paths = [] |
850 | # set up paths for eggs, if given |
851 | - if 'eggs' in self.options: |
852 | - relative_paths = self.options.get( |
853 | - 'relative-paths', |
854 | - buildout['buildout'].get('relative-paths', 'false') |
855 | - ) |
856 | - if relative_paths not in ('true', 'false'): |
857 | - self._user_error( |
858 | - 'The relative-paths option must have the value of ' |
859 | - 'true or false.') |
860 | - relative_paths = relative_paths == 'true' |
861 | - if relative_paths: |
862 | - raise NotImplementedError # XXX |
863 | - self.eggs = zc.recipe.egg.Scripts(buildout, name, options) |
864 | - orig_distributions, ws = self.eggs.working_set() |
865 | - # we want ws, eggs.extra_paths, eggs._relative_paths |
866 | - all_paths = [ |
867 | + if 'eggs' in options: |
868 | + eggs = zc.recipe.egg.Scripts(buildout, name, options) |
869 | + orig_distributions, ws = eggs.working_set() |
870 | + paths.extend( |
871 | zc.buildout.easy_install.realpath(dist.location) |
872 | - for dist in ws] |
873 | - all_paths.extend( |
874 | + for dist in ws) |
875 | + paths.extend( |
876 | zc.buildout.easy_install.realpath(path) |
877 | - for path in self.eggs.extra_paths) |
878 | + for path in eggs.extra_paths) |
879 | else: |
880 | - all_paths = [] |
881 | - paths = [path for path in all_paths if not path.endswith('.zip')] |
882 | - self.options['os-paths'] = (os.pathsep).join(paths) |
883 | - self.options['string-paths'] = ', '.join(repr(p) for p in all_paths) |
884 | - self.options['space-paths'] = ' '.join(paths) |
885 | + paths.extend( |
886 | + os.path.join(buildout.options['directory'], p.strip()) |
887 | + for p in options.get('extra-paths', '').split('\n') |
888 | + if p.strip() |
889 | + ) |
890 | + options['_paths'] = '\n'.join(paths) |
891 | # get and check the files to be created |
892 | self.filenames = self.options.get('files', '*').split() |
893 | self.source_dir = self.options.get('source-directory', '').strip() |
894 | @@ -164,7 +168,7 @@ |
895 | if interpreted: |
896 | globs = {'__builtins__': __builtins__, 'os': os, 'sys': sys} |
897 | locs = {'name': name, 'options': options, 'buildout': buildout, |
898 | - 'paths': paths, 'all_paths': all_paths} |
899 | + 'paths': paths, 'all_paths': paths} |
900 | for value in interpreted.split('\n'): |
901 | if value: |
902 | value = value.split('=', 1) |
903 | @@ -207,18 +211,20 @@ |
904 | 'Destinations already exist: %s. Please make sure that ' |
905 | 'you really want to generate these automatically. Then ' |
906 | 'move them away.', ', '.join(already_exists)) |
907 | - seen = [] # We throw this away right now, but could move template |
908 | - # processing up to __init__ if valuable. That would mean that templates |
909 | - # would be rewritten even if a value in another section had been |
910 | - # referenced; however, it would also mean that __init__ would do |
911 | - # virtually all of the work, with install only doing the writing. |
912 | + self.seen = [] |
913 | + # We throw ``seen`` away right now, but could move template |
914 | + # processing up to __init__ if valuable. That would mean that |
915 | + # templates would be rewritten even if a value in another |
916 | + # section had been referenced; however, it would also mean that |
917 | + # __init__ would do virtually all of the work, with install only |
918 | + # doing the writing. |
919 | for rel_path, last_mod, st_mode in self.actions: |
920 | source = os.path.join(self.source_dir, rel_path) |
921 | dest = os.path.join(self.destination_dir, rel_path[:-3]) |
922 | mode=stat.S_IMODE(st_mode) |
923 | # we process the file first so that it won't be created if there |
924 | # is a problem. |
925 | - processed = Template(source).substitute(self, seen) |
926 | + processed = Template(source, dest, self).substitute() |
927 | self._create_paths(os.path.dirname(dest)) |
928 | result=open(dest, "wt") |
929 | result.write(processed) |
930 | @@ -233,71 +239,119 @@ |
931 | os.mkdir(path) |
932 | self.options.created(path) |
933 | |
934 | + def _call_and_log(self, callable, args, message_generator): |
935 | + try: |
936 | + return callable(*args) |
937 | + except (KeyboardInterrupt, SystemExit): |
938 | + raise |
939 | + except: |
940 | + # Argh. Would like to raise wrapped exception. |
941 | + colno, lineno = self.get_colno_lineno(start) |
942 | + msg = message_generator(lineno, colno) |
943 | + self.logger.error(msg, exc_info=True) |
944 | + raise |
945 | + |
946 | def update(self): |
947 | pass |
948 | |
949 | |
950 | class Template: |
951 | - # hacked from string.Template |
952 | + # Heavily hacked from--"inspired by"?--string.Template |
953 | pattern = re.compile(r""" |
954 | \$(?: |
955 | - \${(?P<escaped>[^}]*)} | # Escape sequence of two delimiters. |
956 | - {(?P<braced_single>[-a-z0-9 ._]+)} | |
957 | - # Delimiter and a braced local option |
958 | - {(?P<braced_double>[-a-z0-9 ._]+:[-a-z0-9 ._]+)} | |
959 | - # Delimiter and a braced fully |
960 | - # qualified option (that is, with |
961 | - # explicit section). |
962 | + \${(?P<escaped>[^}]*)} | # Escape sequence of two delimiters. |
963 | + |
964 | + {((?P<section>[-a-z0-9 ._]+):)? # Optional section name. |
965 | + (?P<option>[-a-z0-9 ._]+) # Required option name. |
966 | + (?P<path_extension>/[^|}]+/?)? # Optional path extensions. |
967 | + ([ ]*(?P<filters>(\|[ ]*[-a-z0-9._]+[ ]*)+))? |
968 | + # Optional filters. |
969 | + } | |
970 | + |
971 | {(?P<invalid>[^}]*}) # Other ill-formed delimiter exprs. |
972 | ) |
973 | """, re.IGNORECASE | re.VERBOSE) |
974 | |
975 | - def __init__(self, source): |
976 | + def __init__(self, source, destination, recipe): |
977 | self.source = source |
978 | + self.destination = zc.buildout.easy_install.realpath(destination) |
979 | + self.recipe = recipe |
980 | self.template = open(source).read() |
981 | |
982 | - def _get_colno_lineno(self, i): |
983 | + def get_colno_lineno(self, i): |
984 | lines = self.template[:i].splitlines(True) |
985 | if not lines: |
986 | colno = 1 |
987 | lineno = 1 |
988 | else: |
989 | - colno = i - len(''.join(lines[:-1])) |
990 | + colno = len(lines[-1]) + 1 |
991 | lineno = len(lines) |
992 | return colno, lineno |
993 | |
994 | - def _get(self, options, section, option, seen, start): |
995 | - value = options.get(option, None, seen) |
996 | + def _get(self, section, option, start): |
997 | + if section is None: |
998 | + section = self.recipe.name # This sets up error messages properly. |
999 | + if section == self.recipe.name: |
1000 | + factory = self.recipe.dynamic_options.get(option) |
1001 | + if factory is not None: |
1002 | + return self.recipe._call_and_log( |
1003 | + factory, (self, start, option), |
1004 | + lambda lineno, colno: ( |
1005 | + 'Dynamic option %r in line %d, col %d of %s ' |
1006 | + 'crashed.') % (option, lineno, colno, self.source)) |
1007 | + # else... |
1008 | + options = self.recipe.options |
1009 | + elif section in self.recipe.buildout: |
1010 | + options = self.recipe.buildout[section] |
1011 | + else: |
1012 | + value = options = None |
1013 | + if options is not None: |
1014 | + value = options.get(option, None, self.recipe.seen) |
1015 | if value is None: |
1016 | - colno, lineno = self._get_colno_lineno(start) |
1017 | + colno, lineno = self.get_colno_lineno(start) |
1018 | raise zc.buildout.buildout.MissingOption( |
1019 | "Option '%s:%s', referenced in line %d, col %d of %s, " |
1020 | "does not exist." % |
1021 | (section, option, lineno, colno, self.source)) |
1022 | return value |
1023 | |
1024 | - def substitute(self, recipe, seen): |
1025 | + def substitute(self): |
1026 | def convert(mo): |
1027 | + start = mo.start() |
1028 | # Check the most common path first. |
1029 | - option = mo.group('braced_single') |
1030 | + option = mo.group('option') |
1031 | if option is not None: |
1032 | - val = self._get(recipe.options, recipe.name, option, seen, |
1033 | - mo.start('braced_single')) |
1034 | + section = mo.group('section') |
1035 | + val = self._get(section, option, start) |
1036 | + path_extension = mo.group('path_extension') |
1037 | + filters = mo.group('filters') |
1038 | + if path_extension is not None: |
1039 | + val = os.path.join(val, *path_extension.split('/')[1:]) |
1040 | + if filters is not None: |
1041 | + for filter_name in filters.split('|')[1:]: |
1042 | + filter_name = filter_name.strip() |
1043 | + filter = self.recipe.filters.get(filter_name) |
1044 | + if filter is None: |
1045 | + colno, lineno = self.get_colno_lineno(start) |
1046 | + raise ValueError( |
1047 | + 'Unknown filter %r ' |
1048 | + 'in line %d, col %d of %s' % |
1049 | + (filter_name, lineno, colno, self.source)) |
1050 | + val = self.recipe._call_and_log( |
1051 | + filter, (val, self, start, filter_name), |
1052 | + lambda lineno, colno: ( |
1053 | + 'Filter %r in line %d, col %d of %s ' |
1054 | + 'crashed processing value %r') % ( |
1055 | + filter_name, lineno, colno, self.source, val)) |
1056 | # We use this idiom instead of str() because the latter will |
1057 | # fail if val is a Unicode containing non-ASCII characters. |
1058 | return '%s' % (val,) |
1059 | - double = mo.group('braced_double') |
1060 | - if double is not None: |
1061 | - section, option = double.split(':') |
1062 | - val = self._get(recipe.buildout[section], section, option, seen, |
1063 | - mo.start('braced_double')) |
1064 | - return '%s' % (val,) |
1065 | escaped = mo.group('escaped') |
1066 | if escaped is not None: |
1067 | return '${%s}' % (escaped,) |
1068 | invalid = mo.group('invalid') |
1069 | if invalid is not None: |
1070 | - colno, lineno = self._get_colno_lineno(mo.start('invalid')) |
1071 | + colno, lineno = self.get_colno_lineno(mo.start('invalid')) |
1072 | raise ValueError( |
1073 | 'Invalid placeholder %r in line %d, col %d of %s' % |
1074 | (mo.group('invalid'), lineno, colno, self.source)) |
1075 | @@ -305,3 +359,193 @@ |
1076 | self.pattern) # programmer error, AFAICT |
1077 | return self.pattern.sub(convert, self.template) |
1078 | |
1079 | + |
1080 | +############################################################################ |
1081 | +# Filters |
1082 | +def filter(func): |
1083 | + "Helper function to register filter functions." |
1084 | + FileTemplate.filters[func.__name__.replace('_', '-')] = func |
1085 | + return func |
1086 | + |
1087 | +@filter |
1088 | +def capitalize(val, template, start, filter): |
1089 | + return val.capitalize() |
1090 | + |
1091 | +@filter |
1092 | +def title(val, template, start, filter): |
1093 | + return val.title() |
1094 | + |
1095 | +@filter |
1096 | +def upper(val, template, start, filter): |
1097 | + return val.upper() |
1098 | + |
1099 | +@filter |
1100 | +def lower(val, template, start, filter): |
1101 | + return val.lower() |
1102 | + |
1103 | +@filter |
1104 | +def path_repr(val, template, start, filter): |
1105 | + # val is a path. |
1106 | + return _maybe_relativize( |
1107 | + val, template, |
1108 | + lambda p: "_z3c_recipe_filetemplate_path_repr(%r)" % (p,), |
1109 | + repr) |
1110 | + |
1111 | +@filter |
1112 | +def shell_path(val, template, start, filter): |
1113 | + # val is a path. |
1114 | + return _maybe_relativize( |
1115 | + val, template, |
1116 | + lambda p: '"$Z3C_RECIPE_FILETEMPLATE_BASE"/%s' % (p,), |
1117 | + lambda p: p) |
1118 | + |
1119 | +# Helpers hacked from zc.buildout.easy_install. |
1120 | +def _maybe_relativize(path, template, relativize, absolutize): |
1121 | + path = zc.buildout.easy_install.realpath(path) |
1122 | + if template.recipe.relative_paths: |
1123 | + buildout_root = template.recipe.buildout_root |
1124 | + if path == buildout_root: |
1125 | + return relativize(os.curdir) |
1126 | + destination = template.destination |
1127 | + common = os.path.dirname(os.path.commonprefix([path, destination])) |
1128 | + if (common == buildout_root or |
1129 | + common.startswith(os.path.join(buildout_root, '')) |
1130 | + ): |
1131 | + return relativize(_relative_path(common, path)) |
1132 | + return absolutize(path) |
1133 | + |
1134 | +def _relative_path(common, path): |
1135 | + """Return the relative path from ``common`` to ``path``. |
1136 | + |
1137 | + This is a helper for _relativitize, which is a helper to |
1138 | + _relative_path_and_setup. |
1139 | + """ |
1140 | + r = [] |
1141 | + while 1: |
1142 | + dirname, basename = os.path.split(path) |
1143 | + r.append(basename) |
1144 | + if dirname == common: |
1145 | + break |
1146 | + assert dirname != path, "dirname of %s is the same" % dirname |
1147 | + path = dirname |
1148 | + r.reverse() |
1149 | + return os.path.join(*r) |
1150 | + |
1151 | + |
1152 | +############################################################################ |
1153 | +# Dynamic options |
1154 | +def dynamic_option(func): |
1155 | + "Helper function to register dynamic options." |
1156 | + FileTemplate.dynamic_options[func.__name__.replace('_', '-')] = func |
1157 | + return func |
1158 | + |
1159 | +@dynamic_option |
1160 | +def os_paths(template, start, name): |
1161 | + return os.pathsep.join( |
1162 | + shell_path(path, template, start, 'os-paths') |
1163 | + for path in template.recipe.paths) |
1164 | + |
1165 | +@dynamic_option |
1166 | +def string_paths(template, start, name): |
1167 | + colno, lineno = template.get_colno_lineno(start) |
1168 | + separator = ',\n' + ((colno - 1) * ' ') |
1169 | + return separator.join( |
1170 | + path_repr(path, template, start, 'string-paths') |
1171 | + for path in template.recipe.paths) |
1172 | + |
1173 | +@dynamic_option |
1174 | +def space_paths(template, start, name): |
1175 | + return ' '.join( |
1176 | + shell_path(path, template, start, 'space-paths') |
1177 | + for path in template.recipe.paths) |
1178 | + |
1179 | +@dynamic_option |
1180 | +def shell_relative_path_setup(template, start, name): |
1181 | + if template.recipe.relative_paths: |
1182 | + depth = _relative_depth( |
1183 | + template.recipe.buildout['buildout']['directory'], |
1184 | + template.destination) |
1185 | + value = SHELL_RELATIVE_PATH_SETUP |
1186 | + if depth: |
1187 | + value += '# Ascend to buildout root.\n' |
1188 | + value += depth * SHELL_DIRNAME |
1189 | + else: |
1190 | + value += '# This is the buildout root.\n' |
1191 | + return value |
1192 | + else: |
1193 | + return '' |
1194 | + |
1195 | +SHELL_RELATIVE_PATH_SETUP = '''\ |
1196 | +# Get full, non-symbolic-link path to this file. |
1197 | +Z3C_RECIPE_FILETEMPLATE_FILENAME=`\\ |
1198 | + readlink -f "$0" 2>/dev/null || \\ |
1199 | + realpath "$0" 2>/dev/null || \\ |
1200 | + type -P "$0" 2>/dev/null` |
1201 | +# Get directory of file. |
1202 | +Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_FILENAME}` |
1203 | +''' |
1204 | + |
1205 | +SHELL_DIRNAME = '''\ |
1206 | +Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_BASE}` |
1207 | +''' |
1208 | + |
1209 | +@dynamic_option |
1210 | +def python_relative_path_setup(template, start, name): |
1211 | + if template.recipe.relative_paths: |
1212 | + depth = _relative_depth( |
1213 | + template.recipe.buildout['buildout']['directory'], |
1214 | + template.destination) |
1215 | + value = PYTHON_RELATIVE_PATH_SETUP_START |
1216 | + if depth: |
1217 | + value += '# Ascend to buildout root.\n' |
1218 | + value += depth * PYTHON_DIRNAME |
1219 | + else: |
1220 | + value += '# This is the buildout root.\n' |
1221 | + value += PYTHON_RELATIVE_PATH_SETUP_END |
1222 | + return value |
1223 | + else: |
1224 | + return '' |
1225 | + |
1226 | +PYTHON_RELATIVE_PATH_SETUP_START = '''\ |
1227 | +import os, imp |
1228 | +# Get path to this file. |
1229 | +if __name__ == '__main__': |
1230 | + _z3c_recipe_filetemplate_filename = __file__ |
1231 | +else: |
1232 | + # If this is an imported module, we want the location of the .py |
1233 | + # file, not the .pyc, because the .py file may have been symlinked. |
1234 | + _z3c_recipe_filetemplate_filename = imp.find_module(__name__)[1] |
1235 | +# Get the full, non-symbolic-link directory for this file. |
1236 | +_z3c_recipe_filetemplate_base = os.path.dirname( |
1237 | + os.path.abspath(os.path.realpath(_z3c_recipe_filetemplate_filename))) |
1238 | +''' |
1239 | + |
1240 | +PYTHON_DIRNAME = '''\ |
1241 | +_z3c_recipe_filetemplate_base = os.path.dirname( |
1242 | + _z3c_recipe_filetemplate_base) |
1243 | +''' |
1244 | + |
1245 | +PYTHON_RELATIVE_PATH_SETUP_END = '''\ |
1246 | +def _z3c_recipe_filetemplate_path_repr(path): |
1247 | + "Return absolute version of buildout-relative path." |
1248 | + return os.path.join(_z3c_recipe_filetemplate_base, path) |
1249 | +''' |
1250 | + |
1251 | +def _relative_depth(common, path): |
1252 | + # Helper ripped from zc.buildout.easy_install. |
1253 | + """Return number of dirs separating ``path`` from ancestor, ``common``. |
1254 | + |
1255 | + For instance, if path is /foo/bar/baz/bing, and common is /foo, this will |
1256 | + return 2--in UNIX, the number of ".." to get from bing's directory |
1257 | + to foo. |
1258 | + """ |
1259 | + n = 0 |
1260 | + while 1: |
1261 | + dirname = os.path.dirname(path) |
1262 | + if dirname == path: |
1263 | + raise AssertionError("dirname of %s is the same" % dirname) |
1264 | + if dirname == common: |
1265 | + break |
1266 | + n += 1 |
1267 | + path = dirname |
1268 | + return n |
1269 | |
1270 | === modified file 'z3c/recipe/filetemplate/tests.txt' |
1271 | --- z3c/recipe/filetemplate/tests.txt 2010-04-20 19:34:27 +0000 |
1272 | +++ z3c/recipe/filetemplate/tests.txt 2010-04-20 19:34:27 +0000 |
1273 | @@ -14,7 +14,7 @@ |
1274 | ... """ |
1275 | ... Hello ${world}! |
1276 | ... """) |
1277 | - |
1278 | + |
1279 | >>> write(sample_buildout, 'goodbyeworld.txt.in', |
1280 | ... """ |
1281 | ... Goodbye ${world}! |
1282 | @@ -56,7 +56,7 @@ |
1283 | ... files = /etc/passwd.in |
1284 | ... root = me |
1285 | ... """) |
1286 | - |
1287 | + |
1288 | >>> print system(buildout) |
1289 | evil: /etc/passwd.in is an absolute path. Paths must be relative to the buildout directory. |
1290 | While: |
1291 | @@ -80,7 +80,7 @@ |
1292 | ... recipe = z3c.recipe.filetemplate |
1293 | ... files = doesntexist |
1294 | ... """) |
1295 | - |
1296 | + |
1297 | >>> print system(buildout) |
1298 | notthere: No template found for these file names: doesntexist.in |
1299 | While: |
1300 | @@ -99,12 +99,12 @@ |
1301 | ... """ |
1302 | ... I'm already here |
1303 | ... """) |
1304 | - |
1305 | + |
1306 | >>> write(sample_buildout, 'alreadyhere.txt.in', |
1307 | ... """ |
1308 | ... I'm the template that's supposed to replace the file above. |
1309 | ... """) |
1310 | - |
1311 | + |
1312 | >>> write(sample_buildout, 'buildout.cfg', |
1313 | ... """ |
1314 | ... [buildout] |
1315 | @@ -137,7 +137,7 @@ |
1316 | ... """ |
1317 | ... Hello ${world}! |
1318 | ... """) |
1319 | - |
1320 | + |
1321 | >>> write(sample_buildout, 'buildout.cfg', |
1322 | ... """ |
1323 | ... [buildout] |
1324 | @@ -147,12 +147,12 @@ |
1325 | ... recipe = z3c.recipe.filetemplate |
1326 | ... files = missing.txt |
1327 | ... """) |
1328 | - |
1329 | + |
1330 | >>> print system(buildout) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE |
1331 | Installing missing. |
1332 | While: |
1333 | Installing missing. |
1334 | - Error: Option 'missing:world', referenced in line 2, col 8 of |
1335 | + Error: Option 'missing:world', referenced in line 2, col 7 of |
1336 | .../sample-buildout/missing.txt.in, does not exist. |
1337 | |
1338 | No changes means just an update |
1339 | @@ -366,3 +366,70 @@ |
1340 | d parts |
1341 | d template |
1342 | |
1343 | +Specifying files with relative paths at the buildout root |
1344 | +--------------------------------------------------------- |
1345 | + |
1346 | +Working at the buildout root follows some different code paths with relative |
1347 | +paths so we explore those here. We also evaluate paths at the directory root. |
1348 | + |
1349 | + >>> rmdir(sample_buildout, 'template') |
1350 | + >>> mkdir(sample_buildout, 'template') |
1351 | + >>> write(sample_buildout, 'template', 'dosomething.py.in', '''\ |
1352 | + ... #!${buildout:executable} |
1353 | + ... ${python-relative-path-setup} |
1354 | + ... root = ${buildout:directory|path-repr} |
1355 | + ... ''') |
1356 | + |
1357 | + >>> write(sample_buildout, 'template', 'dosomething.sh.in', '''\ |
1358 | + ... #!/bin/sh |
1359 | + ... ${shell-relative-path-setup} |
1360 | + ... cat ${buildout:directory|shell-path} |
1361 | + ... ''') |
1362 | + |
1363 | + >>> write(sample_buildout, 'buildout.cfg', |
1364 | + ... """ |
1365 | + ... [buildout] |
1366 | + ... parts = message |
1367 | + ... relative-paths = true |
1368 | + ... |
1369 | + ... [message] |
1370 | + ... recipe = z3c.recipe.filetemplate |
1371 | + ... source-directory = template |
1372 | + ... """) |
1373 | + |
1374 | + >>> print system(buildout) |
1375 | + Uninstalling message. |
1376 | + Installing message. |
1377 | + |
1378 | + >>> cat(sample_buildout, 'dosomething.py') # doctest: +ELLIPSIS |
1379 | + #!... |
1380 | + import os, imp |
1381 | + # Get path to this file. |
1382 | + if __name__ == '__main__': |
1383 | + _z3c_recipe_filetemplate_filename = __file__ |
1384 | + else: |
1385 | + # If this is an imported module, we want the location of the .py |
1386 | + # file, not the .pyc, because the .py file may have been symlinked. |
1387 | + _z3c_recipe_filetemplate_filename = imp.find_module(__name__)[1] |
1388 | + # Get the full, non-symbolic-link directory for this file. |
1389 | + _z3c_recipe_filetemplate_base = os.path.dirname( |
1390 | + os.path.abspath(os.path.realpath(_z3c_recipe_filetemplate_filename))) |
1391 | + # This is the buildout root. |
1392 | + def _z3c_recipe_filetemplate_path_repr(path): |
1393 | + "Return absolute version of buildout-relative path." |
1394 | + return os.path.join(_z3c_recipe_filetemplate_base, path) |
1395 | + <BLANKLINE> |
1396 | + root = _z3c_recipe_filetemplate_path_repr('.') |
1397 | + |
1398 | + >>> cat(sample_buildout, 'dosomething.sh') # doctest: +ELLIPSIS |
1399 | + #!/bin/sh |
1400 | + # Get full, non-symbolic-link path to this file. |
1401 | + Z3C_RECIPE_FILETEMPLATE_FILENAME=`\ |
1402 | + readlink -f "$0" 2>/dev/null || \ |
1403 | + realpath "$0" 2>/dev/null || \ |
1404 | + type -P "$0" 2>/dev/null` |
1405 | + # Get directory of file. |
1406 | + Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_FILENAME}` |
1407 | + # This is the buildout root. |
1408 | + <BLANKLINE> |
1409 | + cat "$Z3C_RECIPE_FILETEMPLATE_BASE"/. |
For reference, this is a patch to Launchpad that takes advantage of this new feature.
http:// pastebin. ubuntu. com/418823/
After this patch and lp~gary/ zc.buildout/ python- support- 9-relative- paths, absolute paths based on the build are only found in Launchpad in three locations:
- in scripts generated by z3c.recipe.tag (would be easy to fix)
- in scripts generated by z3c.recipe.i18n (would be easy to fix)
- in code generated by our Mailman integration (would be hard to fix, but after we move to Python 2.6 Barry Warsaw may be willing and able to help us move to the new Mailman code base).