Merge lp:~gary/z3c.recipe.filetemplate/relative-paths into lp:z3c.recipe.filetemplate

Proposed by Gary Poster
Status: Needs review
Proposed branch: lp:~gary/z3c.recipe.filetemplate/relative-paths
Merge into: lp:z3c.recipe.filetemplate
Prerequisite: lp:~gary/z3c.recipe.filetemplate/cleanup
Diff against target: 1409 lines (+954/-192)
7 files modified
.bzrignore (+7/-0)
CHANGES.txt (+37/-4)
MANIFEST.in (+3/-0)
setup.py (+1/-1)
z3c/recipe/filetemplate/README.txt (+531/-123)
z3c/recipe/filetemplate/__init__.py (+300/-56)
z3c/recipe/filetemplate/tests.txt (+75/-8)
To merge this branch: bzr merge lp:~gary/z3c.recipe.filetemplate/relative-paths
Reviewer Review Type Date Requested Status
Francis J. Lacoste (community) Approve
Review via email: mp+23701@code.launchpad.net

Description of the change

This branch adds support for the buildout relative-paths option to z3c.recipe.filetemplate. Pre-imp call was with flacoste.

The approach chosen adds two features to the recipe: ``path extensions`` and ``filters``. With these features, and some magic variables, I was able to provide a workable solution to the problem of relative paths.

As noted in the MP fields, this builds on the "cleanup" branch.

To post a comment you must log in.
Revision history for this message
Gary Poster (gary) wrote :

For reference, this is a patch to Launchpad that takes advantage of this new feature.

http://pastebin.ubuntu.com/418823/

After this patch and lp~gary/zc.buildout/python-support-9-relative-paths, absolute paths based on the build are only found in Launchpad in three locations:

- in scripts generated by z3c.recipe.tag (would be easy to fix)
- in scripts generated by z3c.recipe.i18n (would be easy to fix)
- in code generated by our Mailman integration (would be hard to fix, but after we move to Python 2.6 Barry Warsaw may be willing and able to help us move to the new Mailman code base).

Revision history for this message
Francis J. Lacoste (flacoste) wrote :
Download full text (4.2 KiB)

I have a couple of minor comments and questions. Should be good to go.

> === modified file 'z3c/recipe/filetemplate/README.txt'

> + >>> cat(sample_buildout, 'bin', 'dosomething.sh') # doctest: +ELLIPSIS
> + #!/bin/sh
> + Z3C_RECIPE_FILETEMPLATE_BASE=`\
> + readlink -f "$0" 2>/dev/null || \
> + realpath "$0" 2>/dev/null || \
> + type -P "$0" 2>/dev/null`
> + Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_BASE}`
> + Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_BASE}`
> + cat "$Z3C_RECIPE_FILETEMPLATE_BASE"/data/info.csv
> +

The multiple mutation of Z3C_RECIPE_FILETEMPLATE_BASE are confusing.
Especially, the two identical Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ...`
I assume that's the way to get to the parent directory of the script. Would be
clearer to use multiple variables, at the detriment of more namespace
polluting. At the very least a comment would help the reader. Your call.

> === modified file 'z3c/recipe/filetemplate/__init__.py'

> + def _get(self, section, option, start):
> + if section is None:
> + section = self.recipe.name # This sets up error messages properly.
> + if section == self.recipe.name:
> + factory = self.recipe.dynamic_options.get(option)
> + if factory is not None:
> + try:
> + return factory(self, start, option)
> + except:
> + # Argh. Would like to raise wrapped exception.
> + colno, lineno = self.get_colno_lineno(start)
> + msg = ('Dynamic option %r in line %d, col %d of %s '
> + 'crashed.') % (option, lineno, colno, self.source)
> + self.recipe.logger.error(msg, exc_info=True)
> + raise
> + # else...
> + options = self.recipe.options
>

Shouldn't you catch and re-raise SystemExit and KeyboardInterrupt? Or was this
fixed in Python 2.5?
> - def substitute(self, recipe, seen):
> + def substitute(self):
> def convert(mo):
> + start = mo.start()
> # Check the most common path first.
> - option = mo.group('braced_single')
> + option = mo.group('option')
> if option is not None:
> - val = self._get(recipe.options, recipe.name, option, seen,
> - mo.start('braced_single'))
> + section = mo.group('section')
> + val = self._get(section, option, start)
> + path_extension = mo.group('path_extension')
> + filters = mo.group('filters')
> + if path_extension is not None:
> + val = os.path.join(val, *path_extension.split('/')[1:])
> + if filters is not None:
> + for filter in filters.split('|')[1:]:
> + filter = filter.strip()
> + if filter not in self.recipe.filters:
> + colno, lineno = self.get_colno_lineno(start)
> + raise ValueError(
> + '...

Read more...

review: Approve
Revision history for this message
Francis J. Lacoste (flacoste) wrote :

On April 19, 2010, Gary Poster wrote:
> For reference, this is a patch to Launchpad that takes advantage of this
> new feature.
>
> http://pastebin.ubuntu.com/418823/
>
> After this patch and lp~gary/zc.buildout/python-support-9-relative-paths,
> absolute paths based on the build are only found in Launchpad in three
> locations:
>
> - in scripts generated by z3c.recipe.tag (would be easy to fix)

We don't care about that one in deployment.

> - in scripts generated by z3c.recipe.i18n (would be easy to fix)

Nor about this one.

> - in code generated by our Mailman integration (would be hard to fix, but
> after we move to Python 2.6 Barry Warsaw may be willing and able to help
> us move to the new Mailman code base).

That one means that we'll probably still need to run make clean on the mailman
machine at least?

--
Francis J. Lacoste
<email address hidden>

22. By Gary Poster

add comments to generated code; only log a problem in filters and dynamic options if they are not stopped because of SystemExit and KeyboardInterrupt.

Revision history for this message
Gary Poster (gary) wrote :
Download full text (4.6 KiB)

On Apr 19, 2010, at 4:28 PM, Francis J. Lacoste wrote:

> Review: Approve
> I have a couple of minor comments and questions. Should be good to go.
>
>> === modified file 'z3c/recipe/filetemplate/README.txt'
>
>> + >>> cat(sample_buildout, 'bin', 'dosomething.sh') # doctest: +ELLIPSIS
>> + #!/bin/sh
>> + Z3C_RECIPE_FILETEMPLATE_BASE=`\
>> + readlink -f "$0" 2>/dev/null || \
>> + realpath "$0" 2>/dev/null || \
>> + type -P "$0" 2>/dev/null`
>> + Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_BASE}`
>> + Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_BASE}`
>> + cat "$Z3C_RECIPE_FILETEMPLATE_BASE"/data/info.csv
>> +
>
> The multiple mutation of Z3C_RECIPE_FILETEMPLATE_BASE are confusing.
> Especially, the two identical Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ...`
> I assume that's the way to get to the parent directory of the script. Would be
> clearer to use multiple variables, at the detriment of more namespace
> polluting. At the very least a comment would help the reader. Your call.

Done, with variable name changes and comments, for both shell and Python versions.

>
>> === modified file 'z3c/recipe/filetemplate/__init__.py'
>
>> + def _get(self, section, option, start):
>> + if section is None:
>> + section = self.recipe.name # This sets up error messages properly.
>> + if section == self.recipe.name:
>> + factory = self.recipe.dynamic_options.get(option)
>> + if factory is not None:
>> + try:
>> + return factory(self, start, option)
>> + except:
>> + # Argh. Would like to raise wrapped exception.
>> + colno, lineno = self.get_colno_lineno(start)
>> + msg = ('Dynamic option %r in line %d, col %d of %s '
>> + 'crashed.') % (option, lineno, colno, self.source)
>> + self.recipe.logger.error(msg, exc_info=True)
>> + raise
>> + # else...
>> + options = self.recipe.options
>>
>
> Shouldn't you catch and re-raise SystemExit and KeyboardInterrupt? Or was this
> fixed in Python 2.5?

Changed.

>> - def substitute(self, recipe, seen):
>> + def substitute(self):
>> def convert(mo):
>> + start = mo.start()
>> # Check the most common path first.
>> - option = mo.group('braced_single')
>> + option = mo.group('option')
>> if option is not None:
>> - val = self._get(recipe.options, recipe.name, option, seen,
>> - mo.start('braced_single'))
>> + section = mo.group('section')
>> + val = self._get(section, option, start)
>> + path_extension = mo.group('path_extension')
>> + filters = mo.group('filters')
>> + if path_extension is not None:
>> + val = os.path.join(val, *path_extension.split('/')[1:])
>> + if filters is not None:
>> + for filter in filters.split('|')[1:]:
>> + filte...

Read more...

Unmerged revisions

22. By Gary Poster

add comments to generated code; only log a problem in filters and dynamic options if they are not stopped because of SystemExit and KeyboardInterrupt.

21. By Gary Poster

remove comment that turned out not to be true.

20. By Gary Poster

tweak based on usage

19. By Gary Poster

add support for relative paths

18. By Gary Poster

re-commit the pertinent work from lp:~gary/z3c.recipe.filetemplate/support-system-python

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file '.bzrignore'
2--- .bzrignore 1970-01-01 00:00:00 +0000
3+++ .bzrignore 2010-04-20 19:34:27 +0000
4@@ -0,0 +1,7 @@
5+.installed.cfg
6+bin
7+develop-eggs
8+eggs
9+parts
10+z3c.recipe.filetemplate.egg-info
11+dist
12
13=== modified file 'CHANGES.txt'
14--- CHANGES.txt 2010-04-20 19:34:27 +0000
15+++ CHANGES.txt 2010-04-20 19:34:27 +0000
16@@ -9,10 +9,35 @@
17 Features
18 --------
19
20-- Support escaping "${...}" with "$${...}" in templates. This is particularly
21- useful for *NIX shell scripts.
22-
23-- Support the relative-paths buildout option. XXX describe design
24+- Enable cross-platform paths by allowing an extended syntax for path
25+ suffixes. Example: If ``${buildout:directory}`` resolves to
26+ ``/sample_buildout`` on a POSIX system and ``C:\sample_buildout`` in
27+ Windows, ``${buildout:directory/foo.txt}`` will resolve to
28+ ``/sample_buildout/foo.txt`` and ``C:\sample_buildout\foo.txt``,
29+ respectively.
30+
31+- Add filters via a pipe syntax, reminiscent of UNIX pipes or Django template
32+ filters. Simple example: if ``${name}`` resolves to ``harry`` then
33+ ``${name|upper}`` resolves to ``HARRY``. Simple string filters are
34+ upper, lower, title, and capitalize, just like the Python string
35+ methods. Also see the next bullet.
36+
37+- Added support for the buildout relative-paths option. Shell scripts should
38+ include ``${shell-relative-path-setup}`` before commands with
39+ buildout-generated paths are executed. Python scripts should use
40+ ``${python-relative-path-setup}`` similarly. ``${os-paths}`` (shell),
41+ ``${space-paths}`` (shell), and ``${string-paths}`` (Python) will have
42+ relative paths if the buildout relative-paths option is used. To convert
43+ individual absolute paths to relative paths, use the ``path-repr`` filter
44+ in Python scripts and the ``shell-path`` filter in shell scripts. Path
45+ suffixes can be combined with these filters, so, if buildout's
46+ relative-paths option is true, ``${buildout:directory/foo.txt|path-repr}``
47+ will produce a buildout-relative, platform appropriate path to
48+ foo.txt. Note that for shell scripts, Windows is not supported at
49+ this time.
50+
51+- Support escaping ``${...}`` with ``$${...}`` in templates. This is
52+ particularly useful for *NIX shell scripts.
53
54 -----
55 Fixes
56@@ -20,6 +45,14 @@
57
58 - Make tests less susceptible to timing errors.
59
60+-------
61+Changes
62+-------
63+
64+- ``${os-paths}`` and ``${space-paths}`` no longer filter out .zip paths.
65+
66+- The entries in ``${string-paths}`` now are separated by newlines. Each
67+ entry is indented to the level of the initial placement of the marker.
68
69 2.0.3 (2009-07-02)
70 ==================
71
72=== added file 'MANIFEST.in'
73--- MANIFEST.in 1970-01-01 00:00:00 +0000
74+++ MANIFEST.in 2010-04-20 19:34:27 +0000
75@@ -0,0 +1,3 @@
76+include *.txt
77+recursive-include z3c *.txt
78+exclude MANIFEST.in buildout.cfg .bzrignore
79
80=== modified file 'setup.py'
81--- setup.py 2009-05-04 17:31:38 +0000
82+++ setup.py 2010-04-20 19:34:27 +0000
83@@ -19,7 +19,7 @@
84 return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
85
86 setup(name='z3c.recipe.filetemplate',
87- version = '2.1dev',
88+ version = '2.1',
89 license='ZPL 2.1',
90 url='http://pypi.python.org/pypi/z3c.recipe.filetemplate',
91 description="zc.buildout recipe for creating files from file templates",
92
93=== modified file 'z3c/recipe/filetemplate/README.txt'
94--- z3c/recipe/filetemplate/README.txt 2010-04-20 19:34:27 +0000
95+++ z3c/recipe/filetemplate/README.txt 2010-04-20 19:34:27 +0000
96@@ -224,6 +224,12 @@
97 Also note that, if you use a source directory and your ``files`` specify a
98 directory, the directory must match precisely.
99
100+ >>> # Clean up for later test.
101+ >>> import shutil
102+ >>> shutil.rmtree(os.path.join(sample_buildout, 'template', 'etc'))
103+ >>> os.remove(os.path.join(
104+ ... sample_buildout, 'template', 'bin', 'helloworld.sh.in'))
105+
106 ==============
107 Advanced Usage
108 ==============
109@@ -235,7 +241,7 @@
110 standard buildout syntax, but used in the template. Notice
111 ``${buildout:parts}`` in the template below.
112
113- >>> write(sample_buildout, 'helloworld.txt.in',
114+ >>> update_file(sample_buildout, 'helloworld.txt.in',
115 ... """
116 ... Hello ${world}. I used these parts: ${buildout:parts}.
117 ... """)
118@@ -257,11 +263,108 @@
119 >>> cat(sample_buildout, 'helloworld.txt')
120 Hello Philipp. I used these parts: message.
121
122-Sharing variables
123+Path Extensions
124+===============
125+
126+Substitutions can have path suffixes using the POSIX "/" path separator.
127+The template will convert these to the proper path separator for the current
128+OS. They also then are part of the value passed to filters, the feature
129+described next. Notice ``${buildout:directory/foo/bar.txt}`` in the template
130+below.
131+
132+ >>> update_file(sample_buildout, 'helloworld.txt.in',
133+ ... """
134+ ... Here's foo/bar.txt in the buildout:
135+ ... ${buildout:directory/foo/bar.txt}
136+ ... """)
137+
138+ >>> print system(buildout)
139+ Uninstalling message.
140+ Installing message.
141+
142+ >>> cat(sample_buildout, 'helloworld.txt') # doctest: +ELLIPSIS
143+ Here's foo/bar.txt in the buildout:
144+ /.../sample-buildout/foo/bar.txt
145+
146+Filters
147+=======
148+
149+You can use pipes within a substitution to filter the original value. This
150+recipe provides several filters for you to use. The syntax is reminiscent of
151+(and inspired by) POSIX pipes and Django template filters. For example,
152+if world = Philipp, ``HELLO ${world|upper}!`` would result in ``HELLO
153+PHILIPP!``.
154+
155+A few simple Python string methods are exposed as filters right now:
156+
157+- capitalize: First letter in string is capitalized.
158+- lower: All letters in string are lowercase.
159+- title: First letter of each word in string is capitalized.
160+- upper: All letters in string are uppercase.
161+
162+Other filters are important for handling paths if buildout's relative-paths
163+option is true. See `Working with Paths`_ for more details.
164+
165+- path-repr: Converts the path to a Python expression for the path. If
166+ buildout's relative-paths option is false, this will simply be a repr
167+ of the absolute path. If relative-paths is true, this will be a
168+ function call to convert a buildout-relative path to an absolute path;
169+ it requires that ``${python-relative-path-setup}`` be included earlier
170+ in the template.
171+
172+- shell-path: Converts the path to a shell expression for the path. Only
173+ POSIX is supported at this time. If buildout's relative-paths option
174+ is false, this will simply be the absolute path. If relative-paths is
175+ true, this will be an expression to convert a buildout-relative path
176+ to an absolute path; it requires that ``${shell-relative-path-setup}``
177+ be included earlier in the template.
178+
179+Combining the three advanced features described so far, then, if the
180+buildout relative-paths option were false, we were in a POSIX system, and
181+the sample buildout were in the root of the system, the template
182+expression ``${buildout:bin-directory/data/initial.csv|path-repr}``
183+would result in ``'/sample-buildout/bin/data/initial.csv'``.
184+
185+Here's a real, working example of the string method filters. We'll have
186+examples of the path filters in the `Working with Paths`_ section.
187+
188+ >>> update_file(sample_buildout, 'helloworld.txt.in',
189+ ... """
190+ ... HELLO ${world|upper}!
191+ ... hello ${world|lower}.
192+ ... ${name|title} and the Chocolate Factory
193+ ... ${sentence|capitalize}
194+ ... """)
195+
196+ >>> write(sample_buildout, 'buildout.cfg',
197+ ... """
198+ ... [buildout]
199+ ... parts = message
200+ ...
201+ ... [message]
202+ ... recipe = z3c.recipe.filetemplate
203+ ... files = helloworld.txt
204+ ... world = Philipp
205+ ... name = willy wonka
206+ ... sentence = that is a good book.
207+ ... """)
208+
209+ >>> print system(buildout)
210+ Uninstalling message.
211+ Installing message.
212+
213+ >>> cat(sample_buildout, 'helloworld.txt') # doctest: +ELLIPSIS
214+ HELLO PHILIPP!
215+ hello philipp.
216+ Willy Wonka and the Chocolate Factory
217+ That is a good book.
218+
219+Sharing Variables
220 =================
221
222-The recipe allows extending one or more sections, to decrease repetition, using
223-the ``extends`` option. For instance, consider the following buildout.
224+The recipe allows extending one or more sections, to decrease
225+repetition, using the ``extends`` option. For instance, consider the
226+following buildout.
227
228 >>> write(sample_buildout, 'buildout.cfg',
229 ... """
230@@ -284,7 +387,7 @@
231 section, and overwritten locally. A template of
232 ``${mygreeting}, ${myaudience}!``...
233
234- >>> write(sample_buildout, 'helloworld.txt.in',
235+ >>> update_file(sample_buildout, 'helloworld.txt.in',
236 ... """
237 ... ${mygreeting}, ${myaudience}!
238 ... """)
239@@ -298,114 +401,15 @@
240 >>> cat(sample_buildout, 'helloworld.txt')
241 Hi, everybody!
242
243-Specifying paths
244-================
245-
246-You can specify eggs and extra-paths in the recipe. If you do, three
247-predefined options will be available in the recipe's options for the template.
248-If "paths" are the non-zip paths, and "all_paths" are all paths, then the
249-options would be defined roughly as given here:
250-
251-``os-paths``
252- ``(os.pathsep).join(paths)``
253-
254-``string-paths``
255- ``', '.join(repr(p) for p in all_paths)``
256-
257-``space-paths``
258- ``' '.join(paths)``
259-
260-For instance, consider this example.
261-
262- >>> write(sample_buildout, 'buildout.cfg',
263- ... """
264- ... [buildout]
265- ... parts = message
266- ...
267- ... [message]
268- ... recipe = z3c.recipe.filetemplate
269- ... files = helloworld.txt
270- ... eggs = demo<0.3
271- ...
272- ... find-links = %(server)s
273- ... index = %(server)s/index
274- ... """ % dict(server=link_server))
275-
276-
277- >>> write(sample_buildout, 'helloworld.txt.in',
278- ... """
279- ... Hello! Here are the paths for the ${eggs} eggs.
280- ... OS paths:
281- ... ${os-paths}
282- ... ---
283- ... String paths:
284- ... ${string-paths}
285- ... ---
286- ... Space paths:
287- ... ${space-paths}
288- ... """)
289-
290- >>> print system(buildout)
291- Getting distribution for 'demo<0.3'.
292- Got demo 0.2.
293- Getting distribution for 'demoneeded'.
294- Got demoneeded 1.2c1.
295- Uninstalling message.
296- Installing message.
297-
298- >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS
299- Hello! Here are the paths for the demo<0.3 eggs.
300- OS paths:
301- .../eggs/demo-0.2...egg:.../eggs/demoneeded-1.2c1...egg
302- ---
303- String paths:
304- '.../eggs/demo-0.2...egg', '.../eggs/demoneeded-1.2c1...egg'
305- ---
306- Space paths:
307- .../eggs/demo-0.2...egg .../eggs/demoneeded-1.2c1...egg
308-
309-You can specify extra-paths as well, which will go at the end of the egg paths.
310-
311- >>> write(sample_buildout, 'buildout.cfg',
312- ... """
313- ... [buildout]
314- ... parts = message
315- ...
316- ... [message]
317- ... recipe = z3c.recipe.filetemplate
318- ... files = helloworld.txt
319- ... eggs = demo<0.3
320- ... extra-paths = ${buildout:directory}/foo
321- ...
322- ... find-links = %(server)s
323- ... index = %(server)s/index
324- ... """ % dict(server=link_server))
325-
326- >>> print system(buildout)
327- Uninstalling message.
328- Installing message.
329-
330- >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS
331- Hello! Here are the paths for the demo<0.3 eggs.
332- OS paths:
333- ...demo...:...demoneeded...:.../sample-buildout/foo
334- ---
335- String paths:
336- '...demo...', '...demoneeded...', '.../sample-buildout/foo'
337- ---
338- Space paths:
339- ...demo... ...demoneeded... .../sample-buildout/foo
340-
341 Defining options in Python
342 ==========================
343
344 You can specify that certain variables should be interpreted as Python using
345 ``interpreted-options``. This takes zero or more lines. Each line should
346-specify an option. It can define immediately (see ``duplicate-os-paths``,
347-``foo-paths``, and ``silly-range`` in the example below) or point to an option
348-to be interepreted, which can be useful if you want to define a
349-multi-line expression (see ``first-interpreted-option`` and
350-``message-reversed-is-egassem``).
351+specify an option. It can define immediately (see ``silly-range`` in
352+the example below) or point to an option to be interepreted, which can
353+be useful if you want to define a multi-line expression (see
354+``first-interpreted-option`` and ``message-reversed-is-egassem``).
355
356 >>> write(sample_buildout, 'buildout.cfg',
357 ... """
358@@ -415,28 +419,20 @@
359 ... [message]
360 ... recipe = z3c.recipe.filetemplate
361 ... files = helloworld.txt
362- ... eggs = demo<0.3
363- ... interpreted-options = duplicate-os-paths=(os.pathsep).join(paths)
364- ... foo-paths='FOO'.join(all_paths)
365- ... silly-range = repr(range(5))
366+ ... interpreted-options = silly-range = repr(range(5))
367 ... first-interpreted-option
368 ... message-reversed-is-egassem
369 ... first-interpreted-option =
370- ... options['interpreted-options'].split()[0].strip()
371+ ... options['interpreted-options'].splitlines()[0].strip()
372 ... message-reversed-is-egassem=
373 ... ''.join(
374 ... reversed(
375 ... buildout['buildout']['parts']))
376 ... not-interpreted=hello world
377- ...
378- ... find-links = %(server)s
379- ... index = %(server)s/index
380- ... """ % dict(server=link_server))
381+ ... """)
382
383- >>> write(sample_buildout, 'helloworld.txt.in', """\
384+ >>> update_file(sample_buildout, 'helloworld.txt.in', """\
385 ... ${not-interpreted}!
386- ... duplicate-os-paths: ${duplicate-os-paths}
387- ... foo-paths: ${foo-paths}
388 ... silly-range: ${silly-range}
389 ... first-interpreted-option: ${first-interpreted-option}
390 ... message-reversed-is-egassem: ${message-reversed-is-egassem}
391@@ -448,9 +444,421 @@
392
393 >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS
394 hello world!
395- duplicate-os-paths: ...demo-0.2...egg:...demoneeded-1.2c1...egg
396- foo-paths: ...demo-0.2...eggFOO...demoneeded-1.2c1...egg
397 silly-range: [0, 1, 2, 3, 4]
398- first-interpreted-option: duplicate-os-paths=(os.pathsep).join(paths)
399+ first-interpreted-option: silly-range = repr(range(5))
400 message-reversed-is-egassem: egassem
401
402+Working with Paths
403+==================
404+
405+We've already mentioned how to handle buildout's relative-paths option
406+in the discussion of filters. This section has some concrete examples
407+and discussion of that. It also introduces how to get a set of paths
408+from specifying dependencies.
409+
410+Here are concrete examples of the path-repr and shell-path filters.
411+We'll show results when relative-paths is true and when it is false.
412+
413+------------------------------
414+Demonstration of ``path-repr``
415+------------------------------
416+
417+Let's say we want to make a custom Python script in the bin directory.
418+It will print some information from a file in a ``data`` directory
419+within the buildout root. Here's the template.
420+
421+ >>> write(sample_buildout, 'template', 'bin', 'dosomething.py.in', '''\
422+ ... #!${buildout:executable}
423+ ... ${python-relative-path-setup}
424+ ... f = open(${buildout:directory/data/info.csv|path-repr})
425+ ... print f.read()
426+ ... ''')
427+ >>> os.chmod(
428+ ... os.path.join(
429+ ... sample_buildout, 'template', 'bin', 'dosomething.py.in'),
430+ ... 0711)
431+
432+If we evaluate that template with relative-paths set to false, the results
433+shouldn't be too surprising.
434+
435+ >>> write(sample_buildout, 'buildout.cfg',
436+ ... """
437+ ... [buildout]
438+ ... parts = message
439+ ...
440+ ... [message]
441+ ... recipe = z3c.recipe.filetemplate
442+ ... source-directory = template
443+ ... """)
444+
445+ >>> print system(buildout)
446+ Uninstalling message.
447+ Installing message.
448+
449+ >>> cat(sample_buildout, 'bin', 'dosomething.py') # doctest: +ELLIPSIS
450+ #!...
451+ <BLANKLINE>
452+ f = open('/.../sample-buildout/data/info.csv')
453+ print f.read()
454+
455+``${python-relative-path-setup}`` evaluated to an empty string. The path
456+is absolute and quoted.
457+
458+If we evaluate it with relative-paths set to true, the results are much...
459+bigger.
460+
461+ >>> write(sample_buildout, 'buildout.cfg',
462+ ... """
463+ ... [buildout]
464+ ... parts = message
465+ ... relative-paths = true
466+ ...
467+ ... [message]
468+ ... recipe = z3c.recipe.filetemplate
469+ ... source-directory = template
470+ ... """)
471+
472+ >>> print system(buildout)
473+ Uninstalling message.
474+ Installing message.
475+
476+ >>> cat(sample_buildout, 'bin', 'dosomething.py') # doctest: +ELLIPSIS
477+ #!...
478+ import os, imp
479+ # Get path to this file.
480+ if __name__ == '__main__':
481+ _z3c_recipe_filetemplate_filename = __file__
482+ else:
483+ # If this is an imported module, we want the location of the .py
484+ # file, not the .pyc, because the .py file may have been symlinked.
485+ _z3c_recipe_filetemplate_filename = imp.find_module(__name__)[1]
486+ # Get the full, non-symbolic-link directory for this file.
487+ _z3c_recipe_filetemplate_base = os.path.dirname(
488+ os.path.abspath(os.path.realpath(_z3c_recipe_filetemplate_filename)))
489+ # Ascend to buildout root.
490+ _z3c_recipe_filetemplate_base = os.path.dirname(
491+ _z3c_recipe_filetemplate_base)
492+ def _z3c_recipe_filetemplate_path_repr(path):
493+ "Return absolute version of buildout-relative path."
494+ return os.path.join(_z3c_recipe_filetemplate_base, path)
495+ <BLANKLINE>
496+ f = open(_z3c_recipe_filetemplate_path_repr('data/info.csv'))
497+ print f.read()
498+
499+That's quite a bit of code. You might wonder why we don't just use '..' for
500+parent directories. The reason is that we want our scripts to be usable
501+from any place on the filesystem. If we used '..' to construct paths
502+relative to the generated file, then the paths would only work from
503+certain directories.
504+
505+So that's how path-repr works. It can really come in handy if you want
506+to support relative paths in buildout. Now let's look at the shell-path
507+filter.
508+
509+-------------------------------
510+Demonstration of ``shell-path``
511+-------------------------------
512+
513+Maybe you want to write some shell scripts. The shell-path filter will help
514+you support buildout relative-paths fairly painlessly.
515+
516+Right now, only POSIX is supported with the shell-path filter, as mentioned
517+before.
518+
519+Usage is very similar to the ``path-repr`` filter. You need to include
520+``${shell-relative-path-setup}`` before you use it, just as you include
521+``${python-relative-path-setup}`` before using ``path-repr``.
522+
523+Let's say we want to make a custom shell script in the bin directory.
524+It will print some information from a file in a ``data`` directory
525+within the buildout root. Here's the template.
526+
527+ >>> write(sample_buildout, 'template', 'bin', 'dosomething.sh.in', '''\
528+ ... #!/bin/sh
529+ ... ${shell-relative-path-setup}
530+ ... cat ${buildout:directory/data/info.csv|shell-path}
531+ ... ''')
532+ >>> os.chmod(
533+ ... os.path.join(
534+ ... sample_buildout, 'template', 'bin', 'dosomething.sh.in'),
535+ ... 0711)
536+
537+If relative-paths is set to false (the default), the results are simple.
538+
539+ >>> write(sample_buildout, 'buildout.cfg',
540+ ... """
541+ ... [buildout]
542+ ... parts = message
543+ ...
544+ ... [message]
545+ ... recipe = z3c.recipe.filetemplate
546+ ... source-directory = template
547+ ... """)
548+
549+ >>> print system(buildout)
550+ Uninstalling message.
551+ Installing message.
552+
553+ >>> cat(sample_buildout, 'bin', 'dosomething.sh') # doctest: +ELLIPSIS
554+ #!/bin/sh
555+ <BLANKLINE>
556+ cat /.../sample-buildout/data/info.csv
557+
558+``${shell-relative-path-setup}`` evaluated to an empty string. The path
559+is absolute.
560+
561+Now let's look at the larger code when relative-paths is set to true.
562+
563+ >>> write(sample_buildout, 'buildout.cfg',
564+ ... """
565+ ... [buildout]
566+ ... parts = message
567+ ... relative-paths = true
568+ ...
569+ ... [message]
570+ ... recipe = z3c.recipe.filetemplate
571+ ... source-directory = template
572+ ... """)
573+
574+ >>> print system(buildout)
575+ Uninstalling message.
576+ Installing message.
577+
578+ >>> cat(sample_buildout, 'bin', 'dosomething.sh') # doctest: +ELLIPSIS
579+ #!/bin/sh
580+ # Get full, non-symbolic-link path to this file.
581+ Z3C_RECIPE_FILETEMPLATE_FILENAME=`\
582+ readlink -f "$0" 2>/dev/null || \
583+ realpath "$0" 2>/dev/null || \
584+ type -P "$0" 2>/dev/null`
585+ # Get directory of file.
586+ Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_FILENAME}`
587+ # Ascend to buildout root.
588+ Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_BASE}`
589+ <BLANKLINE>
590+ cat "$Z3C_RECIPE_FILETEMPLATE_BASE"/data/info.csv
591+
592+As with the Python code, we don't just use '..' for
593+parent directories because we want our scripts to be usable
594+from any place on the filesystem.
595+
596+----------------------------------
597+Getting Arbitrary Dependency Paths
598+----------------------------------
599+
600+You can specify ``eggs`` and ``extra-paths`` in the recipe. The
601+mechanism is the same as the one provided by the zc.recipe.egg, so
602+pertinent options such as find-links and index are available.
603+
604+If you do, the paths for the dependencies will be calculated. They will
605+be available as a list in the namespace of the interpreted options as
606+``paths``. Also, three predefined options will be available in the
607+recipe's options for the template.
608+
609+If ``paths`` are the paths, ``shell_path`` is the ``shell-path`` filter, and
610+``path_repr`` is the ``path-repr`` filter, then the pre-defined options
611+would be defined roughly as given here:
612+
613+``os-paths`` (for shell scripts)
614+ ``(os.pathsep).join(shell_path(path) for path in paths)``
615+
616+``string-paths`` (for Python scripts)
617+ ``',\n '.join(path_repr(path) for path in paths)``
618+
619+``space-paths`` (for shell scripts)
620+ ``' '.join(shell_path(path) for path in paths)``
621+
622+Therefore, if you want to support the relative-paths option, you should
623+include ``${shell-relative-path-setup}`` (for ``os-paths`` and
624+``space-paths``) or ``${python-relative-path-setup}`` (for ``string-paths``)
625+as appropriate at the top of your template.
626+
627+Let's consider a simple example.
628+
629+ >>> write(sample_buildout, 'buildout.cfg',
630+ ... """
631+ ... [buildout]
632+ ... parts = message
633+ ...
634+ ... [message]
635+ ... recipe = z3c.recipe.filetemplate
636+ ... files = helloworld.txt
637+ ... eggs = demo<0.3
638+ ...
639+ ... find-links = %(server)s
640+ ... index = %(server)s/index
641+ ... """ % dict(server=link_server))
642+
643+The relative-paths option is false, the default.
644+
645+ >>> write(sample_buildout, 'helloworld.txt.in',
646+ ... """
647+ ... Hello! Here are the paths for the ${eggs} eggs.
648+ ... OS paths:
649+ ... ${os-paths}
650+ ... ---
651+ ... String paths:
652+ ... ${string-paths}
653+ ... ---
654+ ... Space paths:
655+ ... ${space-paths}
656+ ... """)
657+
658+ >>> print system(buildout)
659+ Getting distribution for 'demo<0.3'.
660+ Got demo 0.2.
661+ Getting distribution for 'demoneeded'.
662+ Got demoneeded 1.2c1.
663+ Uninstalling message.
664+ Installing message.
665+
666+ >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS
667+ Hello! Here are the paths for the demo<0.3 eggs.
668+ OS paths:
669+ /.../eggs/demo-0.2...egg:/.../eggs/demoneeded-1.2c1...egg
670+ ---
671+ String paths:
672+ '/.../eggs/demo-0.2...egg',
673+ '/.../eggs/demoneeded-1.2c1...egg'
674+ ---
675+ Space paths:
676+ /.../eggs/demo-0.2...egg /.../eggs/demoneeded-1.2c1...egg
677+
678+You can specify extra-paths as well, which will go at the end of the egg
679+paths.
680+
681+ >>> write(sample_buildout, 'buildout.cfg',
682+ ... """
683+ ... [buildout]
684+ ... parts = message
685+ ...
686+ ... [message]
687+ ... recipe = z3c.recipe.filetemplate
688+ ... files = helloworld.txt
689+ ... eggs = demo<0.3
690+ ... extra-paths = ${buildout:directory}/foo
691+ ...
692+ ... find-links = %(server)s
693+ ... index = %(server)s/index
694+ ... """ % dict(server=link_server))
695+
696+ >>> print system(buildout)
697+ Uninstalling message.
698+ Installing message.
699+
700+ >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS
701+ Hello! Here are the paths for the demo<0.3 eggs.
702+ OS paths:
703+ /...demo...:/...demoneeded...:/.../sample-buildout/foo
704+ ---
705+ String paths:
706+ '/...demo...',
707+ '/...demoneeded...',
708+ '/.../sample-buildout/foo'
709+ ---
710+ Space paths:
711+ /...demo... /...demoneeded... .../sample-buildout/foo
712+
713+To emphasize the effect of the relative-paths option, let's see what it looks
714+like when we set relative-paths to True.
715+
716+ >>> write(sample_buildout, 'buildout.cfg',
717+ ... """
718+ ... [buildout]
719+ ... parts = message
720+ ... relative-paths = true
721+ ...
722+ ... [message]
723+ ... recipe = z3c.recipe.filetemplate
724+ ... files = helloworld.txt
725+ ... eggs = demo<0.3
726+ ... extra-paths = ${buildout:directory}/foo
727+ ...
728+ ... find-links = %(server)s
729+ ... index = %(server)s/index
730+ ... """ % dict(server=link_server))
731+
732+ >>> print system(buildout)
733+ Uninstalling message.
734+ Installing message.
735+
736+ >>> cat(sample_buildout, 'helloworld.txt') # doctest:+ELLIPSIS
737+ Hello! Here are the paths for the demo<0.3 eggs.
738+ OS paths:
739+ "$Z3C_RECIPE_FILETEMPLATE_BASE"/eggs/demo-0.2-py...egg:"$Z3C_RECIPE_FILETEMPLATE_BASE"/eggs/demoneeded-1.2c1-py...egg:"$Z3C_RECIPE_FILETEMPLATE_BASE"/foo
740+ ---
741+ String paths:
742+ _z3c_recipe_filetemplate_path_repr('eggs/demo-0.2-py...egg'),
743+ _z3c_recipe_filetemplate_path_repr('eggs/demoneeded-1.2c1-py...egg'),
744+ _z3c_recipe_filetemplate_path_repr('foo')
745+ ---
746+ Space paths:
747+ "$Z3C_RECIPE_FILETEMPLATE_BASE"/eggs/demo-0.2-py...egg "$Z3C_RECIPE_FILETEMPLATE_BASE"/eggs/demoneeded-1.2c1-py...egg "$Z3C_RECIPE_FILETEMPLATE_BASE"/foo
748+
749+
750+Remember, your script won't really work unless you include
751+``${shell-relative-path-setup}`` (for ``os-paths`` and ``space-paths``)
752+or ``${python-relative-path-setup}`` (for ``string-paths``) as
753+appropriate at the top of your template.
754+
755+Getting Dependency Paths from ``zc.recipe.egg``
756+-----------------------------------------------
757+
758+You can get the ``eggs`` and ``extra-paths`` from another section using
759+zc.recipe.egg by using the ``extends`` option from the `Sharing Variables`_
760+section above. Then you can use the template options described above to
761+build your paths in your templates.
762+
763+Getting Dependency Paths from ``z3c.recipe.scripts``
764+----------------------------------------------------
765+
766+If, like the Launchpad project, you are using Gary Poster's unreleased
767+package ``z3c.recipe.scripts`` to generate your scripts, and you want to
768+have your scripts use the same Python environment as generated by that
769+recipe, you can just use the path-repr and shell-path filters with standard
770+buildout directories. Here is an example buildout.cfg.
771+
772+::
773+
774+ [buildout]
775+ parts = scripts message
776+ relative-paths = true
777+
778+ [scripts]
779+ recipe = z3c.recipe.scripts
780+ eggs = demo<0.3
781+
782+ [message]
783+ recipe = z3c.recipe.filetemplate
784+ files = helloworld.py
785+
786+Then the template to use this would want to simply put
787+``${scripts:parts-directory|path-repr}`` at the beginning of Python's path.
788+
789+You can do this for subprocesses with PYTHONPATH.
790+
791+ ${python-relative-path-setup}
792+ import os
793+ import subprocess
794+ env = os.environ.copy()
795+ env['PYTHONPATH'] = ${scripts:parts-directory|path-repr}
796+ subprocess.call('myscript', env=env)
797+
798+That's it.
799+
800+Similarly, here's an approach to making a script that will have the
801+right environment. You want to put the parts directory of the
802+z3c.recipe.scripts section in the sys.path before site.py is loaded.
803+This is usually handled by z3c.recipe.scripts itself, but sometimes you
804+may want to write Python scripts in your template for some reason.
805+
806+ #!/usr/bin/env python -S
807+ ${python-relative-path-setup}
808+ import sys
809+ sys.path.insert(0, ${scripts:parts-directory|path-repr})
810+ import site
811+ # do stuff...
812+
813+If you do this for many scripts, put this entire snippet in an option in the
814+recipe and use this snippet as a single substitution in the top of your
815+scripts.
816
817=== modified file 'z3c/recipe/filetemplate/__init__.py'
818--- z3c/recipe/filetemplate/__init__.py 2010-04-20 19:34:27 +0000
819+++ z3c/recipe/filetemplate/__init__.py 2010-04-20 19:34:27 +0000
820@@ -29,10 +29,15 @@
821
822 class FileTemplate(object):
823
824+ filters = {}
825+ dynamic_options = {}
826+
827 def __init__(self, buildout, name, options):
828 self.buildout = buildout
829 self.name = name
830 self.options = options
831+ self.buildout_root = zc.buildout.easy_install.realpath(
832+ buildout['buildout']['directory'])
833 self.logger=logging.getLogger(self.name)
834 # get defaults from extended sections
835 defaults = {}
836@@ -42,34 +47,33 @@
837 defaults.update(self.buildout[section_name])
838 for key, value in defaults.items():
839 self.options.setdefault(key, value)
840+ relative_paths = self.options.setdefault(
841+ 'relative-paths',
842+ buildout['buildout'].get('relative-paths', 'false')
843+ )
844+ if relative_paths not in ('true', 'false'):
845+ self._user_error(
846+ 'The relative-paths option must have the value of '
847+ 'true or false.')
848+ self.relative_paths = relative_paths = (relative_paths == 'true')
849+ self.paths = paths = []
850 # set up paths for eggs, if given
851- if 'eggs' in self.options:
852- relative_paths = self.options.get(
853- 'relative-paths',
854- buildout['buildout'].get('relative-paths', 'false')
855- )
856- if relative_paths not in ('true', 'false'):
857- self._user_error(
858- 'The relative-paths option must have the value of '
859- 'true or false.')
860- relative_paths = relative_paths == 'true'
861- if relative_paths:
862- raise NotImplementedError # XXX
863- self.eggs = zc.recipe.egg.Scripts(buildout, name, options)
864- orig_distributions, ws = self.eggs.working_set()
865- # we want ws, eggs.extra_paths, eggs._relative_paths
866- all_paths = [
867+ if 'eggs' in options:
868+ eggs = zc.recipe.egg.Scripts(buildout, name, options)
869+ orig_distributions, ws = eggs.working_set()
870+ paths.extend(
871 zc.buildout.easy_install.realpath(dist.location)
872- for dist in ws]
873- all_paths.extend(
874+ for dist in ws)
875+ paths.extend(
876 zc.buildout.easy_install.realpath(path)
877- for path in self.eggs.extra_paths)
878+ for path in eggs.extra_paths)
879 else:
880- all_paths = []
881- paths = [path for path in all_paths if not path.endswith('.zip')]
882- self.options['os-paths'] = (os.pathsep).join(paths)
883- self.options['string-paths'] = ', '.join(repr(p) for p in all_paths)
884- self.options['space-paths'] = ' '.join(paths)
885+ paths.extend(
886+ os.path.join(buildout.options['directory'], p.strip())
887+ for p in options.get('extra-paths', '').split('\n')
888+ if p.strip()
889+ )
890+ options['_paths'] = '\n'.join(paths)
891 # get and check the files to be created
892 self.filenames = self.options.get('files', '*').split()
893 self.source_dir = self.options.get('source-directory', '').strip()
894@@ -164,7 +168,7 @@
895 if interpreted:
896 globs = {'__builtins__': __builtins__, 'os': os, 'sys': sys}
897 locs = {'name': name, 'options': options, 'buildout': buildout,
898- 'paths': paths, 'all_paths': all_paths}
899+ 'paths': paths, 'all_paths': paths}
900 for value in interpreted.split('\n'):
901 if value:
902 value = value.split('=', 1)
903@@ -207,18 +211,20 @@
904 'Destinations already exist: %s. Please make sure that '
905 'you really want to generate these automatically. Then '
906 'move them away.', ', '.join(already_exists))
907- seen = [] # We throw this away right now, but could move template
908- # processing up to __init__ if valuable. That would mean that templates
909- # would be rewritten even if a value in another section had been
910- # referenced; however, it would also mean that __init__ would do
911- # virtually all of the work, with install only doing the writing.
912+ self.seen = []
913+ # We throw ``seen`` away right now, but could move template
914+ # processing up to __init__ if valuable. That would mean that
915+ # templates would be rewritten even if a value in another
916+ # section had been referenced; however, it would also mean that
917+ # __init__ would do virtually all of the work, with install only
918+ # doing the writing.
919 for rel_path, last_mod, st_mode in self.actions:
920 source = os.path.join(self.source_dir, rel_path)
921 dest = os.path.join(self.destination_dir, rel_path[:-3])
922 mode=stat.S_IMODE(st_mode)
923 # we process the file first so that it won't be created if there
924 # is a problem.
925- processed = Template(source).substitute(self, seen)
926+ processed = Template(source, dest, self).substitute()
927 self._create_paths(os.path.dirname(dest))
928 result=open(dest, "wt")
929 result.write(processed)
930@@ -233,71 +239,119 @@
931 os.mkdir(path)
932 self.options.created(path)
933
934+ def _call_and_log(self, callable, args, message_generator):
935+ try:
936+ return callable(*args)
937+ except (KeyboardInterrupt, SystemExit):
938+ raise
939+ except:
940+ # Argh. Would like to raise wrapped exception.
941+ colno, lineno = self.get_colno_lineno(start)
942+ msg = message_generator(lineno, colno)
943+ self.logger.error(msg, exc_info=True)
944+ raise
945+
946 def update(self):
947 pass
948
949
950 class Template:
951- # hacked from string.Template
952+ # Heavily hacked from--"inspired by"?--string.Template
953 pattern = re.compile(r"""
954 \$(?:
955- \${(?P<escaped>[^}]*)} | # Escape sequence of two delimiters.
956- {(?P<braced_single>[-a-z0-9 ._]+)} |
957- # Delimiter and a braced local option
958- {(?P<braced_double>[-a-z0-9 ._]+:[-a-z0-9 ._]+)} |
959- # Delimiter and a braced fully
960- # qualified option (that is, with
961- # explicit section).
962+ \${(?P<escaped>[^}]*)} | # Escape sequence of two delimiters.
963+
964+ {((?P<section>[-a-z0-9 ._]+):)? # Optional section name.
965+ (?P<option>[-a-z0-9 ._]+) # Required option name.
966+ (?P<path_extension>/[^|}]+/?)? # Optional path extensions.
967+ ([ ]*(?P<filters>(\|[ ]*[-a-z0-9._]+[ ]*)+))?
968+ # Optional filters.
969+ } |
970+
971 {(?P<invalid>[^}]*}) # Other ill-formed delimiter exprs.
972 )
973 """, re.IGNORECASE | re.VERBOSE)
974
975- def __init__(self, source):
976+ def __init__(self, source, destination, recipe):
977 self.source = source
978+ self.destination = zc.buildout.easy_install.realpath(destination)
979+ self.recipe = recipe
980 self.template = open(source).read()
981
982- def _get_colno_lineno(self, i):
983+ def get_colno_lineno(self, i):
984 lines = self.template[:i].splitlines(True)
985 if not lines:
986 colno = 1
987 lineno = 1
988 else:
989- colno = i - len(''.join(lines[:-1]))
990+ colno = len(lines[-1]) + 1
991 lineno = len(lines)
992 return colno, lineno
993
994- def _get(self, options, section, option, seen, start):
995- value = options.get(option, None, seen)
996+ def _get(self, section, option, start):
997+ if section is None:
998+ section = self.recipe.name # This sets up error messages properly.
999+ if section == self.recipe.name:
1000+ factory = self.recipe.dynamic_options.get(option)
1001+ if factory is not None:
1002+ return self.recipe._call_and_log(
1003+ factory, (self, start, option),
1004+ lambda lineno, colno: (
1005+ 'Dynamic option %r in line %d, col %d of %s '
1006+ 'crashed.') % (option, lineno, colno, self.source))
1007+ # else...
1008+ options = self.recipe.options
1009+ elif section in self.recipe.buildout:
1010+ options = self.recipe.buildout[section]
1011+ else:
1012+ value = options = None
1013+ if options is not None:
1014+ value = options.get(option, None, self.recipe.seen)
1015 if value is None:
1016- colno, lineno = self._get_colno_lineno(start)
1017+ colno, lineno = self.get_colno_lineno(start)
1018 raise zc.buildout.buildout.MissingOption(
1019 "Option '%s:%s', referenced in line %d, col %d of %s, "
1020 "does not exist." %
1021 (section, option, lineno, colno, self.source))
1022 return value
1023
1024- def substitute(self, recipe, seen):
1025+ def substitute(self):
1026 def convert(mo):
1027+ start = mo.start()
1028 # Check the most common path first.
1029- option = mo.group('braced_single')
1030+ option = mo.group('option')
1031 if option is not None:
1032- val = self._get(recipe.options, recipe.name, option, seen,
1033- mo.start('braced_single'))
1034+ section = mo.group('section')
1035+ val = self._get(section, option, start)
1036+ path_extension = mo.group('path_extension')
1037+ filters = mo.group('filters')
1038+ if path_extension is not None:
1039+ val = os.path.join(val, *path_extension.split('/')[1:])
1040+ if filters is not None:
1041+ for filter_name in filters.split('|')[1:]:
1042+ filter_name = filter_name.strip()
1043+ filter = self.recipe.filters.get(filter_name)
1044+ if filter is None:
1045+ colno, lineno = self.get_colno_lineno(start)
1046+ raise ValueError(
1047+ 'Unknown filter %r '
1048+ 'in line %d, col %d of %s' %
1049+ (filter_name, lineno, colno, self.source))
1050+ val = self.recipe._call_and_log(
1051+ filter, (val, self, start, filter_name),
1052+ lambda lineno, colno: (
1053+ 'Filter %r in line %d, col %d of %s '
1054+ 'crashed processing value %r') % (
1055+ filter_name, lineno, colno, self.source, val))
1056 # We use this idiom instead of str() because the latter will
1057 # fail if val is a Unicode containing non-ASCII characters.
1058 return '%s' % (val,)
1059- double = mo.group('braced_double')
1060- if double is not None:
1061- section, option = double.split(':')
1062- val = self._get(recipe.buildout[section], section, option, seen,
1063- mo.start('braced_double'))
1064- return '%s' % (val,)
1065 escaped = mo.group('escaped')
1066 if escaped is not None:
1067 return '${%s}' % (escaped,)
1068 invalid = mo.group('invalid')
1069 if invalid is not None:
1070- colno, lineno = self._get_colno_lineno(mo.start('invalid'))
1071+ colno, lineno = self.get_colno_lineno(mo.start('invalid'))
1072 raise ValueError(
1073 'Invalid placeholder %r in line %d, col %d of %s' %
1074 (mo.group('invalid'), lineno, colno, self.source))
1075@@ -305,3 +359,193 @@
1076 self.pattern) # programmer error, AFAICT
1077 return self.pattern.sub(convert, self.template)
1078
1079+
1080+############################################################################
1081+# Filters
1082+def filter(func):
1083+ "Helper function to register filter functions."
1084+ FileTemplate.filters[func.__name__.replace('_', '-')] = func
1085+ return func
1086+
1087+@filter
1088+def capitalize(val, template, start, filter):
1089+ return val.capitalize()
1090+
1091+@filter
1092+def title(val, template, start, filter):
1093+ return val.title()
1094+
1095+@filter
1096+def upper(val, template, start, filter):
1097+ return val.upper()
1098+
1099+@filter
1100+def lower(val, template, start, filter):
1101+ return val.lower()
1102+
1103+@filter
1104+def path_repr(val, template, start, filter):
1105+ # val is a path.
1106+ return _maybe_relativize(
1107+ val, template,
1108+ lambda p: "_z3c_recipe_filetemplate_path_repr(%r)" % (p,),
1109+ repr)
1110+
1111+@filter
1112+def shell_path(val, template, start, filter):
1113+ # val is a path.
1114+ return _maybe_relativize(
1115+ val, template,
1116+ lambda p: '"$Z3C_RECIPE_FILETEMPLATE_BASE"/%s' % (p,),
1117+ lambda p: p)
1118+
1119+# Helpers hacked from zc.buildout.easy_install.
1120+def _maybe_relativize(path, template, relativize, absolutize):
1121+ path = zc.buildout.easy_install.realpath(path)
1122+ if template.recipe.relative_paths:
1123+ buildout_root = template.recipe.buildout_root
1124+ if path == buildout_root:
1125+ return relativize(os.curdir)
1126+ destination = template.destination
1127+ common = os.path.dirname(os.path.commonprefix([path, destination]))
1128+ if (common == buildout_root or
1129+ common.startswith(os.path.join(buildout_root, ''))
1130+ ):
1131+ return relativize(_relative_path(common, path))
1132+ return absolutize(path)
1133+
1134+def _relative_path(common, path):
1135+ """Return the relative path from ``common`` to ``path``.
1136+
1137+ This is a helper for _relativitize, which is a helper to
1138+ _relative_path_and_setup.
1139+ """
1140+ r = []
1141+ while 1:
1142+ dirname, basename = os.path.split(path)
1143+ r.append(basename)
1144+ if dirname == common:
1145+ break
1146+ assert dirname != path, "dirname of %s is the same" % dirname
1147+ path = dirname
1148+ r.reverse()
1149+ return os.path.join(*r)
1150+
1151+
1152+############################################################################
1153+# Dynamic options
1154+def dynamic_option(func):
1155+ "Helper function to register dynamic options."
1156+ FileTemplate.dynamic_options[func.__name__.replace('_', '-')] = func
1157+ return func
1158+
1159+@dynamic_option
1160+def os_paths(template, start, name):
1161+ return os.pathsep.join(
1162+ shell_path(path, template, start, 'os-paths')
1163+ for path in template.recipe.paths)
1164+
1165+@dynamic_option
1166+def string_paths(template, start, name):
1167+ colno, lineno = template.get_colno_lineno(start)
1168+ separator = ',\n' + ((colno - 1) * ' ')
1169+ return separator.join(
1170+ path_repr(path, template, start, 'string-paths')
1171+ for path in template.recipe.paths)
1172+
1173+@dynamic_option
1174+def space_paths(template, start, name):
1175+ return ' '.join(
1176+ shell_path(path, template, start, 'space-paths')
1177+ for path in template.recipe.paths)
1178+
1179+@dynamic_option
1180+def shell_relative_path_setup(template, start, name):
1181+ if template.recipe.relative_paths:
1182+ depth = _relative_depth(
1183+ template.recipe.buildout['buildout']['directory'],
1184+ template.destination)
1185+ value = SHELL_RELATIVE_PATH_SETUP
1186+ if depth:
1187+ value += '# Ascend to buildout root.\n'
1188+ value += depth * SHELL_DIRNAME
1189+ else:
1190+ value += '# This is the buildout root.\n'
1191+ return value
1192+ else:
1193+ return ''
1194+
1195+SHELL_RELATIVE_PATH_SETUP = '''\
1196+# Get full, non-symbolic-link path to this file.
1197+Z3C_RECIPE_FILETEMPLATE_FILENAME=`\\
1198+ readlink -f "$0" 2>/dev/null || \\
1199+ realpath "$0" 2>/dev/null || \\
1200+ type -P "$0" 2>/dev/null`
1201+# Get directory of file.
1202+Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_FILENAME}`
1203+'''
1204+
1205+SHELL_DIRNAME = '''\
1206+Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_BASE}`
1207+'''
1208+
1209+@dynamic_option
1210+def python_relative_path_setup(template, start, name):
1211+ if template.recipe.relative_paths:
1212+ depth = _relative_depth(
1213+ template.recipe.buildout['buildout']['directory'],
1214+ template.destination)
1215+ value = PYTHON_RELATIVE_PATH_SETUP_START
1216+ if depth:
1217+ value += '# Ascend to buildout root.\n'
1218+ value += depth * PYTHON_DIRNAME
1219+ else:
1220+ value += '# This is the buildout root.\n'
1221+ value += PYTHON_RELATIVE_PATH_SETUP_END
1222+ return value
1223+ else:
1224+ return ''
1225+
1226+PYTHON_RELATIVE_PATH_SETUP_START = '''\
1227+import os, imp
1228+# Get path to this file.
1229+if __name__ == '__main__':
1230+ _z3c_recipe_filetemplate_filename = __file__
1231+else:
1232+ # If this is an imported module, we want the location of the .py
1233+ # file, not the .pyc, because the .py file may have been symlinked.
1234+ _z3c_recipe_filetemplate_filename = imp.find_module(__name__)[1]
1235+# Get the full, non-symbolic-link directory for this file.
1236+_z3c_recipe_filetemplate_base = os.path.dirname(
1237+ os.path.abspath(os.path.realpath(_z3c_recipe_filetemplate_filename)))
1238+'''
1239+
1240+PYTHON_DIRNAME = '''\
1241+_z3c_recipe_filetemplate_base = os.path.dirname(
1242+ _z3c_recipe_filetemplate_base)
1243+'''
1244+
1245+PYTHON_RELATIVE_PATH_SETUP_END = '''\
1246+def _z3c_recipe_filetemplate_path_repr(path):
1247+ "Return absolute version of buildout-relative path."
1248+ return os.path.join(_z3c_recipe_filetemplate_base, path)
1249+'''
1250+
1251+def _relative_depth(common, path):
1252+ # Helper ripped from zc.buildout.easy_install.
1253+ """Return number of dirs separating ``path`` from ancestor, ``common``.
1254+
1255+ For instance, if path is /foo/bar/baz/bing, and common is /foo, this will
1256+ return 2--in UNIX, the number of ".." to get from bing's directory
1257+ to foo.
1258+ """
1259+ n = 0
1260+ while 1:
1261+ dirname = os.path.dirname(path)
1262+ if dirname == path:
1263+ raise AssertionError("dirname of %s is the same" % dirname)
1264+ if dirname == common:
1265+ break
1266+ n += 1
1267+ path = dirname
1268+ return n
1269
1270=== modified file 'z3c/recipe/filetemplate/tests.txt'
1271--- z3c/recipe/filetemplate/tests.txt 2010-04-20 19:34:27 +0000
1272+++ z3c/recipe/filetemplate/tests.txt 2010-04-20 19:34:27 +0000
1273@@ -14,7 +14,7 @@
1274 ... """
1275 ... Hello ${world}!
1276 ... """)
1277-
1278+
1279 >>> write(sample_buildout, 'goodbyeworld.txt.in',
1280 ... """
1281 ... Goodbye ${world}!
1282@@ -56,7 +56,7 @@
1283 ... files = /etc/passwd.in
1284 ... root = me
1285 ... """)
1286-
1287+
1288 >>> print system(buildout)
1289 evil: /etc/passwd.in is an absolute path. Paths must be relative to the buildout directory.
1290 While:
1291@@ -80,7 +80,7 @@
1292 ... recipe = z3c.recipe.filetemplate
1293 ... files = doesntexist
1294 ... """)
1295-
1296+
1297 >>> print system(buildout)
1298 notthere: No template found for these file names: doesntexist.in
1299 While:
1300@@ -99,12 +99,12 @@
1301 ... """
1302 ... I'm already here
1303 ... """)
1304-
1305+
1306 >>> write(sample_buildout, 'alreadyhere.txt.in',
1307 ... """
1308 ... I'm the template that's supposed to replace the file above.
1309 ... """)
1310-
1311+
1312 >>> write(sample_buildout, 'buildout.cfg',
1313 ... """
1314 ... [buildout]
1315@@ -137,7 +137,7 @@
1316 ... """
1317 ... Hello ${world}!
1318 ... """)
1319-
1320+
1321 >>> write(sample_buildout, 'buildout.cfg',
1322 ... """
1323 ... [buildout]
1324@@ -147,12 +147,12 @@
1325 ... recipe = z3c.recipe.filetemplate
1326 ... files = missing.txt
1327 ... """)
1328-
1329+
1330 >>> print system(buildout) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
1331 Installing missing.
1332 While:
1333 Installing missing.
1334- Error: Option 'missing:world', referenced in line 2, col 8 of
1335+ Error: Option 'missing:world', referenced in line 2, col 7 of
1336 .../sample-buildout/missing.txt.in, does not exist.
1337
1338 No changes means just an update
1339@@ -366,3 +366,70 @@
1340 d parts
1341 d template
1342
1343+Specifying files with relative paths at the buildout root
1344+---------------------------------------------------------
1345+
1346+Working at the buildout root follows some different code paths with relative
1347+paths so we explore those here. We also evaluate paths at the directory root.
1348+
1349+ >>> rmdir(sample_buildout, 'template')
1350+ >>> mkdir(sample_buildout, 'template')
1351+ >>> write(sample_buildout, 'template', 'dosomething.py.in', '''\
1352+ ... #!${buildout:executable}
1353+ ... ${python-relative-path-setup}
1354+ ... root = ${buildout:directory|path-repr}
1355+ ... ''')
1356+
1357+ >>> write(sample_buildout, 'template', 'dosomething.sh.in', '''\
1358+ ... #!/bin/sh
1359+ ... ${shell-relative-path-setup}
1360+ ... cat ${buildout:directory|shell-path}
1361+ ... ''')
1362+
1363+ >>> write(sample_buildout, 'buildout.cfg',
1364+ ... """
1365+ ... [buildout]
1366+ ... parts = message
1367+ ... relative-paths = true
1368+ ...
1369+ ... [message]
1370+ ... recipe = z3c.recipe.filetemplate
1371+ ... source-directory = template
1372+ ... """)
1373+
1374+ >>> print system(buildout)
1375+ Uninstalling message.
1376+ Installing message.
1377+
1378+ >>> cat(sample_buildout, 'dosomething.py') # doctest: +ELLIPSIS
1379+ #!...
1380+ import os, imp
1381+ # Get path to this file.
1382+ if __name__ == '__main__':
1383+ _z3c_recipe_filetemplate_filename = __file__
1384+ else:
1385+ # If this is an imported module, we want the location of the .py
1386+ # file, not the .pyc, because the .py file may have been symlinked.
1387+ _z3c_recipe_filetemplate_filename = imp.find_module(__name__)[1]
1388+ # Get the full, non-symbolic-link directory for this file.
1389+ _z3c_recipe_filetemplate_base = os.path.dirname(
1390+ os.path.abspath(os.path.realpath(_z3c_recipe_filetemplate_filename)))
1391+ # This is the buildout root.
1392+ def _z3c_recipe_filetemplate_path_repr(path):
1393+ "Return absolute version of buildout-relative path."
1394+ return os.path.join(_z3c_recipe_filetemplate_base, path)
1395+ <BLANKLINE>
1396+ root = _z3c_recipe_filetemplate_path_repr('.')
1397+
1398+ >>> cat(sample_buildout, 'dosomething.sh') # doctest: +ELLIPSIS
1399+ #!/bin/sh
1400+ # Get full, non-symbolic-link path to this file.
1401+ Z3C_RECIPE_FILETEMPLATE_FILENAME=`\
1402+ readlink -f "$0" 2>/dev/null || \
1403+ realpath "$0" 2>/dev/null || \
1404+ type -P "$0" 2>/dev/null`
1405+ # Get directory of file.
1406+ Z3C_RECIPE_FILETEMPLATE_BASE=`dirname ${Z3C_RECIPE_FILETEMPLATE_FILENAME}`
1407+ # This is the buildout root.
1408+ <BLANKLINE>
1409+ cat "$Z3C_RECIPE_FILETEMPLATE_BASE"/.

Subscribers

People subscribed via source and target branches

to all changes: