Merge lp:~gary/z3c.recipe.filetemplate/cleanup into lp:z3c.recipe.filetemplate

Proposed by Gary Poster on 2010-04-19
Status: Needs review
Proposed branch: lp:~gary/z3c.recipe.filetemplate/cleanup
Merge into: lp:z3c.recipe.filetemplate
Diff against target: 343 lines (+167/-18)
6 files modified
CHANGES.txt (+10/-1)
buildout.cfg (+7/-0)
z3c/recipe/filetemplate/README.txt (+38/-10)
z3c/recipe/filetemplate/__init__.py (+89/-5)
z3c/recipe/filetemplate/tests.py (+20/-0)
z3c/recipe/filetemplate/tests.txt (+3/-2)
To merge this branch: bzr merge lp:~gary/z3c.recipe.filetemplate/cleanup
Reviewer Review Type Date Requested Status
Francis J. Lacoste (community) 2010-04-19 Approve on 2010-04-19
Review via email: mp+23697@code.launchpad.net

Description of the change

This branch takes the parts of lp:~gary/z3c.recipe.filetemplate/support-system-python that are still pertinent after factoring out the parts that only had to do with the legacy system python support branches I had produced for zc.buildout. What remains has been already reviewed (https://code.edge.launchpad.net/~gary/z3c.recipe.filetemplate/support-system-python/+merge/8616). The changes support escaping "$" with "$$" in templates, and make tests less susceptible to timing errors.

To post a comment you must log in.
Gary Poster (gary) wrote :

Note that the XXX in the CHANGES file will be fixed up in the next branch.

review: Approve

Unmerged revisions

18. By Gary Poster on 2010-04-09

re-commit the pertinent work from lp:~gary/z3c.recipe.filetemplate/support-system-python

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'CHANGES.txt'
2--- CHANGES.txt 2009-07-02 18:29:12 +0000
3+++ CHANGES.txt 2010-04-19 19:40:32 +0000
4@@ -9,7 +9,16 @@
5 Features
6 --------
7
8-- None yet.
9+- Support escaping "${...}" with "$${...}" in templates. This is particularly
10+ useful for *NIX shell scripts.
11+
12+- Support the relative-paths buildout option. XXX describe design
13+
14+-----
15+Fixes
16+-----
17+
18+- Make tests less susceptible to timing errors.
19
20
21 2.0.3 (2009-07-02)
22
23=== modified file 'buildout.cfg'
24--- buildout.cfg 2007-09-30 19:14:00 +0000
25+++ buildout.cfg 2010-04-19 19:40:32 +0000
26@@ -1,7 +1,14 @@
27 [buildout]
28 develop = .
29 parts = test
30+ interpreter
31
32 [test]
33 recipe = zc.recipe.testrunner
34 eggs = z3c.recipe.filetemplate
35+
36+[interpreter]
37+recipe = zc.recipe.egg
38+interpreter = py
39+eggs = z3c.recipe.filetemplate
40+
41
42=== modified file 'z3c/recipe/filetemplate/README.txt'
43--- z3c/recipe/filetemplate/README.txt 2009-04-30 21:38:40 +0000
44+++ z3c/recipe/filetemplate/README.txt 2010-04-19 19:40:32 +0000
45@@ -40,7 +40,7 @@
46 ... world = Philipp
47 ... """)
48
49-After executing buildout, we can see that ``$world`` has indeed been
50+After executing buildout, we can see that ``${world}`` has indeed been
51 replaced by ``Philipp``:
52
53 >>> print system(buildout)
54@@ -49,6 +49,35 @@
55 >>> cat(sample_buildout, 'helloworld.txt')
56 Hello Philipp!
57
58+If you need to escape the ${...} pattern, you can do so by repeating the dollar
59+sign.
60+
61+ >>> update_file(sample_buildout, 'helloworld.txt.in',
62+ ... """
63+ ... Hello world! The double $${dollar-sign} escapes!
64+ ... """)
65+
66+ >>> print system(buildout)
67+ Uninstalling message.
68+ Installing message.
69+
70+ >>> cat(sample_buildout, 'helloworld.txt')
71+ Hello world! The double ${dollar-sign} escapes!
72+
73+Note that dollar signs alone, without curly braces, are not parsed.
74+
75+ >>> update_file(sample_buildout, 'helloworld.txt.in',
76+ ... """
77+ ... $Hello $$world! $$$profit!
78+ ... """)
79+
80+ >>> print system(buildout)
81+ Uninstalling message.
82+ Installing message.
83+
84+ >>> cat(sample_buildout, 'helloworld.txt')
85+ $Hello $$world! $$$profit!
86+
87 Note that the output file uses the same permission bits as found on the input
88 file.
89
90@@ -238,11 +267,11 @@
91 ... """
92 ... [buildout]
93 ... parts = message
94- ...
95+ ...
96 ... [template_defaults]
97 ... mygreeting = Hi
98 ... myaudience = World
99- ...
100+ ...
101 ... [message]
102 ... recipe = z3c.recipe.filetemplate
103 ... files = helloworld.txt
104@@ -279,7 +308,7 @@
105
106 ``os-paths``
107 ``(os.pathsep).join(paths)``
108-
109+
110 ``string-paths``
111 ``', '.join(repr(p) for p in all_paths)``
112
113@@ -292,7 +321,7 @@
114 ... """
115 ... [buildout]
116 ... parts = message
117- ...
118+ ...
119 ... [message]
120 ... recipe = z3c.recipe.filetemplate
121 ... files = helloworld.txt
122@@ -341,7 +370,7 @@
123 ... """
124 ... [buildout]
125 ... parts = message
126- ...
127+ ...
128 ... [message]
129 ... recipe = z3c.recipe.filetemplate
130 ... files = helloworld.txt
131@@ -382,7 +411,7 @@
132 ... """
133 ... [buildout]
134 ... parts = message
135- ...
136+ ...
137 ... [message]
138 ... recipe = z3c.recipe.filetemplate
139 ... files = helloworld.txt
140@@ -392,7 +421,7 @@
141 ... silly-range = repr(range(5))
142 ... first-interpreted-option
143 ... message-reversed-is-egassem
144- ... first-interpreted-option =
145+ ... first-interpreted-option =
146 ... options['interpreted-options'].split()[0].strip()
147 ... message-reversed-is-egassem=
148 ... ''.join(
149@@ -404,8 +433,7 @@
150 ... index = %(server)s/index
151 ... """ % dict(server=link_server))
152
153- >>> write(sample_buildout, 'helloworld.txt.in',
154- ... """
155+ >>> write(sample_buildout, 'helloworld.txt.in', """\
156 ... ${not-interpreted}!
157 ... duplicate-os-paths: ${duplicate-os-paths}
158 ... foo-paths: ${foo-paths}
159
160=== modified file 'z3c/recipe/filetemplate/__init__.py'
161--- z3c/recipe/filetemplate/__init__.py 2009-06-06 12:31:22 +0000
162+++ z3c/recipe/filetemplate/__init__.py 2010-04-19 19:40:32 +0000
163@@ -44,6 +44,17 @@
164 self.options.setdefault(key, value)
165 # set up paths for eggs, if given
166 if 'eggs' in self.options:
167+ relative_paths = self.options.get(
168+ 'relative-paths',
169+ buildout['buildout'].get('relative-paths', 'false')
170+ )
171+ if relative_paths not in ('true', 'false'):
172+ self._user_error(
173+ 'The relative-paths option must have the value of '
174+ 'true or false.')
175+ relative_paths = relative_paths == 'true'
176+ if relative_paths:
177+ raise NotImplementedError # XXX
178 self.eggs = zc.recipe.egg.Scripts(buildout, name, options)
179 orig_distributions, ws = self.eggs.working_set()
180 # we want ws, eggs.extra_paths, eggs._relative_paths
181@@ -179,6 +190,7 @@
182 'a %s.',
183 key, expression, evaluated, type(evaluated))
184 options[key] = evaluated
185+
186 def _user_error(self, msg, *args):
187 msg = msg % args
188 self.logger.error(msg)
189@@ -195,17 +207,19 @@
190 'Destinations already exist: %s. Please make sure that '
191 'you really want to generate these automatically. Then '
192 'move them away.', ', '.join(already_exists))
193+ seen = [] # We throw this away right now, but could move template
194+ # processing up to __init__ if valuable. That would mean that templates
195+ # would be rewritten even if a value in another section had been
196+ # referenced; however, it would also mean that __init__ would do
197+ # virtually all of the work, with install only doing the writing.
198 for rel_path, last_mod, st_mode in self.actions:
199 source = os.path.join(self.source_dir, rel_path)
200 dest = os.path.join(self.destination_dir, rel_path[:-3])
201 mode=stat.S_IMODE(st_mode)
202- template=open(source).read()
203- template=re.sub(r"\$\{([^:]+?)\}", r"${%s:\1}" % self.name,
204- template)
205- self._create_paths(os.path.dirname(dest))
206 # we process the file first so that it won't be created if there
207 # is a problem.
208- processed = self.options._sub(template, [])
209+ processed = Template(source).substitute(self, seen)
210+ self._create_paths(os.path.dirname(dest))
211 result=open(dest, "wt")
212 result.write(processed)
213 result.close()
214@@ -221,3 +235,73 @@
215
216 def update(self):
217 pass
218+
219+
220+class Template:
221+ # hacked from string.Template
222+ pattern = re.compile(r"""
223+ \$(?:
224+ \${(?P<escaped>[^}]*)} | # Escape sequence of two delimiters.
225+ {(?P<braced_single>[-a-z0-9 ._]+)} |
226+ # Delimiter and a braced local option
227+ {(?P<braced_double>[-a-z0-9 ._]+:[-a-z0-9 ._]+)} |
228+ # Delimiter and a braced fully
229+ # qualified option (that is, with
230+ # explicit section).
231+ {(?P<invalid>[^}]*}) # Other ill-formed delimiter exprs.
232+ )
233+ """, re.IGNORECASE | re.VERBOSE)
234+
235+ def __init__(self, source):
236+ self.source = source
237+ self.template = open(source).read()
238+
239+ def _get_colno_lineno(self, i):
240+ lines = self.template[:i].splitlines(True)
241+ if not lines:
242+ colno = 1
243+ lineno = 1
244+ else:
245+ colno = i - len(''.join(lines[:-1]))
246+ lineno = len(lines)
247+ return colno, lineno
248+
249+ def _get(self, options, section, option, seen, start):
250+ value = options.get(option, None, seen)
251+ if value is None:
252+ colno, lineno = self._get_colno_lineno(start)
253+ raise zc.buildout.buildout.MissingOption(
254+ "Option '%s:%s', referenced in line %d, col %d of %s, "
255+ "does not exist." %
256+ (section, option, lineno, colno, self.source))
257+ return value
258+
259+ def substitute(self, recipe, seen):
260+ def convert(mo):
261+ # Check the most common path first.
262+ option = mo.group('braced_single')
263+ if option is not None:
264+ val = self._get(recipe.options, recipe.name, option, seen,
265+ mo.start('braced_single'))
266+ # We use this idiom instead of str() because the latter will
267+ # fail if val is a Unicode containing non-ASCII characters.
268+ return '%s' % (val,)
269+ double = mo.group('braced_double')
270+ if double is not None:
271+ section, option = double.split(':')
272+ val = self._get(recipe.buildout[section], section, option, seen,
273+ mo.start('braced_double'))
274+ return '%s' % (val,)
275+ escaped = mo.group('escaped')
276+ if escaped is not None:
277+ return '${%s}' % (escaped,)
278+ invalid = mo.group('invalid')
279+ if invalid is not None:
280+ colno, lineno = self._get_colno_lineno(mo.start('invalid'))
281+ raise ValueError(
282+ 'Invalid placeholder %r in line %d, col %d of %s' %
283+ (mo.group('invalid'), lineno, colno, self.source))
284+ raise ValueError('Unrecognized named group in pattern',
285+ self.pattern) # programmer error, AFAICT
286+ return self.pattern.sub(convert, self.template)
287+
288
289=== modified file 'z3c/recipe/filetemplate/tests.py'
290--- z3c/recipe/filetemplate/tests.py 2009-04-30 17:56:10 +0000
291+++ z3c/recipe/filetemplate/tests.py 2010-04-19 19:40:32 +0000
292@@ -12,12 +12,32 @@
293 #
294 ##############################################################################
295
296+import os
297+import time
298 import zc.buildout.testing
299 import zc.buildout.tests
300 from zope.testing import doctest
301
302+
303+def update_file(dir, *args):
304+ """Update a file.
305+
306+ Make sure that the mtime of the file is updated so that buildout notices
307+ the changes. The resolution of mtime is system dependent, so we keep
308+ trying to write until mtime has actually changed."""
309+ path = os.path.join(dir, *(args[:-1]))
310+ original = os.stat(path).st_mtime
311+ while True:
312+ f = open(path, 'w')
313+ f.write(args[-1])
314+ f.flush()
315+ if os.stat(path).st_mtime != original:
316+ break
317+ time.sleep(0.2)
318+
319 def setUp(test):
320 zc.buildout.tests.easy_install_SetUp(test)
321+ test.globs['update_file'] = update_file
322 zc.buildout.testing.install_develop('z3c.recipe.filetemplate', test)
323
324 def test_suite():
325
326=== modified file 'z3c/recipe/filetemplate/tests.txt'
327--- z3c/recipe/filetemplate/tests.txt 2009-04-30 21:54:08 +0000
328+++ z3c/recipe/filetemplate/tests.txt 2010-04-19 19:40:32 +0000
329@@ -148,11 +148,12 @@
330 ... files = missing.txt
331 ... """)
332
333- >>> print system(buildout)
334+ >>> print system(buildout) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
335 Installing missing.
336 While:
337 Installing missing.
338- Error: Referenced option does not exist: missing world
339+ Error: Option 'missing:world', referenced in line 2, col 8 of
340+ .../sample-buildout/missing.txt.in, does not exist.
341
342 No changes means just an update
343 -------------------------------

Subscribers

People subscribed via source and target branches

to all changes: