Merge lp:~saviq/charm-helpers/confg-patterns into lp:charm-helpers
- confg-patterns
- Merge into devel
Status: | Rejected |
---|---|
Rejected by: | Stuart Bishop |
Proposed branch: | lp:~saviq/charm-helpers/confg-patterns |
Merge into: | lp:charm-helpers |
Diff against target: |
391 lines (+217/-15) 3 files modified
charmhelpers/contrib/openstack/context.py (+9/-0) charmhelpers/contrib/openstack/templating.py (+96/-11) tests/contrib/openstack/test_os_templating.py (+112/-4) |
To merge this branch: | bzr merge lp:~saviq/charm-helpers/confg-patterns |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Stuart Bishop (community) | Disapprove | ||
Review via email: mp+260750@code.launchpad.net |
Commit message
[saviq] Add support for config file patterns
Description of the change
- 384. By Michał Sawicz
-
Fix lint
- 385. By Michał Sawicz
-
Fix for python3
Corey Bryant (corey.bryant) wrote : | # |
Michał Sawicz (saviq) wrote : | # |
> I know the initial target is for the horizon subordinate plugin charms
> to use this, but would you mind giving an example of how horizon would use it?
Sure. The charm would register a pattern config:
> register_
Then,
> class RouterSettingCo
> def __call__(self):
> ctxt = {
> (40, 'router'): {'DISABLED': False if config('profile') in ['cisco'] else True}
> }
> return ctxt
and, for example
> class PluginContext(
> def __call__(self):
> # ...
> ctxt = {}
> for rid in relation_
> # ...
> rdata = relation_
> ctxt[(rdata[
> 'DISABLED': rdata['DISABLED'],
> # ...
> return ctxt
This would result in generating files, say:
> /path/to/
> /path/to/
In PluginContext I can then warn about conflicts etc. Multiple subordinates could even override one another.
HTH, let me know if I should explain more of this in the docstring.
- 386. By Michał Sawicz
-
Fix a bug in pattern context generation and add some more comments
Corey Bryant (corey.bryant) wrote : | # |
> > I know the initial target is for the horizon subordinate plugin charms
> > to use this, but would you mind giving an example of how horizon would use
> it?
>
> Sure. The charm would register a pattern config:
>
> > register_
>
> Then,
>
> > class RouterSettingCo
> > def __call__(self):
> > ctxt = {
> > (40, 'router'): {'DISABLED': False if config('profile') in
> ['cisco'] else True}
> > }
> > return ctxt
>
> and, for example
> > class PluginContext(
> > def __call__(self):
> > # ...
> > ctxt = {}
> > for rid in relation_
> > # ...
> > rdata = relation_
> > ctxt[(rdata[
> rdata['DASHBOARD'],
> > 'DISABLED': rdata['DISABLED'],
> > # ...
> > return ctxt
>
> This would result in generating files, say:
>
> > /path/to/
> > /path/to/
>
> In PluginContext I can then warn about conflicts etc. Multiple subordinates
> could even override one another.
>
> HTH, let me know if I should explain more of this in the docstring.
Thanks for the explanation. I'm wondering if this can be solved without adding the pattern context code. I was just looking at neutron-
BASE_RESOURCE_MAP = OrderedDict([
(NEUTRON_CONF, {
}),
...
Can you do something similar for subordinate horizon charms? In other words, the subordinate charms would write the plugin file themselves.
For example. horizon subordinate 1 could have:
BASE_RESOURCE_MAP = OrderedDict([
(/
}),
...
and horizon subordinate 2 could have:
BASE_RESOURCE_MAP = OrderedDict([
(/
}),
...
You could make the precedence (ie. 40 and 99) more flexibly in the subordinate charm and allow them to be changed via a config option.
Michał Sawicz (saviq) wrote : | # |
@Corey,
Sure, the subordinate could write that file itself, if that's preferable. Unfortunately the remaining problem for the contrail use case is that before Juno there's no way to update HORIZON_CONFIG from the split files, so we need to amend the main settings.py file. But this isn't really related to this MP...
Michał Sawicz (saviq) wrote : | # |
I will update the horizon-contrail charm to do this, so this MP can be dropped, really.
Michał Sawicz (saviq) wrote : | # |
Hah, I can't change the status of this MP to Rejected, can you?
Stuart Bishop (stub) : | # |
Unmerged revisions
- 386. By Michał Sawicz
-
Fix a bug in pattern context generation and add some more comments
- 385. By Michał Sawicz
-
Fix for python3
- 384. By Michał Sawicz
-
Fix lint
- 383. By Michał Sawicz
-
Refactor test to disallow additional calls
- 382. By Michał Sawicz
-
Implement support for generating multiple config files based on a pattern
- 381. By Michał Sawicz
-
Add context and templating classes to support generating multiple files based on a pattern
Preview Diff
1 | === modified file 'charmhelpers/contrib/openstack/context.py' |
2 | --- charmhelpers/contrib/openstack/context.py 2015-04-16 19:19:18 +0000 |
3 | +++ charmhelpers/contrib/openstack/context.py 2015-06-01 22:06:07 +0000 |
4 | @@ -194,6 +194,15 @@ |
5 | raise NotImplementedError |
6 | |
7 | |
8 | +class OSPatternContextGenerator(OSContextGenerator): |
9 | + """Base class for pattern context generators. |
10 | + |
11 | + __call__ should return a dictionary of { tuple: dict }, where the tuple |
12 | + will be used as input for format() for the filename pattern as registered |
13 | + by OSConfigRenderer.register_pattern(). |
14 | + """ |
15 | + |
16 | + |
17 | class SharedDBContext(OSContextGenerator): |
18 | interfaces = ['shared-db'] |
19 | |
20 | |
21 | === modified file 'charmhelpers/contrib/openstack/templating.py' |
22 | --- charmhelpers/contrib/openstack/templating.py 2015-01-22 06:06:03 +0000 |
23 | +++ charmhelpers/contrib/openstack/templating.py 2015-06-01 22:06:07 +0000 |
24 | @@ -14,7 +14,9 @@ |
25 | # You should have received a copy of the GNU Lesser General Public License |
26 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
27 | |
28 | +import glob |
29 | import os |
30 | +import string |
31 | |
32 | import six |
33 | |
34 | @@ -25,6 +27,7 @@ |
35 | INFO |
36 | ) |
37 | from charmhelpers.contrib.openstack.utils import OPENSTACK_CODENAMES |
38 | +from charmhelpers.contrib.openstack.context import OSPatternContextGenerator |
39 | |
40 | try: |
41 | from jinja2 import FileSystemLoader, ChoiceLoader, Environment, exceptions |
42 | @@ -96,7 +99,7 @@ |
43 | else: |
44 | self.contexts = contexts |
45 | |
46 | - self._complete_contexts = [] |
47 | + self._complete_contexts = set() |
48 | |
49 | def context(self): |
50 | ctxt = {} |
51 | @@ -105,9 +108,7 @@ |
52 | if _ctxt: |
53 | ctxt.update(_ctxt) |
54 | # track interfaces for every complete context. |
55 | - [self._complete_contexts.append(interface) |
56 | - for interface in context.interfaces |
57 | - if interface not in self._complete_contexts] |
58 | + self._complete_contexts.update(context.interfaces) |
59 | return ctxt |
60 | |
61 | def complete_contexts(self): |
62 | @@ -120,6 +121,41 @@ |
63 | return self._complete_contexts |
64 | |
65 | |
66 | +class OSPatternConfigTemplate(OSConfigTemplate): |
67 | + """ |
68 | + Associates a config pattern template with a list of context generators. |
69 | + Responsible for constructing a template context based on those generators. |
70 | + """ |
71 | + def __init__(self, pattern, contexts): |
72 | + self.pattern = pattern |
73 | + super(OSPatternConfigTemplate, self).__init__(config_file=None, |
74 | + contexts=contexts) |
75 | + |
76 | + def context(self): |
77 | + base_ctxt = {} |
78 | + ctxt = {} |
79 | + for context in self.contexts: |
80 | + _ctxt = context() |
81 | + if not _ctxt: |
82 | + continue |
83 | + elif isinstance(context, OSPatternContextGenerator): |
84 | + # for each returned key initialize its context with base_ctxt |
85 | + # if not defined before and update with new data |
86 | + for k, v in _ctxt.items(): |
87 | + if k not in ctxt: |
88 | + ctxt[k] = base_ctxt.copy() |
89 | + ctxt[k].update(v) |
90 | + else: |
91 | + # update the base context and all pre-existing file-specific |
92 | + # contexts |
93 | + base_ctxt.update(_ctxt) |
94 | + for key in ctxt: |
95 | + ctxt[key].update(_ctxt) |
96 | + # track interfaces for every complete context. |
97 | + self._complete_contexts.update(context.interfaces) |
98 | + return ctxt |
99 | + |
100 | + |
101 | class OSConfigRenderer(object): |
102 | """ |
103 | This class provides a common templating system to be used by OpenStack |
104 | @@ -132,6 +168,13 @@ |
105 | # import some common context generates from charmhelpers |
106 | from charmhelpers.contrib.openstack import context |
107 | |
108 | + # or create your own |
109 | + class SimpleContextGenerator(OSContextGenerator): |
110 | + def __call__(): |
111 | + return { |
112 | + 'key': 'value' |
113 | + } |
114 | + |
115 | # Create a renderer object for a specific OS release. |
116 | configs = OSConfigRenderer(templates_dir='/tmp/templates', |
117 | openstack_release='folsom') |
118 | @@ -142,12 +185,31 @@ |
119 | configs.register(config_file='/etc/nova/api-paste.ini', |
120 | contexts=[context.IdentityServiceContext()]) |
121 | configs.register(config_file='/etc/haproxy/haproxy.conf', |
122 | - contexts=[context.HAProxyContext()]) |
123 | + contexts=[context.HAProxyContext(), |
124 | + SimpleContextGenerator()]) |
125 | # write out a single config |
126 | configs.write('/etc/nova/nova.conf') |
127 | # write out all registered configs |
128 | configs.write_all() |
129 | |
130 | + Using patterns:: |
131 | + class DashboardContextGenerator(OSPatternContextGenerator): |
132 | + def __call__(): |
133 | + return { |
134 | + (40, 'router'): { 'DISABLED': True } |
135 | + } |
136 | + |
137 | + configs.register_pattern( |
138 | + pattern='/usr/share/openstack-dashboard/openstack_dashboard' |
139 | + '/enabled/_{}_juju_{}.py', |
140 | + contexts=[ |
141 | + DashboardContextGenerator() |
142 | + ]) |
143 | + # delete all files matching the pattern and |
144 | + # write _40_juju_router.py anew |
145 | + configs.write('/usr/share/openstack-dashboard/openstack_dashboard' |
146 | + '/enabled/_{}_juju_{}.py') |
147 | + |
148 | **OpenStack Releases and template loading** |
149 | |
150 | When the object is instantiated, it is associated with a specific OS |
151 | @@ -219,6 +281,15 @@ |
152 | contexts=contexts) |
153 | log('Registered config file: %s' % config_file, level=INFO) |
154 | |
155 | + def register_pattern(self, pattern, contexts): |
156 | + """ |
157 | + Register a config file name pattern with a list of context generators |
158 | + to be called during rendering. Use standard format() specification. |
159 | + """ |
160 | + self.templates[pattern] = OSPatternConfigTemplate(pattern=pattern, |
161 | + contexts=contexts) |
162 | + log('Registered config pattern: %s' % pattern, level=INFO) |
163 | + |
164 | def _get_tmpl_env(self): |
165 | if not self._tmpl_env: |
166 | loader = get_loader(self.templates_dir, self.openstack_release) |
167 | @@ -234,7 +305,8 @@ |
168 | if config_file not in self.templates: |
169 | log('Config not registered: %s' % config_file, level=ERROR) |
170 | raise OSConfigException |
171 | - ctxt = self.templates[config_file].context() |
172 | + ostemplate = self.templates[config_file] |
173 | + ctxt = ostemplate.context() |
174 | |
175 | _tmpl = os.path.basename(config_file) |
176 | try: |
177 | @@ -253,7 +325,14 @@ |
178 | raise e |
179 | |
180 | log('Rendering from template: %s' % _tmpl, level=INFO) |
181 | - return template.render(ctxt) |
182 | + |
183 | + if not isinstance(ostemplate, OSPatternConfigTemplate): |
184 | + ctxt = {(): ctxt} |
185 | + |
186 | + renders = {} |
187 | + for args, file_ctxt in ctxt.items(): |
188 | + renders[args] = template.render(file_ctxt) |
189 | + return renders |
190 | |
191 | def write(self, config_file): |
192 | """ |
193 | @@ -263,10 +342,16 @@ |
194 | log('Config not registered: %s' % config_file, level=ERROR) |
195 | raise OSConfigException |
196 | |
197 | - _out = self.render(config_file) |
198 | - |
199 | - with open(config_file, 'wb') as out: |
200 | - out.write(_out) |
201 | + renders = self.render(config_file) |
202 | + |
203 | + files = glob.glob(''.join([t[0] + ('' if t[1] is None else '*') |
204 | + for t in string.Formatter().parse(config_file)])) |
205 | + for name in files: |
206 | + os.unlink(name) |
207 | + |
208 | + for args, render in renders.items(): |
209 | + with open(config_file.format(*args), 'wb') as out: |
210 | + out.write(render) |
211 | |
212 | log('Wrote template %s.' % config_file, level=INFO) |
213 | |
214 | |
215 | === modified file 'tests/contrib/openstack/test_os_templating.py' |
216 | --- tests/contrib/openstack/test_os_templating.py 2015-02-11 21:41:57 +0000 |
217 | +++ tests/contrib/openstack/test_os_templating.py 2015-06-01 22:06:07 +0000 |
218 | @@ -2,9 +2,11 @@ |
219 | import os |
220 | import unittest |
221 | |
222 | -from mock import patch, call, MagicMock |
223 | +from mock import patch, call, Mock, MagicMock |
224 | |
225 | import charmhelpers.contrib.openstack.templating as templating |
226 | +from charmhelpers.contrib.openstack.context import OSPatternContextGenerator,\ |
227 | + OSContextGenerator |
228 | |
229 | from jinja2.exceptions import TemplateNotFound |
230 | |
231 | @@ -15,7 +17,7 @@ |
232 | builtin_open = 'builtins.open' |
233 | |
234 | |
235 | -class FakeContextGenerator(object): |
236 | +class FakeContextGenerator(OSContextGenerator): |
237 | interfaces = None |
238 | |
239 | def set(self, interfaces, context): |
240 | @@ -26,6 +28,10 @@ |
241 | return self.context |
242 | |
243 | |
244 | +class FakePatternContextGenerator(FakeContextGenerator, OSPatternContextGenerator): |
245 | + pass |
246 | + |
247 | + |
248 | class FakeLoader(object): |
249 | def set(self, template): |
250 | self.template = template |
251 | @@ -55,6 +61,7 @@ |
252 | path = os.path.dirname(__file__) |
253 | self.loader = FakeLoader() |
254 | self.context = FakeContextGenerator() |
255 | + self.pattern_context = FakePatternContextGenerator() |
256 | |
257 | self.addCleanup(patch.object(templating, 'apt_install').start().stop()) |
258 | self.addCleanup(patch.object(templating, 'log').start().stop()) |
259 | @@ -167,9 +174,11 @@ |
260 | '''It renders template if it finds it by config file basename''' |
261 | |
262 | @patch(builtin_open) |
263 | + @patch('glob.glob') |
264 | @patch.object(templating, 'get_loader') |
265 | - def test_write_out_config(self, loader, _open): |
266 | + def test_write_out_config(self, loader, _glob, _open): |
267 | '''It writes a templated config when provided a complete context''' |
268 | + _glob.return_value = [] |
269 | self.context.set(interfaces=['fooservice'], context={'foo': 'bar'}) |
270 | self.renderer.register('/tmp/foo', [self.context]) |
271 | with patch.object(self.renderer, '_get_template') as _get_t: |
272 | @@ -178,8 +187,10 @@ |
273 | self.renderer.write('/tmp/foo') |
274 | _open.assert_called_with('/tmp/foo', 'wb') |
275 | |
276 | - def test_write_all(self): |
277 | + @patch('glob.glob') |
278 | + def test_write_all(self, _glob): |
279 | '''It writes out all configuration files at once''' |
280 | + _glob.return_value = [] |
281 | self.context.set(interfaces=['fooservice'], context={'foo': 'bar'}) |
282 | self.renderer.register('/tmp/foo', [self.context]) |
283 | self.renderer.register('/tmp/bar', [self.context]) |
284 | @@ -192,6 +203,53 @@ |
285 | self.assertEquals(sorted(ex_calls), sorted(_write.call_args_list)) |
286 | pass |
287 | |
288 | + @patch(builtin_open) |
289 | + @patch('glob.glob') |
290 | + @patch.object(templating, 'get_loader') |
291 | + def test_write_out_config_pattern(self, loader, _glob, _open): |
292 | + '''It writes a templated pattern config when provided a complete context''' |
293 | + self.pattern_context.set(interfaces=[], context={(1,): {'foo': 'bar'}, |
294 | + (2,): {'foo': 'baz'}}) |
295 | + self.renderer.register_pattern('/tmp/foo_{}', [self.pattern_context]) |
296 | + _glob.return_value = [] |
297 | + |
298 | + ex_calls = [ |
299 | + call('/tmp/foo_1', 'wb'), |
300 | + call('/tmp/foo_2', 'wb') |
301 | + ] |
302 | + |
303 | + with patch.object(self.renderer, '_get_template') as _get_t: |
304 | + fake_tmpl = MockTemplate() |
305 | + _get_t.return_value = fake_tmpl |
306 | + self.renderer.write('/tmp/foo_{}') |
307 | + self.assertEquals(sorted(ex_calls), sorted(_open.call_args_list)) |
308 | + |
309 | + @patch(builtin_open) |
310 | + @patch('os.unlink') |
311 | + @patch('glob.glob') |
312 | + @patch.object(templating, 'get_loader') |
313 | + def test_write_out_config_pattern_clean(self, loader, _glob, _unlink, _open): |
314 | + '''It deletes all matching files prior to writing new ones''' |
315 | + self.pattern_context.set(interfaces=[], context={(1,): {'foo': 'bar'}}) |
316 | + self.renderer.register_pattern('/tmp/foo_{}', [self.pattern_context]) |
317 | + _glob.return_value = ['/tmp/foo_1', '/tmp/foo_2'] |
318 | + |
319 | + m = Mock() |
320 | + m.attach_mock(_open, 'open') |
321 | + m.attach_mock(_unlink, 'unlink') |
322 | + m.attach_mock(_glob, 'glob') |
323 | + |
324 | + with patch.object(self.renderer, '_get_template') as _get_t: |
325 | + fake_tmpl = MockTemplate() |
326 | + _get_t.return_value = fake_tmpl |
327 | + self.renderer.write('/tmp/foo_{}') |
328 | + m.assert_has_calls([ |
329 | + call.glob('/tmp/foo_*'), |
330 | + call.unlink('/tmp/foo_1'), |
331 | + call.unlink('/tmp/foo_2'), |
332 | + call.open('/tmp/foo_1', 'wb') |
333 | + ]) |
334 | + |
335 | @patch.object(templating, 'get_loader') |
336 | def test_reset_template_loader_for_new_os_release(self, loader): |
337 | self.loader.set('') |
338 | @@ -282,3 +340,53 @@ |
339 | tmpl = templating.OSConfigTemplate(config_file='/tmp/foo', |
340 | contexts=_c1) |
341 | self.assertEquals(tmpl.contexts, [_c1]) |
342 | + |
343 | + def test_generate_context(self): |
344 | + '''Ensure context is generated correctly''' |
345 | + def _c1(): |
346 | + return {'a': 1, 'b': 2} |
347 | + _c1.interfaces = [] |
348 | + |
349 | + def _c2(): |
350 | + return {'a': 3} |
351 | + _c2.interfaces = [] |
352 | + tmpl = templating.OSConfigTemplate(config_file='/tmp/foo', |
353 | + contexts=[_c1, _c2]) |
354 | + self.assertEqual(tmpl.context(), {'a': 3, 'b': 2}) |
355 | + |
356 | + def test_generate_pattern_context(self): |
357 | + class _c1(OSContextGenerator): |
358 | + def __call__(self): |
359 | + return {'a': 1, 'b': 2, 'c': 3, 'd': 4} |
360 | + |
361 | + class _c2(OSPatternContextGenerator): |
362 | + def __call__(self): |
363 | + return { |
364 | + ('foo',): {'a': 4}, |
365 | + ('bar',): {'b': 5} |
366 | + } |
367 | + |
368 | + class _c3(OSContextGenerator): |
369 | + def __call__(self): |
370 | + return {'d': 6} |
371 | + |
372 | + class _c4(OSPatternContextGenerator): |
373 | + def __call__(self): |
374 | + return { |
375 | + ('bar',): {'a': 7}, |
376 | + ('baz',): {} |
377 | + } |
378 | + |
379 | + class _c5(OSContextGenerator): |
380 | + def __call__(self): |
381 | + return {'e': 0} |
382 | + |
383 | + tmpl = templating.OSPatternConfigTemplate(pattern='/tmp/foo_{}', |
384 | + contexts=[_c1(), _c2(), |
385 | + _c3(), _c4(), |
386 | + _c5()]) |
387 | + self.assertEqual(tmpl.context(), { |
388 | + ('foo',): {'a': 4, 'b': 2, 'c': 3, 'd': 6, 'e': 0}, |
389 | + ('bar',): {'a': 7, 'b': 5, 'c': 3, 'd': 6, 'e': 0}, |
390 | + ('baz',): {'a': 1, 'b': 2, 'c': 3, 'd': 6, 'e': 0}, |
391 | + }) |
Hi Michal,
I know the initial target is for the horizon subordinate plugin charms to use this, but would you mind giving an example of how horizon would use it?
I have a couple of comments inline below.
Corey