Merge lp:~michael.nelson/charm-helpers/ansible-support into lp:charm-helpers

Proposed by Michael Nelson
Status: Merged
Merged at revision: 65
Proposed branch: lp:~michael.nelson/charm-helpers/ansible-support
Merge into: lp:charm-helpers
Prerequisite: lp:~michael.nelson/charm-helpers/namespace-relation-data
Diff against target: 545 lines (+346/-62)
7 files modified
charmhelpers/contrib/ansible/__init__.py (+101/-0)
charmhelpers/contrib/saltstack/__init__.py (+17/-14)
charmhelpers/fetch/__init__.py (+1/-1)
setup.py (+1/-0)
tests/contrib/ansible/test_ansible.py (+136/-0)
tests/contrib/saltstack/test_saltstates.py (+87/-46)
tests/fetch/test_fetch.py (+3/-1)
To merge this branch: bzr merge lp:~michael.nelson/charm-helpers/ansible-support
Reviewer Review Type Date Requested Status
Matthew Wedgwood (community) Approve
Review via email: mp+176973@code.launchpad.net

Commit message

Add support for using ansible within charms.

Description of the change

Adds ansible support to charm-helpers.

I've converted a (private - sorry) charm to test out this support [1] - all works fine.

The juju config data, as well as any relation data is available to the ansible playbooks and templates using standard jinja2 syntax: {{ instance_type }}, where relation data is namespaced by the relation type, using django-like syntax: {{ solr__hostname }}.

One question I have is whether there's a better place for the juju_state_to_yaml() helper - it's now being used by both the saltstack and ansible support modules, and I assume it would be useful outside of those contexts also (to cache the juju config+relation data etc).

[1] https://code.launchpad.net/~michael.nelson/canonical-marshal/click-package-index-ansible/+merge/176972

To post a comment you must log in.
Revision history for this message
Matthew Wedgwood (mew) wrote :

Hey, Michael,

This looks awesome, thanks! I can't wait to try it out.

Regarding juju_state_to_yaml(), I'd say it belongs in core.hookenv. Bonus points for including a larger chunk of execution_environment().

-Matthew

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added directory 'charmhelpers/contrib/ansible'
2=== added file 'charmhelpers/contrib/ansible/__init__.py'
3--- charmhelpers/contrib/ansible/__init__.py 1970-01-01 00:00:00 +0000
4+++ charmhelpers/contrib/ansible/__init__.py 2013-07-25 15:55:36 +0000
5@@ -0,0 +1,101 @@
6+# Copyright 2013 Canonical Ltd.
7+#
8+# Authors:
9+# Charm Helpers Developers <juju@lists.ubuntu.com>
10+"""Charm Helpers ansible - declare the state of your machines.
11+
12+This helper enables you to declare your machine state, rather than
13+program it procedurally (and have to test each change to your procedures).
14+Your install hook can be as simple as:
15+
16+{{{
17+import charmhelpers.contrib.ansible
18+
19+
20+def install():
21+ charmhelpers.contrib.ansible.install_ansible_support()
22+ charmhelpers.contrib.ansible.apply_playbook('playbooks/install.yaml')
23+}}}
24+
25+and won't need to change (nor will its tests) when you change the machine
26+state.
27+
28+All of your juju config and relation-data are available as template
29+variables within your playbooks and templates. An install playbook looks
30+something like:
31+
32+{{{
33+---
34+- hosts: localhost
35+ user: root
36+
37+ tasks:
38+ - name: Add private repositories.
39+ template:
40+ src: ../templates/private-repositories.list.jinja2
41+ dest: /etc/apt/sources.list.d/private.list
42+
43+ - name: Update the cache.
44+ apt: update_cache=yes
45+
46+ - name: Install dependencies.
47+ apt: pkg={{ item }}
48+ with_items:
49+ - python-mimeparse
50+ - python-webob
51+ - sunburnt
52+
53+ - name: Setup groups.
54+ group: name={{ item.name }} gid={{ item.gid }}
55+ with_items:
56+ - { name: 'deploy_user', gid: 1800 }
57+ - { name: 'service_user', gid: 1500 }
58+
59+ ...
60+}}}
61+
62+Read more online about playbooks[1] and standard ansible modules[2].
63+
64+[1] http://www.ansibleworks.com/docs/playbooks.html
65+[2] http://www.ansibleworks.com/docs/modules.html
66+"""
67+import os
68+import subprocess
69+
70+import charmhelpers.contrib.saltstack
71+import charmhelpers.core.host
72+import charmhelpers.core.hookenv
73+import charmhelpers.fetch
74+
75+
76+charm_dir = os.environ.get('CHARM_DIR', '')
77+ansible_hosts_path = '/etc/ansible/hosts'
78+# Ansible will automatically include any vars in the following
79+# file in its inventory when run locally.
80+ansible_vars_path = '/etc/ansible/host_vars/localhost'
81+
82+
83+def install_ansible_support(from_ppa=True):
84+ """Installs the ansible package.
85+
86+ By default it is installed from the PPA [1] linked from
87+ the ansible website [2].
88+
89+ [1] https://launchpad.net/~rquillo/+archive/ansible
90+ [2] http://www.ansibleworks.com/docs/gettingstarted.html#ubuntu-and-debian
91+
92+ If from_ppa is false, you must ensure that the package is available
93+ from a configured repository.
94+ """
95+ if from_ppa:
96+ charmhelpers.fetch.add_source('ppa:rquillo/ansible')
97+ charmhelpers.core.host.apt_update(fatal=True)
98+ charmhelpers.core.host.apt_install('ansible')
99+ with open(ansible_hosts_path, 'w+') as hosts_file:
100+ hosts_file.write('localhost ansible_connection=local')
101+
102+
103+def apply_playbook(playbook):
104+ charmhelpers.contrib.saltstack.juju_state_to_yaml(
105+ ansible_vars_path, namespace_separator='__')
106+ subprocess.check_call(['ansible-playbook', '-c', 'local', playbook])
107
108=== modified file 'charmhelpers/contrib/saltstack/__init__.py'
109--- charmhelpers/contrib/saltstack/__init__.py 2013-07-11 13:38:36 +0000
110+++ charmhelpers/contrib/saltstack/__init__.py 2013-07-25 15:55:36 +0000
111@@ -94,7 +94,7 @@
112
113 def update_machine_state(state_path):
114 """Update the machine state using the provided state declaration."""
115- juju_config_2_grains()
116+ juju_state_to_yaml(salt_grains_path)
117 subprocess.check_call([
118 'salt-call',
119 '--local',
120@@ -103,8 +103,8 @@
121 ])
122
123
124-def juju_config_2_grains():
125- """Insert the juju config as salt grains for use in state templates.
126+def juju_state_to_yaml(yaml_path, namespace_separator=':'):
127+ """Update the juju config and state in a yaml file.
128
129 This includes any current relation-get data, and the charm
130 directory.
131@@ -121,7 +121,10 @@
132 if relation_type is not None:
133 relation_data = charmhelpers.core.hookenv.relation_get()
134 relation_data = dict(
135- ("{}:{}".format(relation_type, key), val)
136+ ("{relation_type}{namespace_separator}{key}".format(
137+ relation_type=relation_type.replace('-', '_'),
138+ key=key,
139+ namespace_separator=namespace_separator), val)
140 for key, val in relation_data.items())
141 config.update(relation_data)
142
143@@ -131,16 +134,16 @@
144 value: dumper.represent_scalar(
145 u'tag:yaml.org,2002:str', value))
146
147- grains_dir = os.path.dirname(salt_grains_path)
148- if not os.path.exists(grains_dir):
149- os.makedirs(grains_dir)
150+ yaml_dir = os.path.dirname(yaml_path)
151+ if not os.path.exists(yaml_dir):
152+ os.makedirs(yaml_dir)
153
154- if os.path.exists(salt_grains_path):
155- with open(salt_grains_path, "r") as grain_file:
156- grains = yaml.load(grain_file.read())
157+ if os.path.exists(yaml_path):
158+ with open(yaml_path, "r") as existing_vars_file:
159+ existing_vars = yaml.load(existing_vars_file.read())
160 else:
161- grains = {}
162+ existing_vars = {}
163
164- grains.update(config)
165- with open(salt_grains_path, "w+") as fp:
166- fp.write(yaml.dump(grains))
167+ existing_vars.update(config)
168+ with open(yaml_path, "w+") as fp:
169+ fp.write(yaml.dump(existing_vars))
170
171=== modified file 'charmhelpers/fetch/__init__.py'
172--- charmhelpers/fetch/__init__.py 2013-07-03 11:42:37 +0000
173+++ charmhelpers/fetch/__init__.py 2013-07-25 15:55:36 +0000
174@@ -27,7 +27,7 @@
175 def add_source(source, key=None):
176 if ((source.startswith('ppa:') or
177 source.startswith('http:'))):
178- subprocess.check_call(['add-apt-repository', source])
179+ subprocess.check_call(['add-apt-repository', '--yes', source])
180 elif source.startswith('cloud:'):
181 apt_install(filter_installed_packages(['ubuntu-cloud-keyring']),
182 fatal=True)
183
184=== modified file 'setup.py'
185--- setup.py 2013-06-19 09:54:19 +0000
186+++ setup.py 2013-07-25 15:55:36 +0000
187@@ -20,6 +20,7 @@
188 "charmhelpers.fetch",
189 "charmhelpers.payload",
190 "charmhelpers.contrib",
191+ "charmhelpers.contrib.ansible",
192 "charmhelpers.contrib.charmhelpers",
193 "charmhelpers.contrib.charmsupport",
194 "charmhelpers.contrib.saltstack",
195
196=== added directory 'tests/contrib/ansible'
197=== added file 'tests/contrib/ansible/__init__.py'
198=== added file 'tests/contrib/ansible/test_ansible.py'
199--- tests/contrib/ansible/test_ansible.py 1970-01-01 00:00:00 +0000
200+++ tests/contrib/ansible/test_ansible.py 2013-07-25 15:55:36 +0000
201@@ -0,0 +1,136 @@
202+# Copyright 2013 Canonical Ltd.
203+#
204+# Authors:
205+# Charm Helpers Developers <juju@lists.ubuntu.com>
206+import mock
207+import os
208+import shutil
209+import tempfile
210+import unittest
211+import yaml
212+
213+
214+import charmhelpers.contrib.ansible
215+
216+
217+class InstallAnsibleSupportTestCase(unittest.TestCase):
218+
219+ def setUp(self):
220+ super(InstallAnsibleSupportTestCase, self).setUp()
221+
222+ patcher = mock.patch('charmhelpers.fetch')
223+ self.mock_fetch = patcher.start()
224+ self.addCleanup(patcher.stop)
225+
226+ patcher = mock.patch('charmhelpers.core')
227+ self.mock_core = patcher.start()
228+ self.addCleanup(patcher.stop)
229+
230+
231+ hosts_file = tempfile.NamedTemporaryFile()
232+ self.ansible_hosts_path = hosts_file.name
233+ self.addCleanup(hosts_file.close)
234+ patcher = mock.patch.object(charmhelpers.contrib.ansible,
235+ 'ansible_hosts_path',
236+ self.ansible_hosts_path)
237+ patcher.start()
238+ self.addCleanup(patcher.stop)
239+
240+ def test_adds_ppa_by_default(self):
241+ charmhelpers.contrib.ansible.install_ansible_support()
242+
243+ self.mock_fetch.add_source.assert_called_once_with(
244+ 'ppa:rquillo/ansible')
245+ self.mock_core.host.apt_update.assert_called_once_with(fatal=True)
246+ self.mock_core.host.apt_install.assert_called_once_with(
247+ 'ansible')
248+
249+ def test_no_ppa(self):
250+ charmhelpers.contrib.ansible.install_ansible_support(
251+ from_ppa=False)
252+
253+ self.assertEqual(self.mock_fetch.add_source.call_count, 0)
254+ self.mock_core.host.apt_install.assert_called_once_with(
255+ 'ansible')
256+
257+ def test_writes_ansible_hosts(self):
258+ with open(self.ansible_hosts_path) as hosts_file:
259+ self.assertEqual(hosts_file.read(), '')
260+
261+ charmhelpers.contrib.ansible.install_ansible_support()
262+
263+ with open(self.ansible_hosts_path) as hosts_file:
264+ self.assertEqual(hosts_file.read(),
265+ 'localhost ansible_connection=local')
266+
267+
268+class ApplyPlaybookTestCases(unittest.TestCase):
269+
270+ def setUp(self):
271+ super(ApplyPlaybookTestCases, self).setUp()
272+
273+ # Hookenv patches (a single patch to hookenv doesn't work):
274+ patcher = mock.patch('charmhelpers.core.hookenv.config')
275+ self.mock_config = patcher.start()
276+ self.addCleanup(patcher.stop)
277+ Serializable = charmhelpers.core.hookenv.Serializable
278+ self.mock_config.return_value = Serializable({})
279+ patcher = mock.patch('charmhelpers.core.hookenv.relation_get')
280+ self.mock_relation_get = patcher.start()
281+ self.mock_relation_get.return_value = {}
282+ self.addCleanup(patcher.stop)
283+ patcher = mock.patch('charmhelpers.core.hookenv.relation_type')
284+ self.mock_relation_type = patcher.start()
285+ self.mock_relation_type.return_value = None
286+ self.addCleanup(patcher.stop)
287+ patcher = mock.patch('charmhelpers.core.hookenv.local_unit')
288+ self.mock_local_unit = patcher.start()
289+ self.addCleanup(patcher.stop)
290+ self.mock_local_unit.return_value = {}
291+
292+ patcher = mock.patch('charmhelpers.contrib.ansible.subprocess')
293+ self.mock_subprocess = patcher.start()
294+ self.addCleanup(patcher.stop)
295+
296+ etc_dir = tempfile.mkdtemp()
297+ self.addCleanup(shutil.rmtree, etc_dir)
298+ self.vars_path = os.path.join(etc_dir, 'ansible', 'vars.yaml')
299+ patcher = mock.patch.object(charmhelpers.contrib.ansible,
300+ 'ansible_vars_path', self.vars_path)
301+ patcher.start()
302+ self.addCleanup(patcher.stop)
303+
304+
305+ def test_calls_ansible_playbook(self):
306+ charmhelpers.contrib.ansible.apply_playbook(
307+ 'playbooks/dependencies.yaml')
308+
309+ self.mock_subprocess.check_call.assert_called_once_with([
310+ 'ansible-playbook', '-c', 'local', 'playbooks/dependencies.yaml'])
311+
312+ def test_writes_vars_file(self):
313+ self.assertFalse(os.path.exists(self.vars_path))
314+ self.mock_config.return_value = charmhelpers.core.hookenv.Serializable({
315+ 'group_code_owner': 'webops_deploy',
316+ 'user_code_runner': 'ubunet',
317+ })
318+ self.mock_relation_type.return_value = 'wsgi-file'
319+ self.mock_relation_get.return_value = {
320+ 'relation_key1': 'relation_value1',
321+ 'relation_key2': 'relation_value2',
322+ }
323+
324+ charmhelpers.contrib.ansible.apply_playbook(
325+ 'playbooks/dependencies.yaml')
326+
327+ self.assertTrue(os.path.exists(self.vars_path))
328+ with open(self.vars_path, 'r') as vars_file:
329+ result = yaml.load(vars_file.read())
330+ self.assertEqual({
331+ "group_code_owner": "webops_deploy",
332+ "user_code_runner": "ubunet",
333+ "charm_dir": "",
334+ "local_unit": {},
335+ "wsgi_file__relation_key1": "relation_value1",
336+ "wsgi_file__relation_key2": "relation_value2",
337+ }, result)
338
339=== modified file 'tests/contrib/saltstack/test_saltstates.py'
340--- tests/contrib/saltstack/test_saltstates.py 2013-07-11 13:38:36 +0000
341+++ tests/contrib/saltstack/test_saltstates.py 2013-07-25 15:55:36 +0000
342@@ -9,7 +9,6 @@
343 import unittest
344 import yaml
345
346-import charmhelpers.core.hookenv
347 import charmhelpers.contrib.saltstack
348
349
350@@ -62,7 +61,7 @@
351 self.addCleanup(patcher.stop)
352
353 patcher = mock.patch('charmhelpers.contrib.saltstack.'
354- 'juju_config_2_grains')
355+ 'juju_state_to_yaml')
356 self.mock_config_2_grains = patcher.start()
357 self.addCleanup(patcher.stop)
358
359@@ -81,7 +80,7 @@
360 charmhelpers.contrib.saltstack.update_machine_state(
361 'states/install.yaml')
362
363- self.mock_config_2_grains.assert_called_once_with()
364+ self.mock_config_2_grains.assert_called_once_with('/etc/salt/grains')
365
366
367 class JujuConfig2GrainsTestCase(unittest.TestCase):
368@@ -108,56 +107,98 @@
369 etc_dir = tempfile.mkdtemp()
370 self.addCleanup(shutil.rmtree, etc_dir)
371 self.grain_path = os.path.join(etc_dir, 'salt', 'grains')
372- patcher = mock.patch.object(charmhelpers.contrib.saltstack,
373- 'salt_grains_path', self.grain_path)
374- patcher.start()
375- self.addCleanup(patcher.stop)
376
377 patcher = mock.patch.object(charmhelpers.contrib.saltstack,
378 'charm_dir', '/tmp/charm_dir')
379 patcher.start()
380 self.addCleanup(patcher.stop)
381
382- def test_output_without_relation(self):
383- self.mock_config.return_value = charmhelpers.core.hookenv.Serializable({
384- 'group_code_owner': 'webops_deploy',
385- 'user_code_runner': 'ubunet',
386- })
387- self.mock_local_unit.return_value = "click-index/3"
388-
389- charmhelpers.contrib.saltstack.juju_config_2_grains()
390-
391- with open(self.grain_path, 'r') as grain_file:
392- result = yaml.load(grain_file.read())
393- self.assertEqual({
394- "charm_dir": "/tmp/charm_dir",
395- "group_code_owner": "webops_deploy",
396- "user_code_runner": "ubunet",
397- "local_unit": "click-index/3",
398- }, result)
399+ def test_output_with_empty_relation(self):
400+ self.mock_config.return_value = {
401+ 'group_code_owner': 'webops_deploy',
402+ 'user_code_runner': 'ubunet',
403+ }
404+ self.mock_local_unit.return_value = "click-index/3"
405+
406+ charmhelpers.contrib.saltstack.juju_state_to_yaml(self.grain_path)
407+
408+ with open(self.grain_path, 'r') as grain_file:
409+ result = yaml.load(grain_file.read())
410+ self.assertEqual({
411+ "charm_dir": "/tmp/charm_dir",
412+ "group_code_owner": "webops_deploy",
413+ "user_code_runner": "ubunet",
414+ "local_unit": "click-index/3",
415+ }, result)
416+
417+ def test_output_with_no_relation(self):
418+ self.mock_config.return_value = {
419+ 'group_code_owner': 'webops_deploy',
420+ 'user_code_runner': 'ubunet',
421+ }
422+ self.mock_local_unit.return_value = "click-index/3"
423+ self.mock_relation_get.return_value = None
424+
425+ charmhelpers.contrib.saltstack.juju_state_to_yaml(self.grain_path)
426+
427+ with open(self.grain_path, 'r') as grain_file:
428+ result = yaml.load(grain_file.read())
429+ self.assertEqual({
430+ "charm_dir": "/tmp/charm_dir",
431+ "group_code_owner": "webops_deploy",
432+ "user_code_runner": "ubunet",
433+ "local_unit": "click-index/3",
434+ }, result)
435+
436
437 def test_output_with_relation(self):
438- self.mock_config.return_value = charmhelpers.core.hookenv.Serializable({
439- 'group_code_owner': 'webops_deploy',
440- 'user_code_runner': 'ubunet',
441- })
442- self.mock_relation_type.return_value = 'wsgi-file'
443- self.mock_relation_get.return_value = {
444- 'relation_key1': 'relation_value1',
445- 'relation_key2': 'relation_value2',
446- }
447- self.mock_local_unit.return_value = "click-index/3"
448-
449- charmhelpers.contrib.saltstack.juju_config_2_grains()
450-
451- with open(self.grain_path, 'r') as grain_file:
452- result = yaml.load(grain_file.read())
453- self.assertEqual({
454- "charm_dir": "/tmp/charm_dir",
455- "group_code_owner": "webops_deploy",
456- "user_code_runner": "ubunet",
457- "wsgi-file:relation_key1": "relation_value1",
458- "wsgi-file:relation_key2": "relation_value2",
459+ self.mock_config.return_value = {
460+ 'group_code_owner': 'webops_deploy',
461+ 'user_code_runner': 'ubunet',
462+ }
463+ self.mock_relation_type.return_value = 'wsgi-file'
464+ self.mock_relation_get.return_value = {
465+ 'relation_key1': 'relation_value1',
466+ 'relation_key2': 'relation_value2',
467+ }
468+ self.mock_local_unit.return_value = "click-index/3"
469+
470+ charmhelpers.contrib.saltstack.juju_state_to_yaml(self.grain_path)
471+
472+ with open(self.grain_path, 'r') as grain_file:
473+ result = yaml.load(grain_file.read())
474+ self.assertEqual({
475+ "charm_dir": "/tmp/charm_dir",
476+ "group_code_owner": "webops_deploy",
477+ "user_code_runner": "ubunet",
478+ "wsgi_file:relation_key1": "relation_value1",
479+ "wsgi_file:relation_key2": "relation_value2",
480+ "local_unit": "click-index/3",
481+ }, result)
482+
483+ def test_relation_with_separator(self):
484+ self.mock_config.return_value = {
485+ 'group_code_owner': 'webops_deploy',
486+ 'user_code_runner': 'ubunet',
487+ }
488+ self.mock_relation_type.return_value = 'wsgi-file'
489+ self.mock_relation_get.return_value = {
490+ 'relation_key1': 'relation_value1',
491+ 'relation_key2': 'relation_value2',
492+ }
493+ self.mock_local_unit.return_value = "click-index/3"
494+
495+ charmhelpers.contrib.saltstack.juju_state_to_yaml(
496+ self.grain_path, namespace_separator='__')
497+
498+ with open(self.grain_path, 'r') as grain_file:
499+ result = yaml.load(grain_file.read())
500+ self.assertEqual({
501+ "charm_dir": "/tmp/charm_dir",
502+ "group_code_owner": "webops_deploy",
503+ "user_code_runner": "ubunet",
504+ "wsgi_file__relation_key1": "relation_value1",
505+ "wsgi_file__relation_key2": "relation_value2",
506 "local_unit": "click-index/3",
507 }, result)
508
509@@ -180,7 +221,7 @@
510 })
511 self.mock_local_unit.return_value = "click-index/3"
512
513- charmhelpers.contrib.saltstack.juju_config_2_grains()
514+ charmhelpers.contrib.saltstack.juju_state_to_yaml(self.grain_path)
515
516 with open(self.grain_path, 'r') as grain_file:
517 result = yaml.load(grain_file.read())
518
519=== modified file 'tests/fetch/test_fetch.py'
520--- tests/fetch/test_fetch.py 2013-07-03 11:42:37 +0000
521+++ tests/fetch/test_fetch.py 2013-07-25 15:55:36 +0000
522@@ -34,6 +34,7 @@
523 source = "ppa:test-ppa"
524 fetch.add_source(source=source)
525 check_call.assert_called_with(['add-apt-repository',
526+ '--yes',
527 source])
528
529 @patch('subprocess.check_call')
530@@ -41,6 +42,7 @@
531 source = "http://archive.ubuntu.com/ubuntu raring-backports main"
532 fetch.add_source(source=source)
533 check_call.assert_called_with(['add-apt-repository',
534+ '--yes',
535 source])
536
537 @patch.object(fetch, 'filter_installed_packages')
538@@ -72,7 +74,7 @@
539 key = "akey"
540 fetch.add_source(source=source, key=key)
541 check_call.assert_has_calls([
542- call(['add-apt-repository', source]),
543+ call(['add-apt-repository', '--yes', source]),
544 call(['apt-key', 'import', key])
545 ])
546

Subscribers

People subscribed via source and target branches