Merge lp:~avishai-ish-shalom/cloud-init/chef-refactor into lp:~cloud-init-dev/cloud-init/trunk

Proposed by Avishai Ish-Shalom on 2013-05-15
Status: Rejected
Rejected by: Scott Moser on 2017-06-06
Proposed branch: lp:~avishai-ish-shalom/cloud-init/chef-refactor
Merge into: lp:~cloud-init-dev/cloud-init/trunk
Diff against target: 359 lines (+212/-45)
3 files modified
cloudinit/config/cc_chef.py (+189/-42)
doc/examples/cloud-config-chef.txt (+17/-2)
templates/chef_client.rb.tmpl (+6/-1)
To merge this branch: bzr merge lp:~avishai-ish-shalom/cloud-init/chef-refactor
Reviewer Review Type Date Requested Status
Scott Moser 2013-05-15 Needs Fixing on 2013-07-17
Review via email: mp+164009@code.launchpad.net

Description of the change

Refactored cc_chef, added omnibus and chef-solo support

To post a comment you must log in.
Scott Moser (smoser) wrote :

Avishai,
  Some comments. Sorry for taking so long to see this.
  * BIN_PATHS: I would really prefer that we just use PATH to look for executables. If the system is not set up to have PATH set correctly, the user should just use a full path. ie, the right way to get your executable found is to put it in PATH.
  * repo_path': '/var/lib/cloud/chef_repo'
    doesn't this fit better somewhere else ? I"m not entirely opposed to it, but it just seems like there is likely somewhere else to look. Also, if you're using /var/lib/cloud, please get it through 'cloud_dir' in the Paths object (cloud.paths).
  * some tests would be nice.

Thanks for the submission, sorry its taken so long to get around to it.

review: Needs Fixing

i'll review my patch and get back to you.

On Wed, Jul 17, 2013 at 11:09 PM, Scott Moser <email address hidden> wrote:

> Review: Needs Fixing
>
> Avishai,
> Some comments. Sorry for taking so long to see this.
> * BIN_PATHS: I would really prefer that we just use PATH to look for
> executables. If the system is not set up to have PATH set correctly, the
> user should just use a full path. ie, the right way to get your executable
> found is to put it in PATH.
> * repo_path': '/var/lib/cloud/chef_repo'
> doesn't this fit better somewhere else ? I"m not entirely opposed to
> it, but it just seems like there is likely somewhere else to look. Also,
> if you're using /var/lib/cloud, please get it through 'cloud_dir' in the
> Paths object (cloud.paths).
> * some tests would be nice.
>
> Thanks for the submission, sorry its taken so long to get around to it.
>
> --
>
> https://code.launchpad.net/~avishai-ish-shalom/cloud-init/chef-refactor/+merge/164009
> You are the owner of lp:~avishai-ish-shalom/cloud-init/chef-refactor.
>

Scott Moser (smoser) wrote :

Hello,
Thank you for taking the time to contribute to cloud-init. Cloud-init has moved its revision control system to git. As a result, we are marking all bzr merge proposals as 'rejected'. If you would like to re-submit this proposal for review, please do so by following the current HACKING documentation at http://cloudinit.readthedocs.io/en/latest/topics/hacking.html .

Unmerged revisions

818. By Avishai Ish-Shalom on 2013-05-15

cc_chef: gem install method should install build-essential

817. By Avishai Ish-Shalom on 2013-05-15

cc_chef: get_cookbooks from tarball or git

816. By Avishai Ish-Shalom on 2013-05-15

cc_chef: refactored, implemented omnibus, initial chef-solo support

TODO: Add chef-solo get_cookbooks variants

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'cloudinit/config/cc_chef.py'
2--- cloudinit/config/cc_chef.py 2012-12-12 15:39:43 +0000
3+++ cloudinit/config/cc_chef.py 2013-05-15 17:20:33 +0000
4@@ -20,10 +20,15 @@
5
6 import json
7 import os
8+import urllib
9
10 from cloudinit import templater
11 from cloudinit import url_helper
12 from cloudinit import util
13+from cloudinit.settings import PER_INSTANCE
14+from urlparse import urlparse
15+
16+frequency = PER_INSTANCE
17
18 RUBY_VERSION_DEFAULT = "1.8"
19
20@@ -38,20 +43,51 @@
21
22 OMNIBUS_URL = "https://www.opscode.com/chef/install.sh"
23
24+BIN_PATHS = (
25+ "/usr/bin", "/usr/local/bin",
26+ "/var/lib/gems/2.0.0/bin",
27+ "/var/lib/gems/1.9.1/bin",
28+ "/var/lib/gems/1.9/bin",
29+ "/var/lib/gems/1.8/bin"
30+)
31+DEFAULT_ARGS = {
32+ 'interval': ('-i', 1800),
33+ 'splay': ('-s', 300),
34+ 'daemonize': ('-d', True),
35+ 'fork': ('-f', True)
36+}
37+DEFAULT_CFG = {
38+ 'repo_path': '/var/lib/cloud/chef_repo',
39+ 'repo_source_type': 'tarball'
40+}
41+JSON_ATTRIB_FILE = '/etc/chef/firstboot.json'
42+
43
44 def handle(name, cfg, cloud, log, _args):
45-
46+ log.info("Starting cc_chef")
47 # If there isn't a chef key in the configuration don't do anything
48 if 'chef' not in cfg:
49 log.debug(("Skipping module named %s,"
50 " no 'chef' key in configuration"), name)
51 return
52- chef_cfg = cfg['chef']
53
54+ chef_cfg = dict(DEFAULT_CFG.items() + cfg['chef'].items())
55+ log.debug("Chef config: %r", chef_cfg)
56 # Ensure the chef directories we use exist
57 for d in CHEF_DIRS:
58 util.ensure_dir(d)
59
60+ if 'mode' in chef_cfg:
61+ chef_mode = chef_cfg['mode']
62+ else:
63+ if 'server_url' in chef_cfg and \
64+ ('validation_key' in chef_cfg or 'validation_cert' in chef_cfg):
65+ chef_mode = "client"
66+ else:
67+ chef_mode = "solo"
68+
69+ log.debug("Chef mode is %s", chef_mode)
70+
71 # Set the validation key based on the presence of either 'validation_key'
72 # or 'validation_cert'. In the case where both exist, 'validation_key'
73 # takes precedence
74@@ -61,80 +97,191 @@
75 break
76
77 # Create the chef config from template
78- template_fn = cloud.get_template_filename('chef_client.rb')
79- if template_fn:
80- iid = str(cloud.datasource.get_instance_id())
81- params = {
82- 'server_url': chef_cfg['server_url'],
83- 'node_name': util.get_cfg_option_str(chef_cfg, 'node_name', iid),
84- 'environment': util.get_cfg_option_str(chef_cfg, 'environment',
85- '_default'),
86- 'validation_name': chef_cfg['validation_name']
87- }
88- templater.render_to_file(template_fn, '/etc/chef/client.rb', params)
89- else:
90- log.warn("No template found, not rendering to /etc/chef/client.rb")
91+ if not write_chef_config(chef_cfg, cloud, chef_mode, log):
92+ return False
93
94 # set the firstboot json
95- initial_json = {}
96- if 'run_list' in chef_cfg:
97- initial_json['run_list'] = chef_cfg['run_list']
98- if 'initial_attributes' in chef_cfg:
99- initial_attributes = chef_cfg['initial_attributes']
100- for k in list(initial_attributes.keys()):
101- initial_json[k] = initial_attributes[k]
102- util.write_file('/etc/chef/firstboot.json', json.dumps(initial_json))
103+ write_json_attrib_file(chef_cfg)
104
105 # If chef is not installed, we install chef based on 'install_type'
106+ install_chef(chef_cfg, cloud)
107+
108+ if chef_mode == 'solo':
109+ get_cookbooks(chef_cfg, cloud)
110+
111+ chef_args = chef_args_from_cfg(chef_cfg, ['-j', JSON_ATTRIB_FILE])
112+ if util.get_cfg_option_bool(chef_cfg, 'autostart', default=True):
113+ run_chef(log, chef_mode, chef_args)
114+
115+
116+def install_chef(chef_cfg, cloud):
117 if (not os.path.isfile('/usr/bin/chef-client') or
118- util.get_cfg_option_bool(chef_cfg, 'force_install', default=False)):
119+ util.get_cfg_option_bool(chef_cfg, 'force_install', default=False)):
120
121 install_type = util.get_cfg_option_str(chef_cfg, 'install_type',
122 'packages')
123+ chef_version = util.get_cfg_option_str(chef_cfg, 'version', None)
124+
125 if install_type == "gems":
126 # this will install and run the chef-client from gems
127- chef_version = util.get_cfg_option_str(chef_cfg, 'version', None)
128 ruby_version = util.get_cfg_option_str(chef_cfg, 'ruby_version',
129 RUBY_VERSION_DEFAULT)
130- install_chef_from_gems(cloud.distro, ruby_version, chef_version)
131- # and finally, run chef-client
132- log.debug('Running chef-client')
133- util.subp(['/usr/bin/chef-client',
134- '-d', '-i', '1800', '-s', '20'], capture=False)
135+ ohai_version = util.get_cfg_option_str(chef_cfg, 'ohai_version', None)
136+ install_chef_from_gems(cloud.distro, ruby_version, chef_version, ohai_version)
137 elif install_type == 'packages':
138 # this will install and run the chef-client from packages
139 cloud.distro.install_packages(('chef',))
140 elif install_type == 'omnibus':
141 url = util.get_cfg_option_str(chef_cfg, "omnibus_url", OMNIBUS_URL)
142- content = url_helper.readurl(url=url, retries=5)
143+ content = url_helper.readurl(url=url, retries=5).contents
144 with util.tempdir() as tmpd:
145 # use tmpd over tmpfile to avoid 'Text file busy' on execute
146 tmpf = "%s/chef-omnibus-install" % tmpd
147 util.write_file(tmpf, content, mode=0700)
148- util.subp([tmpf], capture=False)
149+ args = []
150+ if chef_version:
151+ args.append("-v")
152+ args.append(chef_version)
153+ util.subp([tmpf] + args, capture=False)
154 else:
155- log.warn("Unknown chef install type %s", install_type)
156+ raise RuntimeError("Unknown chef install type %s" % install_type)
157+
158+
159+def write_chef_config(chef_cfg, cloud, chef_mode, log):
160+ "Write chef config file from template"
161+ template_fn = cloud.get_template_filename('chef_client.rb')
162+ cfg_filename = "/etc/chef/" + ("client.rb" if chef_mode == "client" else "solo.rb")
163+ if template_fn:
164+ iid = str(cloud.datasource.get_instance_id())
165+ params = {
166+ 'node_name': util.get_cfg_option_str(chef_cfg, 'node_name', iid),
167+ 'environment': util.get_cfg_option_str(chef_cfg, 'environment',
168+ '_default'),
169+ 'mode': chef_mode
170+ }
171+
172+ if chef_mode == "client":
173+ # server_url is required
174+ params['server_url'] = chef_cfg['server_url']
175+ params['validation_name'] = chef_cfg.get('validation_name', None)
176+ elif chef_mode == "solo":
177+ params['repo_path'] = chef_cfg['repo_path']
178+
179+ templater.render_to_file(template_fn, cfg_filename, params)
180+ return True
181+ else:
182+ log.warn("No template found, not rendering to %s", cfg_filename)
183+ return False
184+
185+
186+def write_json_attrib_file(chef_cfg):
187+ initial_json = {}
188+ if 'run_list' in chef_cfg:
189+ initial_json['run_list'] = chef_cfg['run_list']
190+ if 'initial_attributes' in chef_cfg:
191+ initial_attributes = chef_cfg['initial_attributes']
192+ for k in list(initial_attributes.keys()):
193+ initial_json[k] = initial_attributes[k]
194+ util.write_file(JSON_ATTRIB_FILE, json.dumps(initial_json))
195+
196+
197+def run_chef(log, chef_type, chef_args):
198+ chef_bin = "chef-%s" % chef_type
199+ chef_exec = None
200+ for path in BIN_PATHS:
201+ f = os.path.join(path, chef_bin)
202+ if os.path.isfile(f) and os.access(f, os.X_OK):
203+ chef_exec = f
204+ break
205+ if chef_exec is None:
206+ raise RuntimeError("Couldn't find chef executable for %s" % chef_bin)
207+ log.debug("Running %s", chef_exec)
208+ util.subp([chef_exec] + chef_args, capture=False)
209
210
211 def get_ruby_packages(version):
212 # return a list of packages needed to install ruby at version
213- pkgs = ['ruby%s' % version, 'ruby%s-dev' % version]
214+ pkgs = ['ruby%s' % version, 'ruby%s-dev' % version, 'build-essential']
215 if version == "1.8":
216 pkgs.extend(('libopenssl-ruby1.8', 'rubygems1.8'))
217 return pkgs
218
219
220-def install_chef_from_gems(ruby_version, chef_version, distro):
221+def install_chef_from_gems(distro, ruby_version, chef_version, ohai_version=None):
222 distro.install_packages(get_ruby_packages(ruby_version))
223+
224+ def gem_install(gem, version=None):
225+ cmd_args = ['/usr/bin/gem', 'install', gem]
226+ if version is not None:
227+ cmd_args.append('-v')
228+ cmd_args.append(version)
229+
230+ cmd_args += ['--no-rdoc', '--bindir', '/usr/bin', '-q']
231+
232+ return util.subp(cmd_args, capture=False)
233+
234 if not os.path.exists('/usr/bin/gem'):
235 util.sym_link('/usr/bin/gem%s' % ruby_version, '/usr/bin/gem')
236 if not os.path.exists('/usr/bin/ruby'):
237 util.sym_link('/usr/bin/ruby%s' % ruby_version, '/usr/bin/ruby')
238- if chef_version:
239- util.subp(['/usr/bin/gem', 'install', 'chef',
240- '-v %s' % chef_version, '--no-ri',
241- '--no-rdoc', '--bindir', '/usr/bin', '-q'], capture=False)
242+ if ohai_version:
243+ gem_install('ohai', ohai_version)
244+ gem_install('chef', chef_version)
245+
246+
247+def chef_args_from_cfg(cfg, extra_args=[]):
248+ merged_args = {}
249+ for (k, v) in DEFAULT_ARGS.iteritems():
250+ merged_args[k] = (v[0], cfg.get(k, v[1]))
251+
252+ if merged_args['daemonize'][1] is False:
253+ del merged_args['interval']
254+ del merged_args['splay']
255+
256+ args = []
257+ for (k, (arg, v)) in merged_args.iteritems():
258+ if type(v) == bool:
259+ if util.get_cfg_option_bool(cfg, k, v):
260+ args.append(arg)
261+ else:
262+ args.append(arg)
263+ args.append(str(cfg.get(k, v)))
264+
265+ return args + extra_args
266+
267+
268+def get_cookbooks(cfg, cloud):
269+ repo_source_type = util.get_cfg_option_str(
270+ cfg, 'repo_source_type', default='tarball')
271+ if repo_source_type == 'git':
272+ get_cookbooks_from_git(cfg, cloud)
273+ elif repo_source_type == 'tarball':
274+ get_cookbooks_from_tarball(cfg)
275 else:
276- util.subp(['/usr/bin/gem', 'install', 'chef',
277- '--no-ri', '--no-rdoc', '--bindir',
278- '/usr/bin', '-q'], capture=False)
279+ raise RuntimeError("Unknown cookbooks source type %s" % repo_source_type)
280+
281+
282+def get_cookbooks_from_git(cfg, cloud):
283+ cloud.distro.install_packages('git')
284+ enclosing_dir = os.path.dirname(cfg['repo_path'])
285+ if not os.path.isdir(enclosing_dir):
286+ os.makedirs(enclosing_dir)
287+ util.subp(['git', 'clone', '--recurse-submodules', cfg['repo_source'], cfg['repo_path']])
288+
289+
290+def get_cookbooks_from_tarball(cfg):
291+ with util.tempdir() as tmpd:
292+ filename = os.path.basename(urlparse(cfg['repo_source']).path)
293+ tmpfile = os.path.join(tmpd, filename)
294+ urllib.urlretrieve(cfg['repo_source'], tmpfile)
295+ if not os.path.isdir(cfg['repo_path']):
296+ os.makedirs(cfg['repo_path'])
297+ util.subp(['tar', '-xf', tmpfile, '-C', cfg['repo_path']])
298+
299+
300+def get_cookbooks_from_berkshelf(cfg):
301+ raise NotImplementedError()
302+
303+
304+def get_cookbooks_from_librarian(cfg):
305+ raise NotImplementedError()
306
307=== modified file 'doc/examples/cloud-config-chef.txt'
308--- doc/examples/cloud-config-chef.txt 2012-12-12 15:39:43 +0000
309+++ doc/examples/cloud-config-chef.txt 2013-05-15 17:20:33 +0000
310@@ -84,8 +84,23 @@
311 maxclients: 100
312 keepalive: "off"
313
314- # if install_type is 'omnibus', change the url to download
315- omnibus_url: "https://www.opscode.com/chef/install.sh"
316+ # For chef-solo, we want to download cookbooks. Currently git and tarball can be used
317+ # tarball/git should contain cookbooks, roles and data_bags folders
318+ #repo_source_type: git
319+ #repo_source: https://github.com/some-org/cookbooks-repo.git
320+ #repo_path: /var/lib/cloud/chef_repo
321+
322+ # if install_type is 'omnibus', change the url to the download script
323+ #omnibus_url: "https://www.opscode.com/chef/install.sh"
324+
325+ # Daemonize and run every 'interval'
326+ #daemonize: true
327+ #interval: 1800
328+ # Random wait before starting the run
329+ #splay: 300
330+
331+ # fork off a worker for every chef run, effective against memory leaks
332+ fork: true
333
334
335 # Capture all subprocess output into a logfile
336
337=== modified file 'templates/chef_client.rb.tmpl'
338--- templates/chef_client.rb.tmpl 2012-07-09 20:45:26 +0000
339+++ templates/chef_client.rb.tmpl 2013-05-15 17:20:33 +0000
340@@ -11,13 +11,18 @@
341 log_level :info
342 log_location "/var/log/chef/client.log"
343 ssl_verify_mode :verify_none
344+#if $mode == 'client'
345 validation_client_name "$validation_name"
346 validation_key "/etc/chef/validation.pem"
347 client_key "/etc/chef/client.pem"
348 chef_server_url "$server_url"
349 environment "$environment"
350+#else if $mode == 'solo'
351+cookbook_path "$repo_path/cookbooks"
352+data_bag_path "$repo_path/data_bags"
353+role_path "$repo_path/roles"
354+#end if
355 node_name "$node_name"
356-json_attribs "/etc/chef/firstboot.json"
357 file_cache_path "/var/cache/chef"
358 file_backup_path "/var/backups/chef"
359 pid_file "/var/run/chef/client.pid"