Merge lp:~avishai-ish-shalom/cloud-init/chef into lp:~cloud-init-dev/cloud-init/trunk

[Avishai Ish-Shalom]

Added hostname prefix option, custom hostname attribute.
Added chef plugin.

[Scott Moser (smoser)]

Avishai,
  Thank you very much for taking the time to put this together. I'd like to merge this for Oneiric.

  I have a couple comments
a.) Why the change to cloudinit/CloudConfig/cc_set_hostname.py ?
    You summarized that as "Allow configurable hostname prefix and hostname attribute", but I don't initially see how its related to the rest of the commits. If its valuable, we can merge it, but I'd like that to be done separately if it is not related.
b.) You added 'tools/write-mime-multipart.py'
   This file has been moved to cloud-utils (http://launchpad.net/cloud-utils) .

  Thanks again, I'm looking forward to hearing back from you. Sorry for the slow reply.

review: Needs Information

[Avishai Ish-Shalom (avishai-ish-shalom)]

A. it's an unrelated feature. being a git man i thought a branch can be
done in the local repo with bazzar, then i got lazy and just left the
first commit. i've since saw the error of my ways and will branch a
different repo in the future (unless you can tell me what's the bazzar
equivalent of feature branch is).

On 18/07/11 18:29, Scott Moser wrote:
> Review: Needs Information
> Avishai,
> Thank you very much for taking the time to put this together. I'd like to merge this for Oneiric.
>
> I have a couple comments
> a.) Why the change to cloudinit/CloudConfig/cc_set_hostname.py ?
> You summarized that as "Allow configurable hostname prefix and hostname attribute", but I don't initially see how its related to the rest of the commits. If its valuable, we can merge it, but I'd like that to be done separately if it is not related.
> b.) You added 'tools/write-mime-multipart.py'
> This file has been moved to cloud-utils (http://launchpad.net/cloud-utils) .
>
> Thanks again, I'm looking forward to hearing back from you. Sorry for the slow reply.
>

[Scott Moser (smoser)]

On Mon, 18 Jul 2011, Avishai Ish-Shalom wrote:

> A. it's an unrelated feature.

OK, then I can just drop it in the merge, or you can modify it back.

[Mike Moulton (mmoulton)]

I seem to be unable to use the chef feature as included in the Oneiric Beta 1. I have noticed the following errors with this:

- If 'install_type' is undefined, the process fails

- 'firstboot.json' fails to write as it does not yet exist. Assume the missing 'w' param is needed.

- 'validation_cert' writes to '/etc/chef/validation.cert', however to the best of my knowledge chef-client does not know about this file. Instead, it's looking for '/etc/chef/validation.pem' as defined in the client.rb template.

- There seems to be a problem with 'subprocess.check_call([gem_bin', this failed for me, saying 'gem_bin' could not be found. Things work better if you do not install from gems.

I'm not familiar with how to submit patches via launchpad.net, otherwise I would have included fixes for these items.

I would love to get these items fixed for potential inclusion in Oneiric before it goes stable. Let me know what I can do to help.

review: Needs Fixing

[Avishai Ish-Shalom (avishai-ish-shalom)]

Fixed review rejects in 397.

review: Needs Resubmitting

[Mike Moulton (mmoulton)]

Avishai, thank you for the quick turn around. Your changes look like they will address all of my issues.

Scott, is it too late to get this into Oneiric?

review: Approve

[Scott Moser (smoser)]

On Thu, 8 Sep 2011, Mike Moulton wrote:

> Review: Approve
>
> Avishai, thank you for the quick turn around. Your changes look like they will address all of my issues.
>

Mike sorry for the run-around, but can you
a.) please actually test the code path. I can make you a cloud-init deb if needed.
  The cleanest path to do that is probably to
  - launch instance with no user-data
  - install new cloud-init
  - rm -Rf /var/lib/cloud
  - mkdir -p /var/lib/cloud/data/seed/nocloud-net
  - populate /var/lib/cloud/data/seed/nocloud-net/meta-data and user-data
    with the same user data you would have used
  - reboot
  - check
b.) open a bug for this? and describe what user-data you gave.

We will get this into oneiric.

[Mike Moulton (mmoulton)]

Scott, no problem, I will test this. If you could make a deb, that would help.

[Scott Moser (smoser)]

On Thu, 8 Sep 2011, Mike Moulton wrote:

> Scott, no problem, I will test this. If you could make a deb, that would help.

can you please try
http://people.canonical.com/~smoser/cloud-init_0.6.2~chef0_all.deb

[Mike Moulton (mmoulton)]

I was able to get this to work with the following patch to cc_chef.py:

root@ip-10-168-67-228:~# diff /usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/cc_chef.py /tmp/cc_chef.py
35c35
< install_type = cfg.get_cfg_option_str("install_type", "packages")
---
> install_type = util.get_cfg_option_str(chef_cfg, "install_type", "packages")
37c37
< if install_type = "gems":
---
> if install_type == "gems":
65,66c65,66
< firstboot_json_fh.write(runlist_item + "\n")
< firstboot_json_fh.write("]\n\}")
---
> firstboot_json_fh.write("\"" + runlist_item + "\"\n")
> firstboot_json_fh.write("]\n}")

Can either Scott or Avishai apply this?

I will create a bug as requested with the user data I used to get this to work. Please note, the user data will not be testable by others without their own valid chef server and validator key.

review: Needs Fixing

[Mike Moulton (mmoulton)]

I have submitted the following bug https://bugs.launchpad.net/cloud-init/+bug/845161 to address this issue.

=== added file 'cloudinit/CloudConfig/cc_chef.py'
--- cloudinit/CloudConfig/cc_chef.py	1970-01-01 00:00:00 +0000
+++ cloudinit/CloudConfig/cc_chef.py	2011-06-30 22:52:32 +0000
@@ -0,0 +1,80 @@
+# vi: ts=4 expandtab
+#
+#    Author: Avishai Ish-Shalom <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+import os
+import pwd
+import socket
+import subprocess
+import StringIO
+import ConfigParser
+import cloudinit.CloudConfig as cc
+import cloudinit.util as util
+
+ruby_packages = {'1.8': ('ruby', 'rubygems', 'ruby-dev', 'libopenssl-ruby'),
+        '1.9.1': ('ruby1.9.1', 'ruby1.9.1-dev', 'libruby1.9.1'),
+        '1.9': ('ruby1.9', 'ruby1.9-dev', 'libruby1.9') }
+
+def handle(name,cfg,cloud,log,args):
+    # If there isn't a chef key in the configuration don't do anything
+    if not cfg.has_key('chef'): return
+    chef_cfg = cfg['chef']
+
+    # Install chef packages from selected source
+    if not os.path.isfile('/usr/bin/chef-client'):
+        if chef_cfg['install_type'] == "gems":
+            if chef_cfg.has_key('version'):
+                chef_version = chef_cfg['version']
+            else:
+                chef_version = None
+            install_chef_from_gems(
+                    util.get_cfg_option_str(chef_cfg, 'ruby_version', '1.8'),
+                    chef_version)
+        else:
+            cc.install_packages(('chef',))
+
+    # set the validation cert
+    if chef_cfg.has_key('validation_cert'):
+        with open('/etc/chef/validation.cert', 'w') as validation_cert_fh:
+            validation_cert_fh.write(chef_cfg['validation_cert'])
+    
+    # create the chef config from template
+    util.render_to_file('chef_client.rb', '/etc/chef/client.rb',
+            {'server_url': chef_cfg['server_url'], 'validation_name': chef_cfg['validation_name'] || 'chef-validator'})
+
+    chef_args = ['-d']
+    # set the firstboot json
+    if chef_cfg.has_key('run_list'):
+        with open('/etc/chef/firstboot.json') as firstboot_json_fh:
+            firstboot_json_fh.write("{\n\"run_list\":\n[\n")
+            for runlist_item in chef_cfg['run_list']:
+                firstboot_json_fh.write(runlist_item + "\n")
+            firstboot_json_fh.write("]\n\}")
+        chef_args.append('-j /etc/chef/firstboot.json')
+
+    # and finally, run chef
+    subprocess.check_call(['/usr/bin/chef-client'] + chef_args)
+
+def install_chef_from_gems(ruby_version, chef_version = None):
+    cc.install_packages(ruby_packages[ruby_version])
+    chef_version_arg = ""
+    if chef_version: chef_version_arg = "-v %s" % chef_version
+    subprocess.check_call([gem_bin,'install','chef',chef_version_arg, '--no-ri','--no-rdoc','--no-test','-q'])
+    os.mkdirs('/etc/chef', '/var/log/chef', '/var/lib/chef', '/var/cache/chef', '/var/backups/chef', '/var/run/chef')
+    os.symlink('/var/lib/gem/%s/bin/chef-client' % ruby_version, '/usr/bin/chef-client')
+    # Ohai ruby plugin breaks if there is no ruby or gem binaries at /usr/bin, so
+    try: os.symlink('/usr/bin/gem%s' % ruby_version, '/usr/bin/gem')
+    except: pass
+    try: os.symlink('/usr/bin/ruby%s' % ruby_version, '/usr/bin/ruby')
+    except: pass

=== modified file 'cloudinit/CloudConfig/cc_set_hostname.py'
--- cloudinit/CloudConfig/cc_set_hostname.py	2011-01-26 14:03:46 +0000
+++ cloudinit/CloudConfig/cc_set_hostname.py	2011-06-30 22:52:32 +0000
@@ -24,7 +24,12 @@
         return(True)
 
     try:
-        hostname = util.get_cfg_option_str(cfg,"hostname",cloud.get_hostname())
+        hostname_prefix = util.get_cfg_option_str(cfg, "hostname_prefix", None)
+        hostname_attr = util.get_cfg_option_str(cfg, "hostname_attribute", "hostname")
+        hostname_function = getattr(cloud, 'get_' + hostname_attr, None)
+        if hostname_fucntion is None: hostname_fucntion = cloud.get_hostname
+        hostname = util.get_cfg_option_str(cfg,"hostname", hostname_function)
+        if hostname_prefix: hostname = hostname_prefix + "-" + hostname
         set_hostname(hostname, log)
     except Exception as e:
         util.logexc(log)

=== added file 'doc/examples/cloud-config-chef.txt'
--- doc/examples/cloud-config-chef.txt	1970-01-01 00:00:00 +0000
+++ doc/examples/cloud-config-chef.txt	2011-06-30 22:52:32 +0000
@@ -0,0 +1,38 @@
+#cloud-config
+#
+# This is an example file to automatically setup and run puppetd
+# when the instance boots for the first time.
+# Make sure that this file is valid yaml before starting instances.
+# It should be passed as user-data when starting the instance.
+
+# The default is to install from packages. If you want the latest packages from Opscode, be sure to add their repo:
+apt_mirror: http://apt.opscode.com/
+
+chef:
+ # If you want to install from rubygems:
+ install_type: "gems"
+
+ # Chef settings
+ server_url: "https://chef.yourorg.com:4000"
+
+ # Default validation name is chef-validator
+ validation_name: "yourorg-validator"
+ validation_cert: |
+     -----BEGIN CERTIFICATE-----
+     MIICCTCCAXKgAwIBAgIBATANBgkqhkiG9w0BAQUFADANMQswCQYDVQQDDAJjYTAe
+     Fw0xMDAyMTUxNzI5MjFaFw0xNTAyMTQxNzI5MjFaMA0xCzAJBgNVBAMMAmNhMIGf
+     MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCu7Q40sm47/E1Pf+r8AYb/V/FWGPgc
+     b014OmNoX7dgCxTDvps/h8Vw555PdAFsW5+QhsGr31IJNI3kSYprFQcYf7A8tNWu
+     1MASW2CfaEiOEi9F1R3R4Qlz4ix+iNoHiUDTjazw/tZwEdxaQXQVLwgTGRwVa+aA
+     qbutJKi93MILLwIDAQABo3kwdzA4BglghkgBhvhCAQ0EKxYpUHVwcGV0IFJ1Ynkv
+     T3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwDwYDVR0TAQH/BAUwAwEB/zAd
+     BgNVHQ4EFgQUu4+jHB+GYE5Vxo+ol1OAhevspjAwCwYDVR0PBAQDAgEGMA0GCSqG
+     SIb3DQEBBQUAA4GBAH/rxlUIjwNb3n7TXJcDJ6MMHUlwjr03BDJXKb34Ulndkpaf
+     +GAlzPXWa7bO908M9I8RnPfvtKnteLbvgTK+h+zX1XCty+S2EQWk29i2AdoqOTxb
+     hppiGMp0tT5Havu4aceCXiy2crVcudj3NFciy8X66SoECemW9UYDCb9T5D0d
+     -----END CERTIFICATE-----
+
+ # A run list for a first boot json
+ run_list:
+  - "recipe[apache2]"
+  - "role[db]"

=== added file 'templates/chef_client.rb.tmpl'
--- templates/chef_client.rb.tmpl	1970-01-01 00:00:00 +0000
+++ templates/chef_client.rb.tmpl	2011-06-30 22:52:32 +0000
@@ -0,0 +1,12 @@
+log_level          :info
+log_location       "/var/log/chef/client.log"
+ssl_verify_mode    :verify_none
+validation_client_name "$validation_name"
+validation_key         "/etc/chef/validation.pem"
+client_key               "/etc/chef/client.pem"
+chef_server_url    "$server_url"
+file_cache_path    "/var/cache/chef"
+file_backup_path  "/var/backups/chef"
+pid_file           "/var/run/chef/client.pid"
+Chef::Log::Formatter.show_time = true
+

=== added file 'tools/write-mime-multipart.py'
--- tools/write-mime-multipart.py	1970-01-01 00:00:00 +0000
+++ tools/write-mime-multipart.py	2011-06-30 22:52:32 +0000
@@ -0,0 +1,51 @@
+#! /usr/bin/env python
+
+import sys, os
+import email
+import mimetypes
+import re
+
+mimetypes.types_map['.sh'] = 'text/x-shellscript'
+cloud_config_mark_strings = { '#!': 'text/x-shellscript', '#include': 'text/x-include-url',
+    '#cloud-config': 'text/cloud-config', '#upstart-job': 'text/upstart-job',
+    '#cloud-boothook': 'text/cloud-boothook'
+    }
+def write_mime_multipart():
+    multipart_msg = email.mime.Multipart.MIMEMultipart()
+    for arg in sys.argv[1:]:
+        if ',' in arg:
+            (msg_file, msg_type) = arg.split(',')
+        else:
+            msg_file = arg
+            msg_type = None
+
+        msg_file = os.path.expanduser(msg_file)
+        if not os.path.isfile(msg_file):
+            print >> sys.stderr, "Can't find file %s" % arg
+            exit(1)
+
+        if not msg_type: msg_type = get_type_from_file(arg)
+        msg = email.mime.base.MIMEBase(*msg_type.split('/'))
+        msg.set_payload(open(msg_file, 'r').read())
+        multipart_msg.attach(msg)
+
+    print multipart_msg.as_string()
+
+def get_type_from_file(filename):
+    first_line = open(filename).readline()
+    m = re.match('Content-Type: (\w+/\w+)', first_line)
+    if m:
+        return m.groups[1]
+    else:
+        for mark_string, mime_type in cloud_config_mark_strings.items():
+            if first_line.startswith(mark_string):
+                return mime_type
+    return mimetypes.guess_type(filename)[0] or 'text/plain'
+
+if __name__ == '__main__':
+    if len(sys.argv) == 1 or '-h' in sys.argv or '--help' in sys.argv:
+        print "Usage: %s file1,application/cloud-config file2.sh ..." % os.path.basename(sys.argv[0])
+        print "MIME Multipart message will be written to STDOUT"
+        exit(0)
+    write_mime_multipart()
+