Merge lp:~harlowja/cloud-init/cloud-file-write into lp:~cloud-init-dev/cloud-init/trunk

[Scott Moser]

cloud-config support for file writing

Fixes LP: #1012854 by implementing file writing, adjusts
other code to have user/group parsing in util instead
of in stages.py, renames decomp_str to decomp_gzip since
it is more meaningful when named that (as thats all it can
decompress).

[Scott Moser (smoser)]

Joshua, Thanks for the new feature.
A couple things:
 * I think we might as well just require 'compression' to be 'gzip', rather than supporting anything from is_true. having more things just makes more work in the future, and its no harder for someone to produce 'compression: gzip' than 'compression: 1'
 * it seems like the code right now only base64 decodes if the content is set to compressed. I'd like to do that seperately. It maybe that someone wants to simply produce base64 for all files they send through, rather than only for gzipped.
 * examples added in /doc would be nice.

Other than that, I think it looks great.

[Joshua Harlow (harlowja)]

Sounds good to me.

[lp:~harlowja/cloud-init/cloud-file-write]

597. By Joshua Harlow on 2012-07-11

Adjust the decoding of the files that are given so that a canonicalize
happens first, which will examine the incoming encoding, and decide the
neccasary decoding types needed to get the final resultant string and
then use these normalized decoding types to actually do the final decode.

Also change the name of the config key that is looked up to 'write_files'
since 'files' is pretty generic and could have clashes with other modules.

Add an example that shows how to use this in the different encoding formats
that are supported.

598. By Joshua Harlow on 2012-07-11

Fix log message after 'write_files' key change.

599. By Joshua Harlow on 2012-07-11

Update the write files with the new content/compression handling.

Adjust the examples file to reflect this.

=== added file 'cloudinit/config/cc_write_files.py'
--- cloudinit/config/cc_write_files.py	1970-01-01 00:00:00 +0000
+++ cloudinit/config/cc_write_files.py	2012-07-11 20:04:20 +0000
@@ -0,0 +1,102 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Joshua Harlow <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import base64
+import os
+
+from cloudinit import util
+from cloudinit.settings import PER_INSTANCE
+
+frequency = PER_INSTANCE
+
+DEFAULT_OWNER = "root:root"
+DEFAULT_PERMS = 0644
+UNKNOWN_ENC = 'text/plain'
+
+
+def handle(name, cfg, _cloud, log, _args):
+    files = cfg.get('write_files')
+    if not files:
+        log.debug(("Skipping module named %s,"
+                   " no/empty 'write_files' key in configuration"), name)
+        return
+    write_files(name, files, log)
+
+
+def canonicalize_extraction(compression_type, log):
+    if not compression_type:
+        compression_type = ''
+    compression_type = compression_type.lower().strip()
+    if compression_type in ['gz', 'gzip']:
+        return ['application/x-gzip']
+    if compression_type in ['gz+base64', 'gzip+base64', 'gz+b64', 'gzip+b64']:
+        return ['application/base64', 'application/x-gzip']
+    # Yaml already encodes binary data as base64 if it is given to the 
+    # yaml file as binary, so those will be automatically decoded for you.
+    # But the above b64 is just for people that are more 'comfortable'
+    # specifing it manually (which might be a possiblity)
+    if compression_type in ['b64', 'base64']:
+        return ['application/base64']
+    if compression_type:
+        log.warn("Unknown compression type %s, assuming %s",
+                 compression_type, UNKNOWN_ENC)
+    return [UNKNOWN_ENC]
+
+
+def write_files(name, files, log):
+    if not files:
+        return
+
+    for (i, f_info) in enumerate(files):
+        path = f_info.get('path')
+        if not path:
+            log.warn("No path provided to write for entry %s in module %s",
+                     i + 1, name)
+            continue
+        path = os.path.abspath(path)
+        extractions = canonicalize_extraction(f_info.get('compression'), log)
+        contents = extract_contents(f_info.get('content', ''), extractions)
+        (u, g) = util.extract_usergroup(f_info.get('owner', DEFAULT_OWNER))
+        perms = decode_perms(f_info.get('permissions'), DEFAULT_PERMS, log)
+        util.write_file(path, contents, mode=perms)
+        util.chownbyname(path, u, g)
+
+
+def decode_perms(perm, default, log):
+    try:
+        if isinstance(perm, (int, long, float)):
+            # Just 'downcast' it (if a float)
+            return int(perm)
+        else:
+            # Force to string and try octal conversion
+            return int(str(perm), 8)
+    except (TypeError, ValueError):
+        log.warn("Undecodable permissions %s, assuming %s", perm, default)
+        return default
+
+
+def extract_contents(contents, extraction_types):
+    result = str(contents)
+    for t in extraction_types:
+        if t == 'application/x-gzip':
+            result = util.decomp_gzip(result, quiet=False)
+        elif t == 'application/base64':
+            result = base64.b64decode(result)
+        elif t == UNKNOWN_ENC:
+            pass
+    return result
 
=== modified file 'cloudinit/stages.py'
--- cloudinit/stages.py	2012-07-09 19:08:27 +0000
+++ cloudinit/stages.py	2012-07-11 20:04:20 +0000
@@ -133,18 +133,7 @@
         if log_file:
             util.ensure_file(log_file)
             if perms:
-                perms_parted = perms.split(':', 1)
-                u = perms_parted[0]
-                if len(perms_parted) == 2:
-                    g = perms_parted[1]
-                else:
-                    g = ''
-                u = u.strip()
-                g = g.strip()
-                if u == "-1" or u.lower() == "none":
-                    u = None
-                if g == "-1" or g.lower() == "none":
-                    g = None
+                u, g = util.extract_usergroup(perms)
                 try:
                     util.chownbyname(log_file, u, g)
                 except OSError:
 
=== modified file 'cloudinit/user_data.py'
--- cloudinit/user_data.py	2012-06-23 05:04:37 +0000
+++ cloudinit/user_data.py	2012-07-11 20:04:20 +0000
@@ -227,7 +227,7 @@
         raw_data = ''
     if not headers:
         headers = {}
-    data = util.decomp_str(raw_data)
+    data = util.decomp_gzip(raw_data)
     if "mime-version:" in data[0:4096].lower():
         msg = email.message_from_string(data)
         for (key, val) in headers.iteritems():
 
=== modified file 'cloudinit/util.py'
--- cloudinit/util.py	2012-07-10 21:47:02 +0000
+++ cloudinit/util.py	2012-07-11 20:04:20 +0000
@@ -159,6 +159,10 @@
     pass
 
 
+class DecompressionError(Exception):
+    pass
+
+
 def ExtendedTemporaryFile(**kwargs):
     fh = tempfile.NamedTemporaryFile(**kwargs)
     # Replace its unlink with a quiet version
@@ -256,13 +260,32 @@
     return fn
 
 
-def decomp_str(data):
+def decomp_gzip(data, quiet=True):
     try:
         buf = StringIO(str(data))
         with contextlib.closing(gzip.GzipFile(None, "rb", 1, buf)) as gh:
             return gh.read()
-    except:
-        return data
+    except Exception as e:
+        if quiet:
+            return data
+        else:
+            raise DecompressionError(str(e))
+
+
+def extract_usergroup(ug_pair):
+    if not ug_pair:
+        return (None, None)
+    ug_parted = ug_pair.split(':', 1)
+    u = ug_parted[0].strip()
+    if len(ug_parted) == 2:
+        g = ug_parted[1].strip()
+    else:
+        g = None
+    if not u or u == "-1" or u.lower() == "none":
+        u = None
+    if not g or g == "-1" or g.lower() == "none":
+        g = None
+    return (u, g)
 
 
 def find_modules(root_dir):
 
=== added file 'doc/examples/cloud-config-write-files.txt'
--- doc/examples/cloud-config-write-files.txt	1970-01-01 00:00:00 +0000
+++ doc/examples/cloud-config-write-files.txt	2012-07-11 20:04:20 +0000
@@ -0,0 +1,37 @@
+#cloud-config
+# vim: syntax=yaml
+#
+# This is the configuration syntax that the write_files module
+# will know how to understand, it can be given b64, b32, b16, or 
+# gz (or gz+b64) encoded strings which will be decoded accordingly
+# and then written to the path that is provided. 
+#
+# Note: Content strings here are truncated for example purposes.
+#
+write_files:
+-   compression: b64
+    content: CiMgVGhpcyBmaWxlIGNvbnRyb2xzIHRoZSBzdGF0ZSBvZiBTRUxpbnV4...
+    owner: root:root
+    path: /etc/sysconfig/selinux
+    perms: '0644'
+-   content: '
+
+        # My new /etc/sysconfig/samba file
+
+        SMBDOPTIONS="-D"
+
+        '
+    path: /etc/sysconfig/samba
+-   content: !!binary |
+        f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAwARAAAAAAABAAAAAAAAAAJAVAAAAAAAAAAAAAEAAOAAI
+        AEAAHgAdAAYAAAAFAAAAQAAAAAAAAABAAEAAAAAAAEAAQAAAAAAAwAEAAAAAAADAAQAAAAAAAAgA
+        AAAAAAAAAwAAAAQAAAAAAgAAAAAAAAACQAAAAAAAAAJAAAAAAAAcAAAAAAAAABwAAAAAAAAAAQAA
+        ....
+    path: /bin/arch
+    perms: '0555'
+-   compression: gzip
+    content: !!binary |
+        H4sIAIDb/U8C/1NW1E/KzNMvzuBKTc7IV8hIzcnJVyjPL8pJ4QIA6N+MVxsAAAA=
+    path: /usr/bin/hello
+    perms: '0755'
+