Merge lp:~smoser/cloud-init/changeable-templates into lp:~harlowja/cloud-init/changeable-templates

[Scott Moser]

a couple things here

a.) merge with trunk (you can 'bzr merge lp:cloud-init' and get the same).
b.) use textwrap.dedent
c.) add some tests based on actually shipped templates that will need to pass for basic renderer.

[Joshua Harlow (harlowja)]

Seems pretty ok to me, some small comments that u can adjust if u want.

=== modified file 'ChangeLog'
--- ChangeLog	2014-02-27 15:51:22 +0000
+++ ChangeLog	2014-07-18 13:33:31 +0000
@@ -1,3 +1,12 @@
+0.7.6:
+ - open 0.7.6
+ - Enable vendordata on CloudSigma datasource (LP: #1303986)
+ - Poll on /dev/ttyS1 in CloudSigma datasource only if dmidecode says
+   we're running on cloudsigma (LP: #1316475) [Kiril Vladimiroff]
+ - SmartOS test: do not require existance of /dev/ttyS1. [LP: #1316597]
+ - doc: fix user-groups doc to reference plural ssh-authorized-keys
+   (LP: #1327065) [Joern Heissler]
+ - fix 'make test' in python 2.6
 0.7.5:
  - open 0.7.5
  - Add a debug log message around import failures
@@ -33,6 +42,14 @@
    rather than relying on EC2 data in openstack metadata service.
  - SmartOS, AltCloud: disable running on arm systems due to bug
    (LP: #1243287, #1285686) [Oleg Strikov]
+ - Allow running a command to seed random, default is 'pollinate -q'
+   (LP: #1286316) [Dustin Kirkland]
+ - Write status to /run/cloud-init/status.json for consumption by
+   other programs (LP: #1284439)
+ - Azure: if a reboot causes ephemeral storage to be re-provisioned
+   Then we need to re-format it. (LP: #1292648)
+ - OpenNebula: support base64 encoded user-data
+   [Enol Fernandez, Peter Kotcauer]
 0.7.4:
  - fix issue mounting 'ephemeral0' if ephemeral0 was an alias for a
    partitioned block device with target filesystem on ephemeral0.1.

=== renamed file 'TODO' => 'TODO.rst'
--- TODO	2012-07-10 03:32:50 +0000
+++ TODO.rst	2014-07-18 13:33:31 +0000
@@ -1,46 +1,43 @@
-- Consider a 'failsafe' DataSource
-  If all others fail, setting a default that
-   - sets the user password, writing it to console
-   - logs to console that this happened
-- Consider a 'previous' DataSource
-  If no other data source is found, fall back to the 'previous' one
-  keep a indication of what instance id that is in /var/lib/cloud
-- Rewrite "cloud-init-query" (currently not implemented)
-  Possibly have DataSource and cloudinit expose explicit fields
-   - instance-id
-   - hostname
-   - mirror
-   - release
-   - ssh public keys
+==============================================
+Things that cloud-init may do (better) someday
+==============================================
+
+- Consider making ``failsafe`` ``DataSource``
+  - sets the user password, writing it to console
+
+- Consider a ``previous`` ``DataSource``, if no other data source is
+  found, fall back to the ``previous`` one that worked.
+- Rewrite ``cloud-init-query`` (currently not implemented)
+- Possibly have a ``DataSource`` expose explicit fields:
+
+  - instance-id
+  - hostname
+  - mirror
+  - release
+  - ssh public keys
+
 - Remove the conversion of the ubuntu network interface format conversion
   to a RH/fedora format and replace it with a top level format that uses
   the netcf libraries format instead (which itself knows how to translate
-  into the specific formats)
-- Replace the 'apt*' modules with variants that now use the distro classes
-  to perform distro independent packaging commands (where possible)
-- Canonicalize the semaphore/lock name for modules and user data handlers
-  a. It is most likely a bug that currently exists that if a module in config
-     alters its name and it has already ran, then it will get ran again since
-     the lock name hasn't be canonicalized
+  into the specific formats). See for example `netcf`_ which seems to be
+  an active project that has this capability.
+- Replace the ``apt*`` modules with variants that now use the distro classes
+  to perform distro independent packaging commands (wherever possible).
 - Replace some the LOG.debug calls with a LOG.info where appropriate instead
-  of how right now there is really only 2 levels (WARN and DEBUG)
-- Remove the 'cc_' for config modules, either have them fully specified (ie
-  'cloudinit.config.resizefs') or by default only look in the 'cloudinit.config'
-  for these modules (or have a combination of the above), this avoids having
-  to understand where your modules are coming from (which can be altered by
-  the current python inclusion path)
-- Depending on if people think the wrapper around 'os.path.join' provided
-  by the 'paths' object is useful (allowing us to modify based off a 'read'
-  and 'write' configuration based 'root') or is just to confusing, it might be 
-  something to remove later, and just recommend using 'chroot' instead (or the X 
-  different other options which are similar to 'chroot'), which is might be more 
-  natural and less confusing...
-- Instead of just warning when a module is being ran on a 'unknown' distribution
-  perhaps we should not run that module in that case? Or we might want to start
-  reworking those modules so they will run on all distributions? Or if that is
-  not the case, then maybe we want to allow fully specified python paths for
-  modules and start encouraging packages of 'ubuntu' modules, packages of 'rhel'
-  specific modules that people can add instead of having them all under the 
-  cloud-init 'root' tree? This might encourage more development of other modules
-  instead of having to go edit the cloud-init code to accomplish this.
+  of how right now there is really only 2 levels (``WARN`` and ``DEBUG``)
+- Remove the ``cc_`` prefix for config modules, either have them fully
+  specified (ie ``cloudinit.config.resizefs``) or by default only look in
+  the ``cloudinit.config`` namespace for these modules (or have a combination
+  of the above), this avoids having to understand where your modules are
+  coming from (which can be altered by the current python inclusion path)
+- Instead of just warning when a module is being ran on a ``unknown``
+  distribution perhaps we should not run that module in that case? Or we might
+  want to start reworking those modules so they will run on all
+  distributions? Or if that is not the case, then maybe we want to allow
+  fully specified python paths for modules and start encouraging
+  packages of ``ubuntu`` modules, packages of ``rhel`` specific modules that
+  people can add instead of having them all under the  cloud-init ``root``
+  tree? This might encourage more development of other modules instead of
+  having to go edit the cloud-init code to accomplish this.
 
+.. _netcf: https://fedorahosted.org/netcf/

=== modified file 'bin/cloud-init'
--- bin/cloud-init	2014-01-09 00:16:24 +0000
+++ bin/cloud-init	2014-07-18 13:33:31 +0000
@@ -22,8 +22,11 @@
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import argparse
+import json
 import os
 import sys
+import time
+import tempfile
 import traceback
 
 # This is more just for running from the bin folder so that
@@ -126,11 +129,11 @@
                " under section '%s'") % (action_name, full_section_name)
         sys.stderr.write("%s\n" % (msg))
         LOG.debug(msg)
-        return 0
+        return []
     else:
         LOG.debug("Ran %s modules with %s failures",
                   len(which_ran), len(failures))
-        return len(failures)
+        return failures
 
 
 def main_init(name, args):
@@ -220,7 +223,10 @@
         if existing_files:
             LOG.debug("Exiting early due to the existence of %s files",
                       existing_files)
-            return 0
+            return (None, [])
+        else:
+            LOG.debug("Execution continuing, no previous run detected that"
+                      " would allow us to stop early.")
     else:
         # The cache is not instance specific, so it has to be purged
         # but we want 'start' to benefit from a cache if
@@ -249,9 +255,9 @@
                               " Likely bad things to come!"))
         if not args.force:
             if args.local:
-                return 0
+                return (None, [])
             else:
-                return 1
+                return (None, ["No instance datasource found."])
     # Stage 6
     iid = init.instancify()
     LOG.debug("%s will now be targeting instance id: %s", name, iid)
@@ -274,7 +280,7 @@
             init.consume_data(PER_ALWAYS)
     except Exception:
         util.logexc(LOG, "Consuming user data failed!")
-        return 1
+        return (init.datasource, ["Consuming user data failed!"])
 
     # Stage 8 - re-read and apply relevant cloud-config to include user-data
     mods = stages.Modules(init, extract_fns(args))
@@ -291,7 +297,7 @@
     logging.setupLogging(mods.cfg)
 
     # Stage 10
-    return run_module_section(mods, name, name)
+    return (init.datasource, run_module_section(mods, name, name))
 
 
 def main_modules(action_name, args):
@@ -315,14 +321,12 @@
         init.fetch()
     except sources.DataSourceNotFoundException:
         # There was no datasource found, theres nothing to do
-        util.logexc(LOG, ('Can not apply stage %s, '
-                          'no datasource found!'
-                          " Likely bad things to come!"), name)
-        print_exc(('Can not apply stage %s, '
-                   'no datasource found!'
-                   " Likely bad things to come!") % (name))
+        msg = ('Can not apply stage %s, no datasource found! Likely bad '
+               'things to come!' % name)
+        util.logexc(LOG, msg)
+        print_exc(msg)
         if not args.force:
-            return 1
+            return [(msg)]
     # Stage 3
     mods = stages.Modules(init, extract_fns(args))
     # Stage 4
@@ -419,6 +423,110 @@
         return 0
 
 
+def atomic_write_json(path, data):
+    tf = None
+    try:
+        tf = tempfile.NamedTemporaryFile(dir=os.path.dirname(path),
+                                         delete=False)
+        tf.write(json.dumps(data, indent=1) + "\n")
+        tf.close()
+        os.rename(tf.name, path)
+    except Exception as e:
+        if tf is not None:
+            util.del_file(tf.name)
+        raise e
+
+
+def status_wrapper(name, args, data_d=None, link_d=None):
+    if data_d is None:
+        data_d = os.path.normpath("/var/lib/cloud/data")
+    if link_d is None:
+        link_d = os.path.normpath("/run/cloud-init")
+
+    status_path = os.path.join(data_d, "status.json")
+    status_link = os.path.join(link_d, "status.json")
+    result_path = os.path.join(data_d, "result.json")
+    result_link = os.path.join(link_d, "result.json")
+
+    util.ensure_dirs((data_d, link_d,))
+
+    (_name, functor) = args.action
+
+    if name == "init":
+        if args.local:
+            mode = "init-local"
+        else:
+            mode = "init"
+    elif name == "modules":
+        mode = "modules-%s" % args.mode
+    else:
+        raise ValueError("unknown name: %s" % name)
+
+    modes = ('init', 'init-local', 'modules-config', 'modules-final')
+
+    status = None
+    if mode == 'init-local':
+        for f in (status_link, result_link, status_path, result_path):
+            util.del_file(f)
+    else:
+        try:
+            status = json.loads(util.load_file(status_path))
+        except:
+            pass
+
+    if status is None:
+        nullstatus = {
+            'errors': [],
+            'start': None,
+            'end': None,
+        }
+        status = {'v1': {}}
+        for m in modes:
+            status['v1'][m] = nullstatus.copy()
+        status['v1']['datasource'] = None
+
+    v1 = status['v1']
+    v1['stage'] = mode
+    v1[mode]['start'] = time.time()
+
+    atomic_write_json(status_path, status)
+    util.sym_link(os.path.relpath(status_path, link_d), status_link,
+                  force=True)
+
+    try:
+        ret = functor(name, args)
+        if mode in ('init', 'init-local'):
+            (datasource, errors) = ret
+            if datasource is not None:
+                v1['datasource'] = str(datasource)
+        else:
+            errors = ret
+
+        v1[mode]['errors'] = [str(e) for e in errors]
+
+    except Exception as e:
+        v1[mode]['errors'] = [str(e)]
+
+    v1[mode]['finished'] = time.time()
+    v1['stage'] = None
+
+    atomic_write_json(status_path, status)
+
+    if mode == "modules-final":
+        # write the 'finished' file
+        errors = []
+        for m in modes:
+            if v1[m]['errors']:
+                errors.extend(v1[m].get('errors', []))
+
+        atomic_write_json(result_path,
+            {'v1': {'datasource': v1['datasource'], 'errors': errors}})
+        util.sym_link(os.path.relpath(result_path, link_d), result_link,
+                      force=True)
+
+    return len(v1[mode]['errors'])
+
+
 def main():
     parser = argparse.ArgumentParser()
 
@@ -502,6 +610,8 @@
     signal_handler.attach_handlers()
 
     (name, functor) = args.action
+    if name in ("modules", "init"):
+        functor = status_wrapper
 
     return util.log_time(logfunc=LOG.debug, msg="cloud-init mode '%s'" % name,
                          get_uptime=True, func=functor, args=(name, args))

=== modified file 'cloudinit/config/cc_final_message.py'
--- cloudinit/config/cc_final_message.py	2013-09-25 17:51:52 +0000
+++ cloudinit/config/cc_final_message.py	2014-07-18 13:33:31 +0000
@@ -53,6 +53,7 @@
             'version': cver,
             'datasource': str(cloud.datasource),
         }
+        subs.update(dict([(k.upper(), v) for k, v in subs.items()]))
         util.multi_log("%s\n" % (templater.render_string(msg_in, subs)),
                        console=False, stderr=True, log=log)
     except Exception:

=== modified file 'cloudinit/config/cc_power_state_change.py'
--- cloudinit/config/cc_power_state_change.py	2014-02-03 22:03:14 +0000
+++ cloudinit/config/cc_power_state_change.py	2014-07-18 13:33:31 +0000
@@ -22,7 +22,6 @@
 import errno
 import os
 import re
-import signal
 import subprocess
 import time
 

=== modified file 'cloudinit/config/cc_seed_random.py'
--- cloudinit/config/cc_seed_random.py	2014-02-05 15:36:47 +0000
+++ cloudinit/config/cc_seed_random.py	2014-07-18 13:33:31 +0000
@@ -1,8 +1,11 @@
 # vi: ts=4 expandtab
 #
 #    Copyright (C) 2013 Yahoo! Inc.
+#    Copyright (C) 2014 Canonical, Ltd
 #
 #    Author: Joshua Harlow <[e-mail]>
+#    Author: Dustin Kirkland <[e-mail]>
+#    Author: Scott Moser <[e-mail]>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -17,12 +20,15 @@
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import base64
+import os
 from StringIO import StringIO
 
 from cloudinit.settings import PER_INSTANCE
+from cloudinit import log as logging
 from cloudinit import util
 
 frequency = PER_INSTANCE
+LOG = logging.getLogger(__name__)
 
 
 def _decode(data, encoding=None):
@@ -38,24 +44,50 @@
         raise IOError("Unknown random_seed encoding: %s" % (encoding))
 
 
+def handle_random_seed_command(command, required, env=None):
+    if not command and required:
+        raise ValueError("no command found but required=true")
+    elif not command:
+        LOG.debug("no command provided")
+        return
+
+    cmd = command[0]
+    if not util.which(cmd):
+        if required:
+            raise ValueError("command '%s' not found but required=true", cmd)
+        else:
+            LOG.debug("command '%s' not found for seed_command", cmd)
+            return
+    util.subp(command, env=env, capture=False)
+
+
 def handle(name, cfg, cloud, log, _args):
-    if not cfg or "random_seed" not in cfg:
-        log.debug(("Skipping module named %s, "
-                   "no 'random_seed' configuration found"), name)
-        return
+    mycfg = cfg.get('random_seed', {})
+    seed_path = mycfg.get('file', '/dev/urandom')
+    seed_data = mycfg.get('data', '')
 
-    my_cfg = cfg['random_seed']
-    seed_path = my_cfg.get('file', '/dev/urandom')
     seed_buf = StringIO()
-    seed_buf.write(_decode(my_cfg.get('data', ''),
-                           encoding=my_cfg.get('encoding')))
+    if seed_data:
+        seed_buf.write(_decode(seed_data, encoding=mycfg.get('encoding')))
 
+    # 'random_seed' is set up by Azure datasource, and comes already in
+    # openstack meta_data.json
     metadata = cloud.datasource.metadata
     if metadata and 'random_seed' in metadata:
         seed_buf.write(metadata['random_seed'])
 
     seed_data = seed_buf.getvalue()
     if len(seed_data):
-        log.debug("%s: adding %s bytes of random seed entrophy to %s", name,
+        log.debug("%s: adding %s bytes of random seed entropy to %s", name,
                   len(seed_data), seed_path)
         util.append_file(seed_path, seed_data)
+
+    command = mycfg.get('command', ['pollinate', '-q'])
+    req = mycfg.get('command_required', False)
+    try:
+        env = os.environ.copy()
+        env['RANDOM_SEED_FILE'] = seed_path
+        handle_random_seed_command(command=command, required=req, env=env)
+    except ValueError as e:
+        log.warn("handling random command [%s] failed: %s", command, e)
+        raise e

=== modified file 'cloudinit/cs_utils.py'
--- cloudinit/cs_utils.py	2014-02-12 10:14:49 +0000
+++ cloudinit/cs_utils.py	2014-07-18 13:33:31 +0000
@@ -35,6 +35,10 @@
 
 import serial
 
+# these high timeouts are necessary as read may read a lot of data.
+READ_TIMEOUT = 60
+WRITE_TIMEOUT = 10
+
 SERIAL_PORT = '/dev/ttyS1'
 if platform.system() == 'Windows':
     SERIAL_PORT = 'COM2'
@@ -76,7 +80,9 @@
         self.result = self._marshal(self.raw_result)
 
     def _execute(self):
-        connection = serial.Serial(SERIAL_PORT)
+        connection = serial.Serial(port=SERIAL_PORT,
+                                   timeout=READ_TIMEOUT,
+                                   writeTimeout=WRITE_TIMEOUT)
         connection.write(self.request)
         return connection.readline().strip('\x04\n')
 

=== modified file 'cloudinit/importer.py'
--- cloudinit/importer.py	2013-10-09 19:22:06 +0000
+++ cloudinit/importer.py	2014-07-18 13:33:31 +0000
@@ -45,8 +45,6 @@
         real_path.append(base_name)
         full_path = '.'.join(real_path)
         real_paths.append(full_path)
-    LOG.debug("Looking for modules %s that have attributes %s",
-              real_paths, required_attrs)
     for full_path in real_paths:
         mod = None
         try:
@@ -62,6 +60,4 @@
                 found_attrs += 1
         if found_attrs == len(required_attrs):
             found_places.append(full_path)
-    LOG.debug("Found %s with attributes %s in %s", base_name,
-              required_attrs, found_places)
     return found_places

=== modified file 'cloudinit/mergers/__init__.py'
--- cloudinit/mergers/__init__.py	2013-05-03 21:41:28 +0000
+++ cloudinit/mergers/__init__.py	2014-07-18 13:33:31 +0000
@@ -55,9 +55,6 @@
         if not meth:
             meth = self._handle_unknown
             args.insert(0, method_name)
-        LOG.debug("Merging '%s' into '%s' using method '%s' of '%s'",
-                  type_name, type_utils.obj_name(merge_with),
-                  meth.__name__, self)
         return meth(*args)
 
 
@@ -84,8 +81,6 @@
                 # First one that has that method/attr gets to be
                 # the one that will be called
                 meth = getattr(merger, meth_wanted)
-                LOG.debug(("Merging using located merger '%s'"
-                            " since it had method '%s'"), merger, meth_wanted)
                 break
         if not meth:
             return UnknownMerger._handle_unknown(self, meth_wanted,

=== modified file 'cloudinit/sources/DataSourceAzure.py'
--- cloudinit/sources/DataSourceAzure.py	2014-02-10 20:11:45 +0000
+++ cloudinit/sources/DataSourceAzure.py	2014-07-18 13:33:31 +0000
@@ -18,12 +18,14 @@
 
 import base64
 import crypt
+import fnmatch
 import os
 import os.path
 import time
 from xml.dom import minidom
 
 from cloudinit import log as logging
+from cloudinit.settings import PER_ALWAYS
 from cloudinit import sources
 from cloudinit import util
 
@@ -53,14 +55,15 @@
     'disk_setup': {
         'ephemeral0': {'table_type': 'mbr',
                        'layout': True,
-                       'overwrite': False}
-         },
+                       'overwrite': False},
+        },
     'fs_setup': [{'filesystem': 'ext4',
                   'device': 'ephemeral0.1',
-                  'replace_fs': 'ntfs'}]
+                  'replace_fs': 'ntfs'}],
 }
 
 DS_CFG_PATH = ['datasource', DS_NAME]
+DEF_EPHEMERAL_LABEL = 'Temporary Storage'
 
 
 class DataSourceAzureNet(sources.DataSource):
@@ -189,8 +192,17 @@
                 LOG.warn("failed to get instance id in %s: %s", shcfgxml, e)
 
         pubkeys = pubkeys_from_crt_files(fp_files)
-
         self.metadata['public-keys'] = pubkeys
+
+        found_ephemeral = find_ephemeral_disk()
+        if found_ephemeral:
+            self.ds_cfg['disk_aliases']['ephemeral0'] = found_ephemeral
+            LOG.debug("using detected ephemeral0 of %s", found_ephemeral)
+
+        cc_modules_override = support_new_ephemeral(self.sys_cfg)
+        if cc_modules_override:
+            self.cfg['cloud_config_modules'] = cc_modules_override
+
         return True
 
     def device_name_to_device(self, name):
@@ -200,6 +212,92 @@
         return self.cfg
 
 
+def count_files(mp):
+    return len(fnmatch.filter(os.listdir(mp), '*[!cdrom]*'))
+
+
+def find_ephemeral_part():
+    """
+    Locate the default ephmeral0.1 device. This will be the first device
+    that has a LABEL of DEF_EPHEMERAL_LABEL and is a NTFS device. If Azure
+    gets more ephemeral devices, this logic will only identify the first
+    such device.
+    """
+    c_label_devs = util.find_devs_with("LABEL=%s" % DEF_EPHEMERAL_LABEL)
+    c_fstype_devs = util.find_devs_with("TYPE=ntfs")
+    for dev in c_label_devs:
+        if dev in c_fstype_devs:
+            return dev
+    return None
+
+
+def find_ephemeral_disk():
+    """
+    Get the ephemeral disk.
+    """
+    part_dev = find_ephemeral_part()
+    if part_dev and str(part_dev[-1]).isdigit():
+        return part_dev[:-1]
+    elif part_dev:
+        return part_dev
+    return None
+
+
+def support_new_ephemeral(cfg):
+    """
+    Windows Azure makes ephemeral devices ephemeral to boot; a ephemeral device
+    may be presented as a fresh device, or not.
+
+    Since the knowledge of when a disk is supposed to be plowed under is
+    specific to Windows Azure, the logic resides here in the datasource. When a
+    new ephemeral device is detected, cloud-init overrides the default
+    frequency for both disk-setup and mounts for the current boot only.
+    """
+    device = find_ephemeral_part()
+    if not device:
+        LOG.debug("no default fabric formated ephemeral0.1 found")
+        return None
+    LOG.debug("fabric formated ephemeral0.1 device at %s", device)
+
+    file_count = 0
+    try:
+        file_count = util.mount_cb(device, count_files)
+    except:
+        return None
+    LOG.debug("fabric prepared ephmeral0.1 has %s files on it", file_count)
+
+    if file_count >= 1:
+        LOG.debug("fabric prepared ephemeral0.1 will be preserved")
+        return None
+    else:
+        # if device was already mounted, then we need to unmount it
+        # race conditions could allow for a check-then-unmount
+        # to have a false positive. so just unmount and then check.
+        try:
+            util.subp(['umount', device])
+        except util.ProcessExecutionError as e:
+            if device in util.mounts():
+                LOG.warn("Failed to unmount %s, will not reformat.", device)
+                LOG.debug("Failed umount: %s", e)
+                return None
+
+    LOG.debug("cloud-init will format ephemeral0.1 this boot.")
+    LOG.debug("setting disk_setup and mounts modules 'always' for this boot")
+
+    cc_modules = cfg.get('cloud_config_modules')
+    if not cc_modules:
+        return None
+
+    mod_list = []
+    for mod in cc_modules:
+        if mod in ("disk_setup", "mounts"):
+            mod_list.append([mod, PER_ALWAYS])
+            LOG.debug("set module '%s' to 'always' for this boot", mod)
+        else:
+            mod_list.append(mod)
+    return mod_list
+
+
 def handle_set_hostname(enabled, hostname, cfg):
     if not util.is_true(enabled):
         return

=== modified file 'cloudinit/sources/DataSourceCloudSigma.py'
--- cloudinit/sources/DataSourceCloudSigma.py	2014-02-18 16:58:12 +0000
+++ cloudinit/sources/DataSourceCloudSigma.py	2014-07-18 13:33:31 +0000
@@ -15,10 +15,13 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+from base64 import b64decode
+import os
 import re
 
 from cloudinit import log as logging
 from cloudinit import sources
+from cloudinit import util
 from cloudinit.cs_utils import Cepko
 
 LOG = logging.getLogger(__name__)
@@ -39,12 +42,40 @@
         self.ssh_public_key = ''
         sources.DataSource.__init__(self, sys_cfg, distro, paths)
 
+    def is_running_in_cloudsigma(self):
+        """
+        Uses dmidecode to detect if this instance of cloud-init is running
+        in the CloudSigma's infrastructure.
+        """
+        uname_arch = os.uname()[4]
+        if uname_arch.startswith("arm") or uname_arch == "aarch64":
+            # Disabling because dmidecode in CMD_DMI_SYSTEM crashes kvm process
+            LOG.debug("Disabling CloudSigma datasource on arm (LP: #1243287)")
+            return False
+
+        dmidecode_path = util.which('dmidecode')
+        if not dmidecode_path:
+            return False
+
+        LOG.debug("Determining hypervisor product name via dmidecode")
+        try:
+            cmd = [dmidecode_path, "--string", "system-product-name"]
+            system_product_name, _ = util.subp(cmd)
+            return 'cloudsigma' in system_product_name.lower()
+        except:
+            LOG.warn("Failed to get hypervisor product name via dmidecode")
+
+        return False
+
     def get_data(self):
         """
         Metadata is the whole server context and /meta/cloud-config is used
         as userdata.
         """
         dsmode = None
+        if not self.is_running_in_cloudsigma():
+            return False
+
         try:
             server_context = self.cepko.all().result
             server_meta = server_context['meta']
@@ -61,7 +92,13 @@
         if dsmode == "disabled" or dsmode != self.dsmode:
             return False
 
+        base64_fields = server_meta.get('base64_fields', '').split(',')
         self.userdata_raw = server_meta.get('cloudinit-user-data', "")
+        if 'cloudinit-user-data' in base64_fields:
+            self.userdata_raw = b64decode(self.userdata_raw)
+        if 'cloudinit' in server_context.get('vendor_data', {}):
+            self.vendordata_raw = server_context["vendor_data"]["cloudinit"]
+
         self.metadata = server_context
         self.ssh_public_key = server_meta['ssh_public_key']
 

=== modified file 'cloudinit/sources/DataSourceNoCloud.py'
--- cloudinit/sources/DataSourceNoCloud.py	2014-02-18 17:58:21 +0000
+++ cloudinit/sources/DataSourceNoCloud.py	2014-07-18 13:33:31 +0000
@@ -57,7 +57,7 @@
             md = {}
             if parse_cmdline_data(self.cmdline_id, md):
                 found.append("cmdline")
-            mydata.update(md)
+            mydata['meta-data'].update(md)
         except:
             util.logexc(LOG, "Unable to parse command line data")
             return False

=== modified file 'cloudinit/sources/DataSourceOpenNebula.py'
--- cloudinit/sources/DataSourceOpenNebula.py	2014-01-17 01:11:27 +0000
+++ cloudinit/sources/DataSourceOpenNebula.py	2014-07-18 13:33:31 +0000
@@ -4,11 +4,13 @@
 #    Copyright (C) 2012 Yahoo! Inc.
 #    Copyright (C) 2012-2013 CERIT Scientific Cloud
 #    Copyright (C) 2012-2013 OpenNebula.org
+#    Copyright (C) 2014 Consejo Superior de Investigaciones Cientificas
 #
 #    Author: Scott Moser <[e-mail]>
 #    Author: Joshua Harlow <[e-mail]>
 #    Author: Vlastimil Holer <[e-mail]>
 #    Author: Javier Fontan <[e-mail]>
+#    Author: Enol Fernandez <[e-mail]>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -22,6 +24,7 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import base64
 import os
 import pwd
 import re
@@ -417,6 +420,16 @@
     elif "USERDATA" in context:
         results['userdata'] = context["USERDATA"]
 
+    # b64decode user data if necessary (default)
+    if 'userdata' in results:
+        encoding = context.get('USERDATA_ENCODING',
+                               context.get('USER_DATA_ENCODING'))
+        if encoding == "base64":
+            try:
+                results['userdata'] = base64.b64decode(results['userdata'])
+            except TypeError:
+                LOG.warn("Failed base64 decoding of userdata")
+
     # generate static /etc/network/interfaces
     # only if there are any required context variables
     # http://opennebula.org/documentation:rel3.8:cong#network_configuration

=== modified file 'cloudinit/sources/DataSourceSmartOS.py'
--- cloudinit/sources/DataSourceSmartOS.py	2014-02-26 19:28:46 +0000
+++ cloudinit/sources/DataSourceSmartOS.py	2014-07-18 13:33:31 +0000
@@ -170,8 +170,9 @@
         md = {}
         ud = ""
 
-        if not os.path.exists(self.seed):
-            LOG.debug("Host does not appear to be on SmartOS")
+        if not device_exists(self.seed):
+            LOG.debug("No serial device '%s' found for SmartOS datasource",
+                      self.seed)
             return False
 
         uname_arch = os.uname()[4]
@@ -274,6 +275,11 @@
                           b64=b64)
 
 
+def device_exists(device):
+    """Symplistic method to determine if the device exists or not"""
+    return os.path.exists(device)
+
+
 def get_serial(seed_device, seed_timeout):
     """This is replaced in unit testing, allowing us to replace
         serial.Serial with a mocked class.

=== modified file 'cloudinit/stages.py'
--- cloudinit/stages.py	2014-02-13 18:53:08 +0000
+++ cloudinit/stages.py	2014-07-18 13:33:31 +0000
@@ -397,8 +397,8 @@
                     mod = handlers.fixup_handler(mod)
                     types = c_handlers.register(mod)
                     if types:
-                        LOG.debug("Added custom handler for %s from %s",
-                                  types, fname)
+                        LOG.debug("Added custom handler for %s [%s] from %s",
+                                  types, mod, fname)
                 except Exception:
                     util.logexc(LOG, "Failed to register handler from %s",
                                 fname)
@@ -644,6 +644,8 @@
                     freq = mod.frequency
                 if not freq in FREQUENCIES:
                     freq = PER_INSTANCE
+                LOG.debug("Running module %s (%s) with frequency %s",
+                          name, mod, freq)
 
                 # Use the configs logger and not our own
                 # TODO(harlowja): possibly check the module
@@ -657,7 +659,7 @@
                 run_name = "config-%s" % (name)
                 cc.run(run_name, mod.handle, func_args, freq=freq)
             except Exception as e:
-                util.logexc(LOG, "Running %s (%s) failed", name, mod)
+                util.logexc(LOG, "Running module %s (%s) failed", name, mod)
                 failures.append((name, e))
         return (which_ran, failures)
 

=== modified file 'cloudinit/util.py'
--- cloudinit/util.py	2014-02-13 11:27:22 +0000
+++ cloudinit/util.py	2014-07-18 13:33:31 +0000
@@ -1395,8 +1395,10 @@
     return obj_copy.deepcopy(CFG_BUILTIN)
 
 
-def sym_link(source, link):
+def sym_link(source, link, force=False):
     LOG.debug("Creating symbolic link from %r => %r", link, source)
+    if force and os.path.exists(link):
+        del_file(link)
     os.symlink(source, link)
 
 

=== modified file 'cloudinit/version.py'
--- cloudinit/version.py	2013-11-19 21:49:53 +0000
+++ cloudinit/version.py	2014-07-18 13:33:31 +0000
@@ -20,7 +20,7 @@
 
 
 def version():
-    return vr.StrictVersion("0.7.5")
+    return vr.StrictVersion("0.7.6")
 
 
 def version_string():

=== modified file 'doc/examples/cloud-config-user-groups.txt'
--- doc/examples/cloud-config-user-groups.txt	2013-10-02 13:25:36 +0000
+++ doc/examples/cloud-config-user-groups.txt	2014-07-18 13:33:31 +0000
@@ -69,7 +69,7 @@
 #   no-user-group: When set to true, do not create a group named after the user.
 #   no-log-init: When set to true, do not initialize lastlog and faillog database.
 #   ssh-import-id: Optional. Import SSH ids
-#   ssh-authorized-key: Optional. Add key to user's ssh authorized keys file
+#   ssh-authorized-keys: Optional. [list] Add keys to user's authorized keys file
 #   sudo: Defaults to none. Set to the sudo string you want to use, i.e.
 #           ALL=(ALL) NOPASSWD:ALL. To add multiple rules, use the following
 #           format.

=== modified file 'doc/sources/cloudsigma/README.rst'
--- doc/sources/cloudsigma/README.rst	2014-02-13 15:39:39 +0000
+++ doc/sources/cloudsigma/README.rst	2014-07-18 13:33:31 +0000
@@ -23,6 +23,10 @@
 header could be omitted. However since this is a raw-text field you could provide any of the valid
 `config formats`_.
 
+You have the option to encode your user-data using Base64. In order to do that you have to add the
+``cloudinit-user-data`` field to the ``base64_fields``. The latter is a comma-separated field with
+all the meta fields whit base64 encoded values.
+
 If your user-data does not need an internet connection you can create a
 `meta field`_ in the `server context`_ ``cloudinit-dsmode`` and set "local" as value.
 If this field does not exist the default value is "net".

=== added file 'doc/status.txt'
--- doc/status.txt	1970-01-01 00:00:00 +0000
+++ doc/status.txt	2014-07-18 13:33:31 +0000
@@ -0,0 +1,53 @@
+cloud-init will keep a 'status' file up to date for other applications
+wishing to use it to determine cloud-init status.
+
+It will manage 2 files:
+ status.json
+ result.json
+
+The files will be written to /var/lib/cloud/data/ .
+A symlink will be created in /run/cloud-init. The link from /run is to ensure
+that if the file exists, it is not stale for this boot.
+
+status.json's format is:
+   {
+    'v1': {
+     'init': {
+       errors: []      # list of strings for each error that occurred
+       start: float # time.time() that this stage started or None
+       end: float # time.time() that this stage finished or None
+     },
+     'init-local': {
+       'errors': [], 'start': <float>, 'end' <float> # (same as 'init' above)
+     },
+     'modules-config': {
+       'errors': [], 'start': <float>, 'end' <float> # (same as 'init' above)
+     },
+     'modules-final': {
+       'errors': [], 'start': <float>, 'end' <float> # (same as 'init' above)
+     },
+     'datasource': string describing datasource found or None
+     'stage': string representing stage that is currently running
+              ('init', 'init-local', 'modules-final', 'modules-config', None)
+              if None, then no stage is running.  Reader must read the start/end
+              of each of the above stages to determine the state.
+   }
+
+result.json's format is:
+   {
+    'v1': {
+     'datasource': string describing the datasource found
+     'errors': [] # list of errors reported
+    }
+   }
+
+Thus, to determine if cloud-init is finished:
+ fin = "/run/cloud-init/result.json"
+ if os.path.exists(fin):
+   ret = json.load(open(fin, "r"))
+   if len(ret['v1']['errors']):
+     print "Finished with errors:" + "\n".join(ret['v1']['errors'])
+   else:
+     print "Finished no errors"
+ else:
+   print "Not Finished"

=== modified file 'tests/unittests/helpers.py'
--- tests/unittests/helpers.py	2014-02-08 00:40:51 +0000
+++ tests/unittests/helpers.py	2014-07-18 13:33:31 +0000
@@ -52,6 +52,30 @@
                 standardMsg = standardMsg % (value)
                 self.fail(self._formatMessage(msg, standardMsg))
 
+        def assertDictContainsSubset(self, expected, actual, msg=None):
+            missing = []
+            mismatched = []
+            for k, v in expected.iteritems():
+                if k not in actual:
+                    missing.append(k)
+                elif actual[k] != v:
+                    mismatched.append('%r, expected: %r, actual: %r'
+                                      % (k, v, actual[k]))
+
+            if len(missing) == 0 and len(mismatched) == 0:
+                return
+
+            standardMsg = ''
+            if missing:
+                standardMsg = 'Missing: %r' % ','.join(m for m in missing)
+            if mismatched:
+                if standardMsg:
+                    standardMsg += '; '
+                standardMsg += 'Mismatched values: %s' % ','.join(mismatched)
+
+            self.fail(self._formatMessage(msg, standardMsg))
+
+
 else:
     class TestCase(unittest.TestCase):
         pass

=== modified file 'tests/unittests/test__init__.py'
--- tests/unittests/test__init__.py	2014-01-25 03:31:28 +0000
+++ tests/unittests/test__init__.py	2014-07-18 13:33:31 +0000
@@ -1,14 +1,10 @@
-import logging
 import os
-import StringIO
-import sys
 
-from mocker import MockerTestCase, ANY, ARGS, KWARGS
+from mocker import MockerTestCase, ARGS, KWARGS
 
 from cloudinit import handlers
 from cloudinit import helpers
 from cloudinit import importer
-from cloudinit import log
 from cloudinit import settings
 from cloudinit import url_helper
 from cloudinit import util

=== modified file 'tests/unittests/test_datasource/test_cloudsigma.py'
--- tests/unittests/test_datasource/test_cloudsigma.py	2014-02-12 10:14:49 +0000
+++ tests/unittests/test_datasource/test_cloudsigma.py	2014-07-18 13:33:31 +0000
@@ -1,9 +1,11 @@
 # coding: utf-8
-from unittest import TestCase
+import copy
 
 from cloudinit.cs_utils import Cepko
 from cloudinit.sources import DataSourceCloudSigma
 
+from tests.unittests import helpers as test_helpers
+
 
 SERVER_CONTEXT = {
     "cpu": 1000,
@@ -19,21 +21,27 @@
     "smp": 1,
     "tags": ["much server", "very performance"],
     "uuid": "65b2fb23-8c03-4187-a3ba-8b7c919e8890",
-    "vnc_password": "9e84d6cb49e46379"
+    "vnc_password": "9e84d6cb49e46379",
+    "vendor_data": {
+        "location": "zrh",
+        "cloudinit": "#cloud-config\n\n...",
+    }
 }
 
 
 class CepkoMock(Cepko):
-    result = SERVER_CONTEXT
+    def __init__(self, mocked_context):
+        self.result = mocked_context
 
     def all(self):
         return self
 
 
-class DataSourceCloudSigmaTest(TestCase):
+class DataSourceCloudSigmaTest(test_helpers.TestCase):
     def setUp(self):
         self.datasource = DataSourceCloudSigma.DataSourceCloudSigma("", "", "")
-        self.datasource.cepko = CepkoMock()
+        self.datasource.is_running_in_cloudsigma = lambda: True
+        self.datasource.cepko = CepkoMock(SERVER_CONTEXT)
         self.datasource.get_data()
 
     def test_get_hostname(self):
@@ -57,3 +65,34 @@
     def test_user_data(self):
         self.assertEqual(self.datasource.userdata_raw,
                          SERVER_CONTEXT['meta']['cloudinit-user-data'])
+
+    def test_encoded_user_data(self):
+        encoded_context = copy.deepcopy(SERVER_CONTEXT)
+        encoded_context['meta']['base64_fields'] = 'cloudinit-user-data'
+        encoded_context['meta']['cloudinit-user-data'] = 'aGkgd29ybGQK'
+        self.datasource.cepko = CepkoMock(encoded_context)
+        self.datasource.get_data()
+
+        self.assertEqual(self.datasource.userdata_raw, b'hi world\n')
+
+    def test_vendor_data(self):
+        self.assertEqual(self.datasource.vendordata_raw,
+                         SERVER_CONTEXT['vendor_data']['cloudinit'])
+
+    def test_lack_of_vendor_data(self):
+        stripped_context = copy.deepcopy(SERVER_CONTEXT)
+        del stripped_context["vendor_data"]
+        self.datasource = DataSourceCloudSigma.DataSourceCloudSigma("", "", "")
+        self.datasource.cepko = CepkoMock(stripped_context)
+        self.datasource.get_data()
+
+        self.assertIsNone(self.datasource.vendordata_raw)
+
+    def test_lack_of_cloudinit_key_in_vendor_data(self):
+        stripped_context = copy.deepcopy(SERVER_CONTEXT)
+        del stripped_context["vendor_data"]["cloudinit"]
+        self.datasource = DataSourceCloudSigma.DataSourceCloudSigma("", "", "")
+        self.datasource.cepko = CepkoMock(stripped_context)
+        self.datasource.get_data()
+
+        self.assertIsNone(self.datasource.vendordata_raw)

=== modified file 'tests/unittests/test_datasource/test_gce.py'
--- tests/unittests/test_datasource/test_gce.py	2014-02-13 22:03:12 +0000
+++ tests/unittests/test_datasource/test_gce.py	2014-07-18 13:33:31 +0000
@@ -15,7 +15,6 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import unittest
 import httpretty
 import re
 
@@ -25,6 +24,8 @@
 from cloudinit import helpers
 from cloudinit.sources import DataSourceGCE
 
+from tests.unittests import helpers as test_helpers
+
 GCE_META = {
     'instance/id': '123',
     'instance/zone': 'foo/bar',
@@ -54,7 +55,7 @@
         return (404, headers, '')
 
 
-class TestDataSourceGCE(unittest.TestCase):
+class TestDataSourceGCE(test_helpers.TestCase):
 
     def setUp(self):
         self.ds = DataSourceGCE.DataSourceGCE(

=== modified file 'tests/unittests/test_datasource/test_maas.py'
--- tests/unittests/test_datasource/test_maas.py	2014-01-25 03:31:28 +0000
+++ tests/unittests/test_datasource/test_maas.py	2014-07-18 13:33:31 +0000
@@ -3,7 +3,6 @@
 
 from cloudinit.sources import DataSourceMAAS
 from cloudinit import url_helper
-from cloudinit import util
 from tests.unittests.helpers import populate_dir
 
 import mocker

=== modified file 'tests/unittests/test_datasource/test_opennebula.py'
--- tests/unittests/test_datasource/test_opennebula.py	2014-01-17 16:09:15 +0000
+++ tests/unittests/test_datasource/test_opennebula.py	2014-07-18 13:33:31 +0000
@@ -4,6 +4,7 @@
 from mocker import MockerTestCase
 from tests.unittests.helpers import populate_dir
 
+from base64 import b64encode
 import os
 import pwd
 
@@ -164,10 +165,31 @@
 
             public_keys.append(SSH_KEY % (c + 1,))
 
-    def test_user_data(self):
-        for k in ('USER_DATA', 'USERDATA'):
-            my_d = os.path.join(self.tmp, k)
-            populate_context_dir(my_d, {k: USER_DATA})
+    def test_user_data_plain(self):
+        for k in ('USER_DATA', 'USERDATA'):
+            my_d = os.path.join(self.tmp, k)
+            populate_context_dir(my_d, {k: USER_DATA,
+                                        'USERDATA_ENCODING': ''})
+            results = ds.read_context_disk_dir(my_d)
+
+            self.assertTrue('userdata' in results)
+            self.assertEqual(USER_DATA, results['userdata'])
+
+    def test_user_data_encoding_required_for_decode(self):
+        b64userdata = b64encode(USER_DATA)
+        for k in ('USER_DATA', 'USERDATA'):
+            my_d = os.path.join(self.tmp, k)
+            populate_context_dir(my_d, {k: b64userdata})
+            results = ds.read_context_disk_dir(my_d)
+
+            self.assertTrue('userdata' in results)
+            self.assertEqual(b64userdata, results['userdata'])
+
+    def test_user_data_base64_encoding(self):
+        for k in ('USER_DATA', 'USERDATA'):
+            my_d = os.path.join(self.tmp, k)
+            populate_context_dir(my_d, {k: b64encode(USER_DATA),
+                                        'USERDATA_ENCODING': 'base64'})
             results = ds.read_context_disk_dir(my_d)
 
             self.assertTrue('userdata' in results)

=== modified file 'tests/unittests/test_datasource/test_smartos.py'
--- tests/unittests/test_datasource/test_smartos.py	2014-02-26 19:21:40 +0000
+++ tests/unittests/test_datasource/test_smartos.py	2014-07-18 13:33:31 +0000
@@ -24,10 +24,7 @@
 
 import base64
 from cloudinit import helpers as c_helpers
-from cloudinit import stages
-from cloudinit import util
 from cloudinit.sources import DataSourceSmartOS
-from cloudinit.settings import (PER_INSTANCE)
 from tests.unittests import helpers
 import os
 import os.path
@@ -174,6 +171,7 @@
         self.apply_patches([(mod, 'get_serial', _get_serial)])
         self.apply_patches([(mod, 'dmi_data', _dmi_data)])
         self.apply_patches([(os, 'uname', _os_uname)])
+        self.apply_patches([(mod, 'device_exists', lambda d: True)])
         dsrc = mod.DataSourceSmartOS(sys_cfg, distro=None,
                                      paths=self.paths)
         return dsrc

=== modified file 'tests/unittests/test_handler/test_handler_seed_random.py'
--- tests/unittests/test_handler/test_handler_seed_random.py	2013-10-02 13:28:42 +0000
+++ tests/unittests/test_handler/test_handler_seed_random.py	2014-07-18 13:33:31 +0000
@@ -42,10 +42,32 @@
     def setUp(self):
         super(TestRandomSeed, self).setUp()
         self._seed_file = tempfile.mktemp()
+        self.unapply = []
+
+        # by default 'which' has nothing in its path
+        self.apply_patches([(util, 'which', self._which)])
+        self.apply_patches([(util, 'subp', self._subp)])
+        self.subp_called = []
+        self.whichdata = {}
 
     def tearDown(self):
+        apply_patches([i for i in reversed(self.unapply)])
         util.del_file(self._seed_file)
 
+    def apply_patches(self, patches):
+        ret = apply_patches(patches)
+        self.unapply += ret
+
+    def _which(self, program):
+        return self.whichdata.get(program)
+
+    def _subp(self, *args, **kwargs):
+        # supports subp calling with cmd as args or kwargs
+        if 'args' not in kwargs:
+            kwargs['args'] = args[0]
+        self.subp_called.append(kwargs)
+        return
+
     def _compress(self, text):
         contents = StringIO()
         gz_fh = gzip.GzipFile(mode='wb', fileobj=contents)
@@ -148,3 +170,56 @@
         cc_seed_random.handle('test', cfg, c, LOG, [])
         contents = util.load_file(self._seed_file)
         self.assertEquals('tiny-tim-was-here-so-was-josh', contents)
+
+    def test_seed_command_not_provided_pollinate_available(self):
+        c = self._get_cloud('ubuntu', {})
+        self.whichdata = {'pollinate': '/usr/bin/pollinate'}
+        cc_seed_random.handle('test', {}, c, LOG, [])
+
+        subp_args = [f['args'] for f in self.subp_called]
+        self.assertIn(['pollinate', '-q'], subp_args)
+
+    def test_seed_command_not_provided_pollinate_not_available(self):
+        c = self._get_cloud('ubuntu', {})
+        self.whichdata = {}
+        cc_seed_random.handle('test', {}, c, LOG, [])
+
+        # subp should not have been called as which would say not available
+        self.assertEquals(self.subp_called, list())
+
+    def test_unavailable_seed_command_and_required_raises_error(self):
+        c = self._get_cloud('ubuntu', {})
+        self.whichdata = {}
+        self.assertRaises(ValueError, cc_seed_random.handle,
+            'test', {'random_seed': {'command_required': True}}, c, LOG, [])
+
+    def test_seed_command_and_required(self):
+        c = self._get_cloud('ubuntu', {})
+        self.whichdata = {'foo': 'foo'}
+        cfg = {'random_seed': {'command_required': True, 'command': ['foo']}}
+        cc_seed_random.handle('test', cfg, c, LOG, [])
+
+        self.assertIn(['foo'], [f['args'] for f in self.subp_called])
+
+    def test_file_in_environment_for_command(self):
+        c = self._get_cloud('ubuntu', {})
+        self.whichdata = {'foo': 'foo'}
+        cfg = {'random_seed': {'command_required': True, 'command': ['foo'],
+                               'file': self._seed_file}}
+        cc_seed_random.handle('test', cfg, c, LOG, [])
+
+        # this just instists that the first time subp was called,
+        # RANDOM_SEED_FILE was in the environment set up correctly
+        subp_env = [f['env'] for f in self.subp_called]
+        self.assertEqual(subp_env[0].get('RANDOM_SEED_FILE'), self._seed_file)
+
+
+def apply_patches(patches):
+    ret = []
+    for (ref, name, replace) in patches:
+        if replace is None:
+            continue
+        orig = getattr(ref, name)
+        setattr(ref, name, replace)
+        ret.append((ref, name, orig))
+    return ret

=== modified file 'tests/unittests/test_handler/test_handler_yum_add_repo.py'
--- tests/unittests/test_handler/test_handler_yum_add_repo.py	2014-02-06 15:59:04 +0000
+++ tests/unittests/test_handler/test_handler_yum_add_repo.py	2014-07-18 13:33:31 +0000
@@ -1,4 +1,3 @@
-from cloudinit import helpers
 from cloudinit import util
 
 from cloudinit.config import cc_yum_add_repo

=== modified file 'tests/unittests/test_templating.py'
--- tests/unittests/test_templating.py	2014-07-16 18:31:31 +0000
+++ tests/unittests/test_templating.py	2014-07-18 13:33:31 +0000
@@ -17,26 +17,51 @@
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 from tests.unittests import helpers as test_helpers
+import textwrap
 
 from cloudinit import templater
 
 
 class TestTemplates(test_helpers.TestCase):
     def test_render_basic(self):
-        in_data = """
-${b}
+        in_data = textwrap.dedent("""
+            ${b}
 
-c = d
-"""
+            c = d
+            """)
         in_data = in_data.strip()
-        expected_data = """
-2
+        expected_data = textwrap.dedent("""
+            2
 
-c = d
-"""
+            c = d
+            """)
         out_data = templater.basic_render(in_data, {'b': 2})
         self.assertEqual(expected_data.strip(), out_data)
 
+    def test_render_basic_no_parens(self):
+        hn = "myfoohost"
+        in_data = "h=$hostname\nc=d\n"
+        expected_data = "h=%s\nc=d\n" % hn
+        out_data = templater.basic_render(in_data, {'hostname': hn})
+        self.assertEqual(expected_data, out_data)
+
+    def test_render_basic_parens(self):
+        hn = "myfoohost"
+        in_data = "h = ${hostname}\nc=d\n"
+        expected_data = "h = %s\nc=d\n" % hn
+        out_data = templater.basic_render(in_data, {'hostname': hn})
+        self.assertEqual(expected_data, out_data)
+
+    def test_render_basic2(self):
+        mirror = "mymirror"
+        codename = "zany"
+        in_data = "deb $mirror $codename-updates main contrib non-free"
+        ex_data = "deb %s %s-updates main contrib non-free" % (mirror, codename)
+             
+        out_data = templater.basic_render(in_data,
+            {'mirror': mirror, 'codename': codename})
+        self.assertEqual(ex_data, out_data)
+
     def test_detection(self):
         blob = "## template:cheetah"
 
@@ -53,14 +78,12 @@
         self.assertRaises(ValueError, templater.detect_template, blob)
 
     def test_render_cheetah(self):
-        blob = '''## template:cheetah
-$a,$b'''
+        blob = '\n'.join(['## template:cheetah', '$a,$b'])
         c = templater.render_string(blob, {"a": 1, "b": 2})
         self.assertEquals("1,2", c)
 
     def test_render_jinja(self):
-        blob = '''## template:jinja
-{{a}},{{b}}'''
+        blob = '\n'.join(['## template:jinja', '{{a}},{{b}}'])
         c = templater.render_string(blob, {"a": 1, "b": 2})
         self.assertEquals("1,2", c)