Merge lp:~cloud-init/cloud-init/rework into lp:~cloud-init-dev/cloud-init/trunk

[Joshua Harlow]

[lp:~cloud-init/cloud-init/rework]

992. By Joshua Harlow on 2012-07-06

Updated so that if no mirror is found, the module stops running.

993. By Joshua Harlow on 2012-07-06

Add comment about keeping track of what people think about the 'read'
and 'write' root, and if it confuses them, remove it later and just
recommend a more 'natural' way of doing it (ie 'chroot').

994. By Scott Moser on 2012-07-06

setup.py: rename "daemon type" to "init system"

This brings with it other changes, and also makes an install
install all of the requisite init files. (ie, cloud-init needs the -local and
the non-local)

995. By Joshua Harlow on 2012-07-06

Fix the initsys variable, setuptools/distools will automatically assign
to a variable of the name 'init_system' instead due to the param name being
'init-system'.

=== modified file 'ChangeLog'
--- ChangeLog	2012-06-21 15:37:22 +0000
+++ ChangeLog	2012-07-06 21:16:18 +0000
@@ -1,3 +1,196 @@
+0.7.0:
+ - unified binary that activates the various stages
+   - Now using argparse + subcommands to specify the various CLI options
+ - a stage module that clearly separates the stages of the different
+   components (also described how they are used and in what order in the
+   new unified binary)
+ - user_data is now a module that just does user data processing while the
+   actual activation and 'handling' of the processed user data is done via
+   a separate set of files (and modules) with the main 'init' stage being the
+   controller of this
+   - creation of boot_hook, cloud_config, shell_script, upstart_job version 2
+     modules (with classes that perform there functionality) instead of those
+     having functionality that is attached to the cloudinit object (which
+     reduces reuse and limits future functionality, and makes testing harder)
+ - removal of global config that defined paths, shared config, now this is
+   via objects making unit testing testing and global side-effects a non issue
+ - creation of a 'helpers.py' 
+   - this contains an abstraction for the 'lock' like objects that the various 
+     module/handler running stages use to avoid re-running a given 
+     module/handler for a given frequency. this makes it separated from 
+     the actual usage of that object (thus helpful for testing and clear lines
+     usage and how the actual job is accomplished)
+     - a common 'runner' class is the main entrypoint using these locks to
+       run function objects passed in (along with there arguments) and there
+       frequency
+   - add in a 'paths' object that provides access to the previously global
+     and/or config based paths (thus providing a single entrypoint object/type
+     that provides path information)
+       - this also adds in the ability to change the path when constructing 
+       that path 'object' and adding in additional config that can be used to 
+       alter the root paths of 'joins' (useful for testing or possibly useful
+       in chroots?)
+        - config options now avaiable that can alter the 'write_root' and the 
+         'read_root' when backing code uses the paths join() function
+   - add a config parser subclass that will automatically add unknown sections
+     and return default values (instead of throwing exceptions for these cases)
+   - a new config merging class that will be the central object that knows
+     how to do the common configuration merging from the various configuration
+     sources. The order is the following:
+     - cli config files override environment config files
+       which override instance configs which override datasource
+       configs which override base configuration which overrides
+       default configuration.
+ - remove the passing around of the 'cloudinit' object as a 'cloud' variable
+   and instead pass around an 'interface' object that can be given to modules
+   and handlers as there cloud access layer while the backing of that
+   object can be varied (good for abstraction and testing)
+ - use a single set of functions to do importing of modules
+ - add a function in which will search for a given set of module names with
+   a given set of attributes and return those which are found
+ - refactor logging so that instead of using a single top level 'log' that
+   instead each component/module can use its own logger (if desired), this
+   should be backwards compatible with handlers and config modules that used
+   the passed in logger (its still passed in)
+   - ensure that all places where exception are caught and where applicable
+     that the util logexc() is called, so that no exceptions that may occur
+     are dropped without first being logged (where it makes sense for this 
+     to happen)
+ - add a 'requires' file that lists cloud-init dependencies
+   - applying it in package creation (bdeb and brpm) as well as using it
+     in the modified setup.py to ensure dependencies are installed when
+     using that method of packaging
+ - add a 'version.py' that lists the active version (in code) so that code
+   inside cloud-init can report the version in messaging and other config files
+ - cleanup of subprocess usage so that all subprocess calls go through the
+   subp() utility method, which now has an exception type that will provide
+   detailed information on python 2.6 and 2.7
+ - forced all code loading, moving, chmod, writing files and other system
+   level actions to go through standard set of util functions, this greatly 
+   helps in debugging and determining exactly which system actions cloud-init is
+   performing
+ - switching out the templating engine cheetah for tempita since tempita has
+   no external dependencies (minus python) while cheetah has many dependencies
+   which makes it more difficult to adopt cloud-init in distros that may not
+   have those dependencies
+ - adjust url fetching and url trying to go through a single function that
+   reads urls in the new 'url helper' file, this helps in tracing, debugging
+   and knowing which urls are being called and/or posted to from with-in 
+   cloud-init code
+   - add in the sending of a 'User-Agent' header for all urls fetched that
+     do not provide there own header mapping, derive this user-agent from
+     the following template, 'Cloud-Init/{version}' where the version is the
+     cloud-init version number
+ - using prettytable for netinfo 'debug' printing since it provides a standard
+   and defined output that should be easier to parse than a custom format
+ - add a set of distro specific classes, that handle distro specific actions
+   that modules and or handler code can use as needed, this is organized into
+   a base abstract class with child classes that implement the shared 
+   functionality. config determines exactly which subclass to load, so it can
+   be easily extended as needed.
+   - current functionality
+      - network interface config file writing
+      - hostname setting/updating
+      - locale/timezone/ setting
+      - updating of /etc/hosts (with templates or generically)
+      - package commands (ie installing, removing)/mirror finding
+      - interface up/down activating
+   - implemented a debian + ubuntu subclass
+   - implemented a redhat + fedora subclass
+ - adjust the root 'cloud.cfg' file to now have distrobution/path specific 
+   configuration values in it. these special configs are merged as the normal
+   config is, but the system level config is not passed into modules/handlers
+   - modules/handlers must go through the path and distro object instead
+ - have the cloudstack datasource test the url before calling into boto to 
+   avoid the long wait for boto to finish retrying and finally fail when
+   the gateway meta-data address is unavailable
+ - add a simple mock ec2 meta-data python based http server that can serve a
+   very simple set of ec2 meta-data back to callers
+      - useful for testing or for understanding what the ec2 meta-data 
+        service can provide in terms of data or functionality
+ - for ssh key and authorized key file parsing add in classes and util functions
+   that maintain the state of individual lines, allowing for a clearer 
+   separation of parsing and modification (useful for testing and tracing)
+ - add a set of 'base' init.d scripts that can be used on systems that do
+   not have full upstart or systemd support (or support that does not match
+   the standard fedora/ubuntu implementation)
+   - currently these are being tested on RHEL 6.2
+ - separate the datasources into there own subdirectory (instead of being
+   a top-level item), this matches how config 'modules' and user-data 'handlers'
+   are also in there own subdirectory (thus helping new developers and others
+   understand the code layout in a quicker manner)
+ - add the building of rpms based off a new cli tool and template 'spec' file
+   that will templatize and perform the necessary commands to create a source
+   and binary package to be used with a cloud-init install on a 'rpm' supporting
+   system
+   - uses the new standard set of requires and converts those pypi requirements
+     into a local set of package requirments (that are known to exist on RHEL
+     systems but should also exist on fedora systems)
+ - adjust the bdeb builder to be a python script (instead of a shell script) and
+   make its 'control' file a template that takes in the standard set of pypi 
+   dependencies and uses a local mapping (known to work on ubuntu) to create the
+   packages set of dependencies (that should also work on ubuntu-like systems)
+ - pythonify a large set of various pieces of code
+   - remove wrapping return statements with () when it has no effect
+   - upper case all constants used
+   - correctly 'case' class and method names (where applicable)
+   - use os.path.join (and similar commands) instead of custom path creation
+   - use 'is None' instead of the frowned upon '== None' which picks up a large
+     set of 'true' cases than is typically desired (ie for objects that have
+     there own equality)
+   - use context managers on locks, tempdir, chdir, file, selinux, umask, 
+     unmounting commands so that these actions do not have to be closed and/or
+     cleaned up manually in finally blocks, which is typically not done and will
+     eventually be a bug in the future
+   - use the 'abc' module for abstract classes base where possible
+      - applied in the datasource root class, the distro root class, and the
+        user-data v2 root class
+ - when loading yaml, check that the 'root' type matches a predefined set of
+   valid types (typically just 'dict') and throw a type error if a mismatch
+   occurs, this seems to be a good idea to do when loading user config files
+ - when forking a long running task (ie resizing a filesytem) use a new util
+   function that will fork and then call a callback, instead of having to
+   implement all that code in a non-shared location (thus allowing it to be
+   used by others in the future)
+ - when writing out filenames, go through a util function that will attempt to
+   ensure that the given filename is 'filesystem' safe by replacing '/' with
+   '_' and removing characters which do not match a given whitelist of allowed
+   filename characters
+ - for the varying usages of the 'blkid' command make a function in the util
+   module that can be used as the single point of entry for interaction with
+   that command (and its results) instead of having X separate implementations
+ - place the rfc 8222 time formatting and uptime repeated pieces of code in the
+   util module as a set of function with the name 'time_rfc2822'/'uptime'
+ - separate the pylint+pep8 calling from one tool into two indivudal tools so
+   that they can be called independently, add make file sections that can be 
+   used to call these independently
+ - remove the support for the old style config that was previously located in
+   '/etc/ec2-init/ec2-config.cfg', no longer supported!
+ - instead of using a altered config parser that added its own 'dummy' section
+   on in the 'mcollective' module, use configobj which handles the parsing of
+   config without sections better (and it also maintains comments instead of
+   removing them)
+ - use the new defaulting config parser (that will not raise errors on sections
+   that do not exist or return errors when values are fetched that do not exist)
+   in the 'puppet' module
+ - for config 'modules' add in the ability for the module to provide a list of 
+   distro names which it is known to work with, if when ran and the distro being
+   used name does not match one of those in this list, a warning will be written
+   out saying that this module may not work correctly on this distrobution
+ - for all dynamically imported modules ensure that they are fixed up before 
+   they are used by ensuring that they have certain attributes, if they do not
+   have those attributes they will be set to a sensible set of defaults instead
+ - adjust all 'config' modules and handlers to use the adjusted util functions
+   and the new distro objects where applicable so that those pieces of code can 
+   benefit from the unified and enhanced functionality being provided in that
+   util module
+ - fix a potential bug whereby when a #includeonce was encountered it would
+   enable checking of urls against a cache, if later a #include was encountered
+   it would continue checking against that cache, instead of refetching (which
+   would likely be the expected case)
+ - add a openstack/nova based pep8 extension utility ('hacking.py') that allows
+   for custom checks (along with the standard pep8 checks) to occur when running
+   'make pep8' and its derivatives
 0.6.4:
  - support relative path in AuthorizedKeysFile (LP: #970071).
  - make apt-get update run with --quiet (suitable for logging) (LP: #1012613)
 
=== modified file 'Makefile'
--- Makefile	2012-01-12 15:06:27 +0000
+++ Makefile	2012-07-06 21:16:18 +0000
@@ -1,14 +1,33 @@
+CWD=$(shell pwd)
+PY_FILES=$(shell find cloudinit bin -name "*.py")
+PY_FILES+="bin/cloud-init"
 
 all: test
 
+pep8:
+	$(CWD)/tools/run-pep8 $(PY_FILES)
+
 pylint:
-	pylint cloudinit
+	$(CWD)/tools/run-pylint $(PY_FILES)
 
 pyflakes:
-	pyflakes .
+	pyflakes $(PY_FILES)
 
 test:
-	nosetests tests/unittests/
-
-.PHONY: test pylint pyflakes
+	nosetests $(noseopts) tests/unittests/
+
+2to3:
+	2to3 $(PY_FILES)
+
+clean:
+	rm -rf /var/log/cloud-init.log \
+		   /var/lib/cloud/
+
+rpm:
+	cd packages && ./brpm
+
+deb:
+	cd packages && ./bddeb
+
+.PHONY: test pylint pyflakes 2to3 clean pep8 rpm deb
 
 
=== added file 'Requires'
--- Requires	1970-01-01 00:00:00 +0000
+++ Requires	2012-07-06 21:16:18 +0000
@@ -0,0 +1,30 @@
+# Pypi requirements for cloud-init to work
+
+# Used for templating any files or strings that are considered 
+# to be templates, not cheetah since it pulls in alot of extra libs.
+# This one is pretty dinky and does want we want (var substituion)
+Tempita
+
+# This is used for any pretty printing of tabular data.
+PrettyTable
+
+# This one is currently only used by the MAAS datasource. If that
+# datasource is removed, this is no longer needed
+oauth
+
+# This is used to fetch the ec2 metadata into a easily
+# parseable format, instead of having to have cloud-init perform
+# those same fetchs and decodes and signing (...) that ec2 requires.
+boto
+
+# This is only needed for places where we need to support configs in a manner
+# that the built-in config parser is not sufficent (ie
+# when we need to preserve comments, or do not have a top-level
+# section)...
+configobj
+
+# All new style configurations are in the yaml format
+pyyaml
+
+# The new main entrypoint uses argparse instead of optparse
+argparse
 
=== modified file 'TODO'
--- TODO	2011-02-17 20:48:41 +0000
+++ TODO	2012-07-06 21:16:18 +0000
@@ -1,14 +1,37 @@
-- consider 'failsafe' DataSource
+- Consider a 'failsafe' DataSource
   If all others fail, setting a default that
    - sets the user password, writing it to console
    - logs to console that this happened
-- consider 'previous' DataSource
+- Consider a 'previous' DataSource
   If no other data source is found, fall back to the 'previous' one
   keep a indication of what instance id that is in /var/lib/cloud
-- rewrite "cloud-init-query"
-  have DataSource and cloudinit expose explicit fields
+- Rewrite "cloud-init-query" (currently not implemented)
+  Possibly have DataSource and cloudinit expose explicit fields
    - instance-id
    - hostname
    - mirror
    - release
    - ssh public keys
+- Remove the conversion of the ubuntu network interface format conversion
+  to a RH/fedora format and replace it with a top level format that uses
+  the netcf libraries format instead (which itself knows how to translate
+  into the specific formats)
+- Replace the 'apt*' modules with variants that now use the distro classes
+  to perform distro independent packaging commands (where possible)
+- Canonicalize the semaphore/lock name for modules and user data handlers
+  a. It is most likely a bug that currently exists that if a module in config
+     alters its name and it has already ran, then it will get ran again since
+     the lock name hasn't be canonicalized
+- Replace some the LOG.debug calls with a LOG.info where appropriate instead
+  of how right now there is really only 2 levels (WARN and DEBUG)
+- Remove the 'cc_' for config modules, either have them fully specified (ie
+  'cloudinit.config.resizefs') or by default only look in the 'cloudinit.config'
+  for these modules (or have a combination of the above), this avoids having
+  to understand where your modules are coming from (which can be altered by
+  the current python inclusion path)
+- Depending on if people think the wrapper around 'os.path.join' provided
+  by the 'paths' object is useful (allowing us to modify based off a 'read'
+  and 'write' configuration based 'root') or is just to confusing, it might be 
+  something to remove later, and just recommend using 'chroot' instead (or the X 
+  different other options which are similar to 'chroot'), which is might be more 
+  natural and less confusing...
 
=== added directory 'bin'
=== added file 'bin/cloud-init'
--- bin/cloud-init	1970-01-01 00:00:00 +0000
+++ bin/cloud-init	2012-07-06 21:16:18 +0000
@@ -0,0 +1,474 @@
+#!/usr/bin/python
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <[e-mail]>
+#    Author: Juerg Haefliger <[e-mail]>
+#    Author: Joshua Harlow <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import os
+import sys
+import traceback
+
+# This is more just for running from the bin folder so that
+# cloud-init binary can find the cloudinit module
+possible_topdir = os.path.normpath(os.path.join(os.path.abspath(
+        sys.argv[0]), os.pardir, os.pardir))
+if os.path.exists(os.path.join(possible_topdir, "cloudinit", "__init__.py")):
+    sys.path.insert(0, possible_topdir)
+
+from cloudinit import log as logging
+from cloudinit import netinfo
+from cloudinit import sources
+from cloudinit import stages
+from cloudinit import templater
+from cloudinit import util
+from cloudinit import version
+
+from cloudinit.settings import (PER_INSTANCE, PER_ALWAYS, PER_ONCE,
+                                CLOUD_CONFIG)
+
+
+# Pretty little welcome message template
+WELCOME_MSG_TPL = ("Cloud-init v. {{version}} running '{{action}}' at "
+                   "{{timestamp}}. Up {{uptime}} seconds.")
+
+# Module section template
+MOD_SECTION_TPL = "cloud_%s_modules"
+
+# Things u can query on
+QUERY_DATA_TYPES = [
+    'data',
+    'data_raw',
+    'instance_id',
+]
+
+# Frequency shortname to full name
+# (so users don't have to remember the full name...)
+FREQ_SHORT_NAMES = {
+    'instance': PER_INSTANCE,
+    'always': PER_ALWAYS,
+    'once': PER_ONCE,
+}
+
+LOG = logging.getLogger()
+
+
+# Used for when a logger may not be active
+# and we still want to print exceptions...
+def print_exc(msg=''):
+    if msg:
+        sys.stderr.write("%s\n" % (msg))
+    sys.stderr.write('-' * 60)
+    sys.stderr.write("\n")
+    traceback.print_exc(file=sys.stderr)
+    sys.stderr.write('-' * 60)
+    sys.stderr.write("\n")
+
+
+def welcome(action):
+    tpl_params = {
+        'version': version.version_string(),
+        'uptime': util.uptime(),
+        'timestamp': util.time_rfc2822(),
+        'action': action,
+    }
+    tpl_msg = templater.render_string(WELCOME_MSG_TPL, tpl_params)
+    util.multi_log("%s\n" % (tpl_msg),
+                   console=False, stderr=True)
+
+
+def extract_fns(args):
+    # Files are already opened so lets just pass that along
+    # since it would of broke if it couldn't have
+    # read that file already...
+    fn_cfgs = []
+    if args.files:
+        for fh in args.files:
+            # The realpath is more useful in logging
+            # so lets resolve to that...
+            fn_cfgs.append(os.path.realpath(fh.name))
+    return fn_cfgs
+
+
+def run_module_section(mods, action_name, section):
+    full_section_name = MOD_SECTION_TPL % (section)
+    (which_ran, failures) = mods.run_section(full_section_name)
+    total_attempted = len(which_ran) + len(failures)
+    if total_attempted == 0:
+        msg = ("No '%s' modules to run"
+               " under section '%s'") % (action_name, full_section_name)
+        sys.stderr.write("%s\n" % (msg))
+        LOG.debug(msg)
+        return 0
+    else:
+        LOG.debug("Ran %s modules with %s failures",
+                  len(which_ran), len(failures))
+        return len(failures)
+
+
+def main_init(name, args):
+    deps = [sources.DEP_FILESYSTEM, sources.DEP_NETWORK]
+    if args.local:
+        deps = [sources.DEP_FILESYSTEM]
+
+    if not args.local:
+        # See doc/kernel-cmdline.txt
+        #
+        # This is used in maas datasource, in "ephemeral" (read-only root)
+        # environment where the instance netboots to iscsi ro root.
+        # and the entity that controls the pxe config has to configure
+        # the maas datasource.
+        #
+        # Could be used elsewhere, only works on network based (not local).
+        root_name = "%s.d" % (CLOUD_CONFIG)
+        target_fn = os.path.join(root_name, "91_kernel_cmdline_url.cfg")
+        util.read_write_cmdline_url(target_fn)
+
+    # Cloud-init 'init' stage is broken up into the following sub-stages
+    # 1. Ensure that the init object fetches its config without errors
+    # 2. Setup logging/output redirections with resultant config (if any)
+    # 3. Initialize the cloud-init filesystem
+    # 4. Check if we can stop early by looking for various files
+    # 5. Fetch the datasource
+    # 6. Connect to the current instance location + update the cache
+    # 7. Consume the userdata (handlers get activated here)
+    # 8. Construct the modules object
+    # 9. Adjust any subsequent logging/output redirections using
+    #    the modules objects configuration
+    # 10. Run the modules for the 'init' stage
+    # 11. Done!
+    welcome(name)
+    init = stages.Init(deps)
+    # Stage 1
+    init.read_cfg(extract_fns(args))
+    # Stage 2
+    outfmt = None
+    errfmt = None
+    try:
+        LOG.debug("Closing stdin")
+        util.close_stdin()
+        (outfmt, errfmt) = util.fixup_output(init.cfg, name)
+    except:
+        util.logexc(LOG, "Failed to setup output redirection!")
+        print_exc("Failed to setup output redirection!")
+    if args.debug:
+        # Reset so that all the debug handlers are closed out
+        LOG.debug(("Logging being reset, this logger may no"
+                    " longer be active shortly"))
+        logging.resetLogging()
+    logging.setupLogging(init.cfg)
+    # Stage 3
+    try:
+        init.initialize()
+    except Exception:
+        util.logexc(LOG, "Failed to initialize, likely bad things to come!")
+    # Stage 4
+    path_helper = init.paths
+    if not args.local:
+        sys.stderr.write("%s\n" % (netinfo.debug_info()))
+        LOG.debug(("Checking to see if files that we need already"
+                   " exist from a previous run that would allow us"
+                   " to stop early."))
+        stop_files = [
+            os.path.join(path_helper.get_cpath("data"), "no-net"),
+            path_helper.get_ipath_cur("obj_pkl"),
+        ]
+        existing_files = []
+        for fn in stop_files:
+            try:
+                c = util.load_file(fn)
+                if len(c):
+                    existing_files.append((fn, len(c)))
+            except Exception:
+                pass
+        if existing_files:
+            LOG.debug("Exiting early due to the existence of %s files",
+                      existing_files)
+            return 0
+    else:
+        # The cache is not instance specific, so it has to be purged
+        # but we want 'start' to benefit from a cache if
+        # a previous start-local populated one...
+        manual_clean = util.get_cfg_option_bool(init.cfg,
+                                                'manual_cache_clean', False)
+        if manual_clean:
+            LOG.debug("Not purging instance link, manual cleaning enabled")
+            init.purge_cache(False)
+        else:
+            init.purge_cache()
+        # Delete the non-net file as well
+        util.del_file(os.path.join(path_helper.get_cpath("data"), "no-net"))
+    # Stage 5
+    try:
+        init.fetch()
+    except sources.DataSourceNotFoundException:
+        util.logexc(LOG, ("No instance datasource found!"
+                          " Likely bad things to come!"))
+        # In the case of cloud-init (net mode) it is a bit
+        # more likely that the user would consider it
+        # failure if nothing was found. When using
+        # upstart it will also mentions job failure
+        # in console log if exit code is != 0.
+        if not args.force:
+            if args.local:
+                return 0
+            else:
+                return 1
+    # Stage 6
+    iid = init.instancify()
+    LOG.debug("%s will now be targeting instance id: %s", name, iid)
+    init.update()
+    # Stage 7
+    try:
+        # Attempt to consume the data per instance.
+        # This may run user-data handlers and/or perform
+        # url downloads and such as needed.
+        (ran, _results) = init.cloudify().run('consume_userdata',
+                                             init.consume_userdata,
+                                             args=[PER_INSTANCE],
+                                             freq=PER_INSTANCE)
+        if not ran:
+            # Just consume anything that is set to run per-always
+            # if nothing ran in the per-instance code
+            #
+            # See: https://bugs.launchpad.net/bugs/819507 for a little
+            # reason behind this...
+            init.consume_userdata(PER_ALWAYS)
+    except Exception:
+        util.logexc(LOG, "Consuming user data failed!")
+        return 1
+    # Stage 8 - TODO - do we really need to re-extract our configs?
+    mods = stages.Modules(init, extract_fns(args))
+    # Stage 9 - TODO is this really needed??
+    try:
+        outfmt_orig = outfmt
+        errfmt_orig = errfmt
+        (outfmt, errfmt) = util.get_output_cfg(mods.cfg, name)
+        if outfmt_orig != outfmt or errfmt_orig != errfmt:
+            LOG.warn("Stdout, stderr changing to (%s, %s)", outfmt, errfmt)
+            (outfmt, errfmt) = util.fixup_output(mods.cfg, name)
+    except:
+        util.logexc(LOG, "Failed to re-adjust output redirection!")
+    # Stage 10
+    return run_module_section(mods, name, name)
+
+
+def main_modules(action_name, args):
+    name = args.mode
+    # Cloud-init 'modules' stages are broken up into the following sub-stages
+    # 1. Ensure that the init object fetches its config without errors
+    # 2. Get the datasource from the init object, if it does
+    #    not exist then that means the main_init stage never
+    #    worked, and thus this stage can not run.
+    # 3. Construct the modules object
+    # 4. Adjust any subsequent logging/output redirections using
+    #    the modules objects configuration
+    # 5. Run the modules for the given stage name
+    # 6. Done!
+    welcome("%s:%s" % (action_name, name))
+    init = stages.Init(ds_deps=[])
+    # Stage 1
+    init.read_cfg(extract_fns(args))
+    # Stage 2
+    try:
+        init.fetch()
+    except sources.DataSourceNotFoundException:
+        # There was no datasource found, theres nothing to do
+        util.logexc(LOG, ('Can not apply stage %s, '
+                          'no datasource found!'
+                          " Likely bad things to come!"), name)
+        print_exc(('Can not apply stage %s, '
+                   'no datasource found!'
+                   " Likely bad things to come!") % (name))
+        if not args.force:
+            return 1
+    # Stage 3
+    mods = stages.Modules(init, extract_fns(args))
+    # Stage 4
+    try:
+        LOG.debug("Closing stdin")
+        util.close_stdin()
+        util.fixup_output(mods.cfg, name)
+    except:
+        util.logexc(LOG, "Failed to setup output redirection!")
+    if args.debug:
+        # Reset so that all the debug handlers are closed out
+        LOG.debug(("Logging being reset, this logger may no"
+                    " longer be active shortly"))
+        logging.resetLogging()
+    logging.setupLogging(mods.cfg)
+    # Stage 5
+    return run_module_section(mods, name, name)
+
+
+def main_query(name, _args):
+    raise NotImplementedError(("Action '%s' is not"
+                               " currently implemented") % (name))
+
+
+def main_single(name, args):
+    # Cloud-init single stage is broken up into the following sub-stages
+    # 1. Ensure that the init object fetches its config without errors
+    # 2. Attempt to fetch the datasource (warn if it doesn't work)
+    # 3. Construct the modules object
+    # 4. Adjust any subsequent logging/output redirections using
+    #    the modules objects configuration
+    # 5. Run the single module
+    # 6. Done!
+    mod_name = args.name
+    welcome("%s:%s" % (name, mod_name))
+    init = stages.Init(ds_deps=[])
+    # Stage 1
+    init.read_cfg(extract_fns(args))
+    # Stage 2
+    try:
+        init.fetch()
+    except sources.DataSourceNotFoundException:
+        # There was no datasource found,
+        # that might be bad (or ok) depending on
+        # the module being ran (so continue on)
+        util.logexc(LOG, ("Failed to fetch your datasource,"
+                          " likely bad things to come!"))
+        print_exc(("Failed to fetch your datasource,"
+                   " likely bad things to come!"))
+        if not args.force:
+            return 1
+    # Stage 3
+    mods = stages.Modules(init, extract_fns(args))
+    mod_args = args.module_args
+    if mod_args:
+        LOG.debug("Using passed in arguments %s", mod_args)
+    mod_freq = args.frequency
+    if mod_freq:
+        LOG.debug("Using passed in frequency %s", mod_freq)
+        mod_freq = FREQ_SHORT_NAMES.get(mod_freq)
+    # Stage 4
+    try:
+        LOG.debug("Closing stdin")
+        util.close_stdin()
+        util.fixup_output(mods.cfg, None)
+    except:
+        util.logexc(LOG, "Failed to setup output redirection!")
+    if args.debug:
+        # Reset so that all the debug handlers are closed out
+        LOG.debug(("Logging being reset, this logger may no"
+                   " longer be active shortly"))
+        logging.resetLogging()
+    logging.setupLogging(mods.cfg)
+    # Stage 5
+    (which_ran, failures) = mods.run_single(mod_name,
+                                            mod_args,
+                                            mod_freq)
+    if failures:
+        LOG.warn("Ran %s but it failed!", mod_name)
+        return 1
+    elif not which_ran:
+        LOG.warn("Did not run %s, does it exist?", mod_name)
+        return 1
+    else:
+        # Guess it worked
+        return 0
+
+
+def main():
+    parser = argparse.ArgumentParser()
+
+    # Top level args
+    parser.add_argument('--version', '-v', action='version',
+                        version='%(prog)s ' + (version.version_string()))
+    parser.add_argument('--file', '-f', action='append',
+                        dest='files',
+                        help=('additional yaml configuration'
+                              ' files to use'),
+                        type=argparse.FileType('rb'))
+    parser.add_argument('--debug', '-d', action='store_true',
+                        help=('show additional pre-action'
+                              ' logging (default: %(default)s)'),
+                        default=False)
+    parser.add_argument('--force', action='store_true',
+                        help=('force running even if no datasource is'
+                              ' found (use at your own risk)'),
+                        dest='force',
+                        default=False)
+    subparsers = parser.add_subparsers()
+
+    # Each action and its sub-options (if any)
+    parser_init = subparsers.add_parser('init',
+                                        help=('initializes cloud-init and'
+                                              ' performs initial modules'))
+    parser_init.add_argument("--local", '-l', action='store_true',
+                             help="start in local mode (default: %(default)s)",
+                             default=False)
+    # This is used so that we can know which action is selected +
+    # the functor to use to run this subcommand
+    parser_init.set_defaults(action=('init', main_init))
+
+    # These settings are used for the 'config' and 'final' stages
+    parser_mod = subparsers.add_parser('modules',
+                                      help=('activates modules '
+                                            'using a given configuration key'))
+    parser_mod.add_argument("--mode", '-m', action='store',
+                             help=("module configuration name "
+                                    "to use (default: %(default)s)"),
+                             default='config',
+                             choices=('init', 'config', 'final'))
+    parser_mod.set_defaults(action=('modules', main_modules))
+
+    # These settings are used when you want to query information
+    # stored in the cloud-init data objects/directories/files
+    parser_query = subparsers.add_parser('query',
+                                         help=('query information stored '
+                                               'in cloud-init'))
+    parser_query.add_argument("--name", '-n', action="store",
+                              help="item name to query on",
+                              required=True,
+                              choices=QUERY_DATA_TYPES)
+    parser_query.set_defaults(action=('query', main_query))
+
+    # This subcommand allows you to run a single module
+    parser_single = subparsers.add_parser('single',
+                                         help=('run a single module '))
+    parser_single.set_defaults(action=('single', main_single))
+    parser_single.add_argument("--name", '-n', action="store",
+                              help="module name to run",
+                              required=True)
+    parser_single.add_argument("--frequency", action="store",
+                              help=("frequency of the module"),
+                              required=False,
+                              choices=list(FREQ_SHORT_NAMES.keys()))
+    parser_single.add_argument("module_args", nargs="*",
+                              metavar='argument',
+                              help=('any additional arguments to'
+                                    ' pass to this module'))
+    parser_single.set_defaults(action=('single', main_single))
+
+    args = parser.parse_args()
+
+    # Setup basic logging to start (until reinitialized)
+    # iff in debug mode...
+    if args.debug:
+        logging.setupBasicLogging()
+
+    (name, functor) = args.action
+    return functor(name, args)
+
+
+if __name__ == '__main__':
+    sys.exit(main())
 
=== removed file 'cloud-init-cfg.py'
--- cloud-init-cfg.py	2012-01-18 14:07:33 +0000
+++ cloud-init-cfg.py	1970-01-01 00:00:00 +0000
@@ -1,115 +0,0 @@
-#!/usr/bin/python
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <[e-mail]>
-#    Author: Juerg Haefliger <[e-mail]>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import sys
-import cloudinit
-import cloudinit.util as util
-import cloudinit.CloudConfig as CC
-import logging
-import os
-
-
-def Usage(out=sys.stdout):
-    out.write("Usage: %s name\n" % sys.argv[0])
-
-
-def main():
-    # expect to be called with
-    #   name [ freq [ args ]
-    #   run the cloud-config job 'name' at with given args
-    # or
-    #   read cloud config jobs from config (builtin -> system)
-    #   and run all in order
-
-    util.close_stdin()
-
-    modename = "config"
-
-    if len(sys.argv) < 2:
-        Usage(sys.stderr)
-        sys.exit(1)
-    if sys.argv[1] == "all":
-        name = "all"
-        if len(sys.argv) > 2:
-            modename = sys.argv[2]
-    else:
-        freq = None
-        run_args = []
-        name = sys.argv[1]
-        if len(sys.argv) > 2:
-            freq = sys.argv[2]
-            if freq == "None":
-                freq = None
-        if len(sys.argv) > 3:
-            run_args = sys.argv[3:]
-
-    cfg_path = cloudinit.get_ipath_cur("cloud_config")
-    cfg_env_name = cloudinit.cfg_env_name
-    if cfg_env_name in os.environ:
-        cfg_path = os.environ[cfg_env_name]
-
-    cloud = cloudinit.CloudInit(ds_deps=[])  # ds_deps=[], get only cached
-    try:
-        cloud.get_data_source()
-    except cloudinit.DataSourceNotFoundException as e:
-        # there was no datasource found, theres nothing to do
-        sys.exit(0)
-
-    cc = CC.CloudConfig(cfg_path, cloud)
-
-    try:
-        (outfmt, errfmt) = CC.get_output_cfg(cc.cfg, modename)
-        CC.redirect_output(outfmt, errfmt)
-    except Exception as e:
-        err("Failed to get and set output config: %s\n" % e)
-
-    cloudinit.logging_set_from_cfg(cc.cfg)
-    log = logging.getLogger()
-    log.info("cloud-init-cfg %s" % sys.argv[1:])
-
-    module_list = []
-    if name == "all":
-        modlist_cfg_name = "cloud_%s_modules" % modename
-        module_list = CC.read_cc_modules(cc.cfg, modlist_cfg_name)
-        if not len(module_list):
-            err("no modules to run in cloud_config [%s]" % modename, log)
-            sys.exit(0)
-    else:
-        module_list.append([name, freq] + run_args)
-
-    failures = CC.run_cc_modules(cc, module_list, log)
-    if len(failures):
-        err("errors running cloud_config [%s]: %s" % (modename, failures), log)
-    sys.exit(len(failures))
-
-
-def err(msg, log=None):
-    if log:
-        log.error(msg)
-    sys.stderr.write(msg + "\n")
-
-
-def fail(msg, log=None):
-    err(msg, log)
-    sys.exit(1)
-
-if __name__ == '__main__':
-    main()
 
=== removed file 'cloud-init-query.py'
--- cloud-init-query.py	2012-01-18 14:07:33 +0000
+++ cloud-init-query.py	1970-01-01 00:00:00 +0000
@@ -1,56 +0,0 @@
-#!/usr/bin/python
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <[e-mail]>
-#    Author: Juerg Haefliger <[e-mail]>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import sys
-import cloudinit
-import cloudinit.CloudConfig
-
-
-def Usage(out=sys.stdout):
-    out.write("Usage: %s name\n" % sys.argv[0])
-
-
-def main():
-    # expect to be called with name of item to fetch
-    if len(sys.argv) != 2:
-        Usage(sys.stderr)
-        sys.exit(1)
-
-    cfg_path = cloudinit.get_ipath_cur("cloud_config")
-    cc = cloudinit.CloudConfig.CloudConfig(cfg_path)
-    data = {
-        'user_data': cc.cloud.get_userdata(),
-        'user_data_raw': cc.cloud.get_userdata_raw(),
-        'instance_id': cc.cloud.get_instance_id(),
-    }
-
-    name = sys.argv[1].replace('-', '_')
-
-    if name not in data:
-        sys.stderr.write("unknown name '%s'.  Known values are:\n  %s\n" %
-            (sys.argv[1], ' '.join(data.keys())))
-        sys.exit(1)
-
-    print data[name]
-    sys.exit(0)
-
-if __name__ == '__main__':
-    main()
 
=== removed file 'cloud-init.py'
--- cloud-init.py	2012-04-10 20:08:25 +0000
+++ cloud-init.py	1970-01-01 00:00:00 +0000
@@ -1,229 +0,0 @@
-#!/usr/bin/python
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <[e-mail]>
-#    Author: Juerg Haefliger <[e-mail]>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import subprocess
-import sys
-
-import cloudinit
-import cloudinit.util as util
-import cloudinit.CloudConfig as CC
-import cloudinit.DataSource as ds
-import cloudinit.netinfo as netinfo
-import time
-import traceback
-import logging
-import errno
-import os
-
-
-def warn(wstr):
-    sys.stderr.write("WARN:%s" % wstr)
-
-
-def main():
-    util.close_stdin()
-
-    cmds = ("start", "start-local")
-    deps = {"start": (ds.DEP_FILESYSTEM, ds.DEP_NETWORK),
-            "start-local": (ds.DEP_FILESYSTEM, )}
-
-    cmd = ""
-    if len(sys.argv) > 1:
-        cmd = sys.argv[1]
-
-    cfg_path = None
-    if len(sys.argv) > 2:
-        # this is really for debugging only
-        # but you can invoke on development system with ./config/cloud.cfg
-        cfg_path = sys.argv[2]
-
-    if not cmd in cmds:
-        sys.stderr.write("bad command %s. use one of %s\n" % (cmd, cmds))
-        sys.exit(1)
-
-    now = time.strftime("%a, %d %b %Y %H:%M:%S %z", time.gmtime())
-    try:
-        uptimef = open("/proc/uptime")
-        uptime = uptimef.read().split(" ")[0]
-        uptimef.close()
-    except IOError as e:
-        warn("unable to open /proc/uptime\n")
-        uptime = "na"
-
-    cmdline_msg = None
-    cmdline_exc = None
-    if cmd == "start":
-        target = "%s.d/%s" % (cloudinit.system_config,
-            "91_kernel_cmdline_url.cfg")
-        if os.path.exists(target):
-            cmdline_msg = "cmdline: %s existed" % target
-        else:
-            cmdline = util.get_cmdline()
-            try:
-                (key, url, content) = cloudinit.get_cmdline_url(
-                    cmdline=cmdline)
-                if key and content:
-                    util.write_file(target, content, mode=0600)
-                    cmdline_msg = ("cmdline: wrote %s from %s, %s" %
-                        (target, key, url))
-                elif key:
-                    cmdline_msg = ("cmdline: %s, %s had no cloud-config" %
-                        (key, url))
-            except Exception:
-                cmdline_exc = ("cmdline: '%s' raised exception\n%s" %
-                    (cmdline, traceback.format_exc()))
-                warn(cmdline_exc)
-
-    try:
-        cfg = cloudinit.get_base_cfg(cfg_path)
-    except Exception as e:
-        warn("Failed to get base config. falling back to builtin: %s\n" % e)
-        try:
-            cfg = cloudinit.get_builtin_cfg()
-        except Exception as e:
-            warn("Unable to load builtin config\n")
-            raise
-
-    try:
-        (outfmt, errfmt) = CC.get_output_cfg(cfg, "init")
-        CC.redirect_output(outfmt, errfmt)
-    except Exception as e:
-        warn("Failed to get and set output config: %s\n" % e)
-
-    cloudinit.logging_set_from_cfg(cfg)
-    log = logging.getLogger()
-
-    if cmdline_exc:
-        log.debug(cmdline_exc)
-    elif cmdline_msg:
-        log.debug(cmdline_msg)
-
-    try:
-        cloudinit.initfs()
-    except Exception as e:
-        warn("failed to initfs, likely bad things to come: %s\n" % str(e))
-
-    nonet_path = "%s/%s" % (cloudinit.get_cpath("data"), "no-net")
-
-    if cmd == "start":
-        print netinfo.debug_info()
-
-        stop_files = (cloudinit.get_ipath_cur("obj_pkl"), nonet_path)
-        # if starting as the network start, there are cases
-        # where everything is already done for us, and it makes
-        # most sense to exit early and silently
-        for f in stop_files:
-            try:
-                fp = open(f, "r")
-                fp.close()
-            except:
-                continue
-
-            log.debug("no need for cloud-init start to run (%s)\n", f)
-            sys.exit(0)
-    elif cmd == "start-local":
-        # cache is not instance specific, so it has to be purged
-        # but we want 'start' to benefit from a cache if
-        # a previous start-local populated one
-        manclean = util.get_cfg_option_bool(cfg, 'manual_cache_clean', False)
-        if manclean:
-            log.debug("not purging cache, manual_cache_clean = True")
-        cloudinit.purge_cache(not manclean)
-
-        try:
-            os.unlink(nonet_path)
-        except OSError as e:
-            if e.errno != errno.ENOENT:
-                raise
-
-    msg = "cloud-init %s running: %s. up %s seconds" % (cmd, now, uptime)
-    sys.stderr.write(msg + "\n")
-    sys.stderr.flush()
-
-    log.info(msg)
-
-    cloud = cloudinit.CloudInit(ds_deps=deps[cmd])
-
-    try:
-        cloud.get_data_source()
-    except cloudinit.DataSourceNotFoundException as e:
-        sys.stderr.write("no instance data found in %s\n" % cmd)
-        sys.exit(0)
-
-    # set this as the current instance
-    cloud.set_cur_instance()
-
-    # store the metadata
-    cloud.update_cache()
-
-    msg = "found data source: %s" % cloud.datasource
-    sys.stderr.write(msg + "\n")
-    log.debug(msg)
-
-    # parse the user data (ec2-run-userdata.py)
-    try:
-        ran = cloud.sem_and_run("consume_userdata", cloudinit.per_instance,
-            cloud.consume_userdata, [cloudinit.per_instance], False)
-        if not ran:
-            cloud.consume_userdata(cloudinit.per_always)
-    except:
-        warn("consuming user data failed!\n")
-        raise
-
-    cfg_path = cloudinit.get_ipath_cur("cloud_config")
-    cc = CC.CloudConfig(cfg_path, cloud)
-
-    # if the output config changed, update output and err
-    try:
-        outfmt_orig = outfmt
-        errfmt_orig = errfmt
-        (outfmt, errfmt) = CC.get_output_cfg(cc.cfg, "init")
-        if outfmt_orig != outfmt or errfmt_orig != errfmt:
-            warn("stdout, stderr changing to (%s,%s)" % (outfmt, errfmt))
-            CC.redirect_output(outfmt, errfmt)
-    except Exception as e:
-        warn("Failed to get and set output config: %s\n" % e)
-
-    # send the cloud-config ready event
-    cc_path = cloudinit.get_ipath_cur('cloud_config')
-    cc_ready = cc.cfg.get("cc_ready_cmd",
-        ['initctl', 'emit', 'cloud-config',
-         '%s=%s' % (cloudinit.cfg_env_name, cc_path)])
-    if cc_ready:
-        if isinstance(cc_ready, str):
-            cc_ready = ['sh', '-c', cc_ready]
-        subprocess.Popen(cc_ready).communicate()
-
-    module_list = CC.read_cc_modules(cc.cfg, "cloud_init_modules")
-
-    failures = []
-    if len(module_list):
-        failures = CC.run_cc_modules(cc, module_list, log)
-    else:
-        msg = "no cloud_init_modules to run"
-        sys.stderr.write(msg + "\n")
-        log.debug(msg)
-        sys.exit(0)
-
-    sys.exit(len(failures))
-
-if __name__ == '__main__':
-    main()
 
=== removed file 'cloudinit/DataSource.py'
--- cloudinit/DataSource.py	2012-03-19 17:33:39 +0000
+++ cloudinit/DataSource.py	1970-01-01 00:00:00 +0000
@@ -1,214 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <[e-mail]>
-#    Author: Juerg Hafliger <[e-mail]>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-
-DEP_FILESYSTEM = "FILESYSTEM"
-DEP_NETWORK = "NETWORK"
-
-import cloudinit.UserDataHandler as ud
-import cloudinit.util as util
-import socket
-
-
-class DataSource:
-    userdata = None
-    metadata = None
-    userdata_raw = None
-    cfgname = ""
-    # system config (passed in from cloudinit,
-    # cloud-config before input from the DataSource)
-    sys_cfg = {}
-    # datasource config, the cloud-config['datasource']['__name__']
-    ds_cfg = {}  # datasource config
-
-    def __init__(self, sys_cfg=None):
-        if not self.cfgname:
-            name = str(self.__class__).split(".")[-1]
-            if name.startswith("DataSource"):
-                name = name[len("DataSource"):]
-            self.cfgname = name
-        if sys_cfg:
-            self.sys_cfg = sys_cfg
-
-        self.ds_cfg = util.get_cfg_by_path(self.sys_cfg,
-                          ("datasource", self.cfgname), self.ds_cfg)
-
-    def get_userdata(self):
-        if self.userdata == None:
-            self.userdata = ud.preprocess_userdata(self.userdata_raw)
-        return self.userdata
-
-    def get_userdata_raw(self):
-        return(self.userdata_raw)
-
-    # the data sources' config_obj is a cloud-config formated
-    # object that came to it from ways other than cloud-config
-    # because cloud-config content would be handled elsewhere
-    def get_config_obj(self):
-        return({})
-
-    def get_public_ssh_keys(self):
-        keys = []
-        if 'public-keys' not in self.metadata:
-            return([])
-
-        if isinstance(self.metadata['public-keys'], str):
-            return(str(self.metadata['public-keys']).splitlines())
-
-        if isinstance(self.metadata['public-keys'], list):
-            return(self.metadata['public-keys'])
-
-        for _keyname, klist in self.metadata['public-keys'].items():
-            # lp:506332 uec metadata service responds with
-            # data that makes boto populate a string for 'klist' rather
-            # than a list.
-            if isinstance(klist, str):
-                klist = [klist]
-            for pkey in klist:
-                # there is an empty string at the end of the keylist, trim it
-                if pkey:
-                    keys.append(pkey)
-
-        return(keys)
-
-    def device_name_to_device(self, _name):
-        # translate a 'name' to a device
-        # the primary function at this point is on ec2
-        # to consult metadata service, that has
-        #  ephemeral0: sdb
-        # and return 'sdb' for input 'ephemeral0'
-        return(None)
-
-    def get_locale(self):
-        return('en_US.UTF-8')
-
-    def get_local_mirror(self):
-        return None
-
-    def get_instance_id(self):
-        if 'instance-id' not in self.metadata:
-            return "iid-datasource"
-        return(self.metadata['instance-id'])
-
-    def get_hostname(self, fqdn=False):
-        defdomain = "localdomain"
-        defhost = "localhost"
-
-        domain = defdomain
-        if not 'local-hostname' in self.metadata:
-
-            # this is somewhat questionable really.
-            # the cloud datasource was asked for a hostname
-            # and didn't have one. raising error might be more appropriate
-            # but instead, basically look up the existing hostname
-            toks = []
-
-            hostname = socket.gethostname()
-
-            fqdn = util.get_fqdn_from_hosts(hostname)
-
-            if fqdn and fqdn.find(".") > 0:
-                toks = str(fqdn).split(".")
-            elif hostname:
-                toks = [hostname, defdomain]
-            else:
-                toks = [defhost, defdomain]
-
-        else:
-            # if there is an ipv4 address in 'local-hostname', then
-            # make up a hostname (LP: #475354) in format ip-xx.xx.xx.xx
-            lhost = self.metadata['local-hostname']
-            if is_ipv4(lhost):
-                toks = "ip-%s" % lhost.replace(".", "-")
-            else:
-                toks = lhost.split(".")
-
-        if len(toks) > 1:
-            hostname = toks[0]
-            domain = '.'.join(toks[1:])
-        else:
-            hostname = toks[0]
-
-        if fqdn:
-            return "%s.%s" % (hostname, domain)
-        else:
-            return hostname
-
-
-# return a list of classes that have the same depends as 'depends'
-# iterate through cfg_list, loading "DataSourceCollections" modules
-# and calling their "get_datasource_list".
-# return an ordered list of classes that match
-#
-# - modules must be named "DataSource<item>", where 'item' is an entry
-#   in cfg_list
-# - if pkglist is given, it will iterate try loading from that package
-#   ie, pkglist=[ "foo", "" ]
-#     will first try to load foo.DataSource<item>
-#     then DataSource<item>
-def list_sources(cfg_list, depends, pkglist=None):
-    if pkglist is None:
-        pkglist = []
-    retlist = []
-    for ds_coll in cfg_list:
-        for pkg in pkglist:
-            if pkg:
-                pkg = "%s." % pkg
-            try:
-                mod = __import__("%sDataSource%s" % (pkg, ds_coll))
-                if pkg:
-                    mod = getattr(mod, "DataSource%s" % ds_coll)
-                lister = getattr(mod, "get_datasource_list")
-                retlist.extend(lister(depends))
-                break
-            except:
-                raise
-    return(retlist)
-
-
-# depends is a list of dependencies (DEP_FILESYSTEM)
-# dslist is a list of 2 item lists
-# dslist = [
-#   ( class, ( depends-that-this-class-needs ) )
-# }
-# it returns a list of 'class' that matched these deps exactly
-# it is a helper function for DataSourceCollections
-def list_from_depends(depends, dslist):
-    retlist = []
-    depset = set(depends)
-    for elem in dslist:
-        (cls, deps) = elem
-        if depset == set(deps):
-            retlist.append(cls)
-    return(retlist)
-
-
-def is_ipv4(instr):
-    """ determine if input string is a ipv4 address. return boolean"""
-    toks = instr.split('.')
-    if len(toks) != 4:
-        return False
-
-    try:
-        toks = [x for x in toks if (int(x) < 256 and int(x) > 0)]
-    except:
-        return False
-
-    return (len(toks) == 4)
 
=== removed file 'cloudinit/UserDataHandler.py'
--- cloudinit/UserDataHandler.py	2012-06-21 15:37:22 +0000
+++ cloudinit/UserDataHandler.py	1970-01-01 00:00:00 +0000
@@ -1,262 +0,0 @@
-# vi: ts=4 expandtab
-#
-#    Copyright (C) 2009-2010 Canonical Ltd.
-#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
-#
-#    Author: Scott Moser <[e-mail]>
-#    Author: Juerg Hafliger <[e-mail]>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License version 3, as
-#    published by the Free Software Foundation.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import email
-
-from email.mime.multipart import MIMEMultipart
-from email.mime.text import MIMEText
-from email.mime.base import MIMEBase
-import yaml
-import cloudinit
-import cloudinit.util as util
-import hashlib
-import urllib
-
-
-starts_with_mappings = {
-    '#include': 'text/x-include-url',
-    '#include-once': 'text/x-include-once-url',
-    '#!': 'text/x-shellscript',
-    '#cloud-config': 'text/cloud-config',
-    '#upstart-job': 'text/upstart-job',
-    '#part-handler': 'text/part-handler',
-    '#cloud-boothook': 'text/cloud-boothook',
-    '#cloud-config-archive': 'text/cloud-config-archive',
-}
-
-
-# if 'string' is compressed return decompressed otherwise return it
-def decomp_str(string):
-    import StringIO
-    import gzip
-    try:
-        uncomp = gzip.GzipFile(None, "rb", 1, StringIO.StringIO(string)).read()
-        return(uncomp)
-    except:
-        return(string)
-
-
-def do_include(content, appendmsg):
-    import os
-    # is just a list of urls, one per line
-    # also support '#include <url here>'
-    includeonce = False
-    for line in content.splitlines():
-        if line == "#include":
-            continue
-        if line == "#include-once":
-            includeonce = True
-            continue
-        if line.startswith("#include-once"):
-            line = line[len("#include-once"):].lstrip()
-            includeonce = True
-        elif line.startswith("#include"):
-            line = line[len("#include"):].lstrip()
-        if line.startswith("#"):
-            continue
-        if line.strip() == "":
-            continue
-
-        # urls cannot not have leading or trailing white space
-        msum = hashlib.md5()  # pylint: disable=E1101
-        msum.update(line.strip())
-        includeonce_filename = "%s/urlcache/%s" % (
-            cloudinit.get_ipath_cur("data"), msum.hexdigest())
-        try:
-            if includeonce and os.path.isfile(includeonce_filename):
-                with open(includeonce_filename, "r") as fp:
-                    content = fp.read()
-            else:
-                content = urllib.urlopen(line).read()
-                if includeonce:
-                    util.write_file(includeonce_filename, content, mode=0600)
-        except Exception:
-            raise
-
-        process_includes(message_from_string(decomp_str(content)), appendmsg)
-
-
-def explode_cc_archive(archive, appendmsg):
-    for ent in yaml.safe_load(archive):
-        # ent can be one of:
-        #  dict { 'filename' : 'value', 'content' : 'value', 'type' : 'value' }
-        #    filename and type not be present
-        # or
-        #  scalar(payload)
-
-        def_type = "text/cloud-config"
-        if isinstance(ent, str):
-            ent = {'content': ent}
-
-        content = ent.get('content', '')
-        mtype = ent.get('type', None)
-        if mtype == None:
-            mtype = type_from_startswith(content, def_type)
-
-        maintype, subtype = mtype.split('/', 1)
-        if maintype == "text":
-            msg = MIMEText(content, _subtype=subtype)
-        else:
-            msg = MIMEBase(maintype, subtype)
-            msg.set_payload(content)
-
-        if 'filename' in ent:
-            msg.add_header('Content-Disposition', 'attachment',
-                           filename=ent['filename'])
-
-        for header in ent.keys():
-            if header in ('content', 'filename', 'type'):
-                continue
-            msg.add_header(header, ent['header'])
-
-        _attach_part(appendmsg, msg)
-
-
-def multi_part_count(outermsg, newcount=None):
-    """
-    Return the number of attachments to this MIMEMultipart by looking
-    at its 'Number-Attachments' header.
-    """
-    nfield = 'Number-Attachments'
-    if nfield not in outermsg:
-        outermsg[nfield] = "0"
-
-    if newcount != None:
-        outermsg.replace_header(nfield, str(newcount))
-
-    return(int(outermsg.get('Number-Attachments', 0)))
-
-
-def _attach_part(outermsg, part):
-    """
-    Attach an part to an outer message. outermsg must be a MIMEMultipart.
-    Modifies a header in outermsg to keep track of number of attachments.
-    """
-    cur = multi_part_count(outermsg)
-    if not part.get_filename(None):
-        part.add_header('Content-Disposition', 'attachment',
-            filename='part-%03d' % (cur + 1))
-    outermsg.attach(part)
-    multi_part_count(outermsg, cur + 1)
-
-
-def type_from_startswith(payload, default=None):
-    # slist is sorted longest first
-    slist = sorted(starts_with_mappings.keys(), key=lambda e: 0 - len(e))
-    for sstr in slist:
-        if payload.startswith(sstr):
-            return(starts_with_mappings[sstr])
-    return default
-
-
-def process_includes(msg, appendmsg=None):
-    if appendmsg == None:
-        appendmsg = MIMEMultipart()
-
-    for part in msg.walk():
-        # multipart/* are just containers
-        if part.get_content_maintype() == 'multipart':
-            continue
-
-        ctype = None
-        ctype_orig = part.get_content_type()
-
-        payload = part.get_payload(decode=True)
-
-        if ctype_orig in ("text/plain", "text/x-not-multipart"):
-            ctype = type_from_startswith(payload)
-
-        if ctype is None:
-            ctype = ctype_orig
-
-        if ctype in ('text/x-include-url', 'text/x-include-once-url'):
-            do_include(payload, appendmsg)
-            continue
-
-        if ctype == "text/cloud-config-archive":
-            explode_cc_archive(payload, appendmsg)
-            continue
-
-        if 'Content-Type' in msg:
-            msg.replace_header('Content-Type', ctype)
-        else:
-            msg['Content-Type'] = ctype
-
-        _attach_part(appendmsg, part)
-
-
-def message_from_string(data, headers=None):
-    if headers is None:
-        headers = {}
-    if "mime-version:" in data[0:4096].lower():
-        msg = email.message_from_string(data)
-        for (key, val) in headers.items():
-            if key in msg:
-                msg.replace_header(key, val)
-            else:
-                msg[key] = val
-    else:
-        mtype = headers.get("Content-Type", "text/x-not-multipart")
-        maintype, subtype = mtype.split("/", 1)
-        msg = MIMEBase(maintype, subtype, *headers)
-        msg.set_payload(data)
-
-    return(msg)
-
-
-# this is heavily wasteful, reads through userdata string input
-def preprocess_userdata(data):
-    newmsg = MIMEMultipart()
-    process_includes(message_from_string(decomp_str(data)), newmsg)
-    return(newmsg.as_string())
-
-
-# callback is a function that will be called with (data, content_type,
-# filename, payload)
-def walk_userdata(istr, callback, data=None):
-    partnum = 0
-    for part in message_from_string(istr).walk():
-        # multipart/* are just containers
-        if part.get_content_maintype() == 'multipart':
-            continue
-
-        ctype = part.get_content_type()
-        if ctype is None:
-            ctype = 'application/octet-stream'
-
-        filename = part.get_filename()
-        if not filename:
-            filename = 'part-%03d' % partnum
-
-        callback(data, ctype, filename, part.get_payload(decode=True))
-
-        partnum = partnum + 1
-
-
-if __name__ == "__main__":
-    def main():
-        import sys
-        data = decomp_str(file(sys.argv[1]).read())
-        newmsg = MIMEMultipart()
-        process_includes(message_from_string(data), newmsg)
-        print newmsg
-        print "#found %s parts" % multi_part_count(newmsg)
-
-    main()
 
=== modified file 'cloudinit/__init__.py'
--- cloudinit/__init__.py	2012-06-28 17:10:56 +0000
+++ cloudinit/__init__.py	2012-07-06 21:16:18 +0000
@@ -1,11 +1,12 @@
 # vi: ts=4 expandtab
 #
-#    Common code for the EC2 initialisation scripts in Ubuntu
-#    Copyright (C) 2008-2009 Canonical Ltd
+#    Copyright (C) 2012 Canonical Ltd.
 #    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
 #
-#    Author: Soren Hansen <[e-mail]>
+#    Author: Scott Moser <[e-mail]>
 #    Author: Juerg Haefliger <[e-mail]>
+#    Author: Joshua Harlow <[e-mail]>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License version 3, as
@@ -18,650 +19,3 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-varlibdir = '/var/lib/cloud'
-cur_instance_link = varlibdir + "/instance"
-boot_finished = cur_instance_link + "/boot-finished"
-system_config = '/etc/cloud/cloud.cfg'
-seeddir = varlibdir + "/seed"
-cfg_env_name = "CLOUD_CFG"
-
-cfg_builtin = """
-log_cfgs: []
-datasource_list: ["NoCloud", "ConfigDrive", "OVF", "MAAS", "Ec2", "CloudStack"]
-def_log_file: /var/log/cloud-init.log
-syslog_fix_perms: syslog:adm
-"""
-logger_name = "cloudinit"
-
-pathmap = {
-   "handlers": "/handlers",
-   "scripts": "/scripts",
-   "sem": "/sem",
-   "boothooks": "/boothooks",
-   "userdata_raw": "/user-data.txt",
-   "userdata": "/user-data.txt.i",
-   "obj_pkl": "/obj.pkl",
-   "cloud_config": "/cloud-config.txt",
-   "data": "/data",
-   None: "",
-}
-
-per_instance = "once-per-instance"
-per_always = "always"
-per_once = "once"
-
-parsed_cfgs = {}
-
-import os
-
-import cPickle
-import sys
-import os.path
-import errno
-import subprocess
-import yaml
-import logging
-import logging.config
-import StringIO
-import glob
-import traceback
-
-import cloudinit.util as util
-
-
-class NullHandler(logging.Handler):
-    def emit(self, record):
-        pass
-
-
-log = logging.getLogger(logger_name)
-log.addHandler(NullHandler())
-
-
-def logging_set_from_cfg_file(cfg_file=system_config):
-    logging_set_from_cfg(util.get_base_cfg(cfg_file, cfg_builtin, parsed_cfgs))
-
-
-def logging_set_from_cfg(cfg):
-    log_cfgs = []
-    logcfg = util.get_cfg_option_str(cfg, "log_cfg", False)
-    if logcfg:
-        # if there is a 'logcfg' entry in the config, respect
-        # it, it is the old keyname
-        log_cfgs = [logcfg]
-    elif "log_cfgs" in cfg:
-        for cfg in cfg['log_cfgs']:
-            if isinstance(cfg, list):
-                log_cfgs.append('\n'.join(cfg))
-            else:
-                log_cfgs.append()
-
-    if not len(log_cfgs):
-        sys.stderr.write("Warning, no logging configured\n")
-        return
-
-    for logcfg in log_cfgs:
-        try:
-            logging.config.fileConfig(StringIO.StringIO(logcfg))
-            return
-        except:
-            pass
-
-    raise Exception("no valid logging found\n")
-
-
-import cloudinit.DataSource as DataSource
-import cloudinit.UserDataHandler as UserDataHandler
-
-
-class CloudInit:
-    cfg = None
-    part_handlers = {}
-    old_conffile = '/etc/ec2-init/ec2-config.cfg'
-    ds_deps = [DataSource.DEP_FILESYSTEM, DataSource.DEP_NETWORK]
-    datasource = None
-    cloud_config_str = ''
-    datasource_name = ''
-
-    builtin_handlers = []
-
-    def __init__(self, ds_deps=None, sysconfig=system_config):
-        self.builtin_handlers = [
-            ['text/x-shellscript', self.handle_user_script, per_always],
-            ['text/cloud-config', self.handle_cloud_config, per_always],
-            ['text/upstart-job', self.handle_upstart_job, per_instance],
-            ['text/cloud-boothook', self.handle_cloud_boothook, per_always],
-        ]
-
-        if ds_deps != None:
-            self.ds_deps = ds_deps
-
-        self.sysconfig = sysconfig
-
-        self.cfg = self.read_cfg()
-
-    def read_cfg(self):
-        if self.cfg:
-            return(self.cfg)
-
-        try:
-            conf = util.get_base_cfg(self.sysconfig, cfg_builtin, parsed_cfgs)
-        except Exception:
-            conf = get_builtin_cfg()
-
-        # support reading the old ConfigObj format file and merging
-        # it into the yaml dictionary
-        try:
-            from configobj import ConfigObj
-            oldcfg = ConfigObj(self.old_conffile)
-            if oldcfg is None:
-                oldcfg = {}
-            conf = util.mergedict(conf, oldcfg)
-        except:
-            pass
-
-        return(conf)
-
-    def restore_from_cache(self):
-        try:
-            # we try to restore from a current link and static path
-            # by using the instance link, if purge_cache was called
-            # the file wont exist
-            cache = get_ipath_cur('obj_pkl')
-            f = open(cache, "rb")
-            data = cPickle.load(f)
-            f.close()
-            self.datasource = data
-            return True
-        except:
-            return False
-
-    def write_to_cache(self):
-        cache = self.get_ipath("obj_pkl")
-        try:
-            os.makedirs(os.path.dirname(cache))
-        except OSError as e:
-            if e.errno != errno.EEXIST:
-                return False
-
-        try:
-            f = open(cache, "wb")
-            cPickle.dump(self.datasource, f)
-            f.close()
-            os.chmod(cache, 0400)
-        except:
-            raise
-
-    def get_data_source(self):
-        if self.datasource is not None:
-            return True
-
-        if self.restore_from_cache():
-            log.debug("restored from cache type %s" % self.datasource)
-            return True
-
-        cfglist = self.cfg['datasource_list']
-        dslist = list_sources(cfglist, self.ds_deps)
-        dsnames = [f.__name__ for f in dslist]
-
-        log.debug("searching for data source in %s" % dsnames)
-        for cls in dslist:
-            ds = cls.__name__
-            try:
-                s = cls(sys_cfg=self.cfg)
-                if s.get_data():
-                    self.datasource = s
-                    self.datasource_name = ds
-                    log.debug("found data source %s" % ds)
-                    return True
-            except Exception as e:
-                log.warn("get_data of %s raised %s" % (ds, e))
-                util.logexc(log)
-        msg = "Did not find data source. searched classes: %s" % dsnames
-        log.debug(msg)
-        raise DataSourceNotFoundException(msg)
-
-    def set_cur_instance(self):
-        try:
-            os.unlink(cur_instance_link)
-        except OSError as e:
-            if e.errno != errno.ENOENT:
-                raise
-
-        iid = self.get_instance_id()
-        os.symlink("./instances/%s" % iid, cur_instance_link)
-        idir = self.get_ipath()
-        dlist = []
-        for d in ["handlers", "scripts", "sem"]:
-            dlist.append("%s/%s" % (idir, d))
-
-        util.ensure_dirs(dlist)
-
-        ds = "%s: %s\n" % (self.datasource.__class__, str(self.datasource))
-        dp = self.get_cpath('data')
-        util.write_file("%s/%s" % (idir, 'datasource'), ds)
-        util.write_file("%s/%s" % (dp, 'previous-datasource'), ds)
-        util.write_file("%s/%s" % (dp, 'previous-instance-id'), "%s\n" % iid)
-
-    def get_userdata(self):
-        return(self.datasource.get_userdata())
-
-    def get_userdata_raw(self):
-        return(self.datasource.get_userdata_raw())
-
-    def get_instance_id(self):
-        return(self.datasource.get_instance_id())
-
-    def update_cache(self):
-        self.write_to_cache()
-        self.store_userdata()
-
-    def store_userdata(self):
-        util.write_file(self.get_ipath('userdata_raw'),
-            self.datasource.get_userdata_raw(), 0600)
-        util.write_file(self.get_ipath('userdata'),
-            self.datasource.get_userdata(), 0600)
-
-    def sem_getpath(self, name, freq):
-        if freq == 'once-per-instance':
-            return("%s/%s" % (self.get_ipath("sem"), name))
-        return("%s/%s.%s" % (get_cpath("sem"), name, freq))
-
-    def sem_has_run(self, name, freq):
-        if freq == per_always:
-            return False
-        semfile = self.sem_getpath(name, freq)
-        if os.path.exists(semfile):
-            return True
-        return False
-
-    def sem_acquire(self, name, freq):
-        from time import time
-        semfile = self.sem_getpath(name, freq)
-
-        try:
-            os.makedirs(os.path.dirname(semfile))
-        except OSError as e:
-            if e.errno != errno.EEXIST:
-                raise e
-
-        if os.path.exists(semfile) and freq != per_always:
-            return False
-
-        # race condition
-        try:
-            f = open(semfile, "w")
-            f.write("%s\n" % str(time()))
-            f.close()
-        except:
-            return(False)
-        return(True)
-
-    def sem_clear(self, name, freq):
-        semfile = self.sem_getpath(name, freq)
-        try:
-            os.unlink(semfile)
-        except OSError as e:
-            if e.errno != errno.ENOENT:
-                return False
-
-        return True
-
-    # acquire lock on 'name' for given 'freq'
-    # if that does not exist, then call 'func' with given 'args'
-    # if 'clear_on_fail' is True and func throws an exception
-    #  then remove the lock (so it would run again)
-    def sem_and_run(self, semname, freq, func, args=None, clear_on_fail=False):
-        if args is None:
-            args = []
-        if self.sem_has_run(semname, freq):
-            log.debug("%s already ran %s", semname, freq)
-            return False
-        try:
-            if not self.sem_acquire(semname, freq):
-                raise Exception("Failed to acquire lock on %s" % semname)
-
-            func(*args)
-        except:
-            if clear_on_fail:
-                self.sem_clear(semname, freq)
-            raise
-
-        return True
-
-    # get_ipath : get the instance path for a name in pathmap
-    # (/var/lib/cloud/instances/<instance>/name)<name>)
-    def get_ipath(self, name=None):
-        return("%s/instances/%s%s"
-               % (varlibdir, self.get_instance_id(), pathmap[name]))
-
-    def consume_userdata(self, frequency=per_instance):
-        self.get_userdata()
-        data = self
-
-        cdir = get_cpath("handlers")
-        idir = self.get_ipath("handlers")
-
-        # add the path to the plugins dir to the top of our list for import
-        # instance dir should be read before cloud-dir
-        sys.path.insert(0, cdir)
-        sys.path.insert(0, idir)
-
-        part_handlers = {}
-        # add handlers in cdir
-        for fname in glob.glob("%s/*.py" % cdir):
-            if not os.path.isfile(fname):
-                continue
-            modname = os.path.basename(fname)[0:-3]
-            try:
-                mod = __import__(modname)
-                handler_register(mod, part_handlers, data, frequency)
-                log.debug("added handler for [%s] from %s" % (mod.list_types(),
-                                                              fname))
-            except:
-                log.warn("failed to initialize handler in %s" % fname)
-                util.logexc(log)
-
-        # add the internal handers if their type hasn't been already claimed
-        for (btype, bhand, bfreq) in self.builtin_handlers:
-            if btype in part_handlers:
-                continue
-            handler_register(InternalPartHandler(bhand, [btype], bfreq),
-                part_handlers, data, frequency)
-
-        # walk the data
-        pdata = {'handlers': part_handlers, 'handlerdir': idir,
-                 'data': data, 'frequency': frequency}
-        UserDataHandler.walk_userdata(self.get_userdata(),
-            partwalker_callback, data=pdata)
-
-        # give callbacks opportunity to finalize
-        called = []
-        for (_mtype, mod) in part_handlers.iteritems():
-            if mod in called:
-                continue
-            handler_call_end(mod, data, frequency)
-
-    def handle_user_script(self, _data, ctype, filename, payload, _frequency):
-        if ctype == "__end__":
-            return
-        if ctype == "__begin__":
-            # maybe delete existing things here
-            return
-
-        filename = filename.replace(os.sep, '_')
-        scriptsdir = get_ipath_cur('scripts')
-        util.write_file("%s/%s" %
-            (scriptsdir, filename), util.dos2unix(payload), 0700)
-
-    def handle_upstart_job(self, _data, ctype, filename, payload, frequency):
-        # upstart jobs are only written on the first boot
-        if frequency != per_instance:
-            return
-
-        if ctype == "__end__" or ctype == "__begin__":
-            return
-        if not filename.endswith(".conf"):
-            filename = filename + ".conf"
-
-        util.write_file("%s/%s" % ("/etc/init", filename),
-            util.dos2unix(payload), 0644)
-
-    def handle_cloud_config(self, _data, ctype, filename, payload, _frequency):
-        if ctype == "__begin__":
-            self.cloud_config_str = ""
-            return
-        if ctype == "__end__":
-            cloud_config = self.get_ipath("cloud_config")
-            util.write_file(cloud_config, self.cloud_config_str, 0600)
-
-            ## this could merge the cloud config with the system config
-            ## for now, not doing this as it seems somewhat circular
-            ## as CloudConfig does that also, merging it with this cfg
-            ##
-            # ccfg = yaml.safe_load(self.cloud_config_str)
-            # if ccfg is None: ccfg = {}
-            # self.cfg = util.mergedict(ccfg, self.cfg)
-
-            return
-
-        self.cloud_config_str += "\n#%s\n%s" % (filename, payload)
-
-    def handle_cloud_boothook(self, _data, ctype, filename, payload,
-                              _frequency):
-        if ctype == "__end__":
-            return
-        if ctype == "__begin__":
-            return
-
-        filename = filename.replace(os.sep, '_')
-        payload = util.dos2unix(payload)
-        prefix = "#cloud-boothook"
-        start = 0
-        if payload.startswith(prefix):
-            start = len(prefix) + 1
-
-        boothooks_dir = self.get_ipath("boothooks")
-        filepath = "%s/%s" % (boothooks_dir, filename)
-        util.write_file(filepath, payload[start:], 0700)
-        try:
-            env = os.environ.copy()
-            env['INSTANCE_ID'] = self.datasource.get_instance_id()
-            subprocess.check_call([filepath], env=env)
-        except subprocess.CalledProcessError as e:
-            log.error("boothooks script %s returned %i" %
-                (filepath, e.returncode))
-        except Exception as e:
-            log.error("boothooks unknown exception %s when running %s" %
-                (e, filepath))
-
-    def get_public_ssh_keys(self):
-        return(self.datasource.get_public_ssh_keys())
-
-    def get_locale(self):
-        return(self.datasource.get_locale())
-
-    def get_mirror(self):
-        return(self.datasource.get_local_mirror())
-
-    def get_hostname(self, fqdn=False):
-        return(self.datasource.get_hostname(fqdn=fqdn))
-
-    def device_name_to_device(self, name):
-        return(self.datasource.device_name_to_device(name))
-
-    # I really don't know if this should be here or not, but
-    # I needed it in cc_update_hostname, where that code had a valid 'cloud'
-    # reference, but did not have a cloudinit handle
-    # (ie, no cloudinit.get_cpath())
-    def get_cpath(self, name=None):
-        return(get_cpath(name))
-
-
-def initfs():
-    subds = ['scripts/per-instance', 'scripts/per-once', 'scripts/per-boot',
-             'seed', 'instances', 'handlers', 'sem', 'data']
-    dlist = []
-    for subd in subds:
-        dlist.append("%s/%s" % (varlibdir, subd))
-    util.ensure_dirs(dlist)
-
-    cfg = util.get_base_cfg(system_config, cfg_builtin, parsed_cfgs)
-    log_file = util.get_cfg_option_str(cfg, 'def_log_file', None)
-    perms = util.get_cfg_option_str(cfg, 'syslog_fix_perms', None)
-    if log_file:
-        fp = open(log_file, "ab")
-        fp.close()
-    if log_file and perms:
-        (u, g) = perms.split(':', 1)
-        if u == "-1" or u == "None":
-            u = None
-        if g == "-1" or g == "None":
-            g = None
-        util.chownbyname(log_file, u, g)
-
-
-def purge_cache(rmcur=True):
-    rmlist = [boot_finished]
-    if rmcur:
-        rmlist.append(cur_instance_link)
-    for f in rmlist:
-        try:
-            os.unlink(f)
-        except OSError as e:
-            if e.errno == errno.ENOENT:
-                continue
-            return(False)
-        except:
-            return(False)
-    return(True)
-
-
-# get_ipath_cur: get the current instance path for an item
-def get_ipath_cur(name=None):
-    return("%s/%s%s" % (varlibdir, "instance", pathmap[name]))
-
-
-# get_cpath : get the "clouddir" (/var/lib/cloud/<name>)
-# for a name in dirmap
-def get_cpath(name=None):
-    return("%s%s" % (varlibdir, pathmap[name]))
-
-
-def get_base_cfg(cfg_path=None):
-    if cfg_path is None:
-        cfg_path = system_config
-    return(util.get_base_cfg(cfg_path, cfg_builtin, parsed_cfgs))
-
-
-def get_builtin_cfg():
-    return(yaml.safe_load(cfg_builtin))
-
-
-class DataSourceNotFoundException(Exception):
-    pass
-
-
-def list_sources(cfg_list, depends):
-    return(DataSource.list_sources(cfg_list, depends, ["cloudinit", ""]))
-
-
-def handler_register(mod, part_handlers, data, frequency=per_instance):
-    if not hasattr(mod, "handler_version"):
-        setattr(mod, "handler_version", 1)
-
-    for mtype in mod.list_types():
-        part_handlers[mtype] = mod
-
-    handler_call_begin(mod, data, frequency)
-    return(mod)
-
-
-def handler_call_begin(mod, data, frequency):
-    handler_handle_part(mod, data, "__begin__", None, None, frequency)
-
-
-def handler_call_end(mod, data, frequency):
-    handler_handle_part(mod, data, "__end__", None, None, frequency)
-
-
-def handler_handle_part(mod, data, ctype, filename, payload, frequency):
-    # only add the handler if the module should run
-    modfreq = getattr(mod, "frequency", per_instance)
-    if not (modfreq == per_always or
-            (frequency == per_instance and modfreq == per_instance)):
-        return
-    try:
-        if mod.handler_version == 1:
-            mod.handle_part(data, ctype, filename, payload)
-        else:
-            mod.handle_part(data, ctype, filename, payload, frequency)
-    except:
-        util.logexc(log)
-        traceback.print_exc(file=sys.stderr)
-
-
-def partwalker_handle_handler(pdata, _ctype, _filename, payload):
-    curcount = pdata['handlercount']
-    modname = 'part-handler-%03d' % curcount
-    frequency = pdata['frequency']
-
-    modfname = modname + ".py"
-    util.write_file("%s/%s" % (pdata['handlerdir'], modfname), payload, 0600)
-
-    try:
-        mod = __import__(modname)
-        handler_register(mod, pdata['handlers'], pdata['data'], frequency)
-        pdata['handlercount'] = curcount + 1
-    except:
-        util.logexc(log)
-        traceback.print_exc(file=sys.stderr)
-
-
-def partwalker_callback(pdata, ctype, filename, payload):
-    # data here is the part_handlers array and then the data to pass through
-    if ctype == "text/part-handler":
-        if 'handlercount' not in pdata:
-            pdata['handlercount'] = 0
-        partwalker_handle_handler(pdata, ctype, filename, payload)
-        return
-    if ctype not in pdata['handlers'] and payload:
-        if ctype == "text/x-not-multipart":
-            # Extract the first line or 24 bytes for displaying in the log
-            start = payload.split("\n", 1)[0][:24]
-            if start < payload:
-                details = "starting '%s...'" % start.encode("string-escape")
-            else:
-                details = repr(payload)
-            log.warning("Unhandled non-multipart userdata %s", details)
-        return
-    handler_handle_part(pdata['handlers'][ctype], pdata['data'],
-        ctype, filename, payload, pdata['frequency'])
-
-
-class InternalPartHandler:
-    freq = per_instance
-    mtypes = []
-    handler_version = 1
-    handler = None
-
-    def __init__(self, handler, mtypes, frequency, version=2):
-        self.handler = handler
-        self.mtypes = mtypes
-        self.frequency = frequency
-        self.handler_version = version
-
-    def __repr__(self):
-        return("InternalPartHandler: [%s]" % self.mtypes)
-
-    def list_types(self):
-        return(self.mtypes)
-
-    def handle_part(self, data, ctype, filename, payload, frequency):
-        return(self.handler(data, ctype, filename, payload, frequency))
-
-
-def get_cmdline_url(names=('cloud-config-url', 'url'),
-                    starts="#cloud-config", cmdline=None):
-
-    if cmdline == None:
-        cmdline = util.get_cmdline()
-
-    data = util.keyval_str_to_dict(cmdline)
-    url = None
-    key = None
-    for key in names:
-        if key in data:
-            url = data[key]
-            break
-    if url == None:
-        return (None, None, None)
-
-    contents = util.readurl(url)
-
-    if contents.startswith(starts):
-        return (key, url, contents)
-
-    return (key, url, None)
 
=== added file 'cloudinit/cloud.py'
--- cloudinit/cloud.py	1970-01-01 00:00:00 +0000
+++ cloudinit/cloud.py	2012-07-06 21:16:18 +0000
@@ -0,0 +1,101 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <[e-mail]>
+#    Author: Juerg Haefliger <[e-mail]>
+#    Author: Joshua Harlow <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import copy
+import os
+
+from cloudinit import log as logging
+
+LOG = logging.getLogger(__name__)
+
+# This class is the high level wrapper that provides
+# access to cloud-init objects without exposing the stage objects
+# to handler and or module manipulation. It allows for cloud
+# init to restrict what those types of user facing code may see
+# and or adjust (which helps avoid code messing with each other)
+#
+# It also provides util functions that avoid having to know
+# how to get a certain member from this submembers as well
+# as providing a backwards compatible object that can be maintained
+# while the stages/other objects can be worked on independently...
+
+
+class Cloud(object):
+    def __init__(self, datasource, paths, cfg, distro, runners):
+        self.datasource = datasource
+        self.paths = paths
+        self.distro = distro
+        self._cfg = cfg
+        self._runners = runners
+
+    # If a 'user' manipulates logging or logging services
+    # it is typically useful to cause the logging to be
+    # setup again.
+    def cycle_logging(self):
+        logging.resetLogging()
+        logging.setupLogging(self.cfg)
+
+    @property
+    def cfg(self):
+        # Ensure that not indirectly modified
+        return copy.deepcopy(self._cfg)
+
+    def run(self, name, functor, args, freq=None, clear_on_fail=False):
+        return self._runners.run(name, functor, args, freq, clear_on_fail)
+
+    def get_template_filename(self, name):
+        fn = self.paths.template_tpl % (name)
+        if not os.path.isfile(fn):
+            LOG.warn("No template found at %s for template named %s", fn, name)
+            return None
+        return fn
+
+    # The rest of thes are just useful proxies
+    def get_userdata(self):
+        return self.datasource.get_userdata()
+
+    def get_instance_id(self):
+        return self.datasource.get_instance_id()
+
+    def get_public_ssh_keys(self):
+        return self.datasource.get_public_ssh_keys()
+
+    def get_locale(self):
+        return self.datasource.get_locale()
+
+    def get_local_mirror(self):
+        return self.datasource.get_local_mirror()
+
+    def get_hostname(self, fqdn=False):
+        return self.datasource.get_hostname(fqdn=fqdn)
+
+    def device_name_to_device(self, name):
+        return self.datasource.device_name_to_device(name)
+
+    def get_ipath_cur(self, name=None):
+        return self.paths.get_ipath_cur(name)
+
+    def get_cpath(self, name=None):
+        return self.paths.get_cpath(name)
+
+    def get_ipath(self, name=None):
+        return self.paths.get_ipath(name)
 
=== renamed directory 'cloudinit/CloudConfig' => 'cloudinit/config'
=== modified file 'cloudinit/config/__init__.py'
--- cloudinit/CloudConfig/__init__.py	2012-06-13 13:11:27 +0000
+++ cloudinit/config/__init__.py	2012-07-06 21:16:18 +0000
@@ -19,256 +19,38 @@
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-import yaml
-import cloudinit
-import cloudinit.util as util
-import sys
-import traceback
-import os
-import subprocess
-import time
-
-per_instance = cloudinit.per_instance
-per_always = cloudinit.per_always
-per_once = cloudinit.per_once
-
-
-class CloudConfig():
-    cfgfile = None
-    cfg = None
-
-    def __init__(self, cfgfile, cloud=None, ds_deps=None):
-        if cloud == None:
-            self.cloud = cloudinit.CloudInit(ds_deps)
-            self.cloud.get_data_source()
-        else:
-            self.cloud = cloud
-        self.cfg = self.get_config_obj(cfgfile)
-
-    def get_config_obj(self, cfgfile):
-        try:
-            cfg = util.read_conf(cfgfile)
-        except:
-            # TODO: this 'log' could/should be passed in
-            cloudinit.log.critical("Failed loading of cloud config '%s'. "
-                                   "Continuing with empty config\n" % cfgfile)
-            cloudinit.log.debug(traceback.format_exc() + "\n")
-            cfg = None
-        if cfg is None:
-            cfg = {}
-
-        try:
-            ds_cfg = self.cloud.datasource.get_config_obj()
-        except:
-            ds_cfg = {}
-
-        cfg = util.mergedict(cfg, ds_cfg)
-        return(util.mergedict(cfg, self.cloud.cfg))
-
-    def handle(self, name, args, freq=None):
-        try:
-            mod = __import__("cc_" + name.replace("-", "_"), globals())
-            def_freq = getattr(mod, "frequency", per_instance)
-            handler = getattr(mod, "handle")
-
-            if not freq:
-                freq = def_freq
-
-            self.cloud.sem_and_run("config-" + name, freq, handler,
-                [name, self.cfg, self.cloud, cloudinit.log, args])
-        except:
-            raise
-
-
-# reads a cloudconfig module list, returns
-# a 2 dimensional array suitable to pass to run_cc_modules
-def read_cc_modules(cfg, name):
-    if name not in cfg:
-        return([])
-    module_list = []
-    # create 'module_list', an array of arrays
-    # where array[0] = config
-    #       array[1] = freq
-    #       array[2:] = arguemnts
-    for item in cfg[name]:
-        if isinstance(item, str):
-            module_list.append((item,))
-        elif isinstance(item, list):
-            module_list.append(item)
-        else:
-            raise TypeError("failed to read '%s' item in config")
-    return(module_list)
-
-
-def run_cc_modules(cc, module_list, log):
-    failures = []
-    for cfg_mod in module_list:
-        name = cfg_mod[0]
-        freq = None
-        run_args = []
-        if len(cfg_mod) > 1:
-            freq = cfg_mod[1]
-        if len(cfg_mod) > 2:
-            run_args = cfg_mod[2:]
-
-        try:
-            log.debug("handling %s with freq=%s and args=%s" %
-                (name, freq, run_args))
-            cc.handle(name, run_args, freq=freq)
-        except:
-            log.warn(traceback.format_exc())
-            log.error("config handling of %s, %s, %s failed\n" %
-                (name, freq, run_args))
-            failures.append(name)
-
-    return(failures)
-
-
-# always returns well formated values
-# cfg is expected to have an entry 'output' in it, which is a dictionary
-# that includes entries for 'init', 'config', 'final' or 'all'
-#   init: /var/log/cloud.out
-#   config: [ ">> /var/log/cloud-config.out", /var/log/cloud-config.err ]
-#   final:
-#     output: "| logger -p"
-#     error: "> /dev/null"
-# this returns the specific 'mode' entry, cleanly formatted, with value
-# None if if none is given
-def get_output_cfg(cfg, mode="init"):
-    ret = [None, None]
-    if not 'output' in cfg:
-        return ret
-
-    outcfg = cfg['output']
-    if mode in outcfg:
-        modecfg = outcfg[mode]
+from cloudinit.settings import (PER_INSTANCE, FREQUENCIES)
+
+from cloudinit import log as logging
+
+LOG = logging.getLogger(__name__)
+
+# This prefix is used to make it less
+# of a chance that when importing
+# we will not find something else with the same
+# name in the lookup path...
+MOD_PREFIX = "cc_"
+
+
+def form_module_name(name):
+    canon_name = name.replace("-", "_")
+    if canon_name.lower().endswith(".py"):
+        canon_name = canon_name[0:(len(canon_name) - 3)]
+    canon_name = canon_name.strip()
+    if not canon_name:
+        return None
+    if not canon_name.startswith(MOD_PREFIX):
+        canon_name = '%s%s' % (MOD_PREFIX, canon_name)
+    return canon_name
+
+
+def fixup_module(mod, def_freq=PER_INSTANCE):
+    if not hasattr(mod, 'frequency'):
+        setattr(mod, 'frequency', def_freq)
     else:
-        if 'all' not in outcfg:
-            return ret
-        # if there is a 'all' item in the output list
-        # then it applies to all users of this (init, config, final)
-        modecfg = outcfg['all']
-
-    # if value is a string, it specifies stdout and stderr
-    if isinstance(modecfg, str):
-        ret = [modecfg, modecfg]
-
-    # if its a list, then we expect (stdout, stderr)
-    if isinstance(modecfg, list):
-        if len(modecfg) > 0:
-            ret[0] = modecfg[0]
-        if len(modecfg) > 1:
-            ret[1] = modecfg[1]
-
-    # if it is a dictionary, expect 'out' and 'error'
-    # items, which indicate out and error
-    if isinstance(modecfg, dict):
-        if 'output' in modecfg:
-            ret[0] = modecfg['output']
-        if 'error' in modecfg:
-            ret[1] = modecfg['error']
-
-    # if err's entry == "&1", then make it same as stdout
-    # as in shell syntax of "echo foo >/dev/null 2>&1"
-    if ret[1] == "&1":
-        ret[1] = ret[0]
-
-    swlist = [">>", ">", "|"]
-    for i in range(len(ret)):
-        if not ret[i]:
-            continue
-        val = ret[i].lstrip()
-        found = False
-        for s in swlist:
-            if val.startswith(s):
-                val = "%s %s" % (s, val[len(s):].strip())
-                found = True
-                break
-        if not found:
-            # default behavior is append
-            val = "%s %s" % (">>", val.strip())
-        ret[i] = val
-
-    return(ret)
-
-
-# redirect_output(outfmt, errfmt, orig_out, orig_err)
-#  replace orig_out and orig_err with filehandles specified in outfmt or errfmt
-#  fmt can be:
-#   > FILEPATH
-#   >> FILEPATH
-#   | program [ arg1 [ arg2 [ ... ] ] ]
-#
-#   with a '|', arguments are passed to shell, so one level of
-#   shell escape is required.
-def redirect_output(outfmt, errfmt, o_out=sys.stdout, o_err=sys.stderr):
-    if outfmt:
-        (mode, arg) = outfmt.split(" ", 1)
-        if mode == ">" or mode == ">>":
-            owith = "ab"
-            if mode == ">":
-                owith = "wb"
-            new_fp = open(arg, owith)
-        elif mode == "|":
-            proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE)
-            new_fp = proc.stdin
-        else:
-            raise TypeError("invalid type for outfmt: %s" % outfmt)
-
-        if o_out:
-            os.dup2(new_fp.fileno(), o_out.fileno())
-        if errfmt == outfmt:
-            os.dup2(new_fp.fileno(), o_err.fileno())
-            return
-
-    if errfmt:
-        (mode, arg) = errfmt.split(" ", 1)
-        if mode == ">" or mode == ">>":
-            owith = "ab"
-            if mode == ">":
-                owith = "wb"
-            new_fp = open(arg, owith)
-        elif mode == "|":
-            proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE)
-            new_fp = proc.stdin
-        else:
-            raise TypeError("invalid type for outfmt: %s" % outfmt)
-
-        if o_err:
-            os.dup2(new_fp.fileno(), o_err.fileno())
-    return
-
-
-def run_per_instance(name, func, args, clear_on_fail=False):
-    semfile = "%s/%s" % (cloudinit.get_ipath_cur("data"), name)
-    if os.path.exists(semfile):
-        return
-
-    util.write_file(semfile, str(time.time()))
-    try:
-        func(*args)
-    except:
-        if clear_on_fail:
-            os.unlink(semfile)
-        raise
-
-
-# apt_get top level command (install, update...), and args to pass it
-def apt_get(tlc, args=None):
-    if args is None:
-        args = []
-    e = os.environ.copy()
-    e['DEBIAN_FRONTEND'] = 'noninteractive'
-    cmd = ['apt-get', '--option', 'Dpkg::Options::=--force-confold',
-           '--assume-yes', '--quiet', tlc]
-    cmd.extend(args)
-    subprocess.check_call(cmd, env=e)
-
-
-def update_package_sources():
-    run_per_instance("update-sources", apt_get, ("update",))
-
-
-def install_packages(pkglist):
-    update_package_sources()
-    apt_get("install", pkglist)
+        freq = mod.frequency
+        if freq and freq not in FREQUENCIES:
+            LOG.warn("Module %s has an unknown frequency %s", mod, freq)
+    if not hasattr(mod, 'distros'):
+        setattr(mod, 'distros', None)
+    return mod
 
=== modified file 'cloudinit/config/cc_apt_pipelining.py'
--- cloudinit/CloudConfig/cc_apt_pipelining.py	2012-03-09 15:26:09 +0000
+++ cloudinit/config/cc_apt_pipelining.py	2012-07-06 21:16:18 +0000
@@ -16,38 +16,44 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-from cloudinit.CloudConfig import per_instance
-
-frequency = per_instance
-default_file = "/etc/apt/apt.conf.d/90cloud-init-pipelining"
-
-
-def handle(_name, cfg, _cloud, log, _args):
+from cloudinit import util
+from cloudinit.settings import PER_INSTANCE
+
+frequency = PER_INSTANCE
+
+distros = ['ubuntu', 'debian']
+
+DEFAULT_FILE = "/etc/apt/apt.conf.d/90cloud-init-pipelining"
+
+APT_PIPE_TPL = ("//Written by cloud-init per 'apt_pipelining'\n"
+                'Acquire::http::Pipeline-Depth "%s";\n')
+
+# Acquire::http::Pipeline-Depth can be a value
+# from 0 to 5 indicating how many outstanding requests APT should send.
+# A value of zero MUST be specified if the remote host does not properly linger
+# on TCP connections - otherwise data corruption will occur.
+
+
+def handle(_name, cfg, cloud, log, _args):
 
     apt_pipe_value = util.get_cfg_option_str(cfg, "apt_pipelining", False)
-    apt_pipe_value = str(apt_pipe_value).lower()
-
-    if apt_pipe_value == "false":
-        write_apt_snippet("0", log)
-
-    elif apt_pipe_value in ("none", "unchanged", "os"):
+    apt_pipe_value_s = str(apt_pipe_value).lower().strip()
+
+    if apt_pipe_value_s == "false":
+        write_apt_snippet(cloud, "0", log, DEFAULT_FILE)
+    elif apt_pipe_value_s in ("none", "unchanged", "os"):
         return
-
-    elif apt_pipe_value in str(range(0, 6)):
-        write_apt_snippet(apt_pipe_value, log)
-
+    elif apt_pipe_value_s in [str(b) for b in xrange(0, 6)]:
+        write_apt_snippet(cloud, apt_pipe_value_s, log, DEFAULT_FILE)
     else:
-        log.warn("Invalid option for apt_pipeling: %s" % apt_pipe_value)
-
-
-def write_apt_snippet(setting, log, f_name=default_file):
+        log.warn("Invalid option for apt_pipeling: %s", apt_pipe_value)
+
+
+def write_apt_snippet(cloud, setting, log, f_name):
     """ Writes f_name with apt pipeline depth 'setting' """
 
-    acquire_pipeline_depth = 'Acquire::http::Pipeline-Depth "%s";\n'
-    file_contents = ("//Written by cloud-init per 'apt_pipelining'\n"
-                     + (acquire_pipeline_depth % setting))
-
-    util.write_file(f_name, file_contents)
-
-    log.debug("Wrote %s with APT pipeline setting" % f_name)
+    file_contents = APT_PIPE_TPL % (setting)
+
+    util.write_file(cloud.paths.join(False, f_name), file_contents)
+
+    log.debug("Wrote %s with apt pipeline depth setting %s", f_name, setting)
 
=== modified file 'cloudinit/config/cc_apt_update_upgrade.py'
--- cloudinit/CloudConfig/cc_apt_update_upgrade.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_apt_update_upgrade.py	2012-07-06 21:16:18 +0000
@@ -18,50 +18,73 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import subprocess
-import traceback
+import glob
 import os
-import glob
-import cloudinit.CloudConfig as cc
-
-
-def handle(_name, cfg, cloud, log, _args):
+
+from cloudinit import templater
+from cloudinit import util
+
+distros = ['ubuntu', 'debian']
+
+PROXY_TPL = "Acquire::HTTP::Proxy \"%s\";\n"
+PROXY_FN = "/etc/apt/apt.conf.d/95cloud-init-proxy"
+
+# A temporary shell program to get a given gpg key
+# from a given keyserver
+EXPORT_GPG_KEYID = """
+    k=${1} ks=${2};
+    exec 2>/dev/null
+    [ -n "$k" ] || exit 1;
+    armour=$(gpg --list-keys --armour "${k}")
+    if [ -z "${armour}" ]; then
+       gpg --keyserver ${ks} --recv $k >/dev/null &&
+          armour=$(gpg --export --armour "${k}") &&
+          gpg --batch --yes --delete-keys "${k}"
+    fi
+    [ -n "${armour}" ] && echo "${armour}"
+"""
+
+
+def handle(name, cfg, cloud, log, _args):
     update = util.get_cfg_option_bool(cfg, 'apt_update', False)
     upgrade = util.get_cfg_option_bool(cfg, 'apt_upgrade', False)
 
     release = get_release()
-
     mirror = find_apt_mirror(cloud, cfg)
-
-    log.debug("selected mirror at: %s" % mirror)
-
-    if not util.get_cfg_option_bool(cfg, \
-        'apt_preserve_sources_list', False):
-        generate_sources_list(release, mirror)
-        old_mir = util.get_cfg_option_str(cfg, 'apt_old_mirror', \
-            "archive.ubuntu.com/ubuntu")
+    if not mirror:
+        log.debug(("Skipping module named %s,"
+                   " no package 'mirror' located"), name)
+        return
+
+    log.debug("Selected mirror at: %s" % mirror)
+
+    if not util.get_cfg_option_bool(cfg,
+                                    'apt_preserve_sources_list', False):
+        generate_sources_list(release, mirror, cloud, log)
+        old_mir = util.get_cfg_option_str(cfg, 'apt_old_mirror',
+                                          "archive.ubuntu.com/ubuntu")
         rename_apt_lists(old_mir, mirror)
 
-    # set up proxy
+    # Set up any apt proxy
     proxy = cfg.get("apt_proxy", None)
-    proxy_filename = "/etc/apt/apt.conf.d/95cloud-init-proxy"
+    proxy_filename = PROXY_FN
     if proxy:
         try:
-            contents = "Acquire::HTTP::Proxy \"%s\";\n"
-            with open(proxy_filename, "w") as fp:
-                fp.write(contents % proxy)
+            # See man 'apt.conf'
+            contents = PROXY_TPL % (proxy)
+            util.write_file(cloud.paths.join(False, proxy_filename),
+                            contents)
         except Exception as e:
-            log.warn("Failed to write proxy to %s" % proxy_filename)
+            util.logexc(log, "Failed to write proxy to %s", proxy_filename)
     elif os.path.isfile(proxy_filename):
-        os.unlink(proxy_filename)
+        util.del_file(proxy_filename)
 
-    # process 'apt_sources'
+    # Process 'apt_sources'
     if 'apt_sources' in cfg:
-        errors = add_sources(cfg['apt_sources'],
+        errors = add_sources(cloud, cfg['apt_sources'],
                              {'MIRROR': mirror, 'RELEASE': release})
         for e in errors:
-            log.warn("Source Error: %s\n" % ':'.join(e))
+            log.warn("Source Error: %s", ':'.join(e))
 
     dconf_sel = util.get_cfg_option_str(cfg, 'debconf_selections', False)
     if dconf_sel:
@@ -69,45 +92,51 @@
         try:
             util.subp(('debconf-set-selections', '-'), dconf_sel)
         except:
-            log.error("Failed to run debconf-set-selections")
-            log.debug(traceback.format_exc())
+            util.logexc(log, "Failed to run debconf-set-selections")
 
-    pkglist = util.get_cfg_option_list_or_str(cfg, 'packages', [])
+    pkglist = util.get_cfg_option_list(cfg, 'packages', [])
 
     errors = []
     if update or len(pkglist) or upgrade:
         try:
-            cc.update_package_sources()
-        except subprocess.CalledProcessError as e:
-            log.warn("apt-get update failed")
-            log.debug(traceback.format_exc())
+            cloud.distro.update_package_sources()
+        except Exception as e:
+            util.logexc(log, "Package update failed")
             errors.append(e)
 
     if upgrade:
         try:
-            cc.apt_get("upgrade")
-        except subprocess.CalledProcessError as e:
-            log.warn("apt upgrade failed")
-            log.debug(traceback.format_exc())
+            cloud.distro.package_command("upgrade")
+        except Exception as e:
+            util.logexc(log, "Package upgrade failed")
             errors.append(e)
 
     if len(pkglist):
         try:
-            cc.install_packages(pkglist)
-        except subprocess.CalledProcessError as e:
-            log.warn("Failed to install packages: %s " % pkglist)
-            log.debug(traceback.format_exc())
+            cloud.distro.install_packages(pkglist)
+        except Exception as e:
+            util.logexc(log, "Failed to install packages: %s ", pkglist)
             errors.append(e)
 
     if len(errors):
-        raise errors[0]
-
-    return(True)
+        log.warn("%s failed with exceptions, re-raising the last one",
+                 len(errors))
+        raise errors[-1]
+
+
+# get gpg keyid from keyserver
+def getkeybyid(keyid, keyserver):
+    with util.ExtendedTemporaryFile(suffix='.sh') as fh:
+        fh.write(EXPORT_GPG_KEYID)
+        fh.flush()
+        cmd = ['/bin/sh', fh.name, keyid, keyserver]
+        (stdout, _stderr) = util.subp(cmd)
+        return stdout.strip()
 
 
 def mirror2lists_fileprefix(mirror):
     string = mirror
-    # take of http:// or ftp://
+    # take off http:// or ftp://
     if string.endswith("/"):
         string = string[0:-1]
     pos = string.find("://")
@@ -118,39 +147,44 @@
 
 
 def rename_apt_lists(omirror, new_mirror, lists_d="/var/lib/apt/lists"):
-    oprefix = "%s/%s" % (lists_d, mirror2lists_fileprefix(omirror))
-    nprefix = "%s/%s" % (lists_d, mirror2lists_fileprefix(new_mirror))
-    if(oprefix == nprefix):
+    oprefix = os.path.join(lists_d, mirror2lists_fileprefix(omirror))
+    nprefix = os.path.join(lists_d, mirror2lists_fileprefix(new_mirror))
+    if oprefix == nprefix:
         return
     olen = len(oprefix)
     for filename in glob.glob("%s_*" % oprefix):
-        os.rename(filename, "%s%s" % (nprefix, filename[olen:]))
+        # TODO use the cloud.paths.join...
+        util.rename(filename, "%s%s" % (nprefix, filename[olen:]))
 
 
 def get_release():
-    stdout, _stderr = subprocess.Popen(['lsb_release', '-cs'],
-                                       stdout=subprocess.PIPE).communicate()
-    return(str(stdout).strip())
-
-
-def generate_sources_list(codename, mirror):
-    util.render_to_file('sources.list', '/etc/apt/sources.list', \
-        {'mirror': mirror, 'codename': codename})
-
-
-def add_sources(srclist, searchList=None):
+    (stdout, _stderr) = util.subp(['lsb_release', '-cs'])
+    return stdout.strip()
+
+
+def generate_sources_list(codename, mirror, cloud, log):
+    template_fn = cloud.get_template_filename('sources.list')
+    if template_fn:
+        params = {'mirror': mirror, 'codename': codename}
+        out_fn = cloud.paths.join(False, '/etc/apt/sources.list')
+        templater.render_to_file(template_fn, out_fn, params)
+    else:
+        log.warn("No template found, not rendering /etc/apt/sources.list")
+
+
+def add_sources(cloud, srclist, template_params=None):
     """
     add entries in /etc/apt/sources.list.d for each abbreviated
     sources.list entry in 'srclist'.  When rendering template, also
     include the values in dictionary searchList
     """
-    if searchList is None:
-        searchList = {}
-    elst = []
+    if template_params is None:
+        template_params = {}
 
+    errorlist = []
     for ent in srclist:
         if 'source' not in ent:
-            elst.append(["", "missing source"])
+            errorlist.append(["", "missing source"])
             continue
 
         source = ent['source']
@@ -158,51 +192,48 @@
             try:
                 util.subp(["add-apt-repository", source])
             except:
-                elst.append([source, "add-apt-repository failed"])
+                errorlist.append([source, "add-apt-repository failed"])
             continue
 
-        source = util.render_string(source, searchList)
+        source = templater.render_string(source, template_params)
 
         if 'filename' not in ent:
             ent['filename'] = 'cloud_config_sources.list'
 
         if not ent['filename'].startswith("/"):
-            ent['filename'] = "%s/%s" % \
-                ("/etc/apt/sources.list.d/", ent['filename'])
+            ent['filename'] = os.path.join("/etc/apt/sources.list.d/",
+                                           ent['filename'])
 
         if ('keyid' in ent and 'key' not in ent):
             ks = "keyserver.ubuntu.com"
             if 'keyserver' in ent:
                 ks = ent['keyserver']
             try:
-                ent['key'] = util.getkeybyid(ent['keyid'], ks)
+                ent['key'] = getkeybyid(ent['keyid'], ks)
             except:
-                elst.append([source, "failed to get key from %s" % ks])
+                errorlist.append([source, "failed to get key from %s" % ks])
                 continue
 
         if 'key' in ent:
             try:
                 util.subp(('apt-key', 'add', '-'), ent['key'])
             except:
-                elst.append([source, "failed add key"])
+                errorlist.append([source, "failed add key"])
 
         try:
-            util.write_file(ent['filename'], source + "\n", omode="ab")
+            contents = "%s\n" % (source)
+            util.write_file(cloud.paths.join(False, ent['filename']),
+                            contents, omode="ab")
         except:
-            elst.append([source, "failed write to file %s" % ent['filename']])
+            errorlist.append([source,
+                             "failed write to file %s" % ent['filename']])
 
-    return(elst)
+    return errorlist
 
 
 def find_apt_mirror(cloud, cfg):
     """ find an apt_mirror given the cloud and cfg provided """
 
-    # TODO: distro and defaults should be configurable
-    distro = "ubuntu"
-    defaults = {
-        'ubuntu': "http://archive.ubuntu.com/ubuntu",
-        'debian': "http://archive.debian.org/debian",
-    }
     mirror = None
 
     cfg_mirror = cfg.get("apt_mirror", None)
@@ -211,14 +242,13 @@
     elif "apt_mirror_search" in cfg:
         mirror = util.search_for_mirror(cfg['apt_mirror_search'])
     else:
-        if cloud:
-            mirror = cloud.get_mirror()
+        mirror = cloud.get_local_mirror()
 
         mydom = ""
 
         doms = []
 
-        if not mirror and cloud:
+        if not mirror:
             # if we have a fqdn, then search its domain portion first
             (_hostname, fqdn) = util.get_hostname_fqdn(cfg, cloud)
             mydom = ".".join(fqdn.split(".")[1:])
@@ -229,13 +259,14 @@
             doms.extend((".localdomain", "",))
 
             mirror_list = []
+            distro = cloud.distro.name
             mirrorfmt = "http://%s-mirror%s/%s" % (distro, "%s", distro)
             for post in doms:
-                mirror_list.append(mirrorfmt % post)
+                mirror_list.append(mirrorfmt % (post))
 
             mirror = util.search_for_mirror(mirror_list)
 
     if not mirror:
-        mirror = defaults[distro]
+        mirror = cloud.distro.get_package_mirror()
 
     return mirror
 
=== modified file 'cloudinit/config/cc_bootcmd.py'
--- cloudinit/CloudConfig/cc_bootcmd.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_bootcmd.py	2012-07-06 21:16:18 +0000
@@ -17,32 +17,39 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-import cloudinit.util as util
-import subprocess
-import tempfile
+
 import os
-from cloudinit.CloudConfig import per_always
-frequency = per_always
-
-
-def handle(_name, cfg, cloud, log, _args):
+
+from cloudinit import util
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+
+def handle(name, cfg, cloud, log, _args):
+
     if "bootcmd" not in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'bootcmd' key in configuration"), name)
         return
 
-    try:
-        content = util.shellify(cfg["bootcmd"])
-        tmpf = tempfile.TemporaryFile()
-        tmpf.write(content)
-        tmpf.seek(0)
-    except:
-        log.warn("failed to shellify bootcmd")
-        raise
+    with util.ExtendedTemporaryFile(suffix=".sh") as tmpf:
+        try:
+            content = util.shellify(cfg["bootcmd"])
+            tmpf.write(content)
+            tmpf.flush()
+        except:
+            util.logexc(log, "Failed to shellify bootcmd")
+            raise
 
-    try:
-        env = os.environ.copy()
-        env['INSTANCE_ID'] = cloud.get_instance_id()
-        subprocess.check_call(['/bin/sh'], env=env, stdin=tmpf)
-        tmpf.close()
-    except:
-        log.warn("failed to run commands from bootcmd")
-        raise
+        try:
+            env = os.environ.copy()
+            iid = cloud.get_instance_id()
+            if iid:
+                env['INSTANCE_ID'] = str(iid)
+            cmd = ['/bin/sh', tmpf.name]
+            util.subp(cmd, env=env, capture=False)
+        except:
+            util.logexc(log,
+                        ("Failed to run bootcmd module %s"), name)
+            raise
 
=== modified file 'cloudinit/config/cc_byobu.py'
--- cloudinit/CloudConfig/cc_byobu.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_byobu.py	2012-07-06 21:16:18 +0000
@@ -18,18 +18,19 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import subprocess
-import traceback
-
-
-def handle(_name, cfg, _cloud, log, args):
+from cloudinit import util
+
+distros = ['ubuntu', 'debian']
+
+
+def handle(name, cfg, _cloud, log, args):
     if len(args) != 0:
         value = args[0]
     else:
         value = util.get_cfg_option_str(cfg, "byobu_by_default", "")
 
     if not value:
+        log.debug("Skipping module named %s, no 'byobu' values found", name)
         return
 
     if value == "user" or value == "system":
@@ -38,7 +39,7 @@
     valid = ("enable-user", "enable-system", "enable",
              "disable-user", "disable-system", "disable")
     if not value in valid:
-        log.warn("Unknown value %s for byobu_by_default" % value)
+        log.warn("Unknown value %s for byobu_by_default", value)
 
     mod_user = value.endswith("-user")
     mod_sys = value.endswith("-system")
@@ -65,13 +66,6 @@
 
     cmd = ["/bin/sh", "-c", "%s %s %s" % ("X=0;", shcmd, "exit $X")]
 
-    log.debug("setting byobu to %s" % value)
+    log.debug("Setting byobu to %s", value)
 
-    try:
-        subprocess.check_call(cmd)
-    except subprocess.CalledProcessError as e:
-        log.debug(traceback.format_exc(e))
-        raise Exception("Cmd returned %s: %s" % (e.returncode, cmd))
-    except OSError as e:
-        log.debug(traceback.format_exc(e))
-        raise Exception("Cmd failed to execute: %s" % (cmd))
+    util.subp(cmd, capture=False)
 
=== modified file 'cloudinit/config/cc_ca_certs.py'
--- cloudinit/CloudConfig/cc_ca_certs.py	2012-03-08 12:45:43 +0000
+++ cloudinit/config/cc_ca_certs.py	2012-07-06 21:16:18 +0000
@@ -13,25 +13,27 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
-from subprocess import check_call
-from cloudinit.util import (write_file, get_cfg_option_list_or_str,
-                            delete_dir_contents, subp)
+
+from cloudinit import util
 
 CA_CERT_PATH = "/usr/share/ca-certificates/"
 CA_CERT_FILENAME = "cloud-init-ca-certs.crt"
 CA_CERT_CONFIG = "/etc/ca-certificates.conf"
 CA_CERT_SYSTEM_PATH = "/etc/ssl/certs/"
 
+distros = ['ubuntu', 'debian']
+
 
 def update_ca_certs():
     """
     Updates the CA certificate cache on the current machine.
     """
-    check_call(["update-ca-certificates"])
-
-
-def add_ca_certs(certs):
+    util.subp(["update-ca-certificates"], capture=False)
+
+
+def add_ca_certs(paths, certs):
     """
     Adds certificates to the system. To actually apply the new certificates
     you must also call L{update_ca_certs}.
@@ -39,26 +41,29 @@
     @param certs: A list of certificate strings.
     """
     if certs:
-        cert_file_contents = "\n".join(certs)
+        # First ensure they are strings...
+        cert_file_contents = "\n".join([str(c) for c in certs])
         cert_file_fullpath = os.path.join(CA_CERT_PATH, CA_CERT_FILENAME)
-        write_file(cert_file_fullpath, cert_file_contents, mode=0644)
+        cert_file_fullpath = paths.join(False, cert_file_fullpath)
+        util.write_file(cert_file_fullpath, cert_file_contents, mode=0644)
         # Append cert filename to CA_CERT_CONFIG file.
-        write_file(CA_CERT_CONFIG, "\n%s" % CA_CERT_FILENAME, omode="a")
-
-
-def remove_default_ca_certs():
+        util.write_file(paths.join(False, CA_CERT_CONFIG),
+                        "\n%s" % CA_CERT_FILENAME, omode="ab")
+
+
+def remove_default_ca_certs(paths):
     """
     Removes all default trusted CA certificates from the system. To actually
     apply the change you must also call L{update_ca_certs}.
     """
-    delete_dir_contents(CA_CERT_PATH)
-    delete_dir_contents(CA_CERT_SYSTEM_PATH)
-    write_file(CA_CERT_CONFIG, "", mode=0644)
+    util.delete_dir_contents(paths.join(False, CA_CERT_PATH))
+    util.delete_dir_contents(paths.join(False, CA_CERT_SYSTEM_PATH))
+    util.write_file(paths.join(False, CA_CERT_CONFIG), "", mode=0644)
     debconf_sel = "ca-certificates ca-certificates/trust_new_crts select no"
-    subp(('debconf-set-selections', '-'), debconf_sel)
-
-
-def handle(_name, cfg, _cloud, log, _args):
+    util.subp(('debconf-set-selections', '-'), debconf_sel)
+
+
+def handle(name, cfg, cloud, log, _args):
     """
     Call to handle ca-cert sections in cloud-config file.
 
@@ -70,21 +75,25 @@
     """
     # If there isn't a ca-certs section in the configuration don't do anything
     if "ca-certs" not in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'ca-certs' key in configuration"), name)
         return
+
     ca_cert_cfg = cfg['ca-certs']
 
     # If there is a remove-defaults option set to true, remove the system
     # default trusted CA certs first.
     if ca_cert_cfg.get("remove-defaults", False):
-        log.debug("removing default certificates")
-        remove_default_ca_certs()
+        log.debug("Removing default certificates")
+        remove_default_ca_certs(cloud.paths)
 
     # If we are given any new trusted CA certs to add, add them.
     if "trusted" in ca_cert_cfg:
-        trusted_certs = get_cfg_option_list_or_str(ca_cert_cfg, "trusted")
+        trusted_certs = util.get_cfg_option_list(ca_cert_cfg, "trusted")
         if trusted_certs:
-            log.debug("adding %d certificates" % len(trusted_certs))
-            add_ca_certs(trusted_certs)
+            log.debug("Adding %d certificates" % len(trusted_certs))
+            add_ca_certs(cloud.paths, trusted_certs)
 
     # Update the system with the new cert configuration.
+    log.debug("Updating certificates")
     update_ca_certs()
 
=== modified file 'cloudinit/config/cc_chef.py'
--- cloudinit/CloudConfig/cc_chef.py	2012-03-26 17:49:06 +0000
+++ cloudinit/config/cc_chef.py	2012-07-06 21:16:18 +0000
@@ -18,53 +18,71 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import json
 import os
-import subprocess
-import json
-import cloudinit.CloudConfig as cc
-import cloudinit.util as util
-
-ruby_version_default = "1.8"
-
-
-def handle(_name, cfg, cloud, log, _args):
+
+from cloudinit import templater
+from cloudinit import util
+
+RUBY_VERSION_DEFAULT = "1.8"
+
+
+def handle(name, cfg, cloud, log, _args):
+
     # If there isn't a chef key in the configuration don't do anything
     if 'chef' not in cfg:
+        log.debug(("Skipping module named %s,"
+                  " no 'chef' key in configuration"), name)
         return
     chef_cfg = cfg['chef']
 
-    # ensure the chef directories we use exist
-    mkdirs(['/etc/chef', '/var/log/chef', '/var/lib/chef',
-            '/var/cache/chef', '/var/backups/chef', '/var/run/chef'])
+    # Ensure the chef directories we use exist
+    c_dirs = [
+        '/etc/chef',
+        '/var/log/chef',
+        '/var/lib/chef',
+        '/var/cache/chef',
+        '/var/backups/chef',
+        '/var/run/chef',
+    ]
+    for d in c_dirs:
+        util.ensure_dir(cloud.paths.join(False, d))
 
-    # set the validation key based on the presence of either 'validation_key'
+    # Set the validation key based on the presence of either 'validation_key'
     # or 'validation_cert'. In the case where both exist, 'validation_key'
     # takes precedence
     for key in ('validation_key', 'validation_cert'):
         if key in chef_cfg and chef_cfg[key]:
-            with open('/etc/chef/validation.pem', 'w') as validation_key_fh:
-                validation_key_fh.write(chef_cfg[key])
+            v_fn = cloud.paths.join(False, '/etc/chef/validation.pem')
+            util.write_file(v_fn, chef_cfg[key])
             break
 
-    # create the chef config from template
-    util.render_to_file('chef_client.rb', '/etc/chef/client.rb',
-        {'server_url': chef_cfg['server_url'],
-         'node_name': util.get_cfg_option_str(chef_cfg, 'node_name',
-                                          cloud.datasource.get_instance_id()),
-         'environment': util.get_cfg_option_str(chef_cfg, 'environment',
-                                                '_default'),
-         'validation_name': chef_cfg['validation_name']})
+    # Create the chef config from template
+    template_fn = cloud.get_template_filename('chef_client.rb')
+    if template_fn:
+        iid = str(cloud.datasource.get_instance_id())
+        params = {
+            'server_url': chef_cfg['server_url'],
+            'node_name': util.get_cfg_option_str(chef_cfg, 'node_name', iid),
+            'environment': util.get_cfg_option_str(chef_cfg, 'environment',
+                                                   '_default'),
+            'validation_name': chef_cfg['validation_name']
+        }
+        out_fn = cloud.paths.join(False, '/etc/chef/client.rb')
+        templater.render_to_file(template_fn, out_fn, params)
+    else:
+        log.warn("No template found, not rendering to /etc/chef/client.rb")
 
     # set the firstboot json
-    with open('/etc/chef/firstboot.json', 'w') as firstboot_json_fh:
-        initial_json = {}
-        if 'run_list' in chef_cfg:
-            initial_json['run_list'] = chef_cfg['run_list']
-        if 'initial_attributes' in chef_cfg:
-            initial_attributes = chef_cfg['initial_attributes']
-            for k in initial_attributes.keys():
-                initial_json[k] = initial_attributes[k]
-        firstboot_json_fh.write(json.dumps(initial_json))
+    initial_json = {}
+    if 'run_list' in chef_cfg:
+        initial_json['run_list'] = chef_cfg['run_list']
+    if 'initial_attributes' in chef_cfg:
+        initial_attributes = chef_cfg['initial_attributes']
+        for k in list(initial_attributes.keys()):
+            initial_json[k] = initial_attributes[k]
+    firstboot_fn = cloud.paths.join(False, '/etc/chef/firstboot.json')
+    util.write_file(firstboot_fn, json.dumps(initial_json))
 
     # If chef is not installed, we install chef based on 'install_type'
     if not os.path.isfile('/usr/bin/chef-client'):
@@ -74,15 +92,17 @@
             # this will install and run the chef-client from gems
             chef_version = util.get_cfg_option_str(chef_cfg, 'version', None)
             ruby_version = util.get_cfg_option_str(chef_cfg, 'ruby_version',
-                                                   ruby_version_default)
-            install_chef_from_gems(ruby_version, chef_version)
+                                                   RUBY_VERSION_DEFAULT)
+            install_chef_from_gems(cloud.distro, ruby_version, chef_version)
             # and finally, run chef-client
-            log.debug('running chef-client')
-            subprocess.check_call(['/usr/bin/chef-client', '-d', '-i', '1800',
-                                   '-s', '20'])
+            log.debug('Running chef-client')
+            util.subp(['/usr/bin/chef-client',
+                       '-d', '-i', '1800', '-s', '20'], capture=False)
+        elif install_type == 'packages':
+            # this will install and run the chef-client from packages
+            cloud.distro.install_packages(('chef',))
         else:
-            # this will install and run the chef-client from packages
-            cc.install_packages(('chef',))
+            log.warn("Unknown chef install type %s", install_type)
 
 
 def get_ruby_packages(version):
@@ -90,30 +110,20 @@
     pkgs = ['ruby%s' % version, 'ruby%s-dev' % version]
     if version == "1.8":
         pkgs.extend(('libopenssl-ruby1.8', 'rubygems1.8'))
-    return(pkgs)
-
-
-def install_chef_from_gems(ruby_version, chef_version=None):
-    cc.install_packages(get_ruby_packages(ruby_version))
+    return pkgs
+
+
+def install_chef_from_gems(ruby_version, chef_version, distro):
+    distro.install_packages(get_ruby_packages(ruby_version))
     if not os.path.exists('/usr/bin/gem'):
-        os.symlink('/usr/bin/gem%s' % ruby_version, '/usr/bin/gem')
+        util.sym_link('/usr/bin/gem%s' % ruby_version, '/usr/bin/gem')
     if not os.path.exists('/usr/bin/ruby'):
-        os.symlink('/usr/bin/ruby%s' % ruby_version, '/usr/bin/ruby')
+        util.sym_link('/usr/bin/ruby%s' % ruby_version, '/usr/bin/ruby')
     if chef_version:
-        subprocess.check_call(['/usr/bin/gem', 'install', 'chef',
-                               '-v %s' % chef_version, '--no-ri',
-                               '--no-rdoc', '--bindir', '/usr/bin', '-q'])
+        util.subp(['/usr/bin/gem', 'install', 'chef',
+                  '-v %s' % chef_version, '--no-ri',
+                  '--no-rdoc', '--bindir', '/usr/bin', '-q'], capture=False)
     else:
-        subprocess.check_call(['/usr/bin/gem', 'install', 'chef',
-                               '--no-ri', '--no-rdoc', '--bindir',
-                               '/usr/bin', '-q'])
-
-
-def ensure_dir(d):
-    if not os.path.exists(d):
-        os.makedirs(d)
-
-
-def mkdirs(dirs):
-    for d in dirs:
-        ensure_dir(d)
+        util.subp(['/usr/bin/gem', 'install', 'chef',
+                  '--no-ri', '--no-rdoc', '--bindir',
+                  '/usr/bin', '-q'], capture=False)
 
=== modified file 'cloudinit/config/cc_disable_ec2_metadata.py'
--- cloudinit/CloudConfig/cc_disable_ec2_metadata.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_disable_ec2_metadata.py	2012-07-06 21:16:18 +0000
@@ -17,14 +17,20 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-import cloudinit.util as util
-import subprocess
-from cloudinit.CloudConfig import per_always
-
-frequency = per_always
-
-
-def handle(_name, cfg, _cloud, _log, _args):
-    if util.get_cfg_option_bool(cfg, "disable_ec2_metadata", False):
-        fwall = "route add -host 169.254.169.254 reject"
-        subprocess.call(fwall.split(' '))
+
+from cloudinit import util
+
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+REJECT_CMD = ['route', 'add', '-host', '169.254.169.254', 'reject']
+
+
+def handle(name, cfg, _cloud, log, _args):
+    disabled = util.get_cfg_option_bool(cfg, "disable_ec2_metadata", False)
+    if disabled:
+        util.subp(REJECT_CMD, capture=False)
+    else:
+        log.debug(("Skipping module named %s,"
+                   " disabling the ec2 route not enabled"), name)
 
=== modified file 'cloudinit/config/cc_final_message.py'
--- cloudinit/CloudConfig/cc_final_message.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_final_message.py	2012-07-06 21:16:18 +0000
@@ -18,41 +18,51 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-from cloudinit.CloudConfig import per_always
-import sys
-from cloudinit import util, boot_finished
-import time
-
-frequency = per_always
-
-final_message = "cloud-init boot finished at $TIMESTAMP. Up $UPTIME seconds"
-
-
-def handle(_name, cfg, _cloud, log, args):
+from cloudinit import templater
+from cloudinit import util
+from cloudinit import version
+
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+FINAL_MESSAGE_DEF = ("Cloud-init v. {{version}} finished at {{timestamp}}."
+                     " Up {{uptime}} seconds.")
+
+
+def handle(_name, cfg, cloud, log, args):
+
+    msg_in = None
     if len(args) != 0:
         msg_in = args[0]
     else:
-        msg_in = util.get_cfg_option_str(cfg, "final_message", final_message)
-
-    try:
-        uptimef = open("/proc/uptime")
-        uptime = uptimef.read().split(" ")[0]
-        uptimef.close()
-    except IOError as e:
-        log.warn("unable to open /proc/uptime\n")
-        uptime = "na"
-
-    try:
-        ts = time.strftime("%a, %d %b %Y %H:%M:%S %z", time.gmtime())
+        msg_in = util.get_cfg_option_str(cfg, "final_message")
+
+    if not msg_in:
+        template_fn = cloud.get_template_filename('final_message')
+        if template_fn:
+            msg_in = util.load_file(template_fn)
+
+    if not msg_in:
+        msg_in = FINAL_MESSAGE_DEF
+
+    uptime = util.uptime()
+    ts = util.time_rfc2822()
+    cver = version.version_string()
+    try:
+        subs = {
+            'uptime': uptime,
+            'timestamp': ts,
+            'version': cver,
+        }
+        util.multi_log("%s\n" % (templater.render_string(msg_in, subs)),
+                       console=False, stderr=True)
+    except Exception:
+        util.logexc(log, "Failed to render final message template")
+
+    boot_fin_fn = cloud.paths.boot_finished
+    try:
+        contents = "%s - %s - v. %s\n" % (uptime, ts, cver)
+        util.write_file(boot_fin_fn, contents)
     except:
-        ts = "na"
-
-    try:
-        subs = {'UPTIME': uptime, 'TIMESTAMP': ts}
-        sys.stdout.write("%s\n" % util.render_string(msg_in, subs))
-    except Exception as e:
-        log.warn("failed to render string to stdout: %s" % e)
-
-    fp = open(boot_finished, "wb")
-    fp.write(uptime + "\n")
-    fp.close()
+        util.logexc(log, "Failed to write boot finished file %s", boot_fin_fn)
 
=== modified file 'cloudinit/config/cc_foo.py'
--- cloudinit/CloudConfig/cc_foo.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_foo.py	2012-07-06 21:16:18 +0000
@@ -18,12 +18,35 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-#import cloudinit
-#import cloudinit.util as util
-from cloudinit.CloudConfig import per_instance
-
-frequency = per_instance
-
-
-def handle(_name, _cfg, _cloud, _log, _args):
-    print "hi"
+from cloudinit.settings import PER_INSTANCE
+
+# Modules are expected to have the following attributes.
+# 1. A required 'handle' method which takes the following params.
+#    a) The name will not be this files name, but instead
+#    the name specified in configuration (which is the name
+#    which will be used to find this module).
+#    b) A configuration object that is the result of the merging
+#    of cloud configs configuration with legacy configuration
+#    as well as any datasource provided configuration
+#    c) A cloud object that can be used to access various
+#    datasource and paths for the given distro and data provided
+#    by the various datasource instance types.
+#    d) A argument list that may or may not be empty to this module.
+#    Typically those are from module configuration where the module
+#    is defined with some extra configuration that will eventually
+#    be translated from yaml into arguments to this module.
+# 2. A optional 'frequency' that defines how often this module should be ran.
+#    Typically one of PER_INSTANCE, PER_ALWAYS, PER_ONCE. If not
+#    provided PER_INSTANCE will be assumed.
+#    See settings.py for these constants.
+# 3. A optional 'distros' array/set/tuple that defines the known distros
+#    this module will work with (if not all of them). This is used to write
+#    a warning out if a module is being ran on a untested distribution for
+#    informational purposes. If non existent all distros are assumed and
+#    no warning occurs.
+
+frequency = PER_INSTANCE
+
+
+def handle(name, _cfg, _cloud, log, _args):
+    log.debug("Hi from module %s", name)
 
=== modified file 'cloudinit/config/cc_grub_dpkg.py'
--- cloudinit/CloudConfig/cc_grub_dpkg.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_grub_dpkg.py	2012-07-06 21:16:18 +0000
@@ -18,10 +18,12 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import traceback
 import os
 
+from cloudinit import util
+
+distros = ['ubuntu', 'debian']
+
 
 def handle(_name, cfg, _cloud, log, _args):
     idevs = None
@@ -35,14 +37,14 @@
 
     if ((os.path.exists("/dev/sda1") and not os.path.exists("/dev/sda")) or
         (os.path.exists("/dev/xvda1") and not os.path.exists("/dev/xvda"))):
-        if idevs == None:
+        if idevs is None:
             idevs = ""
-        if idevs_empty == None:
+        if idevs_empty is None:
             idevs_empty = "true"
     else:
-        if idevs_empty == None:
+        if idevs_empty is None:
             idevs_empty = "false"
-        if idevs == None:
+        if idevs is None:
             idevs = "/dev/sda"
             for dev in ("/dev/sda", "/dev/vda", "/dev/sda1", "/dev/vda1"):
                 if os.path.exists(dev):
@@ -52,13 +54,14 @@
     # now idevs and idevs_empty are set to determined values
     # or, those set by user
 
-    dconf_sel = "grub-pc grub-pc/install_devices string %s\n" % idevs + \
-        "grub-pc grub-pc/install_devices_empty boolean %s\n" % idevs_empty
-    log.debug("setting grub debconf-set-selections with '%s','%s'" %
+    dconf_sel = (("grub-pc grub-pc/install_devices string %s\n"
+                 "grub-pc grub-pc/install_devices_empty boolean %s\n") %
+                 (idevs, idevs_empty))
+
+    log.debug("Setting grub debconf-set-selections with '%s','%s'" %
         (idevs, idevs_empty))
 
     try:
-        util.subp(('debconf-set-selections'), dconf_sel)
+        util.subp(['debconf-set-selections'], dconf_sel)
     except:
-        log.error("Failed to run debconf-set-selections for grub-dpkg")
-        log.debug(traceback.format_exc())
+        util.logexc(log, "Failed to run debconf-set-selections for grub-dpkg")
 
=== modified file 'cloudinit/config/cc_keys_to_console.py'
--- cloudinit/CloudConfig/cc_keys_to_console.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_keys_to_console.py	2012-07-06 21:16:18 +0000
@@ -18,25 +18,36 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-from cloudinit.CloudConfig import per_instance
-import cloudinit.util as util
-import subprocess
-
-frequency = per_instance
-
-
-def handle(_name, cfg, _cloud, log, _args):
-    cmd = ['/usr/lib/cloud-init/write-ssh-key-fingerprints']
-    fp_blacklist = util.get_cfg_option_list_or_str(cfg,
-        "ssh_fp_console_blacklist", [])
-    key_blacklist = util.get_cfg_option_list_or_str(cfg,
-        "ssh_key_console_blacklist", ["ssh-dss"])
+import os
+
+from cloudinit.settings import PER_INSTANCE
+from cloudinit import util
+
+frequency = PER_INSTANCE
+
+# This is a tool that cloud init provides
+HELPER_TOOL = '/usr/lib/cloud-init/write-ssh-key-fingerprints'
+
+
+def handle(name, cfg, _cloud, log, _args):
+    if not os.path.exists(HELPER_TOOL):
+        log.warn(("Unable to activate module %s,"
+                  " helper tool not found at %s"), name, HELPER_TOOL)
+        return
+
+    fp_blacklist = util.get_cfg_option_list(cfg,
+                                            "ssh_fp_console_blacklist", [])
+    key_blacklist = util.get_cfg_option_list(cfg,
+                                             "ssh_key_console_blacklist",
+                                              ["ssh-dss"])
+
     try:
-        confp = open('/dev/console', "wb")
+        cmd = [HELPER_TOOL]
         cmd.append(','.join(fp_blacklist))
         cmd.append(','.join(key_blacklist))
-        subprocess.call(cmd, stdout=confp)
-        confp.close()
+        (stdout, _stderr) = util.subp(cmd)
+        util.multi_log("%s\n" % (stdout.strip()),
+                       stderr=False, console=True)
     except:
-        log.warn("writing keys to console value")
+        log.warn("Writing keys to the system console failed!")
         raise
 
=== modified file 'cloudinit/config/cc_landscape.py'
--- cloudinit/CloudConfig/cc_landscape.py	2012-04-10 20:22:47 +0000
+++ cloudinit/config/cc_landscape.py	2012-07-06 21:16:18 +0000
@@ -19,16 +19,23 @@
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import os.path
-from cloudinit.CloudConfig import per_instance
+
+from StringIO import StringIO
+
 from configobj import ConfigObj
 
-frequency = per_instance
-
-lsc_client_cfg_file = "/etc/landscape/client.conf"
+from cloudinit import util
+
+from cloudinit.settings import PER_INSTANCE
+
+frequency = PER_INSTANCE
+
+LSC_CLIENT_CFG_FILE = "/etc/landscape/client.conf"
+
+distros = ['ubuntu']
 
 # defaults taken from stock client.conf in landscape-client 11.07.1.1-0ubuntu2
-lsc_builtincfg = {
+LSC_BUILTIN_CFG = {
   'client': {
     'log_level': "info",
     'url': "https://landscape.canonical.com/message-system",
@@ -38,7 +45,7 @@
 }
 
 
-def handle(_name, cfg, _cloud, log, _args):
+def handle(_name, cfg, cloud, log, _args):
     """
     Basically turn a top level 'landscape' entry with a 'client' dict
     and render it to ConfigObj format under '[client]' section in
@@ -47,27 +54,40 @@
 
     ls_cloudcfg = cfg.get("landscape", {})
 
-    if not isinstance(ls_cloudcfg, dict):
-        raise(Exception("'landscape' existed in config, but not a dict"))
-
-    merged = mergeTogether([lsc_builtincfg, lsc_client_cfg_file, ls_cloudcfg])
-
-    if not os.path.isdir(os.path.dirname(lsc_client_cfg_file)):
-        os.makedirs(os.path.dirname(lsc_client_cfg_file))
-
-    with open(lsc_client_cfg_file, "w") as fp:
-        merged.write(fp)
-
-    log.debug("updated %s" % lsc_client_cfg_file)
-
-
-def mergeTogether(objs):
+    if not isinstance(ls_cloudcfg, (dict)):
+        raise RuntimeError(("'landscape' key existed in config,"
+                            " but not a dictionary type,"
+                            " is a %s instead"), util.obj_name(ls_cloudcfg))
+
+    merge_data = [
+        LSC_BUILTIN_CFG,
+        cloud.paths.join(True, LSC_CLIENT_CFG_FILE),
+        ls_cloudcfg,
+    ]
+    merged = merge_together(merge_data)
+
+    lsc_client_fn = cloud.paths.join(False, LSC_CLIENT_CFG_FILE)
+    lsc_dir = cloud.paths.join(False, os.path.dirname(lsc_client_fn))
+    if not os.path.isdir(lsc_dir):
+        util.ensure_dir(lsc_dir)
+
+    contents = StringIO()
+    merged.write(contents)
+    contents.flush()
+
+    util.write_file(lsc_client_fn, contents.getvalue())
+    log.debug("Wrote landscape config file to %s", lsc_client_fn)
+
+
+def merge_together(objs):
     """
     merge together ConfigObj objects or things that ConfigObj() will take in
     later entries override earlier
     """
     cfg = ConfigObj({})
     for obj in objs:
+        if not obj:
+            continue
         if isinstance(obj, ConfigObj):
             cfg.merge(obj)
         else:
 
=== modified file 'cloudinit/config/cc_locale.py'
--- cloudinit/CloudConfig/cc_locale.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_locale.py	2012-07-06 21:16:18 +0000
@@ -18,37 +18,20 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import os.path
-import subprocess
-import traceback
-
-
-def apply_locale(locale, cfgfile):
-    if os.path.exists('/usr/sbin/locale-gen'):
-        subprocess.Popen(['locale-gen', locale]).communicate()
-    if os.path.exists('/usr/sbin/update-locale'):
-        subprocess.Popen(['update-locale', locale]).communicate()
-
-    util.render_to_file('default-locale', cfgfile, {'locale': locale})
-
-
-def handle(_name, cfg, cloud, log, args):
+from cloudinit import util
+
+
+def handle(name, cfg, cloud, log, args):
     if len(args) != 0:
         locale = args[0]
     else:
         locale = util.get_cfg_option_str(cfg, "locale", cloud.get_locale())
 
-    locale_cfgfile = util.get_cfg_option_str(cfg, "locale_configfile",
-                                             "/etc/default/locale")
-
     if not locale:
+        log.debug(("Skipping module named %s, "
+                   "no 'locale' configuration found"), name)
         return
 
-    log.debug("setting locale to %s" % locale)
-
-    try:
-        apply_locale(locale, locale_cfgfile)
-    except Exception as e:
-        log.debug(traceback.format_exc(e))
-        raise Exception("failed to apply locale %s" % locale)
+    log.debug("Setting locale to %s", locale)
+    locale_cfgfile = util.get_cfg_option_str(cfg, "locale_configfile")
+    cloud.distro.apply_locale(locale, locale_cfgfile)
 
=== modified file 'cloudinit/config/cc_mcollective.py'
--- cloudinit/CloudConfig/cc_mcollective.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_mcollective.py	2012-07-06 21:16:18 +0000
@@ -19,81 +19,73 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
-import subprocess
-import StringIO
-import ConfigParser
-import cloudinit.CloudConfig as cc
-import cloudinit.util as util
-
-pubcert_file = "/etc/mcollective/ssl/server-public.pem"
-pricert_file = "/etc/mcollective/ssl/server-private.pem"
-
-
-# Our fake header section
-class FakeSecHead(object):
-    def __init__(self, fp):
-        self.fp = fp
-        self.sechead = '[nullsection]\n'
-
-    def readline(self):
-        if self.sechead:
-            try:
-                return self.sechead
-            finally:
-                self.sechead = None
-        else:
-            return self.fp.readline()
-
-
-def handle(_name, cfg, _cloud, _log, _args):
+from StringIO import StringIO
+
+# Used since this can maintain comments
+# and doesn't need a top level section
+from configobj import ConfigObj
+
+from cloudinit import util
+
+PUBCERT_FILE = "/etc/mcollective/ssl/server-public.pem"
+PRICERT_FILE = "/etc/mcollective/ssl/server-private.pem"
+
+
+def handle(name, cfg, cloud, log, _args):
+
     # If there isn't a mcollective key in the configuration don't do anything
     if 'mcollective' not in cfg:
+        log.debug(("Skipping module named %s, "
+                   "no 'mcollective' key in configuration"), name)
         return
+
     mcollective_cfg = cfg['mcollective']
+
     # Start by installing the mcollective package ...
-    cc.install_packages(("mcollective",))
+    cloud.distro.install_packages(("mcollective",))
 
     # ... and then update the mcollective configuration
     if 'conf' in mcollective_cfg:
-        # Create object for reading server.cfg values
-        mcollective_config = ConfigParser.ConfigParser()
-        # Read server.cfg values from original file in order to be able to mix
-        # the rest up
-        mcollective_config.readfp(FakeSecHead(open('/etc/mcollective/'
-                                                   'server.cfg')))
-        for cfg_name, cfg in mcollective_cfg['conf'].iteritems():
+        # Read server.cfg values from the
+        # original file in order to be able to mix the rest up
+        server_cfg_fn = cloud.paths.join(True, '/etc/mcollective/server.cfg')
+        mcollective_config = ConfigObj(server_cfg_fn)
+        # See: http://tiny.cc/jh9agw
+        for (cfg_name, cfg) in mcollective_cfg['conf'].iteritems():
             if cfg_name == 'public-cert':
-                util.write_file(pubcert_file, cfg, mode=0644)
-                mcollective_config.set(cfg_name,
-                    'plugin.ssl_server_public', pubcert_file)
-                mcollective_config.set(cfg_name, 'securityprovider', 'ssl')
+                pubcert_fn = cloud.paths.join(True, PUBCERT_FILE)
+                util.write_file(pubcert_fn, cfg, mode=0644)
+                mcollective_config['plugin.ssl_server_public'] = pubcert_fn
+                mcollective_config['securityprovider'] = 'ssl'
             elif cfg_name == 'private-cert':
-                util.write_file(pricert_file, cfg, mode=0600)
-                mcollective_config.set(cfg_name,
-                    'plugin.ssl_server_private', pricert_file)
-                mcollective_config.set(cfg_name, 'securityprovider', 'ssl')
+                pricert_fn = cloud.paths.join(True, PRICERT_FILE)
+                util.write_file(pricert_fn, cfg, mode=0600)
+                mcollective_config['plugin.ssl_server_private'] = pricert_fn
+                mcollective_config['securityprovider'] = 'ssl'
             else:
-                # Iterate throug the config items, we'll use ConfigParser.set
-                # to overwrite or create new items as needed
-                for o, v in cfg.iteritems():
-                    mcollective_config.set(cfg_name, o, v)
+                if isinstance(cfg, (basestring, str)):
+                    # Just set it in the 'main' section
+                    mcollective_config[cfg_name] = cfg
+                elif isinstance(cfg, (dict)):
+                    # Iterate throug the config items, create a section
+                    # if it is needed and then add/or create items as needed
+                    if cfg_name not in mcollective_config.sections:
+                        mcollective_config[cfg_name] = {}
+                    for (o, v) in cfg.iteritems():
+                        mcollective_config[cfg_name][o] = v
+                else:
+                    # Otherwise just try to convert it to a string
+                    mcollective_config[cfg_name] = str(cfg)
         # We got all our config as wanted we'll rename
         # the previous server.cfg and create our new one
-        os.rename('/etc/mcollective/server.cfg',
-                  '/etc/mcollective/server.cfg.old')
-        outputfile = StringIO.StringIO()
-        mcollective_config.write(outputfile)
-        # Now we got the whole file, write to disk except first line
-        # Note below, that we've just used ConfigParser because it generally
-        # works.  Below, we remove the initial 'nullsection' header
-        # and then change 'key = value' to 'key: value'.  The global
-        # search and replace of '=' with ':' could be problematic though.
-        # this most likely needs fixing.
-        util.write_file('/etc/mcollective/server.cfg',
-            outputfile.getvalue().replace('[nullsection]\n', '').replace(' =',
-                                                                         ':'),
-            mode=0644)
+        old_fn = cloud.paths.join(False, '/etc/mcollective/server.cfg.old')
+        util.rename(server_cfg_fn, old_fn)
+        # Now we got the whole file, write to disk...
+        contents = StringIO()
+        mcollective_config.write(contents)
+        contents = contents.getvalue()
+        server_cfg_rw = cloud.paths.join(False, '/etc/mcollective/server.cfg')
+        util.write_file(server_cfg_rw, contents, mode=0644)
 
     # Start mcollective
-    subprocess.check_call(['service', 'mcollective', 'start'])
+    util.subp(['service', 'mcollective', 'start'], capture=False)
 
=== modified file 'cloudinit/config/cc_mounts.py'
--- cloudinit/CloudConfig/cc_mounts.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_mounts.py	2012-07-06 21:16:18 +0000
@@ -18,10 +18,16 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import os
+from string import whitespace  # pylint: disable=W0402
+
 import re
-from string import whitespace  # pylint: disable=W0402
+
+from cloudinit import util
+
+# Shortname matches 'sda', 'sda1', 'xvda', 'hda', 'sdb', xvdb, vda, vdd1
+SHORTNAME_FILTER = r"^[x]{0,1}[shv]d[a-z][0-9]*$"
+SHORTNAME = re.compile(SHORTNAME_FILTER)
+WS = re.compile("[%s]+" % (whitespace))
 
 
 def is_mdname(name):
@@ -49,38 +55,46 @@
     if "mounts" in cfg:
         cfgmnt = cfg["mounts"]
 
-    # shortname matches 'sda', 'sda1', 'xvda', 'hda', 'sdb', xvdb, vda, vdd1
-    shortname_filter = r"^[x]{0,1}[shv]d[a-z][0-9]*$"
-    shortname = re.compile(shortname_filter)
-
     for i in range(len(cfgmnt)):
         # skip something that wasn't a list
         if not isinstance(cfgmnt[i], list):
+            log.warn("Mount option %s not a list, got a %s instead",
+                     (i + 1), util.obj_name(cfgmnt[i]))
             continue
 
+        startname = str(cfgmnt[i][0])
+        log.debug("Attempting to determine the real name of %s", startname)
+
         # workaround, allow user to specify 'ephemeral'
         # rather than more ec2 correct 'ephemeral0'
-        if cfgmnt[i][0] == "ephemeral":
+        if startname == "ephemeral":
             cfgmnt[i][0] = "ephemeral0"
+            log.debug(("Adjusted mount option %s "
+                       "name from ephemeral to ephemeral0"), (i + 1))
 
-        if is_mdname(cfgmnt[i][0]):
-            newname = cloud.device_name_to_device(cfgmnt[i][0])
+        if is_mdname(startname):
+            newname = cloud.device_name_to_device(startname)
             if not newname:
-                log.debug("ignoring nonexistant named mount %s" % cfgmnt[i][0])
+                log.debug("Ignoring nonexistant named mount %s", startname)
                 cfgmnt[i][1] = None
             else:
-                if newname.startswith("/"):
-                    cfgmnt[i][0] = newname
-                else:
-                    cfgmnt[i][0] = "/dev/%s" % newname
+                renamed = newname
+                if not newname.startswith("/"):
+                    renamed = "/dev/%s" % newname
+                cfgmnt[i][0] = renamed
+                log.debug("Mapped metadata name %s to %s", startname, renamed)
         else:
-            if shortname.match(cfgmnt[i][0]):
-                cfgmnt[i][0] = "/dev/%s" % cfgmnt[i][0]
+            if SHORTNAME.match(startname):
+                renamed = "/dev/%s" % startname
+                log.debug("Mapped shortname name %s to %s", startname, renamed)
+                cfgmnt[i][0] = renamed
 
         # in case the user did not quote a field (likely fs-freq, fs_passno)
         # but do not convert None to 'None' (LP: #898365)
         for j in range(len(cfgmnt[i])):
-            if isinstance(cfgmnt[i][j], int):
+            if j is None:
+                continue
+            else:
                 cfgmnt[i][j] = str(cfgmnt[i][j])
 
     for i in range(len(cfgmnt)):
@@ -102,14 +116,18 @@
     # for each of the "default" mounts, add them only if no other
     # entry has the same device name
     for defmnt in defmnts:
-        devname = cloud.device_name_to_device(defmnt[0])
+        startname = defmnt[0]
+        devname = cloud.device_name_to_device(startname)
         if devname is None:
+            log.debug("Ignoring nonexistant named default mount %s", startname)
             continue
         if devname.startswith("/"):
             defmnt[0] = devname
         else:
             defmnt[0] = "/dev/%s" % devname
 
+        log.debug("Mapped default device %s to %s", startname, defmnt[0])
+
         cfgmnt_has = False
         for cfgm in cfgmnt:
             if cfgm[0] == defmnt[0]:
@@ -117,14 +135,22 @@
                 break
 
         if cfgmnt_has:
+            log.debug(("Not including %s, already"
+                       " previously included"), startname)
             continue
         cfgmnt.append(defmnt)
 
     # now, each entry in the cfgmnt list has all fstab values
     # if the second field is None (not the string, the value) we skip it
-    actlist = [x for x in cfgmnt if x[1] is not None]
+    actlist = []
+    for x in cfgmnt:
+        if x[1] is None:
+            log.debug("Skipping non-existent device named %s", x[0])
+        else:
+            actlist.append(x)
 
     if len(actlist) == 0:
+        log.debug("No modifications to fstab needed.")
         return
 
     comment = "comment=cloudconfig"
@@ -133,7 +159,7 @@
     dirs = []
     for line in actlist:
         # write 'comment' in the fs_mntops, entry,  claiming this
-        line[3] = "%s,comment=cloudconfig" % line[3]
+        line[3] = "%s,%s" % (line[3], comment)
         if line[2] == "swap":
             needswap = True
         if line[1].startswith("/"):
@@ -141,11 +167,10 @@
         cc_lines.append('\t'.join(line))
 
     fstab_lines = []
-    fstab = open("/etc/fstab", "r+")
-    ws = re.compile("[%s]+" % whitespace)
-    for line in fstab.read().splitlines():
+    fstab = util.load_file(cloud.paths.join(True, "/etc/fstab"))
+    for line in fstab.splitlines():
         try:
-            toks = ws.split(line)
+            toks = WS.split(line)
             if toks[3].find(comment) != -1:
                 continue
         except:
@@ -153,27 +178,23 @@
         fstab_lines.append(line)
 
     fstab_lines.extend(cc_lines)
-
-    fstab.seek(0)
-    fstab.write("%s\n" % '\n'.join(fstab_lines))
-    fstab.truncate()
-    fstab.close()
+    contents = "%s\n" % ('\n'.join(fstab_lines))
+    util.write_file(cloud.paths.join(False, "/etc/fstab"), contents)
 
     if needswap:
         try:
             util.subp(("swapon", "-a"))
         except:
-            log.warn("Failed to enable swap")
+            util.logexc(log, "Activating swap via 'swapon -a' failed")
 
     for d in dirs:
-        if os.path.exists(d):
-            continue
+        real_dir = cloud.paths.join(False, d)
         try:
-            os.makedirs(d)
+            util.ensure_dir(real_dir)
         except:
-            log.warn("Failed to make '%s' config-mount\n", d)
+            util.logexc(log, "Failed to make '%s' config-mount", d)
 
     try:
         util.subp(("mount", "-a"))
     except:
-        log.warn("'mount -a' failed")
+        util.logexc(log, "Activating mounts via 'mount -a' failed")
 
=== modified file 'cloudinit/config/cc_phone_home.py'
--- cloudinit/CloudConfig/cc_phone_home.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_phone_home.py	2012-07-06 21:16:18 +0000
@@ -17,13 +17,22 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-from cloudinit.CloudConfig import per_instance
-import cloudinit.util as util
-from time import sleep
-
-frequency = per_instance
-post_list_all = ['pub_key_dsa', 'pub_key_rsa', 'pub_key_ecdsa', 'instance_id',
-                 'hostname']
+
+from cloudinit import templater
+from cloudinit import url_helper as uhelp
+from cloudinit import util
+
+from cloudinit.settings import PER_INSTANCE
+
+frequency = PER_INSTANCE
+
+POST_LIST_ALL = [
+    'pub_key_dsa',
+    'pub_key_rsa',
+    'pub_key_ecdsa',
+    'instance_id',
+    'hostname'
+]
 
 
 # phone_home:
@@ -35,29 +44,33 @@
 #  url: http://my.foo.bar/$INSTANCE_ID/
 #  post: [ pub_key_dsa, pub_key_rsa, pub_key_ecdsa, instance_id
 #
-def handle(_name, cfg, cloud, log, args):
+def handle(name, cfg, cloud, log, args):
     if len(args) != 0:
         ph_cfg = util.read_conf(args[0])
     else:
         if not 'phone_home' in cfg:
+            log.debug(("Skipping module named %s, "
+                       "no 'phone_home' configuration found"), name)
             return
         ph_cfg = cfg['phone_home']
 
     if 'url' not in ph_cfg:
-        log.warn("no 'url' token in phone_home")
+        log.warn(("Skipping module named %s, "
+                  "no 'url' found in 'phone_home' configuration"), name)
         return
 
     url = ph_cfg['url']
     post_list = ph_cfg.get('post', 'all')
-    tries = ph_cfg.get('tries', 10)
+    tries = ph_cfg.get('tries')
     try:
         tries = int(tries)
     except:
-        log.warn("tries is not an integer. using 10")
         tries = 10
+        util.logexc(log, ("Configuration entry 'tries'"
+                          " is not an integer, using %s instead"), tries)
 
     if post_list == "all":
-        post_list = post_list_all
+        post_list = POST_LIST_ALL
 
     all_keys = {}
     all_keys['instance_id'] = cloud.get_instance_id()
@@ -69,38 +82,37 @@
         'pub_key_ecdsa': '/etc/ssh/ssh_host_ecdsa_key.pub',
     }
 
-    for n, path in pubkeys.iteritems():
+    for (n, path) in pubkeys.iteritems():
         try:
-            fp = open(path, "rb")
-            all_keys[n] = fp.read()
-            fp.close()
+            all_keys[n] = util.load_file(cloud.paths.join(True, path))
         except:
-            log.warn("%s: failed to open in phone_home" % path)
+            util.logexc(log, ("%s: failed to open, can not"
+                              " phone home that data"), path)
 
     submit_keys = {}
     for k in post_list:
         if k in all_keys:
             submit_keys[k] = all_keys[k]
         else:
-            submit_keys[k] = "N/A"
-            log.warn("requested key %s from 'post' list not available")
-
-    url = util.render_string(url, {'INSTANCE_ID': all_keys['instance_id']})
-
-    null_exc = object()
-    last_e = null_exc
-    for i in range(0, tries):
-        try:
-            util.readurl(url, submit_keys)
-            log.debug("succeeded submit to %s on try %i" % (url, i + 1))
-            return
-        except Exception as e:
-            log.debug("failed to post to %s on try %i" % (url, i + 1))
-            last_e = e
-        sleep(3)
-
-    log.warn("failed to post to %s in %i tries" % (url, tries))
-    if last_e is not null_exc:
-        raise(last_e)
-
-    return
+            submit_keys[k] = None
+            log.warn(("Requested key %s from 'post'"
+                      " configuration list not available"), k)
+
+    # Get them read to be posted
+    real_submit_keys = {}
+    for (k, v) in submit_keys.iteritems():
+        if v is None:
+            real_submit_keys[k] = 'N/A'
+        else:
+            real_submit_keys[k] = str(v)
+
+    # Incase the url is parameterized
+    url_params = {
+        'INSTANCE_ID': all_keys['instance_id'],
+    }
+    url = templater.render_string(url, url_params)
+    try:
+        uhelp.readurl(url, data=real_submit_keys, retries=tries, sec_between=3)
+    except:
+        util.logexc(log, ("Failed to post phone home data to"
+                          " %s in %s tries"), url, tries)
 
=== modified file 'cloudinit/config/cc_puppet.py'
--- cloudinit/CloudConfig/cc_puppet.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_puppet.py	2012-07-06 21:16:18 +0000
@@ -18,91 +18,96 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+from StringIO import StringIO
+
 import os
-import os.path
 import pwd
 import socket
-import subprocess
-import StringIO
-import ConfigParser
-import cloudinit.CloudConfig as cc
-import cloudinit.util as util
-
-
-def handle(_name, cfg, cloud, log, _args):
+
+from cloudinit import helpers
+from cloudinit import util
+
+
+def handle(name, cfg, cloud, log, _args):
     # If there isn't a puppet key in the configuration don't do anything
     if 'puppet' not in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'puppet' configuration found"), name)
         return
+
     puppet_cfg = cfg['puppet']
+
     # Start by installing the puppet package ...
-    cc.install_packages(("puppet",))
+    cloud.distro.install_packages(["puppet"])
 
     # ... and then update the puppet configuration
     if 'conf' in puppet_cfg:
         # Add all sections from the conf object to puppet.conf
-        puppet_conf_fh = open('/etc/puppet/puppet.conf', 'r')
+        puppet_conf_fn = cloud.paths.join(True, '/etc/puppet/puppet.conf')
+        contents = util.load_file(puppet_conf_fn)
         # Create object for reading puppet.conf values
-        puppet_config = ConfigParser.ConfigParser()
+        puppet_config = helpers.DefaultingConfigParser()
         # Read puppet.conf values from original file in order to be able to
-        # mix the rest up
-        puppet_config.readfp(StringIO.StringIO(''.join(i.lstrip() for i in
-                                               puppet_conf_fh.readlines())))
-        # Close original file, no longer needed
-        puppet_conf_fh.close()
-        for cfg_name, cfg in puppet_cfg['conf'].iteritems():
-            # ca_cert configuration is a special case
-            # Dump the puppetmaster ca certificate in the correct place
+        # mix the rest up. First clean them up (TODO is this really needed??)
+        cleaned_lines = [i.lstrip() for i in contents.splitlines()]
+        cleaned_contents = '\n'.join(cleaned_lines)
+        puppet_config.readfp(StringIO(cleaned_contents),
+                             filename=puppet_conf_fn)
+        for (cfg_name, cfg) in puppet_cfg['conf'].iteritems():
+            # Cert configuration is a special case
+            # Dump the puppet master ca certificate in the correct place
             if cfg_name == 'ca_cert':
                 # Puppet ssl sub-directory isn't created yet
                 # Create it with the proper permissions and ownership
-                os.makedirs('/var/lib/puppet/ssl')
-                os.chmod('/var/lib/puppet/ssl', 0771)
-                os.chown('/var/lib/puppet/ssl',
-                         pwd.getpwnam('puppet').pw_uid, 0)
-                os.makedirs('/var/lib/puppet/ssl/certs/')
-                os.chown('/var/lib/puppet/ssl/certs/',
-                         pwd.getpwnam('puppet').pw_uid, 0)
-                ca_fh = open('/var/lib/puppet/ssl/certs/ca.pem', 'w')
-                ca_fh.write(cfg)
-                ca_fh.close()
-                os.chown('/var/lib/puppet/ssl/certs/ca.pem',
-                         pwd.getpwnam('puppet').pw_uid, 0)
-                util.restorecon_if_possible('/var/lib/puppet', recursive=True)
+                pp_ssl_dir = cloud.paths.join(False, '/var/lib/puppet/ssl')
+                util.ensure_dir(pp_ssl_dir, 0771)
+                util.chownbyid(pp_ssl_dir,
+                               pwd.getpwnam('puppet').pw_uid, 0)
+                pp_ssl_certs = cloud.paths.join(False,
+                                                '/var/lib/puppet/ssl/certs/')
+                util.ensure_dir(pp_ssl_certs)
+                util.chownbyid(pp_ssl_certs,
+                               pwd.getpwnam('puppet').pw_uid, 0)
+                pp_ssl_ca_certs = cloud.paths.join(False,
+                                                   ('/var/lib/puppet/'
+                                                    'ssl/certs/ca.pem'))
+                util.write_file(pp_ssl_ca_certs, cfg)
+                util.chownbyid(pp_ssl_ca_certs,
+                               pwd.getpwnam('puppet').pw_uid, 0)
             else:
-                #puppet_conf_fh.write("\n[%s]\n" % (cfg_name))
-                # If puppet.conf already has this section we don't want to
-                # write it again
-                if puppet_config.has_section(cfg_name) == False:
-                    puppet_config.add_section(cfg_name)
                 # Iterate throug the config items, we'll use ConfigParser.set
                 # to overwrite or create new items as needed
-                for o, v in cfg.iteritems():
+                for (o, v) in cfg.iteritems():
                     if o == 'certname':
                         # Expand %f as the fqdn
+                        # TODO should this use the cloud fqdn??
                         v = v.replace("%f", socket.getfqdn())
                         # Expand %i as the instance id
-                        v = v.replace("%i",
-                              cloud.datasource.get_instance_id())
-                        # certname needs to be downcase
+                        v = v.replace("%i", cloud.get_instance_id())
+                        # certname needs to be downcased
                         v = v.lower()
                     puppet_config.set(cfg_name, o, v)
-                    #puppet_conf_fh.write("%s=%s\n" % (o, v))
             # We got all our config as wanted we'll rename
             # the previous puppet.conf and create our new one
-            os.rename('/etc/puppet/puppet.conf', '/etc/puppet/puppet.conf.old')
-            with open('/etc/puppet/puppet.conf', 'wb') as configfile:
-                puppet_config.write(configfile)
-            util.restorecon_if_possible('/etc/puppet/puppet.conf')
+            conf_old_fn = cloud.paths.join(False,
+                                           '/etc/puppet/puppet.conf.old')
+            util.rename(puppet_conf_fn, conf_old_fn)
+            puppet_conf_rw = cloud.paths.join(False, '/etc/puppet/puppet.conf')
+            util.write_file(puppet_conf_rw, puppet_config.stringify())
+
     # Set puppet to automatically start
     if os.path.exists('/etc/default/puppet'):
-        subprocess.check_call(['sed', '-i',
-                               '-e', 's/^START=.*/START=yes/',
-                               '/etc/default/puppet'])
+        util.subp(['sed', '-i',
+                  '-e', 's/^START=.*/START=yes/',
+                  '/etc/default/puppet'], capture=False)
     elif os.path.exists('/bin/systemctl'):
-        subprocess.check_call(['/bin/systemctl', 'enable', 'puppet.service'])
+        util.subp(['/bin/systemctl', 'enable', 'puppet.service'],
+                  capture=False)
     elif os.path.exists('/sbin/chkconfig'):
-        subprocess.check_call(['/sbin/chkconfig', 'puppet', 'on'])
+        util.subp(['/sbin/chkconfig', 'puppet', 'on'], capture=False)
     else:
-        log.warn("Do not know how to enable puppet service on this system")
+        log.warn(("Sorry we do not know how to enable"
+                  " puppet services on this system"))
+
     # Start puppetd
-    subprocess.check_call(['service', 'puppet', 'start'])
+    util.subp(['service', 'puppet', 'start'], capture=False)
 
=== modified file 'cloudinit/config/cc_resizefs.py'
--- cloudinit/CloudConfig/cc_resizefs.py	2012-03-21 20:41:50 +0000
+++ cloudinit/config/cc_resizefs.py	2012-07-06 21:16:18 +0000
@@ -18,91 +18,123 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit.util as util
-import subprocess
 import os
 import stat
-import sys
 import time
-import tempfile
-from cloudinit.CloudConfig import per_always
-
-frequency = per_always
-
-
-def handle(_name, cfg, _cloud, log, args):
-    if len(args) != 0:
-        resize_root = False
-        if str(args[0]).lower() in ['true', '1', 'on', 'yes']:
-            resize_root = True
-    else:
-        resize_root = util.get_cfg_option_str(cfg, "resize_rootfs", True)
-
-    if str(resize_root).lower() in ['false', '0']:
-        return
-
-    # we use mktemp rather than mkstemp because early in boot nothing
-    # else should be able to race us for this, and we need to mknod.
-    devpth = tempfile.mktemp(prefix="cloudinit.resizefs.", dir="/run")
-
+
+from cloudinit import util
+from cloudinit.settings import PER_ALWAYS
+
+frequency = PER_ALWAYS
+
+RESIZE_FS_PREFIXES_CMDS = [
+    ('ext', 'resize2fs'),
+    ('xfs', 'xfs_growfs'),
+]
+
+
+def nodeify_path(devpth, where, log):
     try:
-        st_dev = os.stat("/").st_dev
+        st_dev = os.stat(where).st_dev
         dev = os.makedev(os.major(st_dev), os.minor(st_dev))
         os.mknod(devpth, 0400 | stat.S_IFBLK, dev)
+        return st_dev
     except:
         if util.is_container():
-            log.debug("inside container, ignoring mknod failure in resizefs")
+            log.debug("Inside container, ignoring mknod failure in resizefs")
             return
-        log.warn("Failed to make device node to resize /")
+        log.warn("Failed to make device node to resize %s at %s",
+                 where, devpth)
         raise
 
-    cmd = ['blkid', '-c', '/dev/null', '-sTYPE', '-ovalue', devpth]
+
+def get_fs_type(st_dev, path, log):
     try:
-        (fstype, _err) = util.subp(cmd)
-    except subprocess.CalledProcessError as e:
-        log.warn("Failed to get filesystem type of maj=%s, min=%s via: %s" %
-            (os.major(st_dev), os.minor(st_dev), cmd))
-        log.warn("output=%s\nerror=%s\n", e.output[0], e.output[1])
-        os.unlink(devpth)
+        dev_entries = util.find_devs_with(tag='TYPE', oformat='value',
+                                         no_cache=True, path=path)
+        if not dev_entries:
+            return None
+        return dev_entries[0].strip()
+    except util.ProcessExecutionError:
+        util.logexc(log, ("Failed to get filesystem type"
+                          " of maj=%s, min=%s for path %s"),
+                    os.major(st_dev), os.minor(st_dev), path)
         raise
 
-    if str(fstype).startswith("ext"):
-        resize_cmd = ['resize2fs', devpth]
-    elif fstype == "xfs":
-        resize_cmd = ['xfs_growfs', devpth]
+
+def handle(name, cfg, cloud, log, args):
+    if len(args) != 0:
+        resize_root = args[0]
     else:
-        os.unlink(devpth)
-        log.debug("not resizing unknown filesystem %s" % fstype)
+        resize_root = util.get_cfg_option_str(cfg, "resize_rootfs", True)
+
+    if not util.translate_bool(resize_root):
+        log.debug("Skipping module named %s, resizing disabled", name)
         return
 
+    # TODO is the directory ok to be used??
+    resize_root_d = util.get_cfg_option_str(cfg, "resize_rootfs_tmp", "/run")
+    resize_root_d = cloud.paths.join(False, resize_root_d)
+    util.ensure_dir(resize_root_d)
+
+    # TODO: allow what is to be resized to be configurable??
+    resize_what = cloud.paths.join(False, "/")
+    with util.ExtendedTemporaryFile(prefix="cloudinit.resizefs.",
+                                    dir=resize_root_d, delete=True) as tfh:
+        devpth = tfh.name
+
+        # Delete the file so that mknod will work
+        # but don't change the file handle to know that its
+        # removed so that when a later call that recreates
+        # occurs this temporary file will still benefit from
+        # auto deletion
+        tfh.unlink_now()
+
+        st_dev = nodeify_path(devpth, resize_what, log)
+        fs_type = get_fs_type(st_dev, devpth, log)
+        if not fs_type:
+            log.warn("Could not determine filesystem type of %s", resize_what)
+            return
+
+        resizer = None
+        fstype_lc = fs_type.lower()
+        for (pfix, root_cmd) in RESIZE_FS_PREFIXES_CMDS:
+            if fstype_lc.startswith(pfix):
+                resizer = root_cmd
+                break
+
+        if not resizer:
+            log.warn("Not resizing unknown filesystem type %s for %s",
+                     fs_type, resize_what)
+            return
+
+        log.debug("Resizing %s (%s) using %s", resize_what, fs_type, resizer)
+        resize_cmd = [resizer, devpth]
+
+        if resize_root == "noblock":
+            # Fork to a child that will run
+            # the resize command
+            util.fork_cb(do_resize, resize_cmd, log)
+            # Don't delete the file now in the parent
+            tfh.delete = False
+        else:
+            do_resize(resize_cmd, log)
+
+    action = 'Resized'
     if resize_root == "noblock":
-        fid = os.fork()
-        if fid == 0:
-            try:
-                do_resize(resize_cmd, devpth, log)
-                os._exit(0)  # pylint: disable=W0212
-            except Exception as exc:
-                sys.stderr.write("Failed: %s" % exc)
-                os._exit(1)  # pylint: disable=W0212
-    else:
-        do_resize(resize_cmd, devpth, log)
-
-    log.debug("resizing root filesystem (type=%s, maj=%i, min=%i, val=%s)" %
-        (str(fstype).rstrip("\n"), os.major(st_dev), os.minor(st_dev),
-         resize_root))
-
-    return
-
-
-def do_resize(resize_cmd, devpth, log):
+        action = 'Resizing (via forking)'
+    log.debug("%s root filesystem (type=%s, maj=%i, min=%i, val=%s)",
+              action, fs_type, os.major(st_dev), os.minor(st_dev), resize_root)
+
+
+def do_resize(resize_cmd, log):
+    start = time.time()
     try:
-        start = time.time()
         util.subp(resize_cmd)
-    except subprocess.CalledProcessError as e:
-        log.warn("Failed to resize filesystem (%s)" % resize_cmd)
-        log.warn("output=%s\nerror=%s\n", e.output[0], e.output[1])
-        os.unlink(devpth)
+    except util.ProcessExecutionError:
+        util.logexc(log, "Failed to resize filesystem (cmd=%s)", resize_cmd)
         raise
-
-    os.unlink(devpth)
-    log.debug("resize took %s seconds" % (time.time() - start))
+    tot_time = int(time.time() - start)
+    log.debug("Resizing took %s seconds", tot_time)
+    # TODO: Should we add a fsck check after this to make
+    # sure we didn't corrupt anything?
 
=== modified file 'cloudinit/config/cc_rightscale_userdata.py'
--- cloudinit/CloudConfig/cc_rightscale_userdata.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_rightscale_userdata.py	2012-07-06 21:16:18 +0000
@@ -35,44 +35,68 @@
 ##
 ##
 
-import cloudinit.util as util
-from cloudinit.CloudConfig import per_instance
-from cloudinit import get_ipath_cur
+import os
+
+from cloudinit import url_helper as uhelp
+from cloudinit import util
+from cloudinit.settings import PER_INSTANCE
+
 from urlparse import parse_qs
 
-frequency = per_instance
-my_name = "cc_rightscale_userdata"
-my_hookname = 'CLOUD_INIT_REMOTE_HOOK'
-
-
-def handle(_name, _cfg, cloud, log, _args):
+frequency = PER_INSTANCE
+
+MY_NAME = "cc_rightscale_userdata"
+MY_HOOKNAME = 'CLOUD_INIT_REMOTE_HOOK'
+
+
+def handle(name, _cfg, cloud, log, _args):
     try:
         ud = cloud.get_userdata_raw()
     except:
-        log.warn("failed to get raw userdata in %s" % my_name)
+        log.warn("Failed to get raw userdata in module %s", name)
         return
 
     try:
         mdict = parse_qs(ud)
-        if not my_hookname in mdict:
+        if not mdict or not MY_HOOKNAME in mdict:
+            log.debug(("Skipping module %s, "
+                       "did not find %s in parsed"
+                       " raw userdata"), name, MY_HOOKNAME)
             return
     except:
-        log.warn("failed to urlparse.parse_qa(userdata_raw())")
+        util.logexc(log, ("Failed to parse query string %s"
+                           " into a dictionary"), ud)
         raise
 
-    scripts_d = get_ipath_cur('scripts')
-    i = 0
-    first_e = None
-    for url in mdict[my_hookname]:
-        fname = "%s/rightscale-%02i" % (scripts_d, i)
-        i = i + 1
+    wrote_fns = []
+    captured_excps = []
+
+    # These will eventually be then ran by the cc_scripts_user
+    # TODO: maybe this should just be a new user data handler??
+    # Instead of a late module that acts like a user data handler?
+    scripts_d = cloud.get_ipath_cur('scripts')
+    urls = mdict[MY_HOOKNAME]
+    for (i, url) in enumerate(urls):
+        fname = os.path.join(scripts_d, "rightscale-%02i" % (i))
         try:
-            content = util.readurl(url)
-            util.write_file(fname, content, mode=0700)
+            resp = uhelp.readurl(url)
+            # Ensure its a valid http response (and something gotten)
+            if resp.ok() and resp.contents:
+                util.write_file(fname, str(resp), mode=0700)
+                wrote_fns.append(fname)
         except Exception as e:
-            if not first_e:
-                first_e = None
-            log.warn("%s failed to read %s: %s" % (my_name, url, e))
-
-    if first_e:
-        raise(e)
+            captured_excps.append(e)
+            util.logexc(log, "%s failed to read %s and write %s",
+                        MY_NAME, url, fname)
+
+    if wrote_fns:
+        log.debug("Wrote out rightscale userdata to %s files", len(wrote_fns))
+
+    if len(wrote_fns) != len(urls):
+        skipped = len(urls) - len(wrote_fns)
+        log.debug("%s urls were skipped or failed", skipped)
+
+    if captured_excps:
+        log.warn("%s failed with exceptions, re-raising the last one",
+                 len(captured_excps))
+        raise captured_excps[-1]
 
=== modified file 'cloudinit/config/cc_rsyslog.py'
--- cloudinit/CloudConfig/cc_rsyslog.py	2012-01-18 14:07:33 +0000
+++ cloudinit/config/cc_rsyslog.py	2012-07-06 21:16:18 +0000
@@ -18,16 +18,15 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import cloudinit
-import logging
-import cloudinit.util as util
-import traceback
+import os
+
+from cloudinit import util
 
 DEF_FILENAME = "20-cloud-config.conf"
 DEF_DIR = "/etc/rsyslog.d"
 
 
-def handle(_name, cfg, _cloud, log, _args):
+def handle(name, cfg, cloud, log, _args):
     # rsyslog:
     #  - "*.* @@192.158.1.1"
     #  - content: "*.*   @@192.0.2.1:10514"
@@ -37,17 +36,18 @@
 
     # process 'rsyslog'
     if not 'rsyslog' in cfg:
+        log.debug(("Skipping module named %s,"
+                   " no 'rsyslog' key in configuration"), name)
         return