Merge ~wesley-wiedenmeier/cloud-init:integration-testing-distro-features into cloud-init:master

[Wesley Wiedenmeier]

Integration Testing: Improvments to testing on alternate distros

 - Allow images to override user data settings
   - image config option 'user_data_overrides' accepts dict of
     attributes to update a testcase's user data with before
     launching an image
   - workaround for (LP #1575779)
   - should only be used if this is the only way to get an image
     working
 - Add support for distro feature flags
   - add framework for feature flags to release config with
     feature groups and overrides allowed in any release conf
     override level
   - add support for feature flags in platform and config handling
   - during collect, skip testcases that require features not
     supported by the image with a warning message
 - Add required features to testcase configs
   - skip testcases with distro compatibility issues
   - allow test suite to be run in full on other distros
   - many testcases with compatibility issues can be updated
     to allow them to work across distros
 - Updated documentation
   - updated documentation for testsuite config
     - explain how config is merged
     - describe new image config format
     - explain how to configure an image's system_ready_script
   - add documentation on how feature flags work
   - add documentation on how error handling works
   - add documentation on setup_image options

[Wesley Wiedenmeier (wesley-wiedenmeier)]

This is a continuation of the main integration testing update at:
https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/308218
The main update reworks the image config format and enables use of other distros on lxd, both of which are required for the features in this branch.

Adding support for distro specific user data overrides provides a workaround for the hostnamectl in an unpriv container bug. This allows centos tests to run, and can easily be disabled via image config once this bug is resolved.

Adding the distro feature flags makes the support for other distros added by the main update useful. Without the distro feature flags, a full run of all available testcases on a distro does not work for non-ubuntu distros as many testcases are not compatible with multiple distros. Many of the testcases that do not work outside of ubuntu can be made to work on other distros, but a separate branch based on this one will be used for resolving testcase bugs.

The distro feature flags have been set to skip any tests which do not work on most non-ubuntu distros. There may be some additional minor updates to the feature flags to enable running the full testsuite on centos70 and older releases of ubuntu, debian and centos. All testscases which are expected to run with an up to date cloud-init on centos70 are enabled for it, but the test suite has not yet been run with an up to date rpm for centos, so centos tests are not confirmed to work fully yet.
Current status of working/non-working distros is here:
https://docs.google.com/spreadsheets/d/1DAzBlh-wk-rv-WRjllNRG6nnHtAmD0EFBLEYtu8weII/

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:50aa692d246bc2bfd84b0f423f799e2dbbcfe785
https://jenkins.ubuntu.com/server/job/cloud-init-ci/162/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/162
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/162
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/162
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/162
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/162

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/162/rebuild

review: Approve (continuous-integration)

46d291a... by Wesley Wiedenmeier on 2017-04-02

Integration Testing: add support for overriding distro feature flags

This provides a mechanism to override feature flag settings for a test run via
the command line option '--feature-override <feature flag name>=<true/false>'.
In some situations tests may depend on external resources that are not always
available to the test suite. A feature flag representing that external resource
can be set to false in release config by default, and enabled via the command
line only when the user has ensured that that resource is available.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:a01ef1506e9905f46f942f054223ec8a1c26f477
https://jenkins.ubuntu.com/server/job/cloud-init-ci/203/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/203
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/203
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/203
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/203
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/203

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/203/rebuild

review: Approve (continuous-integration)

685ac9c... by Wesley Wiedenmeier on 2017-04-08

Integration Testing: improve lxd snapshot creation

Avoid a potential race between cloud-init starting and files that need to be
removed while creating a snapshot. This race condition only caused errors when
testing on trusty, but is there on any release. It is fixed by waiting for
cloud-init to start fully before running the boot_clean_script, as this
guarantees that files that should be cleaned will not be written after the
boot_clean_script has run.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:13bdee594be3c1eba1fe17787593efb012bf095a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/219/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/219
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/219
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/219
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/219
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/219

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/219/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

I'm going to mark this 'work in progress'. Josh has integrated this branch and others into his merge proposal at https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324136 . We do plan to pull that all soon.

Thanks for your work, Wesley!

[Scott Moser (smoser)]

Hi.
I've marked this 'merged' as I think it is now in trunk under 76d58265e34851b78e952a7f275340863c90a9f5.
If you disagree, please feel free to re-open.

diff --git a/doc/rtd/topics/tests.rst b/doc/rtd/topics/tests.rst
index 0663811..b6b8da1 100644
--- a/doc/rtd/topics/tests.rst
+++ b/doc/rtd/topics/tests.rst
@@ -47,7 +47,7 @@ The test configuration is a YAML file such as *ntp_server.yaml* below:
         cat /etc/ntp.conf | grep '^server'
 
 
-There are two keys, 1 required and 1 optional, in the YAML file:
+There are several keys, 1 required and some optional, in the YAML file:
 
 1. The required key is ``cloud_config``. This should be a string of valid
    YAML that is exactly what would normally be placed in a cloud-config file,
@@ -61,6 +61,11 @@ There are two keys, 1 required and 1 optional, in the YAML file:
    reported. The name of the sub-key is important. The sub-key is used by
    the verification script to recall the output of the commands ran.
 
+3. The ``required_features`` key may be used to specify a list of features
+   flags that an image must have to be able to run the testcase. For example,
+   if a testcase relies on an image supporting apt, then the config for the
+   testcase should include ``required_features: [ apt ]``.
+
 Default Collect Scripts
 -----------------------
 
@@ -154,6 +159,7 @@ Development Checklist
     * Optionally, commands to capture additional output
     * Valid YAML
     * Placed in the appropriate sub-folder in the configs directory
+    * Any image features required for the test are specified
 * Verification File
     * Named 'your_test_here.py'
     * Valid unit tests validating output collected
@@ -252,6 +258,248 @@ configuration users can run the integration tests via tox:
 Users need to invoke the citest enviornment and then pass any additional
 arguments.
 
+Setup Image
+-----------
+The ``run`` and ``collect`` commands have many options to setup the image
+before running tests in addition to installing a deb in the target. Any
+combination of the following can be used:
+
+* ``--deb``: install a deb into the image
+* ``--rpm``: install a rpm into the image
+* ``--repo``: enable a repository and update cloud-init afterwards
+* ``--ppa``: enable a ppa and update cloud-init afterwards
+* ``--upgrade``: upgrade cloud-init from repos
+* ``--upgrade-full``: run a full system upgrade
+* ``--script``: execute a script in the image. this can perform any setup
+  required that is not covered by the other options
+
+Configuring the Test Suite
+==========================
+
+Most of the behavior of the test suite is configurable through several yaml
+files. These control the behavior of the test suite's platforms, images, and
+tests. The main config files for platforms, images and testcases are
+``platforms.yaml``, ``releases.yaml`` and ``testcases.yaml``.
+
+Config handling
+---------------
+All configurable parts of the test suite use a defaults + overrides system for
+managing config entries. All base config items are dictionaries.
+
+Merging is done on a key by key basis, with all keys in the default and
+overrides represented in the final result. If a key exists both in
+the defaults and the overrides, then behavior depends on the type of data the
+key refers to. If it is atomic data or a list, then the overrides will replace
+the default. If the data is a dictionary then the value will be the result of
+merging that dictionary from the default config and that dictionary from the
+overrides.
+
+Merging is done using the function ``tests.cloud_tests.config.merge_config``,
+which can be examined for more detail on config merging behavior.
+
+The following demonstrates merge behavior:
+
+.. code-block:: yaml
+
+    defaults:
+        list_item:
+         - list_entry_1
+         - list_entry_2
+        int_item_1: 123
+        int_item_2: 234
+        dict_item:
+            subkey_1: 1
+            subkey_2: 2
+            subkey_dict:
+                subsubkey_1: a
+                subsubkey_2: b
+
+    overrides:
+        list_item:
+         - overridden_list_entry
+        int_item_1: 0
+        dict_item:
+            subkey_2: false
+            subkey_dict:
+                subsubkey_2: 'new value'
+
+    result:
+        list_item:
+         - overridden_list_entry
+        int_item_1: 0
+        int_item_2: 234
+        dict_item:
+            subkey_1: 1
+            subkey_2: false
+            subkey_dict:
+                subsubkey_1: a
+                subsubkey_2: 'new value'
+
+
+Image Config Structure
+----------------------
+Image configuration is handled in ``releases.yaml``. The image configuration
+controls how platforms locate and acquire images, how the platforms should
+interact with the images, how platforms should detect when an image has fully
+booted, any options that are required to set the image up, and features that
+the image supports.
+
+Since settings for locating an image and interacting with it differ from
+platform to platform, there are 4 levels of settings available for images on
+top of the default image settings. The structure of the image config file is:
+
+.. code-block:: yaml
+
+    default_release_config:
+        default:
+            ...
+        <platform>:
+            ...
+        <platform>:
+            ...
+
+    releases:
+        <release name>:
+            <default>:
+                ...
+            <platform>:
+                ...
+            <platform>:
+                ...
+
+
+The base config is created from the overall defaults and the overrides for the
+platform. The overrides are created from the default config for the image and
+the platform specific overrides for the image.
+
+Image Config for System Boot
+----------------------------
+The test suite must be able to test if a system has fully booted and if
+cloud-init has finished running, so that running collect scripts does not race
+against the target image booting. This is done using the
+``system_ready_script`` and ``cloud_init_ready_script`` image config keys.
+
+Each of these keys accepts a small bash test statement as a string that must
+return 0 or 1. Since this test statement will be added into a larger bash
+statement it must be a single statement using the ``[`` test syntax.
+
+The default image config provides a system ready script that works for any
+systemd based image. If the iamge is not systmed based, then a different test
+statement must be provided. The default config also provides a test for whether
+or not cloud-init has finished which checks for the file
+``/run/cloud-init/result.json``. This should be sufficient for most systems, as
+writing to this file is one of the last things cloud-init does.
+
+The setting ``boot_timeout`` controls how long, in seconds, the platform should
+wait for an image to boot. If the system ready script has not indicated that
+the system is fully booted within this time an error will be raised.
+
+Image Config Feature Flags
+--------------------------
+Not all testcases can work on all images due to features the testcase requires
+not being present on that image. If a testcase requires features in an image
+that are not likely to be present across all distros and platforms that the
+test suite supports, then the test can be skipped everywhere it is not
+supported.
+
+This is done through feature flags, which are names for features supported on
+some images but not all that may be required by testcases. Configuration for
+feature flags is provided in ``releases.yaml`` under the ``features`` top level
+key. The features config includes a list of all currently defined feature flags
+and their meanings, and a list of feature groups.
+
+Feature groups are groups of features that many images have in common. For
+example, the ``ubuntu_specific`` feature group includes features that should be
+present across most ubuntu releases, but may or may not be for other distros.
+Feature groups are specified for an image as a list under the key
+``feature_groups``.
+
+An image's feature flags are derived from the features groups that that image
+has and any feature overrides provided. Feature overrides can be specified
+under the ``features`` key which accepts a dictionary of
+``{<feature_name>: true/false}`` mappings. If a feature is omitted from an
+image's feature flags or set to false in the overrides then the test suite will
+skip any tests that require that feature when using that image.
+
+Feature flags may be overridden at runtime using the ``--feature-override``
+command line argument. It accepts a feature flag and value to set in the format
+``<feature name>: true/false``. Multiple ``--feature-override`` flags can be
+used, and will all be applied to all feature flags for images used during a
+test.
+
+Image Config Setup Overrides
+----------------------------
+If an image requires some of the options for image setup to be used, then it
+may specify overrides for the command line arguments passed into setup image.
+These may be specified as a dictionary under the ``setup_overrides`` key. When
+an image is set up, the arguments that control how it is set up will be the
+arguments from the command line, with any entries in ``setup_overrides`` used
+to override these arguments.
+
+For example, images that do not come with cloud-init already installed should
+have ``setup_overrides: {upgrade: true}`` specified so that in the event that
+no additional setup options are given, cloud-init will be installed from the
+image's repos before running tests. Note that if other options such as
+``--deb`` are passed in on the command line, these will still work as expected,
+since apt's policy for cloud-init would prefer the locally installed deb over
+an older version from the repos.
+
+Image Config Platform Specific Options
+--------------------------------------
+There are many platform specific options in image configuration that allow
+platforms to locate images and that control additional setup that the platform
+may have to do to make the image useable. For information on how these work,
+please consult the documentation for that platform in the integration testing
+suite and the ``releases.yaml`` file for examples.
+
+Error Handling Behavior
+=======================
+
+The test suite makes an attempt to run as many tests as possible even in the
+event of some failing so that automated runs collect as much data as possible.
+In the event that something goes wrong while setting up for or running a test,
+the test suite will attempt to continue running any tests which have not been
+effected by the error.
+
+For example, if the test suite was told to run tests on one platform for two
+releases and an error occured setting up the first image, all tests for that
+image would be skipped, and the test suite would continue to set up the second
+image and run tests on it. Or, if the system does not start properly for one
+testcase out of many to run on that image, that testcase will be skipped and
+the next one will be run.
+
+Note that if any errors at all occur, the test suite will record the failure
+and where it occurred in the result data and write it out to the specified
+result file.
+
+Exit Codes
+----------
+The test suite counts how many errors occur throughout a run. The exit code
+after a run is the number of errors that occured. If the exit code is non-zero
+than something is wrong either with the test suite, the configuration for an
+image, a testcase, or cloud-init itself.
+
+Note that the exit code does not always direclty correspond to the number
+of failed testcases, since in some cases, a single error during image setup
+can mean that several testcases are not run. If run is used, then the exit code
+will be the sum of the number of errors in the collect and verify stages.
+
+Result Data
+-----------
+The test suite generates result data that includes how long each stage of the
+test suite took and which parts were and were not successful. This data is
+dumped to the log after the collect and verify stages, and may also be written
+out in yaml format to a file. If part of the setup failed, the traceback for
+the failure and the error message will be included in the result file. If a
+test verifier finds a problem with the collected data from a test run, the
+class, test function and test will be recorded in the result data.
+
+Data Dir
+--------
+When using run, the collected data is written into a temporary directory. In
+the even that all tests pass, this directory is deleted. In the even that a
+test fails or an error occurs, this data will be left in place, and a message
+will be written to the log giving the location of the data.
 
 Architecture
 ============
diff --git a/tests/cloud_tests/args.py b/tests/cloud_tests/args.py
index b68cc98..8f28b94 100644
--- a/tests/cloud_tests/args.py
+++ b/tests/cloud_tests/args.py
@@ -9,15 +9,18 @@ ARG_SETS = {
     'COLLECT': (
         (('-p', '--platform'),
          {'help': 'platform(s) to run tests on', 'metavar': 'PLATFORM',
-          'action': 'append', 'choices': config.list_enabled_platforms(),
+          'action': 'append', 'choices': config.ENABLED_PLATFORMS,
           'default': []}),
         (('-n', '--os-name'),
          {'help': 'the name(s) of the OS(s) to test', 'metavar': 'NAME',
-          'action': 'append', 'choices': config.list_enabled_distros(),
+          'action': 'append', 'choices': config.ENABLED_DISTROS,
           'default': []}),
         (('-t', '--test-config'),
          {'help': 'test config file(s) to use', 'metavar': 'FILE',
-          'action': 'append', 'default': []}),),
+          'action': 'append', 'default': []}),
+        (('--feature-override',),
+         {'help': 'feature flags override(s), <flagname>=<true/false>',
+          'action': 'append', 'default': [], 'required': False}),),
     'CREATE': (
         (('-c', '--config'),
          {'help': 'cloud-config yaml for testcase', 'metavar': 'DATA',
@@ -61,8 +64,12 @@ ARG_SETS = {
          {'help': 'ppa to enable (implies -u)', 'metavar': 'NAME',
           'action': 'store'}),
         (('-u', '--upgrade'),
-         {'help': 'upgrade before starting tests', 'action': 'store_true',
-          'default': False}),),
+         {'help': 'upgrade or install cloud-init from repo',
+          'action': 'store_true', 'default': False}),
+        (('--upgrade-full',),
+         {'help': 'do full system upgrade from repo (implies -u)',
+          'action': 'store_true', 'default': False}),),
+
 }
 
 SUBCMDS = {
@@ -121,15 +128,15 @@ def normalize_collect_args(args):
     """
     # platform should default to all supported
     if len(args.platform) == 0:
-        args.platform = config.list_enabled_platforms()
+        args.platform = config.ENABLED_PLATFORMS
     args.platform = util.sorted_unique(args.platform)
 
     # os name should default to all enabled
     # if os name is provided ensure that all provided are supported
     if len(args.os_name) == 0:
-        args.os_name = config.list_enabled_distros()
+        args.os_name = config.ENABLED_DISTROS
     else:
-        supported = config.list_enabled_distros()
+        supported = config.ENABLED_DISTROS
         invalid = [os_name for os_name in args.os_name
                    if os_name not in supported]
         if len(invalid) != 0:
@@ -158,6 +165,17 @@ def normalize_collect_args(args):
             args.test_config = valid
     args.test_config = util.sorted_unique(args.test_config)
 
+    # parse feature flag overrides and ensure all are valid
+    if args.feature_override:
+        overrides = args.feature_override
+        args.feature_override = util.parse_conf_list(
+            overrides, boolean=True, valid=config.list_feature_flags())
+        if not args.feature_override:
+            LOG.error('invalid feature flag override(s): %s', overrides)
+            return None
+    else:
+        args.feature_override = {}
+
     return args
 
 
diff --git a/tests/cloud_tests/collect.py b/tests/cloud_tests/collect.py
index 68b47d7..5b004c3 100644
--- a/tests/cloud_tests/collect.py
+++ b/tests/cloud_tests/collect.py
@@ -18,8 +18,10 @@ def collect_script(instance, base_dir, script, script_name):
     return_value: None, may raise errors
     """
     LOG.debug('running collect script: %s', script_name)
-    util.write_file(os.path.join(base_dir, script_name),
-                    instance.run_script(script))
+    (out, err, exit) = instance.run_script(
+        script, rcs=range(0, 256),
+        description='collect: {}'.format(script_name))
+    util.write_file(os.path.join(base_dir, script_name), out)
 
 
 def collect_test_data(args, snapshot, os_name, test_name):
@@ -39,15 +41,27 @@ def collect_test_data(args, snapshot, os_name, test_name):
     test_scripts = test_config['collect_scripts']
     test_output_dir = os.sep.join(
         (args.data_dir, snapshot.platform_name, os_name, test_name))
-    boot_timeout = (test_config.get('boot_timeout')
-                    if isinstance(test_config.get('boot_timeout'), int) else
-                    snapshot.config.get('timeout'))
 
     # if test is not enabled, skip and return 0 failures
     if not test_config.get('enabled', False):
         LOG.warn('test config %s is not enabled, skipping', test_name)
         return ({}, 0)
 
+    # if testcase requires a feature flag that the image does not support,
+    # skip the testcase with a warning
+    req_features = test_config.get('required_features', [])
+    if any(feature not in snapshot.features for feature in req_features):
+        LOG.warn('test config %s requires features not supported by image, '
+                 'skipping.\nrequired features: %s\nsupported features: %s',
+                 test_name, req_features, snapshot.features)
+        return ({}, 0)
+
+    # if there are user data overrides required for this test case, apply them
+    overrides = snapshot.config.get('user_data_overrides', {})
+    if overrides:
+        LOG.debug('updating user data for collect with: %s', overrides)
+        user_data = util.update_user_data(user_data, overrides)
+
     # create test instance
     component = PlatformComponent(
         partial(instances.get_instance, snapshot, user_data,
@@ -56,7 +70,7 @@ def collect_test_data(args, snapshot, os_name, test_name):
     LOG.info('collecting test data for test: %s', test_name)
     with component as instance:
         start_call = partial(run_single, 'boot instance', partial(
-            instance.start, wait=True, wait_time=boot_timeout))
+            instance.start, wait=True, wait_for_cloud_init=True))
         collect_calls = [partial(run_single, 'script {}'.format(script_name),
                                  partial(collect_script, instance,
                                          test_output_dir, script, script_name))
@@ -100,10 +114,9 @@ def collect_image(args, platform, os_name):
     """
     res = ({}, 1)
 
-    os_config = config.load_os_config(os_name)
-    if not os_config.get('enabled'):
-        raise ValueError('OS {} not enabled'.format(os_name))
-
+    os_config = config.load_os_config(
+        platform.platform_name, os_name, require_enabled=True,
+        feature_overrides=args.feature_override)
     component = PlatformComponent(
         partial(images.get_image, platform, os_config))
 
@@ -126,10 +139,8 @@ def collect_platform(args, platform_name):
     """
     res = ({}, 1)
 
-    platform_config = config.load_platform_config(platform_name)
-    if not platform_config.get('enabled'):
-        raise ValueError('Platform {} not enabled'.format(platform_name))
-
+    platform_config = config.load_platform_config(
+        platform_name, require_enabled=True)
     component = PlatformComponent(
         partial(platforms.get_platform, platform_name, platform_config))
 
diff --git a/tests/cloud_tests/config.py b/tests/cloud_tests/config.py
index f3a13c9..9a0ad4d 100644
--- a/tests/cloud_tests/config.py
+++ b/tests/cloud_tests/config.py
@@ -14,6 +14,20 @@ RELEASES_CONF = os.path.join(BASE_DIR, 'releases.yaml')
 TESTCASE_CONF = os.path.join(BASE_DIR, 'testcases.yaml')
 
 
+def get(base, key):
+    """
+    get config entry 'key' from base, ensuring is dictionary
+    """
+    return base[key] if key in base and base[key] is not None else {}
+
+
+def enabled(config):
+    """
+    test if config item is enabled
+    """
+    return isinstance(config, dict) and config.get('enabled', False)
+
+
 def path_to_name(path):
     """
     convert abs or rel path to test config to path under configs/
@@ -61,22 +75,63 @@ def merge_config(base, override):
     return res
 
 
-def load_platform_config(platform):
+def merge_feature_groups(feature_conf, feature_groups, overrides):
+    """
+    combine feature groups and overrides to construct a supported feature list
+    feature_conf: feature config from releases.yaml
+    feature_groups: feature groups the release is a member of
+    overrides: overrides specified by the release's config
+    return_value: dict of {feature: true/false} settings
+    """
+    res = dict().fromkeys(feature_conf['all'])
+    for group in feature_groups:
+        res.update(feature_conf['groups'][group])
+    res.update(overrides)
+    return res
+
+
+def load_platform_config(platform_name, require_enabled=False):
     """
     load configuration for platform
+    platform_name: name of platform to retrieve config for
+    require_enabled: if true, raise error if 'enabled' not True
+    return_value: config dict
     """
     main_conf = c_util.read_conf(PLATFORM_CONF)
-    return merge_config(main_conf.get('default_platform_config'),
-                        main_conf.get('platforms')[platform])
+    conf = merge_config(main_conf['default_platform_config'],
+                        main_conf['platforms'][platform_name])
+    if require_enabled and not enabled(conf):
+        raise ValueError('Platform is not enabled')
+    return conf
 
 
-def load_os_config(os_name):
+def load_os_config(platform_name, os_name, require_enabled=False,
+                   feature_overrides={}):
     """
     load configuration for os
+    platform_name: platform name to load os config for
+    os_name: name of os to retrieve config for
+    require_enabled: if true, raise error if 'enabled' not True
+    feature_overrides: feature flag overrides to merge with features config
+    return_value: config dict
     """
     main_conf = c_util.read_conf(RELEASES_CONF)
-    return merge_config(main_conf.get('default_release_config'),
-                        main_conf.get('releases')[os_name])
+    default = main_conf['default_release_config']
+    image = main_conf['releases'][os_name]
+    conf = merge_config(merge_config(get(default, 'default'),
+                                     get(default, platform_name)),
+                        merge_config(get(image, 'default'),
+                                     get(image, platform_name)))
+
+    feature_conf = main_conf['features']
+    feature_groups = conf.get('feature_groups', [])
+    overrides = merge_config(get(conf, 'features'), feature_overrides)
+    conf['features'] = merge_feature_groups(
+        feature_conf, feature_groups, overrides)
+
+    if require_enabled and not enabled(conf):
+        raise ValueError('OS is not enabled')
+    return conf
 
 
 def load_test_config(path):
@@ -87,20 +142,34 @@ def load_test_config(path):
                         c_util.read_conf(name_to_path(path)))
 
 
+def list_feature_flags():
+    """
+    list all supported feature flags
+    """
+    feature_conf = get(c_util.read_conf(RELEASES_CONF), 'features')
+    return feature_conf.get('all', [])
+
+
 def list_enabled_platforms():
     """
     list all platforms enabled for testing
     """
-    platforms = c_util.read_conf(PLATFORM_CONF).get('platforms')
-    return [k for k, v in platforms.items() if v.get('enabled')]
+    platforms = get(c_util.read_conf(PLATFORM_CONF), 'platforms')
+    return [k for k, v in platforms.items() if enabled(v)]
 
 
-def list_enabled_distros():
+def list_enabled_distros(platforms):
     """
-    list all distros enabled for testing
+    list all distros enabled for testing on specified platforms
     """
-    releases = c_util.read_conf(RELEASES_CONF).get('releases')
-    return [k for k, v in releases.items() if v.get('enabled')]
+
+    def platform_has_enabled(config):
+        return any(enabled(merge_config(get(config, 'default'),
+                                        get(config, platform)))
+                   for platform in platforms)
+
+    releases = get(c_util.read_conf(RELEASES_CONF), 'releases')
+    return [k for k, v in releases.items() if platform_has_enabled(v)]
 
 
 def list_test_configs():
@@ -110,4 +179,8 @@ def list_test_configs():
     return [os.path.abspath(f) for f in
             glob.glob(os.sep.join((TEST_CONF_DIR, '*', '*.yaml')))]
 
+
+ENABLED_PLATFORMS = list_enabled_platforms()
+ENABLED_DISTROS = list_enabled_distros(ENABLED_PLATFORMS)
+
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/configs/bugs/lp1628337.yaml b/tests/cloud_tests/configs/bugs/lp1628337.yaml
index 1d6bf48..e39b3cd 100644
--- a/tests/cloud_tests/configs/bugs/lp1628337.yaml
+++ b/tests/cloud_tests/configs/bugs/lp1628337.yaml
@@ -1,6 +1,9 @@
 #
 # LP Bug 1628337: cloud-init tries to install NTP before even configuring the archives
 #
+required_features:
+  - apt
+  - lsb_release
 cloud_config: |
   #cloud-config
   ntp:
diff --git a/tests/cloud_tests/configs/examples/add_apt_repositories.yaml b/tests/cloud_tests/configs/examples/add_apt_repositories.yaml
index b896435..4b8575f 100644
--- a/tests/cloud_tests/configs/examples/add_apt_repositories.yaml
+++ b/tests/cloud_tests/configs/examples/add_apt_repositories.yaml
@@ -4,6 +4,8 @@
 # 2016-11-17: Disabled as covered by module based tests
 #
 enabled: False
+required_features:
+  - apt
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/apt_configure_conf.yaml b/tests/cloud_tests/configs/modules/apt_configure_conf.yaml
index 163ae3f..de45300 100644
--- a/tests/cloud_tests/configs/modules/apt_configure_conf.yaml
+++ b/tests/cloud_tests/configs/modules/apt_configure_conf.yaml
@@ -1,6 +1,8 @@
 #
 # Provide a configuration for APT
 #
+required_features:
+  - apt
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/apt_configure_disable_suites.yaml b/tests/cloud_tests/configs/modules/apt_configure_disable_suites.yaml
index 73e4a53..9880067 100644
--- a/tests/cloud_tests/configs/modules/apt_configure_disable_suites.yaml
+++ b/tests/cloud_tests/configs/modules/apt_configure_disable_suites.yaml
@@ -1,6 +1,9 @@
 #
 # Disables everything in sources.list
 #
+required_features:
+  - apt
+  - lsb_release
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/apt_configure_primary.yaml b/tests/cloud_tests/configs/modules/apt_configure_primary.yaml
index 2ec30ca..41bcf2f 100644
--- a/tests/cloud_tests/configs/modules/apt_configure_primary.yaml
+++ b/tests/cloud_tests/configs/modules/apt_configure_primary.yaml
@@ -1,6 +1,9 @@
 #
 # Setup a custome primary sources.list
 #
+required_features:
+  - apt
+  - apt_src_cont
 cloud_config: |
   #cloud-config
   apt:
@@ -16,4 +19,8 @@ collect_scripts:
     #!/bin/bash
     grep -v '^#' /etc/apt/sources.list | sed '/^\s*$/d' | grep -c gtlib.gatech.edu
 
+  sources.list: |
+      #!/bin/bash
+      cat /etc/apt/sources.list
+
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/configs/modules/apt_configure_proxy.yaml b/tests/cloud_tests/configs/modules/apt_configure_proxy.yaml
index e737130..be6c6f8 100644
--- a/tests/cloud_tests/configs/modules/apt_configure_proxy.yaml
+++ b/tests/cloud_tests/configs/modules/apt_configure_proxy.yaml
@@ -1,6 +1,8 @@
 #
 # Set apt proxy
 #
+required_features:
+  - apt
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/apt_configure_security.yaml b/tests/cloud_tests/configs/modules/apt_configure_security.yaml
index f6a2c82..83dd51d 100644
--- a/tests/cloud_tests/configs/modules/apt_configure_security.yaml
+++ b/tests/cloud_tests/configs/modules/apt_configure_security.yaml
@@ -1,6 +1,9 @@
 #
 # Add security to sources.list
 #
+required_features:
+  - apt
+  - ubuntu_repos
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/apt_configure_sources_key.yaml b/tests/cloud_tests/configs/modules/apt_configure_sources_key.yaml
index e7568a6..bde9398 100644
--- a/tests/cloud_tests/configs/modules/apt_configure_sources_key.yaml
+++ b/tests/cloud_tests/configs/modules/apt_configure_sources_key.yaml
@@ -1,6 +1,9 @@
 #
 # Add a sources.list entry with a given key (Debian Jessie)
 #
+required_features:
+  - apt
+  - lsb_release
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/apt_configure_sources_keyserver.yaml b/tests/cloud_tests/configs/modules/apt_configure_sources_keyserver.yaml
index 1a4a238..11da61e 100644
--- a/tests/cloud_tests/configs/modules/apt_configure_sources_keyserver.yaml
+++ b/tests/cloud_tests/configs/modules/apt_configure_sources_keyserver.yaml
@@ -1,6 +1,9 @@
 #
 # Add a sources.list entry with a key from a keyserver
 #
+required_features:
+  - apt
+  - lsb_release
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/apt_configure_sources_list.yaml b/tests/cloud_tests/configs/modules/apt_configure_sources_list.yaml
index 057fc72..143cb08 100644
--- a/tests/cloud_tests/configs/modules/apt_configure_sources_list.yaml
+++ b/tests/cloud_tests/configs/modules/apt_configure_sources_list.yaml
@@ -1,6 +1,9 @@
 #
 # Generate a sources.list
 #
+required_features:
+  - apt
+  - lsb_release
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/apt_configure_sources_ppa.yaml b/tests/cloud_tests/configs/modules/apt_configure_sources_ppa.yaml
index dee9dc7..9efdae5 100644
--- a/tests/cloud_tests/configs/modules/apt_configure_sources_ppa.yaml
+++ b/tests/cloud_tests/configs/modules/apt_configure_sources_ppa.yaml
@@ -1,6 +1,12 @@
 #
 # Add a PPA to source.list
 #
+# NOTE: on older ubuntu releases the sources file added is named
+#       'curtin-dev-test-archive-trusty', without 'ubuntu' in the middle
+required_features:
+  - apt
+  - ppa
+  - ppa_file_name
 cloud_config: |
   #cloud-config
   apt:
@@ -16,5 +22,8 @@ collect_scripts:
   apt-key: |
     #!/bin/bash
     apt-key finger
+  sources_full: |
+    #!/bin/bash
+    cat /etc/apt/sources.list
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/configs/modules/apt_pipelining_disable.yaml b/tests/cloud_tests/configs/modules/apt_pipelining_disable.yaml
index 5fa0cee..bd9b5d0 100644
--- a/tests/cloud_tests/configs/modules/apt_pipelining_disable.yaml
+++ b/tests/cloud_tests/configs/modules/apt_pipelining_disable.yaml
@@ -1,6 +1,8 @@
 #
 # Disable apt pipelining value
 #
+required_features:
+  - apt
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/apt_pipelining_os.yaml b/tests/cloud_tests/configs/modules/apt_pipelining_os.yaml
index 87d183e..cbed3ba 100644
--- a/tests/cloud_tests/configs/modules/apt_pipelining_os.yaml
+++ b/tests/cloud_tests/configs/modules/apt_pipelining_os.yaml
@@ -1,6 +1,8 @@
 #
 # Set apt pipelining value to OS
 #
+required_features:
+  - apt
 cloud_config: |
   #cloud-config
   apt:
diff --git a/tests/cloud_tests/configs/modules/byobu.yaml b/tests/cloud_tests/configs/modules/byobu.yaml
index fd648c7..a9aa1f3 100644
--- a/tests/cloud_tests/configs/modules/byobu.yaml
+++ b/tests/cloud_tests/configs/modules/byobu.yaml
@@ -1,6 +1,8 @@
 #
 # Install and enable byobu system wide and default user
 #
+required_features:
+  - byobu
 cloud_config: |
   #cloud-config
   byobu_by_default: enable
diff --git a/tests/cloud_tests/configs/modules/keys_to_console.yaml b/tests/cloud_tests/configs/modules/keys_to_console.yaml
index a90e42c..5d86e73 100644
--- a/tests/cloud_tests/configs/modules/keys_to_console.yaml
+++ b/tests/cloud_tests/configs/modules/keys_to_console.yaml
@@ -1,6 +1,8 @@
 #
 # Hide printing of ssh key and fingerprints for specific keys
 #
+required_features:
+  - syslog
 cloud_config: |
   #cloud-config
   ssh_fp_console_blacklist: [ssh-dss, ssh-dsa, ecdsa-sha2-nistp256]
diff --git a/tests/cloud_tests/configs/modules/landscape.yaml b/tests/cloud_tests/configs/modules/landscape.yaml
index e6f4955..ed2c37c 100644
--- a/tests/cloud_tests/configs/modules/landscape.yaml
+++ b/tests/cloud_tests/configs/modules/landscape.yaml
@@ -4,6 +4,8 @@
 # 2016-11-17: Disabled due to this not working
 #
 enabled: false
+required_features:
+  - landscape
 cloud_config: |
   #cloud-conifg
   landscape:
diff --git a/tests/cloud_tests/configs/modules/locale.yaml b/tests/cloud_tests/configs/modules/locale.yaml
index af5ad63..e01518a 100644
--- a/tests/cloud_tests/configs/modules/locale.yaml
+++ b/tests/cloud_tests/configs/modules/locale.yaml
@@ -1,6 +1,9 @@
 #
 # Set locale to non-default option and verify
 #
+required_features:
+  - engb_locale
+  - locale_gen
 cloud_config: |
   #cloud-config
   locale: en_GB.UTF-8
diff --git a/tests/cloud_tests/configs/modules/lxd_bridge.yaml b/tests/cloud_tests/configs/modules/lxd_bridge.yaml
index 568bb70..e6b7e76 100644
--- a/tests/cloud_tests/configs/modules/lxd_bridge.yaml
+++ b/tests/cloud_tests/configs/modules/lxd_bridge.yaml
@@ -1,6 +1,8 @@
 #
 # LXD configured with directory backend and IPv4 bridge
 #
+required_features:
+  - lxd
 cloud_config: |
   #cloud-config
   lxd:
diff --git a/tests/cloud_tests/configs/modules/lxd_dir.yaml b/tests/cloud_tests/configs/modules/lxd_dir.yaml
index 99b9219..f93a3fa 100644
--- a/tests/cloud_tests/configs/modules/lxd_dir.yaml
+++ b/tests/cloud_tests/configs/modules/lxd_dir.yaml
@@ -1,6 +1,8 @@
 #
 # LXD configured with directory backend
 #
+required_features:
+  - lxd
 cloud_config: |
   #cloud-config
   lxd:
diff --git a/tests/cloud_tests/configs/modules/ntp.yaml b/tests/cloud_tests/configs/modules/ntp.yaml
index d094157..0d07ef5 100644
--- a/tests/cloud_tests/configs/modules/ntp.yaml
+++ b/tests/cloud_tests/configs/modules/ntp.yaml
@@ -1,6 +1,14 @@
 #
 # Emtpy NTP config to setup using defaults
 #
+# NOTE: this should not require apt feature, use 'which' rather than 'dpkg -l'
+# NOTE: this should not require no_ntpdate feature, use 'which' to check for
+#       installation rather than 'dpkg -l', as 'grep ntp' matches 'ntpdate'
+# NOTE: the verifier should check for any ntp server not 'ubuntu.pool.ntp.org'
+required_features:
+  - apt
+  - no_ntpdate
+  - ubuntu_ntp
 cloud_config: |
   #cloud-config
   ntp:
@@ -16,5 +24,8 @@ collect_scripts:
   ntp_conf_empty: |
     #!/bin/bash
     grep '^pool' /etc/ntp.conf
+  ntp_installed_list: |
+    #!/bin/bash
+    dpkg -l | grep ntp
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/configs/modules/ntp_pools.yaml b/tests/cloud_tests/configs/modules/ntp_pools.yaml
index bd0ac29..6ec1bfe 100644
--- a/tests/cloud_tests/configs/modules/ntp_pools.yaml
+++ b/tests/cloud_tests/configs/modules/ntp_pools.yaml
@@ -1,6 +1,16 @@
 #
 # NTP config using specific pools
 #
+# NOTE: this should not require apt feature, use 'which' rather than 'dpkg -l'
+# NOTE: this should not require no_ntpdate feature, use 'which' to check for 
+#       installation rather than 'dpkg -l', as 'grep ntp' matches 'ntpdate'
+# NOTE: lsb_release listed here because with recent cloud-init deb with
+#       (LP: 1628337) resolved, cloud-init will attempt to configure archives.
+#       this fails without lsb_release as UNAVAILABLE is used for $RELEASE
+required_features:
+  - apt
+  - no_ntpdate
+  - lsb_release
 cloud_config: |
   #cloud-config
   ntp:
diff --git a/tests/cloud_tests/configs/modules/ntp_servers.yaml b/tests/cloud_tests/configs/modules/ntp_servers.yaml
index 934b9c5..b61ec37 100644
--- a/tests/cloud_tests/configs/modules/ntp_servers.yaml
+++ b/tests/cloud_tests/configs/modules/ntp_servers.yaml
@@ -1,6 +1,12 @@
 #
 # NTP config using specific servers
 #
+# NOTE: this should not require apt feature, use 'which' rather than 'dpkg -l'
+# NOTE: this should not require no_ntpdate feature, use 'which' to check for 
+#       installation rather than 'dpkg -l', as 'grep ntp' matches 'ntpdate'
+required_features:
+  - apt
+  - no_ntpdate
 cloud_config: |
   #cloud-config
   ntp:
diff --git a/tests/cloud_tests/configs/modules/package_update_upgrade_install.yaml b/tests/cloud_tests/configs/modules/package_update_upgrade_install.yaml
index d027d54..71d24b8 100644
--- a/tests/cloud_tests/configs/modules/package_update_upgrade_install.yaml
+++ b/tests/cloud_tests/configs/modules/package_update_upgrade_install.yaml
@@ -1,6 +1,17 @@
 #
 # Update/upgrade via apt and then install a pair of packages
 #
+# NOTE: this should not require apt feature, use 'which' rather than 'dpkg -l'
+# NOTE: the testcase for this looks for the command in history.log as
+#       /usr/bin/apt-get..., which is not how it always appears. it should
+#       instead look for just apt-get...
+# NOTE: this testcase should not require 'apt_up_out', and should look for a
+#       call to 'apt-get upgrade' or 'apt-get dist-upgrade' in cloud-init.log
+#       rather than 'Calculating upgrade...' in output
+required_features:
+  - apt
+  - apt_hist_fmt
+  - apt_up_out
 cloud_config: |
   #cloud-config
   packages:
diff --git a/tests/cloud_tests/configs/modules/set_hostname.yaml b/tests/cloud_tests/configs/modules/set_hostname.yaml
index 5aae150..c96344c 100644
--- a/tests/cloud_tests/configs/modules/set_hostname.yaml
+++ b/tests/cloud_tests/configs/modules/set_hostname.yaml
@@ -1,6 +1,8 @@
 #
 # Set the hostname and update /etc/hosts
 #
+required_features:
+  - hostname
 cloud_config: |
   #cloud-config
   hostname: myhostname
diff --git a/tests/cloud_tests/configs/modules/set_hostname_fqdn.yaml b/tests/cloud_tests/configs/modules/set_hostname_fqdn.yaml
index 0014c19..daf7593 100644
--- a/tests/cloud_tests/configs/modules/set_hostname_fqdn.yaml
+++ b/tests/cloud_tests/configs/modules/set_hostname_fqdn.yaml
@@ -1,6 +1,8 @@
 #
 # Set the hostname and update /etc/hosts
 #
+required_features:
+  - hostname
 cloud_config: |
   #cloud-config
   manage_etc_hosts: true
diff --git a/tests/cloud_tests/configs/modules/set_password.yaml b/tests/cloud_tests/configs/modules/set_password.yaml
index 8fa46d9..04d7c58 100644
--- a/tests/cloud_tests/configs/modules/set_password.yaml
+++ b/tests/cloud_tests/configs/modules/set_password.yaml
@@ -1,6 +1,8 @@
 #
 # Set password of default user
 #
+required_features:
+  - ubuntu_user
 cloud_config: |
   #cloud-config
   password: password
diff --git a/tests/cloud_tests/configs/modules/set_password_expire.yaml b/tests/cloud_tests/configs/modules/set_password_expire.yaml
index 926731f..789604b 100644
--- a/tests/cloud_tests/configs/modules/set_password_expire.yaml
+++ b/tests/cloud_tests/configs/modules/set_password_expire.yaml
@@ -1,6 +1,8 @@
 #
 # Expire password for all users
 #
+required_features:
+  - sshd
 cloud_config: |
   #cloud-config
   chpasswd: { expire: True }
diff --git a/tests/cloud_tests/configs/modules/snappy.yaml b/tests/cloud_tests/configs/modules/snappy.yaml
index 923bfe1..030b790 100644
--- a/tests/cloud_tests/configs/modules/snappy.yaml
+++ b/tests/cloud_tests/configs/modules/snappy.yaml
@@ -1,6 +1,8 @@
 #
 # Install snappy
 #
+required_features:
+  - snap
 cloud_config: |
   #cloud-config
   snappy:
diff --git a/tests/cloud_tests/configs/modules/ssh_auth_key_fingerprints_disable.yaml b/tests/cloud_tests/configs/modules/ssh_auth_key_fingerprints_disable.yaml
index 33943bd..746653e 100644
--- a/tests/cloud_tests/configs/modules/ssh_auth_key_fingerprints_disable.yaml
+++ b/tests/cloud_tests/configs/modules/ssh_auth_key_fingerprints_disable.yaml
@@ -1,6 +1,8 @@
 #
 # Disable fingerprint printing
 #
+required_features:
+  - syslog
 cloud_config: |
   #cloud-config
   ssh_genkeytypes: []
diff --git a/tests/cloud_tests/configs/modules/ssh_auth_key_fingerprints_enable.yaml b/tests/cloud_tests/configs/modules/ssh_auth_key_fingerprints_enable.yaml
index 4c97077..9f5dc34 100644
--- a/tests/cloud_tests/configs/modules/ssh_auth_key_fingerprints_enable.yaml
+++ b/tests/cloud_tests/configs/modules/ssh_auth_key_fingerprints_enable.yaml
@@ -1,6 +1,11 @@
 #
 # Print auth keys with different hash than md5
 #
+# NOTE: testcase checks for '256 SHA256:.*(ECDSA)' on output line on trusty
+#       this fails as line in output reads '256:.*(ECDSA)'
+required_features:
+  - syslog
+  - ssh_key_fmt
 cloud_config: |
   #cloud-config
   ssh_genkeytypes:
diff --git a/tests/cloud_tests/configs/modules/ssh_import_id.yaml b/tests/cloud_tests/configs/modules/ssh_import_id.yaml
index 6e5a163..b62d3f6 100644
--- a/tests/cloud_tests/configs/modules/ssh_import_id.yaml
+++ b/tests/cloud_tests/configs/modules/ssh_import_id.yaml
@@ -1,6 +1,9 @@
 #
 # Import a user's ssh key via gh or lp
 #
+required_features:
+  - ubuntu_user
+  - sudo
 cloud_config: |
   #cloud-config
   ssh_import_id:
diff --git a/tests/cloud_tests/configs/modules/ssh_keys_generate.yaml b/tests/cloud_tests/configs/modules/ssh_keys_generate.yaml
index 637d783..659fd93 100644
--- a/tests/cloud_tests/configs/modules/ssh_keys_generate.yaml
+++ b/tests/cloud_tests/configs/modules/ssh_keys_generate.yaml
@@ -1,6 +1,8 @@
 #
 # SSH keys generated using cloud-init
 #
+required_features:
+  - ubuntu_user
 cloud_config: |
   #cloud-config
   ssh_genkeytypes:
diff --git a/tests/cloud_tests/configs/modules/ssh_keys_provided.yaml b/tests/cloud_tests/configs/modules/ssh_keys_provided.yaml
index 25df645..5ceb362 100644
--- a/tests/cloud_tests/configs/modules/ssh_keys_provided.yaml
+++ b/tests/cloud_tests/configs/modules/ssh_keys_provided.yaml
@@ -2,6 +2,9 @@
 # SSH keys provided via cloud config
 #
 enabled: False
+required_features:
+  - ubuntu_user
+  - sudo
 cloud_config: |
   #cloud-config
   disable_root: false
diff --git a/tests/cloud_tests/configs/modules/timezone.yaml b/tests/cloud_tests/configs/modules/timezone.yaml
index 8c96ed4..5112aa9 100644
--- a/tests/cloud_tests/configs/modules/timezone.yaml
+++ b/tests/cloud_tests/configs/modules/timezone.yaml
@@ -1,6 +1,8 @@
 #
 # Set system timezone
 #
+required_features:
+  - daylight_time
 cloud_config: |
   #cloud-config
   timezone: US/Aleutian
diff --git a/tests/cloud_tests/configs/modules/user_groups.yaml b/tests/cloud_tests/configs/modules/user_groups.yaml
index 9265595..71cc9da 100644
--- a/tests/cloud_tests/configs/modules/user_groups.yaml
+++ b/tests/cloud_tests/configs/modules/user_groups.yaml
@@ -1,6 +1,8 @@
 #
 # Create groups and users with various options
 #
+required_features:
+  - ubuntu_user
 cloud_config: |
   #cloud-config
   # Add groups to the system
diff --git a/tests/cloud_tests/configs/modules/write_files.yaml b/tests/cloud_tests/configs/modules/write_files.yaml
index 4bb2991..ce936b7 100644
--- a/tests/cloud_tests/configs/modules/write_files.yaml
+++ b/tests/cloud_tests/configs/modules/write_files.yaml
@@ -1,6 +1,10 @@
 #
 # Write various file types
 #
+# NOTE: on trusty 'file' has an output formatting error for binary files and
+#       has 2 spaces in 'LSB  executable', which causes a failure here
+required_features:
+  - no_file_fmt_e
 cloud_config: |
   #cloud-config
   write_files:
diff --git a/tests/cloud_tests/images/base.py b/tests/cloud_tests/images/base.py
index 394b11f..1f604cf 100644
--- a/tests/cloud_tests/images/base.py
+++ b/tests/cloud_tests/images/base.py
@@ -7,13 +7,14 @@ class Image(object):
     """
     platform_name = None
 
-    def __init__(self, name, config, platform):
+    def __init__(self, platform, config):
         """
-        setup
+        Set up image
+        platform: platform object
+        config: image configuration
         """
-        self.name = name
-        self.config = config
         self.platform = platform
+        self.config = config
 
     def __str__(self):
         """
@@ -28,10 +29,24 @@ class Image(object):
         """
         raise NotImplementedError
 
-    # FIXME: instead of having execute and push_file and other instance methods
-    #        here which pass through to a hidden instance, it might be better
-    #        to expose an instance that the image can be modified through
-    def execute(self, command, stdin=None, stdout=None, stderr=None, env={}):
+    @property
+    def features(self):
+        """
+        feature flags supported by this image
+        return_value: list of feature names
+        """
+        return [k for k, v in self.config.get('features', {}).items() if v]
+
+    @property
+    def setup_overrides(self):
+        """
+        setup options that need to be overridden for the image
+        return_value: dictionary to update args with
+        """
+        # NOTE: more sophisticated options may be requied at some point
+        return self.config.get('setup_overrides', {})
+
+    def execute(self, *args, **kwargs):
         """
         execute command in image, modifying image
         """
@@ -43,7 +58,7 @@ class Image(object):
         """
         raise NotImplementedError
 
-    def run_script(self, script):
+    def run_script(self, *args, **kwargs):
         """
         run script in image, modifying image
         return_value: script output
diff --git a/tests/cloud_tests/images/lxd.py b/tests/cloud_tests/images/lxd.py
index 7a41614..f820582 100644
--- a/tests/cloud_tests/images/lxd.py
+++ b/tests/cloud_tests/images/lxd.py
@@ -2,6 +2,10 @@
 
 from tests.cloud_tests.images import base
 from tests.cloud_tests.snapshots import lxd as lxd_snapshot
+from tests.cloud_tests import util
+
+import os
+import shutil
 
 
 class LXDImage(base.Image):
@@ -10,27 +14,44 @@ class LXDImage(base.Image):
     """
     platform_name = "lxd"
 
-    def __init__(self, name, config, platform, pylxd_image):
+    def __init__(self, platform, config, pylxd_image):
         """
-        setup
+        Set up image
+        platform: platform object
+        config: image configuration
         """
-        self.platform = platform
-        self._pylxd_image = pylxd_image
+        self.modified = False
         self._instance = None
-        super(LXDImage, self).__init__(name, config, platform)
+        self._pylxd_image = None
+        self.pylxd_image = pylxd_image
+        super(LXDImage, self).__init__(platform, config)
 
     @property
     def pylxd_image(self):
-        self._pylxd_image.sync()
+        if self._pylxd_image:
+            self._pylxd_image.sync()
         return self._pylxd_image
 
+    @pylxd_image.setter
+    def pylxd_image(self, pylxd_image):
+        if self._instance:
+            self._instance.destroy()
+            self._instance = None
+        if (self._pylxd_image and
+                (self._pylxd_image is not pylxd_image) and
+                (not self.config.get('cache_base_image') or self.modified)):
+            self._pylxd_image.delete(wait=True)
+        self.modified = False
+        self._pylxd_image = pylxd_image
+
     @property
     def instance(self):
         if not self._instance:
             self._instance = self.platform.launch_container(
-                image=self.pylxd_image.fingerprint,
-                image_desc=str(self), use_desc='image-modification')
-        self._instance.start(wait=True, wait_time=self.config.get('timeout'))
+                self.properties, self.config, self.features,
+                use_desc='image-modification', image_desc=str(self),
+                image=self.pylxd_image.fingerprint)
+        self._instance.start()
         return self._instance
 
     @property
@@ -46,6 +67,78 @@ class LXDImage(base.Image):
             'release': properties.get('release'),
         }
 
+    def export_image(self, output_dir):
+        """
+        export image from lxd image store to (split) tarball on disk
+        output_dir: dir to store tarballs in
+        return_value: tuple of path to metadata tarball and rootfs tarball
+        """
+        # pylxd's image export feature doesn't do split exports, so use cmdline
+        util.subp(['lxc', 'image', 'export', self.pylxd_image.fingerprint,
+                   output_dir], capture=True)
+        tarballs = [p for p in os.listdir(output_dir) if p.endswith('tar.xz')]
+        metadata = os.path.join(
+            output_dir, next(p for p in tarballs if p.startswith('meta-')))
+        rootfs = os.path.join(
+            output_dir, next(p for p in tarballs if not p.startswith('meta-')))
+        return (metadata, rootfs)
+
+    def import_image(self, metadata, rootfs):
+        """
+        import image to lxd image store from (split) tarball on disk
+        note, this will replace and delete the current pylxd_image
+        metadata: metadata tarball
+        rootfs: rootfs tarball
+        return_value: imported image fingerprint
+        """
+        alias = util.gen_instance_name(
+            image_desc=str(self), use_desc='update-metadata')
+        util.subp(['lxc', 'image', 'import', metadata, rootfs,
+                   '--alias', alias], capture=True)
+        self.pylxd_image = self.platform.query_image_by_alias(alias)
+        return self.pylxd_image.fingerprint
+
+    def update_templates(self, template_config, template_data):
+        """
+        update the image's template configuration
+        note, this will replace and delete the current pylxd_image
+        template_config: config overrides for template portion of metadata
+        template_data: template data to place into templates/
+        """
+        # set up tmp files
+        export_dir = util.tmpdir()
+        extract_dir = util.tmpdir()
+        new_metadata = os.path.join(export_dir, 'new-meta.tar.xz')
+        metadata_yaml = os.path.join(extract_dir, 'metadata.yaml')
+        template_dir = os.path.join(extract_dir, 'templates')
+
+        try:
+            # extract old data
+            (metadata, rootfs) = self.export_image(export_dir)
+            shutil.unpack_archive(metadata, extract_dir)
+
+            # update metadata
+            metadata = util.read_conf(metadata_yaml)
+            templates = metadata.get('templates', {})
+            templates.update(template_config)
+            metadata['templates'] = templates
+            util.yaml_dump(metadata, metadata_yaml)
+
+            # write out template files
+            for name, content in template_data.items():
+                path = os.path.join(template_dir, name)
+                util.write_file(path, content)
+
+            # store new data, mark new image as modified
+            util.flat_tar(new_metadata, extract_dir)
+            self.import_image(new_metadata, rootfs)
+            self.modified = True
+
+        finally:
+            # remove tmpfiles
+            shutil.rmtree(export_dir)
+            shutil.rmtree(extract_dir)
+
     def execute(self, *args, **kwargs):
         """
         execute command in image, modifying image
@@ -58,35 +151,43 @@ class LXDImage(base.Image):
         """
         return self.instance.push_file(local_path, remote_path)
 
-    def run_script(self, script):
+    def run_script(self, *args, **kwargs):
         """
         run script in image, modifying image
         return_value: script output
         """
-        return self.instance.run_script(script)
+        return self.instance.run_script(*args, **kwargs)
 
     def snapshot(self):
         """
         create snapshot of image, block until done
         """
-        # clone current instance, start and freeze clone
+        # get empty user data to pass in to instance
+        # if overrides for user data provided, use them
+        empty_userdata = util.update_user_data(
+            {}, self.config.get('user_data_overrides', {}))
+        conf = {'user.user-data': empty_userdata}
+        # clone current instance
         instance = self.platform.launch_container(
+            self.properties, self.config, self.features,
             container=self.instance.name, image_desc=str(self),
-            use_desc='snapshot')
-        instance.start(wait=True, wait_time=self.config.get('timeout'))
+            use_desc='snapshot', container_config=conf)
+        # wait for cloud-init before boot_clean_script is run to ensure
+        # /var/lib/cloud is removed cleanly
+        instance.start(wait=True, wait_for_cloud_init=True)
         if self.config.get('boot_clean_script'):
             instance.run_script(self.config.get('boot_clean_script'))
+        # freeze current instance and return snapshot
         instance.freeze()
         return lxd_snapshot.LXDSnapshot(
-            self.properties, self.config, self.platform, instance)
+            self.platform, self.properties, self.config,
+            self.features, instance)
 
     def destroy(self):
         """
         clean up data associated with image
         """
-        if self._instance:
-            self._instance.destroy()
-        self.pylxd_image.delete(wait=True)
+        self.pylxd_image = None
         super(LXDImage, self).destroy()
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/base.py b/tests/cloud_tests/instances/base.py
index 9559d28..252c4c5 100644
--- a/tests/cloud_tests/instances/base.py
+++ b/tests/cloud_tests/instances/base.py
@@ -1,8 +1,5 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
-import os
-import uuid
-
 
 class Instance(object):
     """
@@ -10,26 +7,39 @@ class Instance(object):
     """
     platform_name = None
 
-    def __init__(self, name):
+    def __init__(self, platform, name, properties, config, features):
         """
-        setup
+        Set up instance
+        platform: platform object
+        name: hostname of instance
+        properties: image properties
+        config: image config
+        features: supported feature flags
         """
+        self.platform = platform
         self.name = name
+        self.properties = properties
+        self.config = config
+        self.features = features
 
-    def execute(self, command, stdin=None, stdout=None, stderr=None, env={}):
+    def execute(self, command, stdout=None, stderr=None, env={},
+                rcs=None, description=None):
         """
+        Execute command in instance, recording output, error and exit code.
+        Assumes functional networking and execution as root with the
+        target filesystem being available at /.
+
         command: the command to execute as root inside the image
-        stdin, stderr, stdout: file handles
+        stdout, stderr: file handles to write output and error to
         env: environment variables
-
-        Execute assumes functional networking and execution as root with the
-        target filesystem being available at /.
+        rcs: allowed return codes from command
+        description: purpose of command
 
         return_value: tuple containing stdout data, stderr data, exit code
         """
         raise NotImplementedError
 
-    def read_data(self, remote_path, encode=False):
+    def read_data(self, remote_path, decode=False):
         """
         read_data from instance filesystem
         remote_path: path in instance
@@ -49,6 +59,8 @@ class Instance(object):
     def pull_file(self, remote_path, local_path):
         """
         copy file at 'remote_path', from instance to 'local_path'
+        remote_path: path on remote instance
+        local_path: path on local instance
         """
         with open(local_path, 'wb') as fp:
             fp.write(self.read_data(remote_path), encode=True)
@@ -56,18 +68,34 @@ class Instance(object):
     def push_file(self, local_path, remote_path):
         """
         copy file at 'local_path' to instance at 'remote_path'
+        local_path: path on local instance
+        remote_path: path on remote instance
         """
         with open(local_path, 'rb') as fp:
             self.write_data(remote_path, fp.read())
 
-    def run_script(self, script):
+    def run_script(self, script, rcs=None, description=None):
         """
         run script in target and return stdout
+        script: script contents
+        rcs: allowed return codes from script
+        description: purpose of script
+        return_value: stdout from script
         """
-        script_path = os.path.join('/tmp', str(uuid.uuid1()))
-        self.write_data(script_path, script)
-        (out, err, exit_code) = self.execute(['/bin/bash', script_path])
-        return out
+        script_path = self.tmpfile()
+        try:
+            self.write_data(script_path, script)
+            return self.execute(
+                ['/bin/bash', script_path], rcs=rcs, description=description)
+        finally:
+            self.execute(['rm', script_path], rcs=rcs)
+
+    def tmpfile(self):
+        """
+        get a tmp file in the target
+        return_value: path to new file in target
+        """
+        return self.execute(['mktemp'])[0].strip()
 
     def console_log(self):
         """
@@ -87,7 +115,7 @@ class Instance(object):
         """
         raise NotImplementedError
 
-    def start(self, wait=True):
+    def start(self, wait=True, wait_for_cloud_init=False):
         """
         start instance
         """
@@ -99,22 +127,32 @@ class Instance(object):
         """
         pass
 
-    def _wait_for_cloud_init(self, wait_time):
+    def _wait_for_system(self, wait_for_cloud_init):
         """
         wait until system has fully booted and cloud-init has finished
+        wait_time: maximum time to wait
+        return_value: None, may raise OSError if wait_time exceeded
         """
-        if not wait_time:
-            return
 
-        found_msg = 'found'
-        cmd = ('for ((i=0;i<{wait};i++)); do [ -f "{file}" ] && '
-               '{{ echo "{msg}";break; }} || sleep 1; done').format(
-            file='/run/cloud-init/result.json',
-            wait=wait_time, msg=found_msg)
+        def clean_test(test):
+            """
+            clean formatting for system ready test testcase
+            """
+            return ' '.join(l for l in test.strip().splitlines()
+                            if not l.lstrip().startswith('#'))
+
+        time = self.config['boot_timeout']
+        tests = [self.config['system_ready_script']]
+        if wait_for_cloud_init:
+            tests.append(self.config['cloud_init_ready_script'])
+
+        formatted_tests = ' && '.join(clean_test(t) for t in tests)
+        test_cmd = ('for ((i=0;i<{time};i++)); do {test} && exit 0; sleep 1; '
+                    'done; exit 1;').format(time=time, test=formatted_tests)
+        cmd = ['/bin/bash', '-c', test_cmd]
+
+        if self.execute(cmd, rcs=(0, 1))[-1] != 0:
+            raise OSError('timeout: after {}s system not started'.format(time))
 
-        (out, err, exit) = self.execute(['/bin/bash', '-c', cmd])
-        if out.strip() != found_msg:
-            raise OSError('timeout: after {}s, cloud-init has not started'
-                          .format(wait_time))
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/lxd.py b/tests/cloud_tests/instances/lxd.py
index f0aa121..dfc8363 100644
--- a/tests/cloud_tests/instances/lxd.py
+++ b/tests/cloud_tests/instances/lxd.py
@@ -1,6 +1,7 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
 from tests.cloud_tests.instances import base
+from tests.cloud_tests import util
 
 
 class LXDInstance(base.Instance):
@@ -9,41 +10,69 @@ class LXDInstance(base.Instance):
     """
     platform_name = "lxd"
 
-    def __init__(self, name, platform, pylxd_container):
+    def __init__(self, platform, name, properties, config, features,
+                 pylxd_container):
         """
-        setup
+        Set up instance
+        platform: platform object
+        name: hostname of instance
+        properties: image properties
+        config: image config
+        features: supported feature flags
         """
-        self.platform = platform
         self._pylxd_container = pylxd_container
-        super(LXDInstance, self).__init__(name)
+        super(LXDInstance, self).__init__(
+            platform, name, properties, config, features)
 
     @property
     def pylxd_container(self):
         self._pylxd_container.sync()
         return self._pylxd_container
 
-    def execute(self, command, stdin=None, stdout=None, stderr=None, env={}):
+    def execute(self, command, stdout=None, stderr=None, env={},
+                rcs=None, description=None):
         """
+        Execute command in instance, recording output, error and exit code.
+        Assumes functional networking and execution as root with the
+        target filesystem being available at /.
+
         command: the command to execute as root inside the image
-        stdin, stderr, stdout: file handles
+        stdout, stderr: file handles to write output and error to
         env: environment variables
-
-        Execute assumes functional networking and execution as root with the
-        target filesystem being available at /.
+        rcs: allowed return codes from command
+        description: purpose of command
 
         return_value: tuple containing stdout data, stderr data, exit code
         """
-        # TODO: the pylxd api handler for container.execute needs to be
-        #       extended to properly pass in stdin
-        # TODO: the pylxd api handler for container.execute needs to be
-        #       extended to get the return code, for now just use 0
+        # ensure instance is running and execute the command
         self.start()
-        if stdin:
-            raise NotImplementedError
         res = self.pylxd_container.execute(command, environment=env)
-        for (f, data) in (i for i in zip((stdout, stderr), res) if i[0]):
-            f.write(data)
-        return res + (0,)
+
+        # get out, exit and err from pylxd return
+        if hasattr(res, 'exit_code'):
+            # pylxd 2.2 returns ContainerExecuteResult, named tuple of
+            # (exit_code, out, err)
+            (exit, out, err) = res
+        else:
+            # pylxd 2.1.3 and earlier only return out and err, no exit
+            # LOG.warning('using pylxd version < 2.2')
+            (out, err) = res
+            exit = 0
+
+        # write data to file descriptors if needed
+        if stdout:
+            stdout.write(out)
+        if stderr:
+            stderr.write(err)
+
+        # if the command exited with a code not allowed in rcs, then fail
+        if exit not in (rcs if rcs else (0,)):
+            error_desc = ('Failed command to: {}'.format(description)
+                          if description else None)
+            raise util.InTargetExecuteError(
+                out, err, exit, command, self.name, error_desc)
+
+        return (out, err, exit)
 
     def read_data(self, remote_path, decode=False):
         """
@@ -83,14 +112,14 @@ class LXDInstance(base.Instance):
         if self.pylxd_container.status != 'Stopped':
             self.pylxd_container.stop(wait=wait)
 
-    def start(self, wait=True, wait_time=None):
+    def start(self, wait=True, wait_for_cloud_init=False):
         """
         start instance
         """
         if self.pylxd_container.status != 'Running':
             self.pylxd_container.start(wait=wait)
-            if wait and isinstance(wait_time, int):
-                self._wait_for_cloud_init(wait_time)
+            if wait:
+                self._wait_for_system(wait_for_cloud_init)
 
     def freeze(self):
         """
diff --git a/tests/cloud_tests/platforms.yaml b/tests/cloud_tests/platforms.yaml
index 5972b32..b91834a 100644
--- a/tests/cloud_tests/platforms.yaml
+++ b/tests/cloud_tests/platforms.yaml
@@ -10,7 +10,55 @@ default_platform_config:
 platforms:
     lxd:
         enabled: true
-        get_image_timeout: 600
+        # overrides for image templates
+        template_overrides:
+            /var/lib/cloud/seed/nocloud-net/meta-data:
+                when:
+                    - create
+                    - copy
+                template: cloud-init-meta.tpl
+            /var/lib/cloud/seed/nocloud-net/network-config:
+                when:
+                    - create
+                    - copy
+                template: cloud-init-network.tpl
+            /var/lib/cloud/seed/nocloud-net/user-data:
+                when:
+                    - create
+                    - copy
+                template: cloud-init-user.tpl
+                properties:
+                    default: |
+                        #cloud-config
+                        {}
+            /var/lib/cloud/seed/nocloud-net/vendor-data:
+                when:
+                    - create
+                    - copy
+                template: cloud-init-vendor.tpl
+                properties:
+                    default: |
+                        #cloud-config
+                        {}
+        # overrides image template files
+        template_files:
+            cloud-init-meta.tpl: |
+                #cloud-config
+                instance-id: {{ container.name }}
+                local-hostname: {{ container.name }}
+                {{ config_get("user.meta-data", "") }}
+            cloud-init-network.tpl: |
+                {% if config_get("user.network-config", "") == "" %}version: 1
+                config:
+                    - type: physical
+                      name: eth0
+                      subnets:
+                          - type: {% if config_get("user.network_mode", "") == "link-local" %}manual{% else %}dhcp{% endif %}
+                            control: auto{% else %}{{ config_get("user.network-config", "") }}{% endif %}
+            cloud-init-user.tpl: |
+                {{ config_get("user.user-data", properties.default) }}
+            cloud-init-vendor.tpl: |
+                {{ config_get("user.vendor-data", properties.default) }}
     ec2: {}
     azure: {}
 
diff --git a/tests/cloud_tests/platforms/base.py b/tests/cloud_tests/platforms/base.py
index 615e2e0..2b6e514 100644
--- a/tests/cloud_tests/platforms/base.py
+++ b/tests/cloud_tests/platforms/base.py
@@ -15,17 +15,7 @@ class Platform(object):
 
     def get_image(self, img_conf):
         """
-        Get image using 'img_conf', where img_conf is a dict containing all
-        image configuration parameters
-
-        in this dict there must be a 'platform_ident' key containing
-        configuration for identifying each image on a per platform basis
-
-        see implementations for get_image() for details about the contents
-        of the platform's config entry
-
-        note: see 'releases' main_config.yaml for example entries
-
+        get image using specified image configuration
         img_conf: configuration for image
         return_value: cloud_tests.images instance
         """
@@ -37,17 +27,4 @@ class Platform(object):
         """
         pass
 
-    def _extract_img_platform_config(self, img_conf):
-        """
-        extract platform configuration for current platform from img_conf
-        """
-        platform_ident = img_conf.get('platform_ident')
-        if not platform_ident:
-            raise ValueError('invalid img_conf, missing \'platform_ident\'')
-        ident = platform_ident.get(self.platform_name)
-        if not ident:
-            raise ValueError('img_conf: {} missing config for platform {}'
-                             .format(img_conf, self.platform_name))
-        return ident
-
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/platforms/lxd.py b/tests/cloud_tests/platforms/lxd.py
index 847cc54..4d8b58c 100644
--- a/tests/cloud_tests/platforms/lxd.py
+++ b/tests/cloud_tests/platforms/lxd.py
@@ -27,28 +27,32 @@ class LXDPlatform(base.Platform):
 
     def get_image(self, img_conf):
         """
-        Get image
-        img_conf: dict containing config for image. platform_ident must have:
-            alias: alias to use for simplestreams server
-            sstreams_server: simplestreams server to use, or None for default
+        get image using specified image configuration
+        img_conf: configuration for image
         return_value: cloud_tests.images instance
         """
-        lxd_conf = self._extract_img_platform_config(img_conf)
-        image = self.client.images.create_from_simplestreams(
-            lxd_conf.get('sstreams_server', DEFAULT_SSTREAMS_SERVER),
-            lxd_conf['alias'])
-        return lxd_image.LXDImage(
-            image.properties['description'], img_conf, self, image)
+        pylxd_image = self.client.images.create_from_simplestreams(
+            img_conf.get('sstreams_server', DEFAULT_SSTREAMS_SERVER),
+            img_conf['alias'])
+        image = lxd_image.LXDImage(self, img_conf, pylxd_image)
+        if img_conf.get('override_templates', False):
+            image.update_templates(self.config.get('template_overrides', {}),
+                                   self.config.get('template_files', {}))
+        return image
 
-    def launch_container(self, image=None, container=None, ephemeral=False,
-                         config=None, block=True,
-                         image_desc=None, use_desc=None):
+    def launch_container(self, properties, config, features,
+                         image=None, container=None, ephemeral=False,
+                         container_config=None, block=True, image_desc=None,
+                         use_desc=None):
         """
         launch a container
+        properties: image properties
+        config: image configuration
+        features: image features
         image: image fingerprint to launch from
         container: container to copy
         ephemeral: delete image after first shutdown
-        config: config options for instance as dict
+        container_config: config options for instance as dict
         block: wait until container created
         image_desc: description of image being launched
         use_desc: description of container's use
@@ -61,11 +65,13 @@ class LXDPlatform(base.Platform):
                                            use_desc=use_desc,
                                            used_list=self.list_containers()),
             'ephemeral': bool(ephemeral),
-            'config': config if isinstance(config, dict) else {},
+            'config': (container_config
+                       if isinstance(container_config, dict) else {}),
             'source': ({'type': 'image', 'fingerprint': image} if image else
                        {'type': 'copy', 'source': container})
         }, wait=block)
-        return lxd_instance.LXDInstance(container.name, self, container)
+        return lxd_instance.LXDInstance(self, container.name, properties,
+                                        config, features, container)
 
     def container_exists(self, container_name):
         """
@@ -88,6 +94,14 @@ class LXDPlatform(base.Platform):
         """
         return [container.name for container in self.client.containers.all()]
 
+    def query_image_by_alias(self, alias):
+        """
+        get image by alias in local image store
+        alias: alias of image
+        return_value: pylxd image (not cloud_tests.images instance)
+        """
+        return self.client.images.get_by_alias(alias)
+
     def destroy(self):
         """
         Clean up platform data
diff --git a/tests/cloud_tests/releases.yaml b/tests/cloud_tests/releases.yaml
index 3ffa68f..4b5d0c3 100644
--- a/tests/cloud_tests/releases.yaml
+++ b/tests/cloud_tests/releases.yaml
@@ -1,79 +1,281 @@
 # ============================= Release Config ================================
 default_release_config:
-    # all are disabled by default
-    enabled: false
-    # timeout for booting image and running cloud init
-    timeout: 120
-    # platform_ident values for the image, with data to identify the image
-    # on that platform. see platforms.base for more information
-    platform_ident: {}
-    # a script to run after a boot that is used to modify an image, before
-    # making a snapshot of the image. may be useful for removing data left
-    # behind from cloud-init booting, such as logs, to ensure that data from
-    # snapshot.launch() will not include a cloud-init.log from a boot used to
-    # create the snapshot, if cloud-init has not run
-    boot_clean_script: |
-        #!/bin/bash
-        rm -rf /var/log/cloud-init.log /var/log/cloud-init-output.log \
-            /var/lib/cloud/ /run/cloud-init/ /var/log/syslog
+    # global default configuration options
+    default:
+        # all are disabled by default
+        enabled: false
+        # timeout for booting image and running cloud init
+        boot_timeout: 120
+        # a script to run after a boot that is used to modify an image, before
+        # making a snapshot of the image. may be useful for removing data left
+        # behind from cloud-init booting, such as logs, to ensure that data
+        # from snapshot.launch() will not include a cloud-init.log from a boot
+        # used to create the snapshot, if cloud-init has not run
+        boot_clean_script: |
+            #!/bin/bash
+            rm -rf /var/log/cloud-init.log /var/log/cloud-init-output.log \
+                /var/lib/cloud/ /run/cloud-init/ /var/log/syslog
+        # test script to determine if system is booted fully
+        system_ready_script: |
+            # permit running or degraded state as both indicate complete boot
+            [ $(systemctl is-system-running) = 'running' -o
+              $(systemctl is-system-running) = 'degraded' ]
+        # test script to determine if cloud-init has finished
+        cloud_init_ready_script: |
+            [ -f '/run/cloud-init/result.json' ]
+        # currently used features and their uses are:
+        # features groups and additional feature settings
+        feature_groups: []
+        features: {}
+
+    # lxd specific default configuration options
+    lxd:
+        # default sstreams server to use for lxd image retrieval
+        sstreams_server: https://us.images.linuxcontainers.org:8443
+        # keep base image, avoids downloading again next run
+        cache_base_image: true
+        # lxd images from linuxcontainers.org do not have the nocloud seed
+        # templates in place, so the image metadata must be modified
+        override_templates: true
+        # arg overrides to set image up
+        setup_overrides:
+            # lxd images from linuxcontainers.org do not come with
+            # cloud-init, so must pull cloud-init in from repo using
+            # setup_image.upgrade
+            upgrade: true
+
+features:
+    # all currently supported feature flags
+    all:
+        - apt       # image supports apt package manager
+        - byobu     # byobu is available in repositories
+        - landscape # landscape-client available in repos
+        - lxd       # lxd is available in the image
+        - ppa       # image supports ppas
+        - rpm       # image supports rpms
+        - snap      # supports snapd
+        # NOTE: the following feature flags are to work around bugs in the
+        #       images, and can be removed when no longer needed
+        - hostname      # setting system hostname works
+        # NOTE: the following feature flags are to work around issues in the
+        #       testcases, and can be removed when no longer needed
+        - apt_src_cont  # default contents and format of sources.list matches
+                        # ubuntu sources.list
+        - apt_hist_fmt  # apt command history entries use full paths to apt
+                        # executable rather than relative paths
+        - daylight_time # timezones are daylight not standard time
+        - apt_up_out    # 'Calculating upgrade..' present in log output from
+                        # apt-get dist-upgrade output
+        - engb_locale   # locale en_GB.UTF-8 is available
+        - locale_gen    # the /etc/locale.gen file exists
+        - no_ntpdate    # 'ntpdate' is not installed by default
+        - no_file_fmt_e # the 'file' utility does not have a formatting error
+        - ppa_file_name # the name of the source file added to sources.list.d has
+                        # the expected format for newer ubuntu releases
+        - sshd          # requires ssh server to be installed by default
+        - ssh_key_fmt   # ssh auth keys printed to console have expected format
+        - syslog        # test case requires syslog to be written by default
+        - ubuntu_ntp    # expect ubuntu.pool.ntp.org to be used as ntp server
+        - ubuntu_repos  # test case requres ubuntu repositories to be used
+        - ubuntu_user   # test case needs user with the name 'ubuntu' to exist
+        # NOTE: the following feature flags are to work around issues that may
+        #       be considered bugs in cloud-init
+        - lsb_release   # image has lsb_release installed, maybe should install
+                        # if missing by default
+        - sudo          # image has sudo installed, should not be required
+    # feature flag groups
+    groups:
+        base:
+            hostname: true
+            no_file_fmt_e: true
+        ubuntu_specific:
+            apt_src_cont: true
+            apt_hist_fmt: true
+            byobu: true
+            daylight_time: true
+            engb_locale: true
+            landscape: true
+            locale_gen: true
+            lsb_release: true
+            lxd: true
+            ppa: true
+            ppa_file_name: true
+            snap: true
+            sshd: true
+            ssh_key_fmt: true
+            sudo: true
+            syslog: true
+            ubuntu_ntp: true
+            ubuntu_repos: true
+            ubuntu_user: true
+        debian_base:
+            apt: true
+            apt_up_out: true
+            no_ntpdate: true
+        rhel_base:
+            rpm: true
 
 releases:
-    trusty:
-        enabled: true
-        platform_ident:
-            lxd:
-                # if sstreams_server is omitted, default is used, defined in
-                # tests.cloud_tests.platforms.lxd.DEFAULT_SSTREAMS_SERVER as:
-                # sstreams_server: https://us.images.linuxcontainers.org:8443
-                #alias: ubuntu/trusty/default
-                alias: t
-                sstreams_server: https://cloud-images.ubuntu.com/daily
-    xenial:
-        enabled: true
-        platform_ident:
-            lxd:
-                #alias: ubuntu/xenial/default
-                alias: x
-                sstreams_server: https://cloud-images.ubuntu.com/daily
-    yakkety:
-        enabled: true
-        platform_ident:
-            lxd:
-                #alias: ubuntu/yakkety/default
-                alias: y
-                sstreams_server: https://cloud-images.ubuntu.com/daily
+    # UBUNTU =================================================================
     zesty:
-        enabled: true
-        platform_ident:
-            lxd:
-                #alias: ubuntu/zesty/default
-                alias: z
-                sstreams_server: https://cloud-images.ubuntu.com/daily
-    jessie:
-        platform_ident:
-            lxd:
-                alias: debian/jessie/default
+        # EOL: Jan 2018
+        default:
+            enabled: true
+            feature_groups:
+                - base
+                - debian_base
+                - ubuntu_specific
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: zesty
+            setup_overrides: null
+            override_templates: false
+    yakkety:
+        # EOL: Jul 2017
+        default:
+            enabled: true
+            feature_groups:
+                - base
+                - debian_base
+                - ubuntu_specific
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: yakkety
+            setup_overrides: null
+            override_templates: false
+    xenial:
+        # EOL: Apr 2021
+        default:
+            enabled: true
+            feature_groups:
+                - base
+                - debian_base
+                - ubuntu_specific
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: xenial
+            setup_overrides: null
+            override_templates: false
+    trusty:
+        # EOL: Apr 2019
+        default:
+            enabled: true
+            feature_groups:
+                - base
+                - debian_base
+                - ubuntu_specific
+            features:
+                apt_up_out: false
+                locale_gen: false
+                lxd: false
+                ppa_file_name: false
+                snap: false
+                ssh_key_fmt: false
+                no_ntpdate: false
+                no_file_fmt_e: false
+            system_ready_script: |
+                #!/bin/bash
+                # upstart based, so use old style runlevels
+                [ $(runlevel | awk '{print $2}') = '2' ]
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: trusty
+            setup_overrides: null
+            override_templates: false
+    precise:
+        # EOL: Apr 2017
+        default:
+            # still supported but not relevant for development, not enabled
+            # tests should still work though unless they use newer features
+            enabled: false
+            feature_groups:
+                - base
+                - debian_base
+                - ubuntu_specific
+            features:
+                lxd: false
+            system_ready_script: |
+                #!/bin/bash
+                # upstart based, so use old style runlevels
+                [ $(runlevel | awk '{print $2}') = '2' ]
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: precise
+            setup_overrides: null
+            override_templates: false
+    # DEBIAN =================================================================
     sid:
-        platform_ident:
-            lxd:
-                alias: debian/sid/default
+        # EOL: N/A
+        default:
+            # tests should work on sid, however it is not always stable
+            enabled: false
+            feature_groups:
+                - base
+                - debian_base
+        lxd:
+            alias: debian/sid/default
     stretch:
-        platform_ident:
-            lxd:
-                alias: debian/stretch/default
+        # EOL: Not yet released
+        default:
+            enabled: true
+            feature_groups:
+                - base
+                - debian_base
+        lxd:
+            alias: debian/stretch/default
+    jessie:
+        # EOL: Jun 2020
+        # NOTE: the cloud-init version shipped with jessie is out of date
+        #       tests work if an up to date deb is used
+        default:
+            enabled: true
+            feature_groups:
+                - base
+                - debian_base
+        lxd:
+            alias: debian/jessie/default
     wheezy:
-        platform_ident:
-            lxd:
-                alias: debian/wheezy/default
+        # EOL: May 2018 (Apr 2016 - end of full updates)
+        default:
+            # this is old enough that it is no longer relevant for development
+            enabled: false
+            feature_groups:
+                - base
+                - debian_base
+        lxd:
+            alias: debian/wheezy/default
+    # CENTOS =================================================================
     centos70:
-        timeout: 180
-        platform_ident:
-            lxd:
-                alias: centos/7/default
+        # EOL: Jun 2024 (2020 - end of full updates)
+        default:
+            enabled: true
+            feature_groups:
+                - base
+                - rhel_base
+            user_data_overrides:
+                preserve_hostname: true
+        lxd:
+            features:
+                # NOTE: (LP: #1575779)
+                hostname: false
+            alias: centos/7/default
     centos66:
-        timeout: 180
-        platform_ident:
-            lxd:
-                alias: centos/6/default
+        # EOL: Nov 2020
+        default:
+            enabled: true
+            feature_groups:
+                - base
+                - rhel_base
+            # still supported, but only bugfixes after may 2017
+            system_ready_script: |
+                #!/bin/bash
+                [ $(runlevel | awk '{print $2}') = '3' ]
+            user_data_overrides:
+                preserve_hostname: true
+        lxd:
+            features:
+                # NOTE: (LP: #1575779)
+                hostname: false
+            alias: centos/6/default
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/setup_image.py b/tests/cloud_tests/setup_image.py
index 5d6c638..1b74ceb 100644
--- a/tests/cloud_tests/setup_image.py
+++ b/tests/cloud_tests/setup_image.py
@@ -7,6 +7,30 @@ from functools import partial
 import os
 
 
+def installed_version(image, package, ensure_installed=True):
+    """
+    get installed version of package
+    image: cloud_tests.images instance to operate on
+    package: name of package
+    ensure_installed: raise error if not installed
+    return_value: cloud-init version string
+    """
+    # get right cmd for os family
+    os_family = util.get_os_family(image.properties['os'])
+    if os_family == 'debian':
+        cmd = ['dpkg-query', '-W', "--showformat='${Version}'", package]
+    elif os_family == 'redhat':
+        cmd = ['rpm', '-q', '--queryformat', "'%{VERSION}'", package]
+    else:
+        raise NotImplementedError
+
+    # query version
+    msg = 'query version for package: {}'.format(package)
+    (out, err, exit) = image.execute(
+        cmd, description=msg, rcs=(0,) if ensure_installed else range(0, 256))
+    return out.strip()
+
+
 def install_deb(args, image):
     """
     install deb into image
@@ -21,20 +45,18 @@ def install_deb(args, image):
                                   'family: {}'.format(args.deb, os_family))
 
     # install deb
-    LOG.debug('installing deb: %s into target', args.deb)
+    msg = 'install deb: "{}" into target'.format(args.deb)
+    LOG.debug(msg)
     remote_path = os.path.join('/tmp', os.path.basename(args.deb))
     image.push_file(args.deb, remote_path)
-    (out, err, exit) = image.execute(['dpkg', '-i', remote_path])
-    if exit != 0:
-        raise OSError('failed install deb: {}\n\tstdout: {}\n\tstderr: {}'
-                      .format(args.deb, out, err))
+    cmd = 'dpkg -i {} || apt-get install --yes -f'.format(remote_path)
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
 
     # check installed deb version matches package
     fmt = ['-W', "--showformat='${Version}'"]
     (out, err, exit) = image.execute(['dpkg-deb'] + fmt + [remote_path])
     expected_version = out.strip()
-    (out, err, exit) = image.execute(['dpkg-query'] + fmt + ['cloud-init'])
-    found_version = out.strip()
+    found_version = installed_version(image, 'cloud-init')
     if expected_version != found_version:
         raise OSError('install deb version "{}" does not match expected "{}"'
                       .format(found_version, expected_version))
@@ -52,24 +74,21 @@ def install_rpm(args, image):
     """
     # ensure system is compatible with package format
     os_family = util.get_os_family(image.properties['os'])
-    if os_family not in ['redhat', 'sles']:
+    if os_family != 'redhat':
         raise NotImplementedError('install rpm: {} not supported on os '
                                   'family: {}'.format(args.rpm, os_family))
 
     # install rpm
-    LOG.debug('installing rpm: %s into target', args.rpm)
+    msg = 'install rpm: "{}" into target'.format(args.rpm)
+    LOG.debug(msg)
     remote_path = os.path.join('/tmp', os.path.basename(args.rpm))
     image.push_file(args.rpm, remote_path)
-    (out, err, exit) = image.execute(['rpm', '-U', remote_path])
-    if exit != 0:
-        raise OSError('failed to install rpm: {}\n\tstdout: {}\n\tstderr: {}'
-                      .format(args.rpm, out, err))
+    image.execute(['rpm', '-U', remote_path], description=msg)
 
     fmt = ['--queryformat', '"%{VERSION}"']
     (out, err, exit) = image.execute(['rpm', '-q'] + fmt + [remote_path])
     expected_version = out.strip()
-    (out, err, exit) = image.execute(['rpm', '-q'] + fmt + ['cloud-init'])
-    found_version = out.strip()
+    found_version = installed_version(image, 'cloud-init')
     if expected_version != found_version:
         raise OSError('install rpm version "{}" does not match expected "{}"'
                       .format(found_version, expected_version))
@@ -80,13 +99,34 @@ def install_rpm(args, image):
 
 def upgrade(args, image):
     """
-    run the system's upgrade command
+    upgrade or install cloud-init from repo
+    args: cmdline arguments
+    image: cloud_tests.images instance to operate on
+    return_value: None, may raise errors
+    """
+    # determine command for os_family
+    os_family = util.get_os_family(image.properties['os'])
+    if os_family == 'debian':
+        cmd = 'apt-get update && apt-get install cloud-init --yes'
+    elif os_family == 'redhat':
+        cmd = 'yum install cloud-init --assumeyes'
+    else:
+        raise NotImplementedError
+
+    # upgrade cloud-init
+    msg = 'upgrading cloud-init'
+    LOG.debug(msg)
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
+
+
+def upgrade_full(args, image):
+    """
+    run the system's full upgrade command
     args: cmdline arguments
     image: cloud_tests.images instance to operate on
     return_value: None, may raise errors
     """
     # determine appropriate upgrade command for os_family
-    # TODO: maybe use cloudinit.distros for this?
     os_family = util.get_os_family(image.properties['os'])
     if os_family == 'debian':
         cmd = 'apt-get update && apt-get upgrade --yes'
@@ -97,11 +137,9 @@ def upgrade(args, image):
                                   'from family: {}'.format(os_family))
 
     # upgrade system
-    LOG.debug('upgrading system')
-    (out, err, exit) = image.execute(['/bin/sh', '-c', cmd])
-    if exit != 0:
-        raise OSError('failed to upgrade system\n\tstdout: {}\n\tstderr:{}'
-                      .format(out, err))
+    msg = 'full system upgrade'
+    LOG.debug(msg)
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
 
 
 def run_script(args, image):
@@ -111,9 +149,9 @@ def run_script(args, image):
     image: cloud_tests.images instance to operate on
     return_value: None, may raise errors
     """
-    # TODO: get exit status back from script and add error handling here
-    LOG.debug('running setup image script in target image')
-    image.run_script(args.script)
+    msg = 'run setup image script in target image'
+    LOG.debug(msg)
+    image.run_script(args.script, description=msg)
 
 
 def enable_ppa(args, image):
@@ -124,17 +162,15 @@ def enable_ppa(args, image):
     return_value: None, may raise errors
     """
     # ppa only supported on ubuntu (maybe debian?)
-    if image.properties['os'] != 'ubuntu':
+    if image.properties['os'].lower() != 'ubuntu':
         raise NotImplementedError('enabling a ppa is only available on ubuntu')
 
     # add ppa with add-apt-repository and update
     ppa = 'ppa:{}'.format(args.ppa)
-    LOG.debug('enabling %s', ppa)
+    msg = 'enable ppa: "{}" in target'.format(ppa)
+    LOG.debug(msg)
     cmd = 'add-apt-repository --yes {} && apt-get update'.format(ppa)
-    (out, err, exit) = image.execute(['/bin/sh', '-c', cmd])
-    if exit != 0:
-        raise OSError('enable ppa for {} failed\n\tstdout: {}\n\tstderr: {}'
-                      .format(ppa, out, err))
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
 
 
 def enable_repo(args, image):
@@ -155,11 +191,9 @@ def enable_repo(args, image):
         raise NotImplementedError('enable repo command not configured for '
                                   'distro from family: {}'.format(os_family))
 
-    LOG.debug('enabling repo: "%s"', args.repo)
-    (out, err, exit) = image.execute(['/bin/sh', '-c', cmd])
-    if exit != 0:
-        raise OSError('enable repo {} failed\n\tstdout: {}\n\tstderr: {}'
-                      .format(args.repo, out, err))
+    msg = 'enable repo: "{}" in target'.format(args.repo)
+    LOG.debug(msg)
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
 
 
 def setup_image(args, image):
@@ -169,6 +203,11 @@ def setup_image(args, image):
     image: cloud_tests.image instance to operate on
     return_value: tuple of results and fail count
     """
+    # update the args if necessary for this image
+    overrides = image.setup_overrides
+    LOG.debug('updating args for setup with: %s', overrides)
+    args = util.update_args(args, overrides, preserve_old=True)
+
     # mapping of setup cmdline arg name to setup function
     # represented as a tuple rather than a dict or odict as lookup by name not
     # needed, and order is important as --script and --upgrade go at the end
@@ -179,17 +218,19 @@ def setup_image(args, image):
         ('repo', enable_repo, 'setup func for --repo, enable repo'),
         ('ppa', enable_ppa, 'setup func for --ppa, enable ppa'),
         ('script', run_script, 'setup func for --script, run script'),
-        ('upgrade', upgrade, 'setup func for --upgrade, upgrade pkgs'),
+        ('upgrade', upgrade, 'setup func for --upgrade, upgrade cloud-init'),
+        ('upgrade-full', upgrade_full, 'setup func for --upgrade-full'),
     )
 
     # determine which setup functions needed
     calls = [partial(stage.run_single, desc, partial(func, args, image))
              for name, func, desc in handlers if getattr(args, name, None)]
 
-    image_name = 'image: distro={}, release={}'.format(
-        image.properties['os'], image.properties['release'])
-    LOG.info('setting up %s', image_name)
-    return stage.run_stage('set up for {}'.format(image_name), calls,
-                           continue_after_error=False)
+    LOG.info('setting up %s', image)
+    res = stage.run_stage(
+        'set up for {}'.format(image), calls, continue_after_error=False)
+    LOG.debug('after setup complete, installed cloud-init version is: %s',
+              installed_version(image, 'cloud-init'))
+    return res
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/snapshots/base.py b/tests/cloud_tests/snapshots/base.py
index d715f03..cbe3f5f 100644
--- a/tests/cloud_tests/snapshots/base.py
+++ b/tests/cloud_tests/snapshots/base.py
@@ -7,12 +7,18 @@ class Snapshot(object):
     """
     platform_name = None
 
-    def __init__(self, properties, config):
+    def __init__(self, platform, properties, config, features):
         """
         Set up snapshot
+        platform: platform object
+        properties: image properties
+        config: image config
+        features: supported feature flags
         """
+        self.platform = platform
         self.properties = properties
         self.config = config
+        self.features = features
 
     def __str__(self):
         """
diff --git a/tests/cloud_tests/snapshots/lxd.py b/tests/cloud_tests/snapshots/lxd.py
index eabbce3..2241035 100644
--- a/tests/cloud_tests/snapshots/lxd.py
+++ b/tests/cloud_tests/snapshots/lxd.py
@@ -9,13 +9,18 @@ class LXDSnapshot(base.Snapshot):
     """
     platform_name = "lxd"
 
-    def __init__(self, properties, config, platform, pylxd_frozen_instance):
+    def __init__(self, platform, properties, config, features,
+                 pylxd_frozen_instance):
         """
         Set up snapshot
+        platform: platform object
+        properties: image properties
+        config: image config
+        features: supported feature flags
         """
-        self.platform = platform
         self.pylxd_frozen_instance = pylxd_frozen_instance
-        super(LXDSnapshot, self).__init__(properties, config)
+        super(LXDSnapshot, self).__init__(
+            platform, properties, config, features)
 
     def launch(self, user_data, meta_data=None, block=True, start=True,
                use_desc=None):
@@ -34,10 +39,11 @@ class LXDSnapshot(base.Snapshot):
         if meta_data:
             inst_config['user.meta-data'] = meta_data
         instance = self.platform.launch_container(
-            container=self.pylxd_frozen_instance.name, config=inst_config,
-            block=block, image_desc=str(self), use_desc=use_desc)
+            self.properties, self.config, self.features, block=block,
+            image_desc=str(self), container=self.pylxd_frozen_instance.name,
+            use_desc=use_desc, container_config=inst_config)
         if start:
-            instance.start(wait=True, wait_time=self.config.get('timeout'))
+            instance.start()
         return instance
 
     def destroy(self):
diff --git a/tests/cloud_tests/testcases.yaml b/tests/cloud_tests/testcases.yaml
index c22b08e..7183e01 100644
--- a/tests/cloud_tests/testcases.yaml
+++ b/tests/cloud_tests/testcases.yaml
@@ -2,6 +2,7 @@
 base_test_data:
     script_timeout: 20
     enabled: True
+    required_features: []
     cloud_config: |
         #cloud-config
     collect_scripts:
diff --git a/tests/cloud_tests/util.py b/tests/cloud_tests/util.py
index 64a8667..52a702e 100644
--- a/tests/cloud_tests/util.py
+++ b/tests/cloud_tests/util.py
@@ -1,5 +1,6 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
+import copy
 import glob
 import os
 import random
@@ -7,10 +8,18 @@ import string
 import tempfile
 import yaml
 
-from cloudinit.distros import OSFAMILIES
 from cloudinit import util as c_util
 from tests.cloud_tests import LOG
 
+OS_FAMILY_MAPPING = {
+    'debian': ['debian', 'ubuntu'],
+    'redhat': ['centos', 'rhel', 'fedora'],
+    'gentoo': ['gentoo'],
+    'freebsd': ['freebsd'],
+    'suse': ['sles'],
+    'arch': ['arch'],
+}
+
 
 def list_test_data(data_dir):
     """
@@ -68,7 +77,7 @@ def gen_instance_name(prefix='cloud-test', image_desc=None, use_desc=None,
         """
         filter bad characters out of elem and trim to length
         """
-        elem = elem[:max_len] if elem else unknown
+        elem = elem.lower()[:max_len] if elem else unknown
         return ''.join(c if c in valid else delim for c in elem)
 
     return next(name for name in
@@ -88,7 +97,8 @@ def get_os_family(os_name):
     """
     get os family type for os_name
     """
-    return next((k for k, v in OSFAMILIES.items() if os_name in v), None)
+    return next((k for k, v in OS_FAMILY_MAPPING.items()
+                 if os_name.lower() in v), None)
 
 
 def current_verbosity():
@@ -127,12 +137,17 @@ def configure_yaml():
         'tag:yaml.org,2002:str', data, style='|' if '\n' in data else '')))
 
 
-def yaml_format(data):
+def yaml_format(data, content_type=None):
     """
     format data as yaml
+    data: data to dump
+    header: is specified, add a header to the dumped data
+    return_value: yaml string
     """
     configure_yaml()
-    return yaml.dump(data, indent=2, default_flow_style=False)
+    content_type = (
+        '#{}\n'.format(content_type.strip('#\n')) if content_type else '')
+    return content_type + yaml.dump(data, indent=2, default_flow_style=False)
 
 
 def yaml_dump(data, path):
@@ -158,6 +173,108 @@ def write_file(*args, **kwargs):
     """
     write a file using cloudinit.util.write_file
     """
-    c_util.write_file(*args, **kwargs)
+    return c_util.write_file(*args, **kwargs)
+
+
+def read_conf(*args, **kwargs):
+    """
+    read configuration using cloudinit.util.read_conf
+    """
+    return c_util.read_conf(*args, **kwargs)
+
+
+def subp(*args, **kwargs):
+    """
+    execute a command on the system shell using cloudinit.util.subp
+    """
+    return c_util.subp(*args, **kwargs)
+
+
+def tmpdir(prefix='cloud_test_util_'):
+    return tempfile.mkdtemp(prefix=prefix)
+
+
+def rel_files(basedir):
+    """
+    list of files under directory by relative path, not including directories
+    return_value: list or relative paths
+    """
+    basedir = os.path.normpath(basedir)
+    return [path[len(basedir) + 1:] for path in
+            glob.glob(os.path.join(basedir, '**'), recursive=True)
+            if not os.path.isdir(path)]
+
+
+def flat_tar(output, basedir, owner='root', group='root'):
+    """
+    create a flat tar archive (no leading ./) from basedir
+    output: output tar file to write
+    basedir: base directory for archive
+    owner: owner of archive files
+    group: group archive files belong to
+    return_value: none
+    """
+    c_util.subp(['tar', 'cf', output, '--owner', owner, '--group', group,
+                 '-C', basedir] + rel_files(basedir), capture=True)
+
+
+def parse_conf_list(entries, valid=None, boolean=False):
+    """
+    parse config in a list of strings in key=value format
+    entries: list of key=value strings
+    valid: list of valid keys in result, return None if invalid input
+    boolean: if true, then interpret all values as booleans where 'true' = True
+    return_value: dict of configuration or None if invalid
+    """
+    res = {key: value.lower() == 'true' if boolean else value
+           for key, value in (i.split('=') for i in entries)}
+    return res if not valid or all(k in valid for k in res.keys()) else None
+
+
+def update_args(args, updates, preserve_old=True):
+    """
+    update cmdline arguments from a dictionary
+    args: cmdline arguments
+    updates: dictionary of {arg_name: new_value} mappings
+    preserve_old: if true, create a deep copy of args before updating
+    return_value: updated cmdline arguments, as new object if preserve_old=True
+    """
+    args = copy.deepcopy(args) if preserve_old else args
+    if updates:
+        vars(args).update(updates)
+    return args
+
+
+def update_user_data(user_data, updates, dump_to_yaml=True):
+    """
+    user_data: user data as yaml string or dict
+    updates: dictionary to merge with user data
+    dump_to_yaml: return as yaml dumped string if true
+    return_value: updated user data, as yaml string if dump_to_yaml is true
+    """
+    user_data = (c_util.load_yaml(user_data)
+                 if isinstance(user_data, str) else copy.deepcopy(user_data))
+    user_data.update(updates)
+    return (yaml_format(user_data, content_type='cloud-config')
+            if dump_to_yaml else user_data)
+
+
+class InTargetExecuteError(c_util.ProcessExecutionError):
+    """
+    Error type for in target commands that fail
+    """
+    default_desc = 'Unexpected error while running command in target instance'
+
+    def __init__(self, stdout, stderr, exit_code, cmd, instance,
+                 description=None):
+        """
+        init error and parent error class
+        """
+        if isinstance(cmd, (tuple, list)):
+            cmd = ' '.join(cmd)
+        super(InTargetExecuteError, self).__init__(
+            stdout=stdout, stderr=stderr, exit_code=exit_code, cmd=cmd,
+            reason="Instance: {}".format(instance),
+            description=description if description else self.default_desc)
 
 # vi: ts=4 expandtab
diff --git a/tox.ini b/tox.ini
index bf9046a..5cf8d22 100644
--- a/tox.ini
+++ b/tox.ini
@@ -101,4 +101,4 @@ basepython = python3
 commands = {envpython} -m tests.cloud_tests {posargs}
 passenv = HOME
 deps =
-    pylxd==2.1.3
+    pylxd==2.2.3