Merge ~powersj/cloud-init:cii-kvm into cloud-init:master

[Joshua Powers]

tests: Enable the NoCloud KVM platform

The NoCloud KVM platform includes:

  * Downloads daily Ubuntu images using streams and store in
    /srv/images
  * Image customization, if required, is done using
    mount-image-callback otherwise image is untouched
  * Launches KVM via the xkvm script, a wrapper around
    qemu-system, and sets custom port for SSH
  * Generation and inject an SSH (RSA 4096) key pair to use for
    communication with the guest to collect test artifacts
  * Add method to produce safe shell strings by base64 encoding
    the command

Additional Changes:

  * Set default backend to use LXD
  * Verify not running script as root in order to prevent images
    from becoming owned by root
  * Removed extra quotes around that were added when collecting
    the cloud-init version from the image
  * Added info about each release as previously the lxd backend
    was able to query that information from pylxd image info,
    however, other backends will not be able to obtain the same
    information as easily

Testing completed:

# Smoke Test: single test on both platforms
python3 -m tests.cloud_tests run --verbose --platform nocloud-kvm --os-name xenial -t modules/locale
python3 -m tests.cloud_tests run --verbose --platform lxd --os-name xenial -t modules/locale

# Specify cloud-init deb to install
python3 -m tests.cloud_tests run --verbose --platform nocloud-kvm --os-name xenial -t modules/ntp --deb *.deb
python3 -m tests.cloud_tests run --verbose --platform lxd --os-name xenial -t modules/ntp --deb *.deb

# Specify repo to pull cloud-init from
python3 -m tests.cloud_tests run --verbose --platform nocloud-kvm --os-name artful -t modules/locale --repo "deb http://us.archive.ubuntu.com/ubuntu/ artful main restricted"
python3 -m tests.cloud_tests run --verbose --platform lxd --os-name artful -t modules/locale --repo "deb http://us.archive.ubuntu.com/ubuntu/ artful main restricted"

# Specify PPA to install from
python3 -m tests.cloud_tests run --verbose --platform nocloud-kvm --os-name xenial -t modules/locale --ppa ppa:cloud-init-dev/daily
python3 -m tests.cloud_tests run --verbose --platform lxd --os-name xenial -t modules/locale --ppa ppa:cloud-init-dev/daily

# Run nightly integration test on LTS and devel releases
python3 -m tests.cloud_tests run --verbose --platform lxd --os-name xenial --os-name artful

# Run KVM on Torkoal QA Slave
python3 -m tests.cloud_tests run --verbose --platform nocloud-kvm --os-name xenial -t modules/locale

# Build local tree and inject deb
# These are blocked right now due to build issues
python3 -m tests.cloud_tests tree_run --verbose --platform nocloud-kvm --os-name zesty -t modules/bootcmd
python3 -m tests.cloud_tests tree_run --verbose --platform lxd --os-name zesty -t modules/bootcmd

Note, not all existing tests work with the nocloud-kvm. This is because lxd runs with the root user, whereas nocloud-kvm runs as a normal 'ubuntu' user. Certain assumptions were made with lxd that do not apply to nocloud-kvm. As such updated test cases will come in a later merge.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:273226e027ba483c3ae26787a2975348ef6b07c5
https://jenkins.ubuntu.com/server/job/cloud-init-ci/52/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/52/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:8625408dd166a837f11381c5aea683ba14898e89
https://jenkins.ubuntu.com/server/job/cloud-init-ci/53/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/53/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:31be94b892c57d57d829b0e2c7de5bf333b53d1b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/54/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/54/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

Looks pretty solid. Just a few in-line comments/fixes

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:42536229033d6580bb8cdcaf1cdbcd8c4afeaf16
https://jenkins.ubuntu.com/server/job/cloud-init-ci/156/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/156/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Looks good, initial comments and running through tests now.

[Chad Smith (chad.smith)]

Additionally, since you are importing paramiko, we probably need a dev dependency added to requirements-test.txt

[Joshua Powers (powersj)]

I have been leaving integration test requirements out of requirements-test.txt. For example, pylxd is not in there either. I consider that file what is required for unit tests only. Thoughts?

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:661af70598fa19c1c5cbb84880fa430b697c6d79
https://jenkins.ubuntu.com/server/job/cloud-init-ci/165/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/165/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:80dd771ae2e9f760de9d72924e0c33381e378471
https://jenkins.ubuntu.com/server/job/cloud-init-ci/166/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/166/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

There are quite a few with open() as fh: fh.read() chunks; if you're using cloudinit.util then you can just use util.load_file()

same for writing (util.write_file())

[Chad Smith (chad.smith)]

Thanks for working this josh. Here are a few more more comments. I'm still seeing spurious issues on my laptop trying to run this (I just hit an OOM as well) and NoValidConnectionsError from paramiko. But I have also seen successful runs. I'm +1 on this iteration with various fixes/comments inline. We can iterate and improve as we have cycles.

[Joshua Powers (powersj)]

Thanks for the reviews!

@chad.smith - It would be helpful to get your command line that you used where things failed or crashed.

For the record there are two design decisions that I want to clarify:

1) LXD still uses pylxd. I would like to change this over to mount-image-callback at some point, but not in this merge. As a result, there are a number of places that take advantage of pylxd operations giving a return code. This is passed around in numerous places. Using subprocess that is in cloud-init already does not have this feature, so I have to fake out the return code in a few places. With the move to MIC I will remove this entirely.

2) I prefer to pass string commands around in functions versus having arrays of strings. It is far easier to develop when modifying a string than an array of strings. It also makes far more sense to have a command as a string, as an array is not a command. The function doing the processing can run shlex.

[Scott Moser (smoser)]

"It also makes far more sense to have a command as a string, as an array is not a command."
That is generally false.

A shell tokenizes a string based on some very complicated rules and turns it into an array which it then passes to execve(2).

If you ever have to pass a command that has a ", ', &, *, , # in it, then you'll end up wishing you could pass an array due to fighting the shell's interpretation.

we should do a better job in cloud-init in subp or this path of printing the command so you can cut and paste it than we do (not just printing the array), but that is fairly easily done.

i'll takea poke at fixing the shell stuff to proper arrays.

[Scott Moser (smoser)]

fyi, regarding mount-image-callback:
 https://bugs.launchpad.net/cloud-init/+bug/1715994
:-(

[Chad Smith (chad.smith)]

+1 on changesets thanks Josh.

review: Approve

[Scott Moser (smoser)]

To satisfy your desire to pass strings (which i agree are shorter)
we can make a requirement for (instance|image).execute to take a string.

that does make simple usage simpler, i agree.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:6abf7c3fcce720fc9343a838d60b10fe6af1b250
https://jenkins.ubuntu.com/server/job/cloud-init-ci/274/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/274/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:ee71778d942f4f55da5407fb0d8e0a9649a3305b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/278/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/278/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

I'm not opposed to the string changes, but it'd be nice if you did them in a separate mp.
see my suggestion to support strings in execute at
 https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/330459

then you could do your string changes and then this, and it will look smaller.

[Joshua Powers (powersj)]

@smoser: few questions and answers below.

[Scott Moser (smoser)]

responded to comments from 09-08.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:f6397f8c2e7c22a7378d5d201cc37f7c6fc478d4
https://jenkins.ubuntu.com/server/job/cloud-init-ci/297/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/297/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:b7c7f3d1053ded4840a8ee3cf683c413666d4b06
https://jenkins.ubuntu.com/server/job/cloud-init-ci/298/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/298/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:376168e251a1d4f2ee3643fed6092b8907f057ec
https://jenkins.ubuntu.com/server/job/cloud-init-ci/299/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/299/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

some comments...
I wont insist on any of these.

there is some cleanup that we need to do though.

[Scott Moser (smoser)]

I guess this is 'Approve' and we can fix things more later.

review: Approve

diff --git a/tests/cloud_tests/__main__.py b/tests/cloud_tests/__main__.py
index 260ddb3..7ee29ca 100644
--- a/tests/cloud_tests/__main__.py
+++ b/tests/cloud_tests/__main__.py
@@ -4,6 +4,7 @@
 
 import argparse
 import logging
+import os
 import sys
 
 from tests.cloud_tests import args, bddeb, collect, manage, run_funcs, verify
@@ -50,7 +51,7 @@ def main():
             return -1
 
     # run handler
-    LOG.debug('running with args: %s\n', parsed)
+    LOG.debug('running with args: %s', parsed)
     return {
         'bddeb': bddeb.bddeb,
         'collect': collect.collect,
@@ -63,6 +64,8 @@ def main():
 
 
 if __name__ == "__main__":
+    if os.geteuid() == 0:
+        sys.exit('Do not run as root')
     sys.exit(main())
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/args.py b/tests/cloud_tests/args.py
index 369d60d..c6c1877 100644
--- a/tests/cloud_tests/args.py
+++ b/tests/cloud_tests/args.py
@@ -170,9 +170,9 @@ def normalize_collect_args(args):
     @param args: parsed args
     @return_value: updated args, or None if errors occurred
     """
-    # platform should default to all supported
+    # platform should default to lxd
     if len(args.platform) == 0:
-        args.platform = config.ENABLED_PLATFORMS
+        args.platform = ['lxd']
     args.platform = util.sorted_unique(args.platform)
 
     # os name should default to all enabled
diff --git a/tests/cloud_tests/collect.py b/tests/cloud_tests/collect.py
index b44e8bd..4a2422e 100644
--- a/tests/cloud_tests/collect.py
+++ b/tests/cloud_tests/collect.py
@@ -120,6 +120,7 @@ def collect_image(args, platform, os_name):
     os_config = config.load_os_config(
         platform.platform_name, os_name, require_enabled=True,
         feature_overrides=args.feature_override)
+    LOG.debug('os config: %s', os_config)
     component = PlatformComponent(
         partial(images.get_image, platform, os_config))
 
@@ -144,6 +145,8 @@ def collect_platform(args, platform_name):
 
     platform_config = config.load_platform_config(
         platform_name, require_enabled=True)
+    platform_config['data_dir'] = args.data_dir
+    LOG.debug('platform config: %s', platform_config)
     component = PlatformComponent(
         partial(platforms.get_platform, platform_name, platform_config))
 
diff --git a/tests/cloud_tests/config.py b/tests/cloud_tests/config.py
index 4d5dc80..52fc2bd 100644
--- a/tests/cloud_tests/config.py
+++ b/tests/cloud_tests/config.py
@@ -112,6 +112,7 @@ def load_os_config(platform_name, os_name, require_enabled=False,
     feature_conf = main_conf['features']
     feature_groups = conf.get('feature_groups', [])
     overrides = merge_config(get(conf, 'features'), feature_overrides)
+    conf['arch'] = c_util.get_architecture()
     conf['features'] = merge_feature_groups(
         feature_conf, feature_groups, overrides)
 
diff --git a/tests/cloud_tests/images/nocloudkvm.py b/tests/cloud_tests/images/nocloudkvm.py
new file mode 100644
index 0000000..a7af0e5
--- /dev/null
+++ b/tests/cloud_tests/images/nocloudkvm.py
@@ -0,0 +1,88 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+"""NoCloud KVM Image Base Class."""
+
+from tests.cloud_tests.images import base
+from tests.cloud_tests.snapshots import nocloudkvm as nocloud_kvm_snapshot
+
+
+class NoCloudKVMImage(base.Image):
+    """NoCloud KVM backed image."""
+
+    platform_name = "nocloud-kvm"
+
+    def __init__(self, platform, config, img_path):
+        """Set up image.
+
+        @param platform: platform object
+        @param config: image configuration
+        @param img_path: path to the image
+        """
+        self.modified = False
+        self._instance = None
+        self._img_path = img_path
+
+        super(NoCloudKVMImage, self).__init__(platform, config)
+
+    @property
+    def instance(self):
+        """Returns an instance of an image."""
+        if not self._instance:
+            if not self._img_path:
+                raise RuntimeError()
+
+            self._instance = self.platform.create_image(
+                self.properties, self.config, self.features, self._img_path,
+                image_desc=str(self), use_desc='image-modification')
+        return self._instance
+
+    @property
+    def properties(self):
+        """Dictionary containing: 'arch', 'os', 'version', 'release'."""
+        return {
+            'arch': self.config['arch'],
+            'os': self.config['family'],
+            'release': self.config['release'],
+            'version': self.config['version'],
+        }
+
+    def execute(self, *args, **kwargs):
+        """Execute command in image, modifying image."""
+        return self.instance.execute(*args, **kwargs)
+
+    def push_file(self, local_path, remote_path):
+        """Copy file at 'local_path' to instance at 'remote_path'."""
+        return self.instance.push_file(local_path, remote_path)
+
+    def run_script(self, *args, **kwargs):
+        """Run script in image, modifying image.
+
+        @return_value: script output
+        """
+        return self.instance.run_script(*args, **kwargs)
+
+    def snapshot(self):
+        """Create snapshot of image, block until done."""
+        if not self._img_path:
+            raise RuntimeError()
+
+        instance = self.platform.create_image(
+            self.properties, self.config, self.features,
+            self._img_path, image_desc=str(self), use_desc='snapshot')
+
+        return nocloud_kvm_snapshot.NoCloudKVMSnapshot(
+            self.platform, self.properties, self.config,
+            self.features, instance)
+
+    def destroy(self):
+        """Unset path to signal image is no longer used.
+
+        The removal of the images and all other items is handled by the
+        framework. In some cases we want to keep the images, so let the
+        framework decide whether to keep or destroy everything.
+        """
+        self._img_path = None
+        self._instance.destroy()
+        super(NoCloudKVMImage, self).destroy()
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/base.py b/tests/cloud_tests/instances/base.py
index 58f45b1..9bdda60 100644
--- a/tests/cloud_tests/instances/base.py
+++ b/tests/cloud_tests/instances/base.py
@@ -90,7 +90,7 @@ class Instance(object):
             return self.execute(
                 ['/bin/bash', script_path], rcs=rcs, description=description)
         finally:
-            self.execute(['rm', script_path], rcs=rcs)
+            self.execute(['rm', '-f', script_path], rcs=rcs)
 
     def tmpfile(self):
         """Get a tmp file in the target.
diff --git a/tests/cloud_tests/instances/nocloudkvm.py b/tests/cloud_tests/instances/nocloudkvm.py
new file mode 100644
index 0000000..7abfe73
--- /dev/null
+++ b/tests/cloud_tests/instances/nocloudkvm.py
@@ -0,0 +1,216 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+"""Base NoCloud KVM instance."""
+
+import os
+import paramiko
+import shlex
+import socket
+import subprocess
+import time
+
+from cloudinit import util as c_util
+from tests.cloud_tests.instances import base
+from tests.cloud_tests import util
+
+
+class NoCloudKVMInstance(base.Instance):
+    """NoCloud KVM backed instance."""
+
+    platform_name = "nocloud-kvm"
+
+    def __init__(self, platform, name, properties, config, features,
+                 user_data, meta_data):
+        """Set up instance.
+
+        @param platform: platform object
+        @param name: image path
+        @param properties: dictionary of properties
+        @param config: dictionary of configuration values
+        @param features: dictionary of supported feature flags
+        """
+        self.user_data = user_data
+        self.meta_data = meta_data
+        self.ssh_key_file = os.path.join(platform.config['data_dir'],
+                                         platform.config['private_key'])
+        self.ssh_port = None
+        self.pid = None
+        self.pid_file = None
+
+        super(NoCloudKVMInstance, self).__init__(
+            platform, name, properties, config, features)
+
+    def destroy(self):
+        """Clean up instance."""
+        if self.pid:
+            try:
+                c_util.subp(['kill', '-9', self.pid])
+            except util.ProcessExectuionError:
+                pass
+
+        if self.pid_file:
+            os.remove(self.pid_file)
+
+        self.pid = None
+        super(NoCloudKVMInstance, self).destroy()
+
+    def execute(self, command, stdout=None, stderr=None, env=None,
+                rcs=None, description=None):
+        """Execute command in instance.
+
+        Assumes functional networking and execution as root with the
+        target filesystem being available at /.
+
+        @param command: the command to execute as root inside the image
+            if command is a string, then it will be executed as:
+            ['sh', '-c', command]
+        @param stdout, stderr: file handles to write output and error to
+        @param env: environment variables
+        @param rcs: allowed return codes from command
+        @param description: purpose of command
+        @return_value: tuple containing stdout data, stderr data, exit code
+        """
+        if env is None:
+            env = {}
+
+        if isinstance(command, str):
+            command = ['sh', '-c', command]
+
+        if self.pid:
+            return self.ssh(command)
+        else:
+            return self.mount_image_callback(command) + (0,)
+
+    def mount_image_callback(self, cmd):
+        """Run mount-image-callback."""
+        mic = ('sudo mount-image-callback --system-mounts --system-resolvconf '
+               '%s -- chroot _MOUNTPOINT_ ' % self.name)
+
+        out, err = c_util.subp(shlex.split(mic) + cmd)
+
+        return out, err
+
+    def generate_seed(self, tmpdir):
+        """Generate nocloud seed from user-data"""
+        seed_file = os.path.join(tmpdir, '%s_seed.img' % self.name)
+        user_data_file = os.path.join(tmpdir, '%s_user_data' % self.name)
+
+        with open(user_data_file, "w") as ud_file:
+            ud_file.write(self.user_data)
+
+        c_util.subp(['cloud-localds', seed_file, user_data_file])
+
+        return seed_file
+
+    def get_free_port(self):
+        """Get a free port assigned by the kernel."""
+        s = socket.socket()
+        s.bind(('', 0))
+        num = s.getsockname()[1]
+        s.close()
+        return num
+
+    def push_file(self, local_path, remote_path):
+        """Copy file at 'local_path' to instance at 'remote_path'.
+
+        If we have a pid then SSH is up, otherwise, use
+        mount-image-callback.
+
+        @param local_path: path on local instance
+        @param remote_path: path on remote instance
+        """
+        if self.pid:
+            super(NoCloudKVMInstance, self).push_file()
+        else:
+            cmd = ("sudo mount-image-callback --system-mounts "
+                   "--system-resolvconf %s -- chroot _MOUNTPOINT_ "
+                   "/bin/sh -c 'cat - > %s'" % (self.name, remote_path))
+            local_file = open(local_path)
+            p = subprocess.Popen(shlex.split(cmd),
+                                 stdin=local_file,
+                                 stdout=subprocess.PIPE,
+                                 stderr=subprocess.PIPE)
+            p.wait()
+
+    def sftp_put(self, path, data):
+        """SFTP put a file."""
+        client = self._ssh_connect()
+        sftp = client.open_sftp()
+
+        with sftp.open(path, 'w') as f:
+            f.write(data)
+
+        client.close()
+
+    def ssh(self, command):
+        """Run a command via SSH."""
+        client = self._ssh_connect()
+
+        try:
+            _, out, err = client.exec_command(util.shell_pack(command))
+        except paramiko.SSHException:
+            raise util.InTargetExecuteError('', '', -1, command, self.name)
+
+        exit = out.channel.recv_exit_status()
+        out = ''.join(out.readlines())
+        err = ''.join(err.readlines())
+        client.close()
+
+        return out, err, exit
+
+    def _ssh_connect(self, hostname='localhost', username='ubuntu',
+                     banner_timeout=120, retry_attempts=30):
+        """Connect via SSH."""
+        private_key = paramiko.RSAKey.from_private_key_file(self.ssh_key_file)
+        client = paramiko.SSHClient()
+        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+        while retry_attempts:
+            try:
+                client.connect(hostname=hostname, username=username,
+                               port=self.ssh_port, pkey=private_key,
+                               banner_timeout=banner_timeout)
+                return client
+            except (paramiko.SSHException, TypeError):
+                time.sleep(1)
+                retry_attempts = retry_attempts - 1
+
+        error_desc = 'Failed command to: %s@%s:%s' % (username, hostname,
+                                                      self.ssh_port)
+        raise util.InTargetExecuteError('', '', -1, 'ssh connect',
+                                        self.name, error_desc)
+
+    def start(self, wait=True, wait_for_cloud_init=False):
+        """Start instance."""
+        tmpdir = self.platform.config['data_dir']
+        seed = self.generate_seed(tmpdir)
+        self.pid_file = os.path.join(tmpdir, '%s.pid' % self.name)
+        self.ssh_port = self.get_free_port()
+
+        cmd = ('./tools/xkvm --disk %s,cache=unsafe --disk %s,cache=unsafe '
+               '--netdev user,hostfwd=tcp::%s-:22 '
+               '-- -pidfile %s -vnc none -m 2G -smp 2'
+               % (self.name, seed, self.ssh_port, self.pid_file))
+
+        subprocess.Popen(shlex.split(cmd), close_fds=True,
+                         stdin=subprocess.PIPE,
+                         stdout=subprocess.PIPE,
+                         stderr=subprocess.PIPE)
+
+        while not os.path.exists(self.pid_file):
+            time.sleep(1)
+
+        with open(self.pid_file, 'r') as pid_f:
+            self.pid = pid_f.readlines()[0].strip()
+
+        if wait:
+            self._wait_for_system(wait_for_cloud_init)
+
+    def write_data(self, remote_path, data):
+        """Write data to instance filesystem.
+
+        @param remote_path: path in instance
+        @param data: data to write, either str or bytes
+        """
+        self.sftp_put(remote_path, data)
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/platforms.yaml b/tests/cloud_tests/platforms.yaml
index b91834a..fa4f845 100644
--- a/tests/cloud_tests/platforms.yaml
+++ b/tests/cloud_tests/platforms.yaml
@@ -59,6 +59,10 @@ platforms:
                 {{ config_get("user.user-data", properties.default) }}
             cloud-init-vendor.tpl: |
                 {{ config_get("user.vendor-data", properties.default) }}
+    nocloud-kvm:
+        enabled: true
+        private_key: id_rsa
+        public_key: id_rsa.pub
     ec2: {}
     azure: {}
 
diff --git a/tests/cloud_tests/platforms/__init__.py b/tests/cloud_tests/platforms/__init__.py
index 443f6d4..3490fe8 100644
--- a/tests/cloud_tests/platforms/__init__.py
+++ b/tests/cloud_tests/platforms/__init__.py
@@ -3,8 +3,10 @@
 """Main init."""
 
 from tests.cloud_tests.platforms import lxd
+from tests.cloud_tests.platforms import nocloudkvm
 
 PLATFORMS = {
+    'nocloud-kvm': nocloudkvm.NoCloudKVMPlatform,
     'lxd': lxd.LXDPlatform,
 }
 
diff --git a/tests/cloud_tests/platforms/nocloudkvm.py b/tests/cloud_tests/platforms/nocloudkvm.py
new file mode 100644
index 0000000..f1f8187
--- /dev/null
+++ b/tests/cloud_tests/platforms/nocloudkvm.py
@@ -0,0 +1,90 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+"""Base NoCloud KVM platform."""
+import glob
+import os
+
+from simplestreams import filters
+from simplestreams import mirrors
+from simplestreams import objectstores
+from simplestreams import util as s_util
+
+from cloudinit import util as c_util
+from tests.cloud_tests.images import nocloudkvm as nocloud_kvm_image
+from tests.cloud_tests.instances import nocloudkvm as nocloud_kvm_instance
+from tests.cloud_tests.platforms import base
+from tests.cloud_tests import util
+
+
+class NoCloudKVMPlatform(base.Platform):
+    """NoCloud KVM test platform."""
+
+    platform_name = 'nocloud-kvm'
+
+    def get_image(self, img_conf):
+        """Get image using specified image configuration.
+
+        @param img_conf: configuration for image
+        @return_value: cloud_tests.images instance
+        """
+        (url, path) = s_util.path_from_mirror_url(img_conf['mirror_url'], None)
+
+        filter = filters.get_filters(['arch=%s' % c_util.get_architecture(),
+                                      'release=%s' % img_conf['release'],
+                                      'ftype=disk1.img'])
+        mirror_config = {'filters': filter,
+                         'keep_items': False,
+                         'max_items': 1,
+                         'checksumming_reader': True,
+                         'item_download': True
+                         }
+
+        def policy(content, path):
+            return s_util.read_signed(content, keyring=img_conf['keyring'])
+
+        smirror = mirrors.UrlMirrorReader(url, policy=policy)
+        tstore = objectstores.FileStore(img_conf['mirror_dir'])
+        tmirror = mirrors.ObjectFilterMirror(config=mirror_config,
+                                             objectstore=tstore)
+        tmirror.sync(smirror, path)
+
+        search_d = os.path.join(img_conf['mirror_dir'], '**',
+                                img_conf['release'], '**', '*.img')
+
+        images = []
+        for fname in glob.iglob(search_d, recursive=True):
+            images.append(fname)
+
+        if len(images) != 1:
+            raise Exception('No unique images found')
+
+        image = nocloud_kvm_image.NoCloudKVMImage(self, img_conf, images[0])
+        if img_conf.get('override_templates', False):
+            image.update_templates(self.config.get('template_overrides', {}),
+                                   self.config.get('template_files', {}))
+        return image
+
+    def create_image(self, properties, config, features,
+                     src_img_path, image_desc=None, use_desc=None,
+                     user_data=None, meta_data=None):
+        """Create an image
+
+        @param src_img_path: image path to launch from
+        @param properties: image properties
+        @param config: image configuration
+        @param features: image features
+        @param image_desc: description of image being launched
+        @param use_desc: description of container's use
+        @return_value: cloud_tests.instances instance
+        """
+        name = util.gen_instance_name(image_desc=image_desc, use_desc=use_desc)
+        img_path = os.path.join(self.config['data_dir'], name + '.qcow2')
+        c_util.subp(['qemu-img', 'create', '-f', 'qcow2',
+                    '-b', src_img_path, img_path])
+
+        return nocloud_kvm_instance.NoCloudKVMInstance(self, img_path,
+                                                       properties, config,
+                                                       features, user_data,
+                                                       meta_data)
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/releases.yaml b/tests/cloud_tests/releases.yaml
index c8dd142..ec7e2d5 100644
--- a/tests/cloud_tests/releases.yaml
+++ b/tests/cloud_tests/releases.yaml
@@ -27,7 +27,12 @@ default_release_config:
         # features groups and additional feature settings
         feature_groups: []
         features: {}
-
+    nocloud-kvm:
+        mirror_url: https://cloud-images.ubuntu.com/daily
+        mirror_dir: '/srv/citest/nocloud-kvm'
+        keyring: /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
+        setup_overrides: null
+        override_templates: false
     # lxd specific default configuration options
     lxd:
         # default sstreams server to use for lxd image retrieval
@@ -121,6 +126,9 @@ releases:
         # EOL: Jul 2018
         default:
             enabled: true
+            release: artful
+            version: 17.10
+            family: ubuntu
             feature_groups:
                 - base
                 - debian_base
@@ -134,6 +142,9 @@ releases:
         # EOL: Jan 2018
         default:
             enabled: true
+            release: zesty
+            version: 17.04
+            family: ubuntu
             feature_groups:
                 - base
                 - debian_base
@@ -147,6 +158,9 @@ releases:
         # EOL: Apr 2021
         default:
             enabled: true
+            release: xenial
+            version: 16.04
+            family: ubuntu
             feature_groups:
                 - base
                 - debian_base
@@ -160,6 +174,9 @@ releases:
         # EOL: Apr 2019
         default:
             enabled: true
+            release: trusty
+            version: 14.04
+            family: ubuntu
             feature_groups:
                 - base
                 - debian_base
diff --git a/tests/cloud_tests/setup_image.py b/tests/cloud_tests/setup_image.py
index 3c0fff6..6672ffb 100644
--- a/tests/cloud_tests/setup_image.py
+++ b/tests/cloud_tests/setup_image.py
@@ -5,6 +5,7 @@
 from functools import partial
 import os
 
+from cloudinit import util as c_util
 from tests.cloud_tests import LOG
 from tests.cloud_tests import stage, util
 
@@ -19,7 +20,7 @@ def installed_package_version(image, package, ensure_installed=True):
     """
     os_family = util.get_os_family(image.properties['os'])
     if os_family == 'debian':
-        cmd = ['dpkg-query', '-W', "--showformat='${Version}'", package]
+        cmd = ['dpkg-query', '-W', "--showformat=${Version}", package]
     elif os_family == 'redhat':
         cmd = ['rpm', '-q', '--queryformat', "'%{VERSION}'", package]
     else:
@@ -53,7 +54,7 @@ def install_deb(args, image):
     image.execute(cmd, description=msg)
 
     # check installed deb version matches package
-    fmt = ['-W', "--showformat='${Version}'"]
+    fmt = ['-W', "--showformat=${Version}"]
     (out, err, exit) = image.execute(['dpkg-deb'] + fmt + [remote_path])
     expected_version = out.strip()
     found_version = installed_package_version(image, 'cloud-init')
@@ -191,6 +192,20 @@ def enable_repo(args, image):
     image.execute(cmd, description=msg)
 
 
+def generate_ssh_keys(data_dir):
+    """Generate SSH keys to be used with image."""
+    LOG.info('generating SSH keys')
+    filename = os.path.join(data_dir, 'id_rsa')
+
+    if os.path.exists(filename):
+        c_util.del_file(filename)
+
+    c_util.subp(['ssh-keygen', '-t', 'rsa', '-b', '4096',
+                 '-f', filename, '-P', '',
+                 '-C', 'ubuntu@cloud_test'],
+                capture=True)
+
+
 def setup_image(args, image):
     """Set up image as specified in args.
 
@@ -226,6 +241,7 @@ def setup_image(args, image):
         'set up for {}'.format(image), calls, continue_after_error=False)
     LOG.debug('after setup complete, installed cloud-init version is: %s',
               installed_package_version(image, 'cloud-init'))
+    generate_ssh_keys(args.data_dir)
     return res
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/snapshots/nocloudkvm.py b/tests/cloud_tests/snapshots/nocloudkvm.py
new file mode 100644
index 0000000..0999834
--- /dev/null
+++ b/tests/cloud_tests/snapshots/nocloudkvm.py
@@ -0,0 +1,74 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+"""Base NoCloud KVM snapshot."""
+import os
+
+from tests.cloud_tests.snapshots import base
+
+
+class NoCloudKVMSnapshot(base.Snapshot):
+    """NoCloud KVM image copy backed snapshot."""
+
+    platform_name = "nocloud-kvm"
+
+    def __init__(self, platform, properties, config, features,
+                 instance):
+        """Set up snapshot.
+
+        @param platform: platform object
+        @param properties: image properties
+        @param config: image config
+        @param features: supported feature flags
+        """
+        self.instance = instance
+
+        super(NoCloudKVMSnapshot, self).__init__(
+            platform, properties, config, features)
+
+    def launch(self, user_data, meta_data=None, block=True, start=True,
+               use_desc=None):
+        """Launch instance.
+
+        @param user_data: user-data for the instance
+        @param instance_id: instance-id for the instance
+        @param block: wait until instance is created
+        @param start: start instance and wait until fully started
+        @param use_desc: description of snapshot instance use
+        @return_value: an Instance
+        """
+        key_file = os.path.join(self.platform.config['data_dir'],
+                                self.platform.config['public_key'])
+        user_data = self.inject_ssh_key(user_data, key_file)
+
+        instance = self.platform.create_image(
+            self.properties, self.config, self.features,
+            self.instance.name, image_desc=str(self), use_desc=use_desc,
+            user_data=user_data, meta_data=meta_data)
+
+        if start:
+            instance.start()
+
+        return instance
+
+    def inject_ssh_key(self, user_data, key_file):
+        """Inject the authorized key into the user_data."""
+        with open(key_file) as f:
+            value = f.read()
+
+        key = 'ssh_authorized_keys:'
+        value = '  - %s' % value.strip()
+        user_data = user_data.split('\n')
+        if key in user_data:
+            user_data.insert(user_data.index(key) + 1, '%s' % value)
+        else:
+            user_data.insert(-1, '%s' % key)
+            user_data.insert(-1, '%s' % value)
+
+        return '\n'.join(user_data)
+
+    def destroy(self):
+        """Clean up snapshot data."""
+        self.instance.destroy()
+        super(NoCloudKVMSnapshot, self).destroy()
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/util.py b/tests/cloud_tests/util.py
index 2bbe21c..4357fbb 100644
--- a/tests/cloud_tests/util.py
+++ b/tests/cloud_tests/util.py
@@ -2,12 +2,14 @@
 
 """Utilities for re-use across integration tests."""
 
+import base64
 import copy
 import glob
 import os
 import random
 import shutil
 import string
+import subprocess
 import tempfile
 import yaml
 
@@ -242,6 +244,47 @@ def update_user_data(user_data, updates, dump_to_yaml=True):
             if dump_to_yaml else user_data)
 
 
+def shell_safe(cmd):
+    """Produce string safe shell string.
+
+    Create a string that can be passed to:
+         set -- <string>
+    to produce the same array that cmd represents.
+
+    Internally we utilize 'getopt's ability/knowledge on how to quote
+    strings to be safe for shell.  This implementation could be changed
+    to be pure python.  It is just a matter of correctly escaping
+    or quoting characters like: ' " ^ & $ ; ( ) ...
+
+    @param cmd: command as a list
+    """
+    out = subprocess.check_output(
+        ["getopt", "--shell", "sh", "--options", "", "--", "--"] + list(cmd))
+    # out contains ' -- <data>\n'. drop the ' -- ' and the '\n'
+    return out[4:-1].decode()
+
+
+def shell_pack(cmd):
+    """Return a string that can shuffled through 'sh' and execute cmd.
+
+    In Python subprocess terms:
+        check_output(cmd) == check_output(shell_pack(cmd), shell=True)
+
+    @param cmd: list or string of command to pack up
+    """
+
+    if isinstance(cmd, str):
+        cmd = [cmd]
+    else:
+        cmd = list(cmd)
+
+    stuffed = shell_safe(cmd)
+    # for whatever reason b64encode returns bytes when it is clearly
+    # representable as a string by nature of being base64 encoded.
+    b64 = base64.b64encode(stuffed.encode()).decode()
+    return 'eval set -- "$(echo %s | base64 --decode)" && exec "$@"' % b64
+
+
 class InTargetExecuteError(c_util.ProcessExecutionError):
     """Error type for in target commands that fail."""