Merge ~smoser/cloud-init:cleanup/cii-cleanup into cloud-init:master

[Scott Moser]

tests: integration test cleanup and full pass of nocloud-kvm.

Integration test harness changes:
 * Enable collection of console log in nocloud-kvm and lxd.
 * Collect the console log to results for all test runs.
 * change 'tmpfile' to pick name locally instead of using 'mktemp'.
 * drop the 'instance' attribute from nocloud-kvm Image and
   demote LXDImage.instance to a private attribute.
   This is because Images do not actually have instances.
   (LXDImage internally uses a booted system to modify the image).
 * Add 'TargetBase' as a superclass of Image and Instance providing
   implementations of execute, read_data, write_data, pull_file,
   and push_file. These all depend on an implementation of _execute.
 * Improve '_execute' implementations to support accepting stdin.
 * execute supports 'rcs=False' meaning 'do not raise exception'.
 * Drop support for pylxd < 2.2. older versions cannot determine
   exit code of 'execute', which makes them unusable.
 * make NoCloudKVMInstance._execute run as root via sudo. This required
   some changes so that 'hostname' could be reverse-looked up in order
   to avoid sudo taking a long time (~20 seconds).

Test changes here:
 * do not use /tmp, but rather /var/tmp (LP: #1707222)
 * make keys_to_console assertions more strict.
 * change user test cases to always add default (ubuntu) user
   so that nocloud-kvm's execute which operates over ssh can work.

Lots of things here.

following onto Josh's branch from
 https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/331736

read commit message for ididivudal things until i write a better commit message.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:e8f618c9c72e723e66f63eaaafafb9d392b15593
https://jenkins.ubuntu.com/server/job/cloud-init-ci/456/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/456/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:4ce1002cb5956e821e27087008221b5ffef0af45
https://jenkins.ubuntu.com/server/job/cloud-init-ci/457/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/457/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:773013c4f058f1f3daec544ff1211eee52933988
https://jenkins.ubuntu.com/server/job/cloud-init-ci/458/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/458/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:3f395a74479c236d50668761f7ff4cf3d8a949d6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/459/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/459/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

I'm not sure I understand the dropping of instance property from the NoCloud KVM class vs. the LXD one.

[Scott Moser (smoser)]

> I'm not sure I understand the dropping of instance property from the NoCloud
> KVM class vs. the LXD one.

An Image does not have an instance associated with it.

An 'Image' is something you can manipulate the contents of (execute,
write_file, read_file) but is not booted. Imagine it being a .img file
that you can mount-image-callback on (as it is in nocloud-kvm).

Currently the lxd Image is something that has to be booted in order to
provide those operations. It uses an actual lxd.Instance to do that.
There is nothing specifically wrong about this, and may well be how
we have to do things on some platforms.

My goal on lxd though would be to replace that with 'lxd-pstart' (or
lxd-pstart like function). That way there is no 'cleaning' required
as there are no side-effects of the original boot.

I believe that its name and "important look" made Josh believe he needed
to implement .instance in nocloudkvm's Image. So, I demoted the .instance
to be private.

8973cb2... by Scott Moser on 2017-11-01

adjust write_data and read_data docstrings.

989b215... by Scott Moser on 2017-11-01

nocloudkvm.Instance: use parent class's write_data drop un-used sftp

write_data was still using 'sftp', which would write the data as
the user we connected to. Thats wrong, in that it is supposed
to write as root.

At some point that would likely be converted to have 'mode' and 'owner'
arguments.

sftp is no longer used, so drop it.

[Ryan Harper (raharper)]

On Wed, Nov 1, 2017 at 1:45 PM, Scott Moser <email address hidden>
wrote:

> > I'm not sure I understand the dropping of instance property from the
> NoCloud
> > KVM class vs. the LXD one.
>
> An Image does not have an instance associated with it.
>
> An 'Image' is something you can manipulate the contents of (execute,
> write_file, read_file) but is not booted. Imagine it being a .img file
> that you can mount-image-callback on (as it is in nocloud-kvm).
>
> Currently the lxd Image is something that has to be booted in order to
> provide those operations. It uses an actual lxd.Instance to do that.
> There is nothing specifically wrong about this, and may well be how
> we have to do things on some platforms.
>

OK, IIUC, since NoCloud uses mount-image-callback, we can modify
the image without the need for an "instance" of it (aka, booted).

> --
> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-
> init/+merge/333059
> Your team cloud-init commiters is requested to review the proposed merge
> of ~smoser/cloud-init:cleanup/cii-cleanup into cloud-init:master.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~cloud-init-dev
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~cloud-init-dev
> More help : https://help.launchpad.net/ListHelp
>

[Ryan Harper (raharper)]

One more round on the @property structure, see inline.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:989b2154a380420b73ed5f098705647882b81735
https://jenkins.ubuntu.com/server/job/cloud-init-ci/460/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/460/rebuild

review: Approve (continuous-integration)

b6409c0... by Scott Moser on 2017-11-01

more info in docstring

f385dc4... by Scott Moser on 2017-11-01

mount-image-callback: do not decode data

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:b4f6b54f3c3e87253cec1d4aef895c613ebed92a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/461/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/461/rebuild

review: Needs Fixing (continuous-integration)

dd3b6c0... by Scott Moser on 2017-11-02

rework of execute and _execute.

Also here:
 * make run_script only use a single execute
 * fix (i think) the paramiko client to correctly send and wait
   on output.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:ac32c998307716217d309da66317055b817daad6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/462/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/462/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:d4b5f240c6cf48fcb34c7671e54b903d0dfe760d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/463/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/463/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:5f6aa3bc02f9b5da251c3df2b0c6b6b3d50c903d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/464/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/464/rebuild

review: Needs Fixing (continuous-integration)

3de1be3... by Scott Moser on 2017-11-02

_execute doesnt need string support anymore

dbb2a8e... by Scott Moser on 2017-11-02

use rcs=False

9b4fbdf... by Scott Moser on 2017-11-02

and no more convert to bytes in ssh

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:3de1be31b1be84b2418bbc7ea17a805837aa1149
https://jenkins.ubuntu.com/server/job/cloud-init-ci/465/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/465/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:7d1c5e8092ecc32521ab560b29df4c23338158dd
https://jenkins.ubuntu.com/server/job/cloud-init-ci/466/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/466/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:9b4fbdf073c38f5284efa05373a616fd25a0fb2b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/467/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/467/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:9b4fbdf073c38f5284efa05373a616fd25a0fb2b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/468/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/468/rebuild

review: Approve (continuous-integration)

5d38855... by Scott Moser on 2017-11-02

minor fix for readability

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:5d38855c6254d82df54ce31eea8863798b892c6c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/469/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/469/rebuild

review: Approve (continuous-integration)

[Joshua Powers (powersj)]

I cannot get the nocloud-kvm datasource to even run with the following commit:

$ git rev-parse HEAD
5d38855c6254d82df54ce31eea8863798b892c6c

One comment below about a debug message and should the commit message more verbosely call out the new TargetBased class?

* PASS - lxd run xenial
* PASS - lxd tree_run xenial
* FAIL - nocloud-kvm run xenial
* FAIL - nocloud-kvm run artful
* FAIL - nocloud-kvm tree_run artful

nocloud-kvm failure:

2017-11-03 18:14:58,081 - tests.cloud_tests - ERROR - stage part: script file_gzip encountered error: [Errno None] Unable to connect to port 47557 on 127.0.0.1
2017-11-03 18:14:58,081 - tests.cloud_tests - ERROR - traceback:
  File "/tmp/cloud-init/tests/cloud_tests/stage.py", line 47, in run_single
    call()
  File "/tmp/cloud-init/tests/cloud_tests/collect.py", line 26, in collect_script
    description='collect: {}'.format(script_name))
  File "/tmp/cloud-init/tests/cloud_tests/util.py", line 422, in run_script
    stdin=script, description=description, rcs=rcs)
  File "/tmp/cloud-init/tests/cloud_tests/util.py", line 327, in execute
    out, err, rc = self._execute(command=command, stdin=stdin, env=env)
  File "/tmp/cloud-init/tests/cloud_tests/instances/nocloudkvm.py", line 62, in _execute
    return self.ssh(['sudo'] + env_args + list(command), stdin=stdin)
  File "/tmp/cloud-init/tests/cloud_tests/instances/nocloudkvm.py", line 86, in ssh
    client = self._ssh_connect()
  File "/tmp/cloud-init/tests/cloud_tests/instances/nocloudkvm.py", line 115, in _ssh_connect
    banner_timeout=banner_timeout)
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 324, in connect
    raise NoValidConnectionsError(errors)

review: Needs Fixing

[Scott Moser (smoser)]

Can you paste a full log of what you did ?
This works fine for me on diglett. here is what I did:

$ tox-venv citest python3 -m tests.cloud_tests run --platform=nocloud-kvm --os-name=xenial --preserve-data --dat
a-dir=results.nocloud-kvm.d --verbose --test=tests/cloud_tests/testcases/modules/final_message.py

http://paste.ubuntu.com/25881300/

ec53bbb... by Scott Moser on 2017-11-03

LXDImage: do not [attempt] start instance everytime property referenced

397ae68... by Scott Moser on 2017-11-03

only log debug of command show command if no description

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:397ae68232a5ef8a00c83f5ad8172047fef6a065
https://jenkins.ubuntu.com/server/job/cloud-init-ci/470/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/470/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Per the issue JoshPowers brought up on Friday, sudo times out after 20 seconds on the kvm instances due to the inability to resolve the 'ubuntu' hostname. That 20 second timeout affects every single instance.execute which runs 'sudo'.

I think smoser & rharper brought up the possible solution to
   1. reuse ssh connection
   2. Possibly add ubuntu to /etc/hosts so we don't hit DNS resolution errors for nocloud.

Here is a patch which takes smoser's ssh connection caching/reuse + my changes to update etc_hosts on nocloudkvm images
http://pastebin.ubuntu.com/25881811/

[Chad Smith (chad.smith)]

After applying the above patch I see all collect artifact calls returning quickly instead of being affected by the sudo timeout

http://pastebin.ubuntu.com/25884331/

[Chad Smith (chad.smith)]

..... haven't run through the full test suite though. it seems some tests are still hitting a timeout at 20 seconds on each collect artifact call. not sure if that's because we've reset the hostname in those tests :/

15549a3... by Scott Moser on 2017-11-06

namespace sudo issues by moving hostnames to i9n.cloud-init.io domain

This just makes each of the hostnames we use under the i9.cloud-init.io
domain. cloud-init.io can be updated to provide these dns records
(or wildcard dns record). At this point the easiest fix is to add
the following lines to your /etc/hosts (line break is just to keep
it less than 80 chars, you could put all hosts on the same long line).

127.0.1.126 cloudinit1.i9n.cloud-init.io cloudinit2.i9n.cloud-init.io
127.0.1.126 ubuntu.i9n.cloud-init.io

7897d64... by Scott Moser on 2017-11-06

re-use ssh connection.

Rather than creating a new ssh connection for each sub-command, re-use
the connection and create a new channel.

In my very un-scientific test with a sample set of 1, and entries in
hosts' /etc/hosts, i found:
 before: 6 commands took 1.277
 after : 6 commands took 0.469

d85fabb... by Scott Moser on 2017-11-06

Do not require modification of /etc/hosts: use actually populated subdomain

I've populated i9n.brickies.net with records for:
   ubuntu, cloudinit1, cloudinit2

Then, the change here was just:
sed s,i9n.cloud-init.io,i9n.brickies.net, -i \
   $(grep -rl --include="*.yaml" --include="*.py" i9n.cloud-init.io tests/)

The result is slower than host having entries in /etc/hosts, which
does make sense (.469 -> .557 for the 6 executes in the test i ran).
But it means the user doesnt have to modify anything in /etc/hosts.

[Chad Smith (chad.smith)]

Let's make that a proper cut-n-paste
http://pastebin.ubuntu.com/25884331/

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:2c6ac709bf42fe7255bea46a10a9960029322ff4
https://jenkins.ubuntu.com/server/job/cloud-init-ci/471/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/471/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:b5aea7d2986d003dc82fde92351e7382703681d0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/472/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/472/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:33d646d4a95b2b00395a4cc83be8f58c73133ef3
https://jenkins.ubuntu.com/server/job/cloud-init-ci/473/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/473/rebuild

review: Needs Fixing (continuous-integration)

[Scott Moser (smoser)]

Josh,
Could you re-review?

I think I addressed your issues, making execute show the description.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:7897d64b94fd3aa6d119d533f808f2f11d6ecd6a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/474/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/474/rebuild

review: Needs Fixing (continuous-integration)

[Chad Smith (chad.smith)]

Performance is all good on my side now that your brickeies A-records are registered. I no longer see 20 second timeouts between collect calls. Minor nits inline. Let's land this and iterate on ec2.

review: Approve

[Chad Smith (chad.smith)]

... once ci passes and torkoal is fixed.

[Joshua Powers (powersj)]

You already pulled this, but it looks good

review: Approve

[Scott Moser (smoser)]

Bah. I saw your approved and didn't read your comments. Sorry for setting presumptuous.

On November 6, 2017 4:42:33 PM EST, Chad Smith <email address hidden> wrote:
>... once ci passes and torkoal is fixed.
>--
>https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/333059
>You are the owner of ~smoser/cloud-init:cleanup/cii-cleanup.

diff --git a/tests/cloud_tests/collect.py b/tests/cloud_tests/collect.py
index 4a2422e..71ee764 100644
--- a/tests/cloud_tests/collect.py
+++ b/tests/cloud_tests/collect.py
@@ -22,11 +22,21 @@ def collect_script(instance, base_dir, script, script_name):
     """
     LOG.debug('running collect script: %s', script_name)
     (out, err, exit) = instance.run_script(
-        script, rcs=range(0, 256),
+        script.encode(), rcs=False,
         description='collect: {}'.format(script_name))
     c_util.write_file(os.path.join(base_dir, script_name), out)
 
 
+def collect_console(instance, base_dir):
+    LOG.debug('getting console log')
+    try:
+        data = instance.console_log()
+    except NotImplementedError as e:
+        data = 'Not Implemented: %s' % e
+    with open(os.path.join(base_dir, 'console.log'), "wb") as fp:
+        fp.write(data)
+
+
 def collect_test_data(args, snapshot, os_name, test_name):
     """Collect data for test case.
 
@@ -79,8 +89,12 @@ def collect_test_data(args, snapshot, os_name, test_name):
                                          test_output_dir, script, script_name))
                          for script_name, script in test_scripts.items()]
 
+        console_log = partial(
+            run_single, 'collect console',
+            partial(collect_console, instance, test_output_dir))
+
         res = run_stage('collect for test: {}'.format(test_name),
-                        [start_call] + collect_calls)
+                        [start_call] + collect_calls + [console_log])
 
     return res
 
diff --git a/tests/cloud_tests/images/base.py b/tests/cloud_tests/images/base.py
index 0a1e056..d503108 100644
--- a/tests/cloud_tests/images/base.py
+++ b/tests/cloud_tests/images/base.py
@@ -2,8 +2,10 @@
 
 """Base class for images."""
 
+from ..util import TargetBase
 
-class Image(object):
+
+class Image(TargetBase):
     """Base class for images."""
 
     platform_name = None
@@ -43,21 +45,6 @@ class Image(object):
         # NOTE: more sophisticated options may be requied at some point
         return self.config.get('setup_overrides', {})
 
-    def execute(self, *args, **kwargs):
-        """Execute command in image, modifying image."""
-        raise NotImplementedError
-
-    def push_file(self, local_path, remote_path):
-        """Copy file at 'local_path' to instance at 'remote_path'."""
-        raise NotImplementedError
-
-    def run_script(self, *args, **kwargs):
-        """Run script in image, modifying image.
-
-        @return_value: script output
-        """
-        raise NotImplementedError
-
     def snapshot(self):
         """Create snapshot of image, block until done."""
         raise NotImplementedError
diff --git a/tests/cloud_tests/images/lxd.py b/tests/cloud_tests/images/lxd.py
index fd4e93c..5caeba4 100644
--- a/tests/cloud_tests/images/lxd.py
+++ b/tests/cloud_tests/images/lxd.py
@@ -24,7 +24,7 @@ class LXDImage(base.Image):
         @param config: image configuration
         """
         self.modified = False
-        self._instance = None
+        self._img_instance = None
         self._pylxd_image = None
         self.pylxd_image = pylxd_image
         super(LXDImage, self).__init__(platform, config)
@@ -38,9 +38,9 @@ class LXDImage(base.Image):
 
     @pylxd_image.setter
     def pylxd_image(self, pylxd_image):
-        if self._instance:
+        if self._img_instance:
             self._instance.destroy()
-            self._instance = None
+            self._img_instance = None
         if (self._pylxd_image and
                 (self._pylxd_image is not pylxd_image) and
                 (not self.config.get('cache_base_image') or self.modified)):
@@ -49,15 +49,19 @@ class LXDImage(base.Image):
         self._pylxd_image = pylxd_image
 
     @property
-    def instance(self):
-        """Property function."""
-        if not self._instance:
-            self._instance = self.platform.launch_container(
+    def _instance(self):
+        """Internal use only, returns a instance
+
+        This starts an lxc instance from the image, so it is "dirty".
+        Better would be some way to modify this "at rest".
+        lxc-pstart would be an option."""
+        if not self._img_instance:
+            self._img_instance = self.platform.launch_container(
                 self.properties, self.config, self.features,
                 use_desc='image-modification', image_desc=str(self),
                 image=self.pylxd_image.fingerprint)
-        self._instance.start()
-        return self._instance
+            self._img_instance.start()
+        return self._img_instance
 
     @property
     def properties(self):
@@ -144,20 +148,20 @@ class LXDImage(base.Image):
             shutil.rmtree(export_dir)
             shutil.rmtree(extract_dir)
 
-    def execute(self, *args, **kwargs):
+    def _execute(self, *args, **kwargs):
         """Execute command in image, modifying image."""
-        return self.instance.execute(*args, **kwargs)
+        return self._instance._execute(*args, **kwargs)
 
     def push_file(self, local_path, remote_path):
         """Copy file at 'local_path' to instance at 'remote_path'."""
-        return self.instance.push_file(local_path, remote_path)
+        return self._instance.push_file(local_path, remote_path)
 
     def run_script(self, *args, **kwargs):
         """Run script in image, modifying image.
 
         @return_value: script output
         """
-        return self.instance.run_script(*args, **kwargs)
+        return self._instance.run_script(*args, **kwargs)
 
     def snapshot(self):
         """Create snapshot of image, block until done."""
@@ -169,7 +173,7 @@ class LXDImage(base.Image):
         # clone current instance
         instance = self.platform.launch_container(
             self.properties, self.config, self.features,
-            container=self.instance.name, image_desc=str(self),
+            container=self._instance.name, image_desc=str(self),
             use_desc='snapshot', container_config=conf)
         # wait for cloud-init before boot_clean_script is run to ensure
         # /var/lib/cloud is removed cleanly
diff --git a/tests/cloud_tests/images/nocloudkvm.py b/tests/cloud_tests/images/nocloudkvm.py
index a7af0e5..1e7962c 100644
--- a/tests/cloud_tests/images/nocloudkvm.py
+++ b/tests/cloud_tests/images/nocloudkvm.py
@@ -2,6 +2,8 @@
 
 """NoCloud KVM Image Base Class."""
 
+from cloudinit import util as c_util
+
 from tests.cloud_tests.images import base
 from tests.cloud_tests.snapshots import nocloudkvm as nocloud_kvm_snapshot
 
@@ -19,24 +21,11 @@ class NoCloudKVMImage(base.Image):
         @param img_path: path to the image
         """
         self.modified = False
-        self._instance = None
         self._img_path = img_path
 
         super(NoCloudKVMImage, self).__init__(platform, config)
 
     @property
-    def instance(self):
-        """Returns an instance of an image."""
-        if not self._instance:
-            if not self._img_path:
-                raise RuntimeError()
-
-            self._instance = self.platform.create_image(
-                self.properties, self.config, self.features, self._img_path,
-                image_desc=str(self), use_desc='image-modification')
-        return self._instance
-
-    @property
     def properties(self):
         """Dictionary containing: 'arch', 'os', 'version', 'release'."""
         return {
@@ -46,20 +35,26 @@ class NoCloudKVMImage(base.Image):
             'version': self.config['version'],
         }
 
-    def execute(self, *args, **kwargs):
+    def _execute(self, command, stdin=None, env=None):
         """Execute command in image, modifying image."""
-        return self.instance.execute(*args, **kwargs)
+        return self.mount_image_callback(command, stdin=stdin, env=env)
 
-    def push_file(self, local_path, remote_path):
-        """Copy file at 'local_path' to instance at 'remote_path'."""
-        return self.instance.push_file(local_path, remote_path)
+    def mount_image_callback(self, command, stdin=None, env=None):
+        """Run mount-image-callback."""
 
-    def run_script(self, *args, **kwargs):
-        """Run script in image, modifying image.
+        env_args = []
+        if env:
+            env_args = ['env'] + ["%s=%s" for k, v in env.items()]
 
-        @return_value: script output
-        """
-        return self.instance.run_script(*args, **kwargs)
+        mic_chroot = ['sudo', 'mount-image-callback', '--system-mounts',
+                      '--system-resolvconf', self._img_path,
+                      '--', 'chroot', '_MOUNTPOINT_']
+        try:
+            out, err = c_util.subp(mic_chroot + env_args + list(command),
+                                   data=stdin, decode=False)
+            return (out, err, 0)
+        except c_util.ProcessExecutionError as e:
+            return (e.stdout, e.stderr, e.exit_code)
 
     def snapshot(self):
         """Create snapshot of image, block until done."""
@@ -82,7 +77,6 @@ class NoCloudKVMImage(base.Image):
         framework decide whether to keep or destroy everything.
         """
         self._img_path = None
-        self._instance.destroy()
         super(NoCloudKVMImage, self).destroy()
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/base.py b/tests/cloud_tests/instances/base.py
index 9bdda60..8c59d62 100644
--- a/tests/cloud_tests/instances/base.py
+++ b/tests/cloud_tests/instances/base.py
@@ -2,8 +2,10 @@
 
 """Base instance."""
 
+from ..util import TargetBase
 
-class Instance(object):
+
+class Instance(TargetBase):
     """Base instance object."""
 
     platform_name = None
@@ -22,82 +24,7 @@ class Instance(object):
         self.properties = properties
         self.config = config
         self.features = features
-
-    def execute(self, command, stdout=None, stderr=None, env=None,
-                rcs=None, description=None):
-        """Execute command in instance, recording output, error and exit code.
-
-        Assumes functional networking and execution as root with the
-        target filesystem being available at /.
-
-        @param command: the command to execute as root inside the image
-            if command is a string, then it will be executed as:
-            ['sh', '-c', command]
-        @param stdout, stderr: file handles to write output and error to
-        @param env: environment variables
-        @param rcs: allowed return codes from command
-        @param description: purpose of command
-        @return_value: tuple containing stdout data, stderr data, exit code
-        """
-        raise NotImplementedError
-
-    def read_data(self, remote_path, decode=False):
-        """Read data from instance filesystem.
-
-        @param remote_path: path in instance
-        @param decode: return as string
-        @return_value: data as str or bytes
-        """
-        raise NotImplementedError
-
-    def write_data(self, remote_path, data):
-        """Write data to instance filesystem.
-
-        @param remote_path: path in instance
-        @param data: data to write, either str or bytes
-        """
-        raise NotImplementedError
-
-    def pull_file(self, remote_path, local_path):
-        """Copy file at 'remote_path', from instance to 'local_path'.
-
-        @param remote_path: path on remote instance
-        @param local_path: path on local instance
-        """
-        with open(local_path, 'wb') as fp:
-            fp.write(self.read_data(remote_path))
-
-    def push_file(self, local_path, remote_path):
-        """Copy file at 'local_path' to instance at 'remote_path'.
-
-        @param local_path: path on local instance
-        @param remote_path: path on remote instance
-        """
-        with open(local_path, 'rb') as fp:
-            self.write_data(remote_path, fp.read())
-
-    def run_script(self, script, rcs=None, description=None):
-        """Run script in target and return stdout.
-
-        @param script: script contents
-        @param rcs: allowed return codes from script
-        @param description: purpose of script
-        @return_value: stdout from script
-        """
-        script_path = self.tmpfile()
-        try:
-            self.write_data(script_path, script)
-            return self.execute(
-                ['/bin/bash', script_path], rcs=rcs, description=description)
-        finally:
-            self.execute(['rm', '-f', script_path], rcs=rcs)
-
-    def tmpfile(self):
-        """Get a tmp file in the target.
-
-        @return_value: path to new file in target
-        """
-        return self.execute(['mktemp'])[0].strip()
+        self._tmp_count = 0
 
     def console_log(self):
         """Instance console.
diff --git a/tests/cloud_tests/instances/lxd.py b/tests/cloud_tests/instances/lxd.py
index a43918c..3b035d8 100644
--- a/tests/cloud_tests/instances/lxd.py
+++ b/tests/cloud_tests/instances/lxd.py
@@ -2,8 +2,11 @@
 
 """Base LXD instance."""
 
-from tests.cloud_tests.instances import base
-from tests.cloud_tests import util
+from . import base
+
+import os
+import shutil
+from tempfile import mkdtemp
 
 
 class LXDInstance(base.Instance):
@@ -24,6 +27,8 @@ class LXDInstance(base.Instance):
         self._pylxd_container = pylxd_container
         super(LXDInstance, self).__init__(
             platform, name, properties, config, features)
+        self.tmpd = mkdtemp(prefix="%s-%s" % (type(self).__name__, name))
+        self._setup_console_log()
 
     @property
     def pylxd_container(self):
@@ -31,74 +36,69 @@ class LXDInstance(base.Instance):
         self._pylxd_container.sync()
         return self._pylxd_container
 
-    def execute(self, command, stdout=None, stderr=None, env=None,
-                rcs=None, description=None):
-        """Execute command in instance, recording output, error and exit code.
-
-        Assumes functional networking and execution as root with the
-        target filesystem being available at /.
-
-        @param command: the command to execute as root inside the image
-            if command is a string, then it will be executed as:
-            ['sh', '-c', command]
-        @param stdout: file handler to write output
-        @param stderr: file handler to write error
-        @param env: environment variables
-        @param rcs: allowed return codes from command
-        @param description: purpose of command
-        @return_value: tuple containing stdout data, stderr data, exit code
-        """
+    def _setup_console_log(self):
+        logf = os.path.join(self.tmpd, "console.log")
+
+        # doing this ensures we can read it. Otherwise it ends up root:root.
+        with open(logf, "w") as fp:
+            fp.write("# %s\n" % self.name)
+
+        cfg = "lxc.console.logfile=%s" % logf
+        orig = self._pylxd_container.config.get('raw.lxc', "")
+        if orig:
+            orig += "\n"
+        self._pylxd_container.config['raw.lxc'] = orig + cfg
+        self._pylxd_container.save()
+        self._console_log_file = logf
+
+    def _execute(self, command, stdin=None, env=None):
         if env is None:
             env = {}
 
-        if isinstance(command, str):
-            command = ['sh', '-c', command]
+        if stdin is not None:
+            # pylxd does not support input to execute.
+            # https://github.com/lxc/pylxd/issues/244
+            #
+            # The solution here is write a tmp file in the container
+            # and then execute a shell that sets it standard in to
+            # be from that file, removes it, and calls the comand.
+            tmpf = self.tmpfile()
+            self.write_data(tmpf, stdin)
+            ncmd = 'exec <"{tmpf}"; rm -f "{tmpf}"; exec "$@"'
+            command = (['sh', '-c', ncmd.format(tmpf=tmpf), 'stdinhack'] +
+                       list(command))
 
         # ensure instance is running and execute the command
         self.start()
+        # execute returns a ContainerExecuteResult, named tuple
+        # (exit_code, stdout, stderr)
         res = self.pylxd_container.execute(command, environment=env)
 
         # get out, exit and err from pylxd return
-        if hasattr(res, 'exit_code'):
-            # pylxd 2.2 returns ContainerExecuteResult, named tuple of
-            # (exit_code, out, err)
-            (exit, out, err) = res
-        else:
+        if not hasattr(res, 'exit_code'):
             # pylxd 2.1.3 and earlier only return out and err, no exit
-            # LOG.warning('using pylxd version < 2.2')
-            (out, err) = res
-            exit = 0
-
-        # write data to file descriptors if needed
-        if stdout:
-            stdout.write(out)
-        if stderr:
-            stderr.write(err)
-
-        # if the command exited with a code not allowed in rcs, then fail
-        if exit not in (rcs if rcs else (0,)):
-            error_desc = ('Failed command to: {}'.format(description)
-                          if description else None)
-            raise util.InTargetExecuteError(
-                out, err, exit, command, self.name, error_desc)
+            raise RuntimeError(
+                "No 'exit_code' in pylxd.container.execute return.\n"
+                "pylxd > 2.2 is required.")
 
-        return (out, err, exit)
+        return res.stdout, res.stderr, res.exit_code
 
     def read_data(self, remote_path, decode=False):
         """Read data from instance filesystem.
 
         @param remote_path: path in instance
-        @param decode: return as string
-        @return_value: data as str or bytes
+        @param decode: decode data before returning.
+        @return_value: content of remote_path as bytes if 'decode' is False,
+                       and as string if 'decode' is True.
         """
         data = self.pylxd_container.files.get(remote_path)
-        return data.decode() if decode and isinstance(data, bytes) else data
+        return data.decode() if decode else data
 
     def write_data(self, remote_path, data):
         """Write data to instance filesystem.
 
         @param remote_path: path in instance
-        @param data: data to write, either str or bytes
+        @param data: data to write in bytes
         """
         self.pylxd_container.files.put(remote_path, data)
 
@@ -107,7 +107,14 @@ class LXDInstance(base.Instance):
 
         @return_value: bytes of this instance’s console
         """
-        raise NotImplementedError
+        if not os.path.exists(self._console_log_file):
+            raise NotImplementedError(
+                "Console log '%s' does not exist. If this is a remote "
+                "lxc, then this is really NotImplementedError.  If it is "
+                "A local lxc, then this is a RuntimeError."
+                "https://github.com/lxc/lxd/issues/1129")
+        with open(self._console_log_file, "rb") as fp:
+            return fp.read()
 
     def reboot(self, wait=True):
         """Reboot instance."""
@@ -144,6 +151,7 @@ class LXDInstance(base.Instance):
         if self.platform.container_exists(self.name):
             raise OSError('container {} was not properly removed'
                           .format(self.name))
+        shutil.rmtree(self.tmpd)
         super(LXDInstance, self).destroy()
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/nocloudkvm.py b/tests/cloud_tests/instances/nocloudkvm.py
index 8a0e531..fbb04aa 100644
--- a/tests/cloud_tests/instances/nocloudkvm.py
+++ b/tests/cloud_tests/instances/nocloudkvm.py
@@ -12,11 +12,19 @@ from cloudinit import util as c_util
 from tests.cloud_tests.instances import base
 from tests.cloud_tests import util
 
+# This domain contains reverse lookups for hostnames that are used.
+# The primary reason is so sudo will return quickly when it attempts
+# to look up the hostname.  i9n is just short for 'integration'.
+# use i9n.brickies.net until i9n.cloud-init.io is popualted:
+#   https://portal.admin.canonical.com/107125
+CI_DOMAIN = "i9n.brickies.net"
+
 
 class NoCloudKVMInstance(base.Instance):
     """NoCloud KVM backed instance."""
 
     platform_name = "nocloud-kvm"
+    _ssh_client = None
 
     def __init__(self, platform, name, properties, config, features,
                  user_data, meta_data):
@@ -35,6 +43,7 @@ class NoCloudKVMInstance(base.Instance):
         self.ssh_port = None
         self.pid = None
         self.pid_file = None
+        self.console_file = None
 
         super(NoCloudKVMInstance, self).__init__(
             platform, name, properties, config, features)
@@ -51,43 +60,18 @@ class NoCloudKVMInstance(base.Instance):
             os.remove(self.pid_file)
 
         self.pid = None
-        super(NoCloudKVMInstance, self).destroy()
-
-    def execute(self, command, stdout=None, stderr=None, env=None,
-                rcs=None, description=None):
-        """Execute command in instance.
-
-        Assumes functional networking and execution as root with the
-        target filesystem being available at /.
-
-        @param command: the command to execute as root inside the image
-            if command is a string, then it will be executed as:
-            ['sh', '-c', command]
-        @param stdout, stderr: file handles to write output and error to
-        @param env: environment variables
-        @param rcs: allowed return codes from command
-        @param description: purpose of command
-        @return_value: tuple containing stdout data, stderr data, exit code
-        """
-        if env is None:
-            env = {}
-
-        if isinstance(command, str):
-            command = ['sh', '-c', command]
+        if self._ssh_client:
+            self._ssh_client.close()
+            self._ssh_client = None
 
-        if self.pid:
-            return self.ssh(command)
-        else:
-            return self.mount_image_callback(command) + (0,)
+        super(NoCloudKVMInstance, self).destroy()
 
-    def mount_image_callback(self, cmd):
-        """Run mount-image-callback."""
-        out, err = c_util.subp(['sudo', 'mount-image-callback',
-                                '--system-mounts', '--system-resolvconf',
-                                self.name, '--', 'chroot',
-                                '_MOUNTPOINT_'] + cmd)
+    def _execute(self, command, stdin=None, env=None):
+        env_args = []
+        if env:
+            env_args = ['env'] + ["%s=%s" for k, v in env.items()]
 
-        return out, err
+        return self.ssh(['sudo'] + env_args + list(command), stdin=stdin)
 
     def generate_seed(self, tmpdir):
         """Generate nocloud seed from user-data"""
@@ -109,57 +93,31 @@ class NoCloudKVMInstance(base.Instance):
         s.close()
         return num
 
-    def push_file(self, local_path, remote_path):
-        """Copy file at 'local_path' to instance at 'remote_path'.
-
-        If we have a pid then SSH is up, otherwise, use
-        mount-image-callback.
-
-        @param local_path: path on local instance
-        @param remote_path: path on remote instance
-        """
-        if self.pid:
-            super(NoCloudKVMInstance, self).push_file()
-        else:
-            local_file = open(local_path)
-            p = subprocess.Popen(['sudo', 'mount-image-callback',
-                                  '--system-mounts', '--system-resolvconf',
-                                  self.name, '--', 'chroot', '_MOUNTPOINT_',
-                                  '/bin/sh', '-c', 'cat - > %s' % remote_path],
-                                 stdin=local_file,
-                                 stdout=subprocess.PIPE,
-                                 stderr=subprocess.PIPE)
-            p.wait()
-
-    def sftp_put(self, path, data):
-        """SFTP put a file."""
-        client = self._ssh_connect()
-        sftp = client.open_sftp()
-
-        with sftp.open(path, 'w') as f:
-            f.write(data)
-
-        client.close()
-
-    def ssh(self, command):
+    def ssh(self, command, stdin=None):
         """Run a command via SSH."""
         client = self._ssh_connect()
 
+        cmd = util.shell_pack(command)
         try:
-            _, out, err = client.exec_command(util.shell_pack(command))
-        except paramiko.SSHException:
-            raise util.InTargetExecuteError('', '', -1, command, self.name)
-
-        exit = out.channel.recv_exit_status()
-        out = ''.join(out.readlines())
-        err = ''.join(err.readlines())
-        client.close()
-
-        return out, err, exit
+            fp_in, fp_out, fp_err = client.exec_command(cmd)
+            channel = fp_in.channel
+            if stdin is not None:
+                fp_in.write(stdin)
+                fp_in.close()
+
+            channel.shutdown_write()
+            rc = channel.recv_exit_status()
+            return (fp_out.read(), fp_err.read(), rc)
+        except paramiko.SSHException as e:
+            raise util.InTargetExecuteError(
+                b'', b'', -1, command, self.name, reason=e)
 
     def _ssh_connect(self, hostname='localhost', username='ubuntu',
                      banner_timeout=120, retry_attempts=30):
         """Connect via SSH."""
+        if self._ssh_client:
+            return self._ssh_client
+
         private_key = paramiko.RSAKey.from_private_key_file(self.ssh_key_file)
         client = paramiko.SSHClient()
         client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
@@ -168,6 +126,7 @@ class NoCloudKVMInstance(base.Instance):
                 client.connect(hostname=hostname, username=username,
                                port=self.ssh_port, pkey=private_key,
                                banner_timeout=banner_timeout)
+                self._ssh_client = client
                 return client
             except (paramiko.SSHException, TypeError):
                 time.sleep(1)
@@ -183,15 +142,19 @@ class NoCloudKVMInstance(base.Instance):
         tmpdir = self.platform.config['data_dir']
         seed = self.generate_seed(tmpdir)
         self.pid_file = os.path.join(tmpdir, '%s.pid' % self.name)
+        self.console_file = os.path.join(tmpdir, '%s-console.log' % self.name)
         self.ssh_port = self.get_free_port()
 
-        subprocess.Popen(['./tools/xkvm',
-                          '--disk', '%s,cache=unsafe' % self.name,
-                          '--disk', '%s,cache=unsafe' % seed,
-                          '--netdev',
-                          'user,hostfwd=tcp::%s-:22' % self.ssh_port,
-                          '--', '-pidfile', self.pid_file, '-vnc', 'none',
-                          '-m', '2G', '-smp', '2'],
+        cmd = ['./tools/xkvm',
+               '--disk', '%s,cache=unsafe' % self.name,
+               '--disk', '%s,cache=unsafe' % seed,
+               '--netdev', ','.join(['user',
+                                     'hostfwd=tcp::%s-:22' % self.ssh_port,
+                                     'dnssearch=%s' % CI_DOMAIN]),
+               '--', '-pidfile', self.pid_file, '-vnc', 'none',
+               '-m', '2G', '-smp', '2', '-nographic',
+               '-serial', 'file:' + self.console_file]
+        subprocess.Popen(cmd,
                          close_fds=True,
                          stdin=subprocess.PIPE,
                          stdout=subprocess.PIPE,
@@ -206,12 +169,10 @@ class NoCloudKVMInstance(base.Instance):
         if wait:
             self._wait_for_system(wait_for_cloud_init)
 
-    def write_data(self, remote_path, data):
-        """Write data to instance filesystem.
-
-        @param remote_path: path in instance
-        @param data: data to write, either str or bytes
-        """
-        self.sftp_put(remote_path, data)
+    def console_log(self):
+        if not self.console_file:
+            return b''
+        with open(self.console_file, "rb") as fp:
+            return fp.read()
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/examples/run_commands.yaml b/tests/cloud_tests/testcases/examples/run_commands.yaml
index b0e311b..f80eb8c 100644
--- a/tests/cloud_tests/testcases/examples/run_commands.yaml
+++ b/tests/cloud_tests/testcases/examples/run_commands.yaml
@@ -7,10 +7,10 @@ enabled: False
 cloud_config: |
   #cloud-config
   runcmd:
-   - echo cloud-init run cmd test > /tmp/run_cmd
+   - echo cloud-init run cmd test > /var/tmp/run_cmd
 collect_scripts:
   run_cmd: |
     #!/bin/bash
-    cat /tmp/run_cmd
+    cat /var/tmp/run_cmd
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/keys_to_console.py b/tests/cloud_tests/testcases/modules/keys_to_console.py
index 88b6812..07f3811 100644
--- a/tests/cloud_tests/testcases/modules/keys_to_console.py
+++ b/tests/cloud_tests/testcases/modules/keys_to_console.py
@@ -10,13 +10,13 @@ class TestKeysToConsole(base.CloudTestCase):
     def test_excluded_keys(self):
         """Test excluded keys missing."""
         out = self.get_data_file('syslog')
-        self.assertNotIn('DSA', out)
-        self.assertNotIn('ECDSA', out)
+        self.assertNotIn('(DSA)', out)
+        self.assertNotIn('(ECDSA)', out)
 
     def test_expected_keys(self):
         """Test expected keys exist."""
         out = self.get_data_file('syslog')
-        self.assertIn('ED25519', out)
-        self.assertIn('RSA', out)
+        self.assertIn('(ED25519)', out)
+        self.assertIn('(RSA)', out)
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/runcmd.yaml b/tests/cloud_tests/testcases/modules/runcmd.yaml
index 04e5a05..8309a88 100644
--- a/tests/cloud_tests/testcases/modules/runcmd.yaml
+++ b/tests/cloud_tests/testcases/modules/runcmd.yaml
@@ -4,10 +4,10 @@
 cloud_config: |
   #cloud-config
   runcmd:
-   - echo cloud-init run cmd test > /tmp/run_cmd
+   - echo cloud-init run cmd test > /var/tmp/run_cmd
 collect_scripts:
   run_cmd: |
     #!/bin/bash
-    cat /tmp/run_cmd
+    cat /var/tmp/run_cmd
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/set_hostname.py b/tests/cloud_tests/testcases/modules/set_hostname.py
index 6e96a75..1dbe64c 100644
--- a/tests/cloud_tests/testcases/modules/set_hostname.py
+++ b/tests/cloud_tests/testcases/modules/set_hostname.py
@@ -7,9 +7,11 @@ from tests.cloud_tests.testcases import base
 class TestHostname(base.CloudTestCase):
     """Test hostname module."""
 
+    ex_hostname = "cloudinit2"
+
     def test_hostname(self):
         """Test hostname command shows correct output."""
         out = self.get_data_file('hostname')
-        self.assertIn('myhostname', out)
+        self.assertIn(self.ex_hostname, out)
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/set_hostname.yaml b/tests/cloud_tests/testcases/modules/set_hostname.yaml
index c96344c..071fb22 100644
--- a/tests/cloud_tests/testcases/modules/set_hostname.yaml
+++ b/tests/cloud_tests/testcases/modules/set_hostname.yaml
@@ -5,7 +5,8 @@ required_features:
   - hostname
 cloud_config: |
   #cloud-config
-  hostname: myhostname
+  hostname: cloudinit2
+
 collect_scripts:
   hosts: |
     #!/bin/bash
diff --git a/tests/cloud_tests/testcases/modules/set_hostname_fqdn.py b/tests/cloud_tests/testcases/modules/set_hostname_fqdn.py
index 398f3d4..08ceae0 100644
--- a/tests/cloud_tests/testcases/modules/set_hostname_fqdn.py
+++ b/tests/cloud_tests/testcases/modules/set_hostname_fqdn.py
@@ -7,20 +7,24 @@ from tests.cloud_tests.testcases import base
 class TestHostnameFqdn(base.CloudTestCase):
     """Test Hostname module."""
 
+    ex_hostname = "cloudinit1"
+    ex_fqdn = "cloudinit2.i9n.brickies.net"
+
     def test_hostname(self):
         """Test hostname output."""
         out = self.get_data_file('hostname')
-        self.assertIn('myhostname', out)
+        self.assertIn(self.ex_hostname, out)
 
     def test_hostname_fqdn(self):
         """Test hostname fqdn output."""
         out = self.get_data_file('fqdn')
-        self.assertIn('host.myorg.com', out)
+        self.assertIn(self.ex_fqdn, out)
 
     def test_hosts(self):
         """Test /etc/hosts file."""
         out = self.get_data_file('hosts')
-        self.assertIn('127.0.1.1 host.myorg.com myhostname', out)
+        self.assertIn('127.0.1.1 %s %s' % (self.ex_fqdn, self.ex_hostname),
+                      out)
         self.assertIn('127.0.0.1 localhost', out)
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/set_hostname_fqdn.yaml b/tests/cloud_tests/testcases/modules/set_hostname_fqdn.yaml
index daf7593..5320ac5 100644
--- a/tests/cloud_tests/testcases/modules/set_hostname_fqdn.yaml
+++ b/tests/cloud_tests/testcases/modules/set_hostname_fqdn.yaml
@@ -6,8 +6,8 @@ required_features:
 cloud_config: |
   #cloud-config
   manage_etc_hosts: true
-  hostname: myhostname
-  fqdn: host.myorg.com
+  hostname: cloudinit1
+  fqdn: cloudinit2.i9n.brickies.net
 collect_scripts:
   hosts: |
     #!/bin/bash
diff --git a/tests/cloud_tests/testcases/modules/set_password_expire.py b/tests/cloud_tests/testcases/modules/set_password_expire.py
index a1c3aa0..967aca7 100644
--- a/tests/cloud_tests/testcases/modules/set_password_expire.py
+++ b/tests/cloud_tests/testcases/modules/set_password_expire.py
@@ -18,6 +18,6 @@ class TestPasswordExpire(base.CloudTestCase):
     def test_sshd_config(self):
         """Test sshd config allows passwords."""
         out = self.get_data_file('sshd_config')
-        self.assertIn('PasswordAuthentication no', out)
+        self.assertIn('PasswordAuthentication yes', out)
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/set_password_expire.yaml b/tests/cloud_tests/testcases/modules/set_password_expire.yaml
index 789604b..ba6344b 100644
--- a/tests/cloud_tests/testcases/modules/set_password_expire.yaml
+++ b/tests/cloud_tests/testcases/modules/set_password_expire.yaml
@@ -6,7 +6,9 @@ required_features:
 cloud_config: |
   #cloud-config
   chpasswd: { expire: True }
+  ssh_pwauth: yes
   users:
+    - default
     - name: tom
       password: $1$xyz$sPMsLNmf66Ohl.ol6JvzE.
       lock_passwd: false
diff --git a/tests/cloud_tests/testcases/modules/set_password_list.yaml b/tests/cloud_tests/testcases/modules/set_password_list.yaml
index a2a89c9..fd3e1e4 100644
--- a/tests/cloud_tests/testcases/modules/set_password_list.yaml
+++ b/tests/cloud_tests/testcases/modules/set_password_list.yaml
@@ -5,6 +5,7 @@ cloud_config: |
   #cloud-config
   ssh_pwauth: yes
   users:
+    - default
     - name: tom
       # md5 gotomgo
       passwd: "$1$S7$tT1BEDIYrczeryDQJfdPe0"
diff --git a/tests/cloud_tests/testcases/modules/set_password_list_string.yaml b/tests/cloud_tests/testcases/modules/set_password_list_string.yaml
index c2a0f63..e9fe54b 100644
--- a/tests/cloud_tests/testcases/modules/set_password_list_string.yaml
+++ b/tests/cloud_tests/testcases/modules/set_password_list_string.yaml
@@ -5,6 +5,7 @@ cloud_config: |
   #cloud-config
   ssh_pwauth: yes
   users:
+    - default
     - name: tom
       # md5 gotomgo
       passwd: "$1$S7$tT1BEDIYrczeryDQJfdPe0"
diff --git a/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_disable.py b/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_disable.py
index 8222321..e7329d4 100644
--- a/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_disable.py
+++ b/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_disable.py
@@ -13,12 +13,4 @@ class TestSshKeyFingerprintsDisable(base.CloudTestCase):
         self.assertIn('Skipping module named ssh-authkey-fingerprints, '
                       'logging of ssh fingerprints disabled', out)
 
-    def test_syslog(self):
-        """Verify output of syslog."""
-        out = self.get_data_file('syslog')
-        self.assertNotRegex(out, r'256 SHA256:.*(ECDSA)')
-        self.assertNotRegex(out, r'256 SHA256:.*(ED25519)')
-        self.assertNotRegex(out, r'1024 SHA256:.*(DSA)')
-        self.assertNotRegex(out, r'2048 SHA256:.*(RSA)')
-
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_disable.yaml b/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_disable.yaml
index 746653e..d93893e 100644
--- a/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_disable.yaml
+++ b/tests/cloud_tests/testcases/modules/ssh_auth_key_fingerprints_disable.yaml
@@ -5,7 +5,6 @@ required_features:
   - syslog
 cloud_config: |
   #cloud-config
-  ssh_genkeytypes: []
   no_ssh_fingerprints: true
 collect_scripts:
   syslog: |
diff --git a/tests/cloud_tests/testcases/modules/ssh_keys_generate.py b/tests/cloud_tests/testcases/modules/ssh_keys_generate.py
index fd6d9ba..b68f556 100644
--- a/tests/cloud_tests/testcases/modules/ssh_keys_generate.py
+++ b/tests/cloud_tests/testcases/modules/ssh_keys_generate.py
@@ -9,11 +9,6 @@ class TestSshKeysGenerate(base.CloudTestCase):
 
     # TODO: Check cloud-init-output for the correct keys being generated
 
-    def test_ubuntu_authorized_keys(self):
-        """Test passed in key is not in list for ubuntu."""
-        out = self.get_data_file('auth_keys_ubuntu')
-        self.assertEqual('', out)
-
     def test_dsa_public(self):
         """Test dsa public key not generated."""
         out = self.get_data_file('dsa_public')
diff --git a/tests/cloud_tests/testcases/modules/ssh_keys_generate.yaml b/tests/cloud_tests/testcases/modules/ssh_keys_generate.yaml
index 659fd93..0a7adf6 100644
--- a/tests/cloud_tests/testcases/modules/ssh_keys_generate.yaml
+++ b/tests/cloud_tests/testcases/modules/ssh_keys_generate.yaml
@@ -10,12 +10,6 @@ cloud_config: |
   - ed25519
   authkey_hash: sha512
 collect_scripts:
-  auth_keys_root: |
-    #!/bin/bash
-    cat /root/.ssh/authorized_keys
-  auth_keys_ubuntu: |
-    #!/bin/bash
-    cat /home/ubuntu/ssh/authorized_keys
   dsa_public: |
     #!/bin/bash
     cat /etc/ssh/ssh_host_dsa_key.pub
diff --git a/tests/cloud_tests/testcases/modules/ssh_keys_provided.py b/tests/cloud_tests/testcases/modules/ssh_keys_provided.py
index 544649d..add3f46 100644
--- a/tests/cloud_tests/testcases/modules/ssh_keys_provided.py
+++ b/tests/cloud_tests/testcases/modules/ssh_keys_provided.py
@@ -7,17 +7,6 @@ from tests.cloud_tests.testcases import base
 class TestSshKeysProvided(base.CloudTestCase):
     """Test ssh keys module."""
 
-    def test_ubuntu_authorized_keys(self):
-        """Test passed in key is not in list for ubuntu."""
-        out = self.get_data_file('auth_keys_ubuntu')
-        self.assertEqual('', out)
-
-    def test_root_authorized_keys(self):
-        """Test passed in key is in authorized list for root."""
-        out = self.get_data_file('auth_keys_root')
-        self.assertIn('lzrkPqONphoZx0LDV86w7RUz1ksDzAdcm0tvmNRFMN1a0frDs50'
-                      '6oA3aWK0oDk4Nmvk8sXGTYYw3iQSkOvDUUlIsqdaO+w==', out)
-
     def test_dsa_public(self):
         """Test dsa public key passed in."""
         out = self.get_data_file('dsa_public')
diff --git a/tests/cloud_tests/testcases/modules/ssh_keys_provided.yaml b/tests/cloud_tests/testcases/modules/ssh_keys_provided.yaml
index 5ceb362..41f6355 100644
--- a/tests/cloud_tests/testcases/modules/ssh_keys_provided.yaml
+++ b/tests/cloud_tests/testcases/modules/ssh_keys_provided.yaml
@@ -71,12 +71,6 @@ cloud_config: |
       -----END EC PRIVATE KEY-----
     ecdsa_public: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFsS5Tvky/IC/dXhE/afxxUG6kdQOvdQJCYGZN42OZqWasYF+L3IG+3/wrV7jOrNrL3AyagHl6+lpPDiSXDMcpQ= root@xenial-lxd
 collect_scripts:
-  auth_keys_root: |
-    #!/bin/bash
-    cat /root/.ssh/authorized_keys
-  auth_keys_ubuntu: |
-    #!/bin/bash
-    cat /home/ubuntu/ssh/authorized_keys
   dsa_public: |
     #!/bin/bash
     cat /etc/ssh/ssh_host_dsa_key.pub
diff --git a/tests/cloud_tests/util.py b/tests/cloud_tests/util.py
index 4357fbb..92c31c3 100644
--- a/tests/cloud_tests/util.py
+++ b/tests/cloud_tests/util.py
@@ -7,6 +7,7 @@ import copy
 import glob
 import os
 import random
+import shlex
 import shutil
 import string
 import subprocess
@@ -285,20 +286,165 @@ def shell_pack(cmd):
     return 'eval set -- "$(echo %s | base64 --decode)" && exec "$@"' % b64
 
 
+def shell_quote(cmd):
+    if isinstance(cmd, (tuple, list)):
+        return ' '.join([shlex.quote(x) for x in cmd])
+    return shlex.quote(cmd)
+
+
+class TargetBase(object):
+    _tmp_count = 0
+
+    def execute(self, command, stdin=None, env=None,
+                rcs=None, description=None):
+        """Execute command in instance, recording output, error and exit code.
+
+        Assumes functional networking and execution as root with the
+        target filesystem being available at /.
+
+        @param command: the command to execute as root inside the image
+            if command is a string, then it will be executed as:
+            ['sh', '-c', command]
+        @param stdin: bytes content for standard in
+        @param env: environment variables
+        @param rcs: return codes.
+                    None (default): non-zero exit code will raise exception.
+                    False: any is allowed (No execption raised).
+                    list of int: any rc not in the list will raise exception.
+        @param description: purpose of command
+        @return_value: tuple containing stdout data, stderr data, exit code
+        """
+        if isinstance(command, str):
+            command = ['sh', '-c', command]
+
+        if rcs is None:
+            rcs = (0,)
+
+        if description:
+            LOG.debug('Executing "%s"', description)
+        else:
+            LOG.debug("Executing command: %s", shell_quote(command))
+
+        out, err, rc = self._execute(command=command, stdin=stdin, env=env)
+
+        # False means accept anything.
+        if (rcs is False or rc in rcs):
+            return out, err, rc
+
+        raise InTargetExecuteError(out, err, rc, command, description)
+
+    def _execute(self, command, stdin=None, env=None):
+        """Execute command in inside, return stdout, stderr and exit code.
+
+        Assumes functional networking and execution as root with the
+        target filesystem being available at /.
+
+        @param stdin: bytes content for standard in
+        @param env: environment variables
+        @return_value: tuple containing stdout data, stderr data, exit code
+
+        This is intended to be implemented by the Image or Instance.
+        Many callers will use the higher level 'execute'."""
+        raise NotImplementedError
+
+    def read_data(self, remote_path, decode=False):
+        """Read data from instance filesystem.
+
+        @param remote_path: path in instance
+        @param decode: decode data before returning.
+        @return_value: content of remote_path as bytes if 'decode' is False,
+                       and as string if 'decode' is True.
+        """
+        # when sh is invoked with '-c', then the first argument is "$0"
+        # which is commonly understood as the "program name".
+        # 'read_data' is the program name, and 'remote_path' is '$1'
+        stdout, stderr, rc = self._execute(
+            ["sh", "-c", 'exec cat "$1"', 'read_data', remote_path])
+        if rc != 0:
+            raise RuntimeError("Failed to read file '%s'" % remote_path)
+
+        if decode:
+            return stdout.decode()
+        return stdout
+
+    def write_data(self, remote_path, data):
+        """Write data to instance filesystem.
+
+        @param remote_path: path in instance
+        @param data: data to write in bytes
+        """
+        # when sh is invoked with '-c', then the first argument is "$0"
+        # which is commonly understood as the "program name".
+        # 'write_data' is the program name, and 'remote_path' is '$1'
+        _, _, rc = self._execute(
+            ["sh", "-c", 'exec cat >"$1"', 'write_data', remote_path],
+            stdin=data)
+
+        if rc != 0:
+            raise RuntimeError("Failed to write to '%s'" % remote_path)
+        return
+
+    def pull_file(self, remote_path, local_path):
+        """Copy file at 'remote_path', from instance to 'local_path'.
+
+        @param remote_path: path on remote instance
+        @param local_path: path on local instance
+        """
+        with open(local_path, 'wb') as fp:
+            fp.write(self.read_data(remote_path))
+
+    def push_file(self, local_path, remote_path):
+        """Copy file at 'local_path' to instance at 'remote_path'.
+
+        @param local_path: path on local instance
+        @param remote_path: path on remote instance"""
+        with open(local_path, "rb") as fp:
+            self.write_data(remote_path, data=fp.read())
+
+    def run_script(self, script, rcs=None, description=None):
+        """Run script in target and return stdout.
+
+        @param script: script contents
+        @param rcs: allowed return codes from script
+        @param description: purpose of script
+        @return_value: stdout from script
+        """
+        # Just write to a file, add execute, run it, then remove it.
+        shblob = '; '.join((
+            'set -e',
+            's="$1"',
+            'shift',
+            'cat > "$s"',
+            'trap "rm -f $s" EXIT',
+            'chmod +x "$s"',
+            '"$s" "$@"'))
+        return self.execute(
+            ['sh', '-c', shblob, 'runscript', self.tmpfile()],
+            stdin=script, description=description, rcs=rcs)
+
+    def tmpfile(self):
+        """Get a tmp file in the target.
+
+        @return_value: path to new file in target
+        """
+        path = "/tmp/%s-%04d" % (type(self).__name__, self._tmp_count)
+        self._tmp_count += 1
+        return path
+
+
 class InTargetExecuteError(c_util.ProcessExecutionError):
     """Error type for in target commands that fail."""
 
-    default_desc = 'Unexpected error while running command in target instance'
+    default_desc = 'Unexpected error while running command.'
 
-    def __init__(self, stdout, stderr, exit_code, cmd, instance,
-                 description=None):
+    def __init__(self, stdout, stderr, exit_code, cmd, description=None,
+                 reason=None):
         """Init error and parent error class."""
-        if isinstance(cmd, (tuple, list)):
-            cmd = ' '.join(cmd)
         super(InTargetExecuteError, self).__init__(
-            stdout=stdout, stderr=stderr, exit_code=exit_code, cmd=cmd,
-            reason="Instance: {}".format(instance),
-            description=description if description else self.default_desc)
+            stdout=stdout, stderr=stderr, exit_code=exit_code,
+            cmd=shell_quote(cmd),
+            description=description if description else self.default_desc,
+            reason=reason)
 
 
 class TempDir(object):