Merge ~smoser/cloud-init:fix/test-run_tree-fixes into cloud-init:master

[Scott Moser]

tests: fix run_tree and bddeb

This was broken probably when we inserted the ssh keys into Platform.
  tox -e citest tree_run
and
  tox -e citest bddeb

would fail with KeyError in Platform.init due to lack of a data_dir.

Also here are a few fixes found from attempting to make it work.

see commit message

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:2abbfe1faef809a56cbfeb6f5f5a50300e6e66c4
https://jenkins.ubuntu.com/server/job/cloud-init-ci/823/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/823/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:27fd0d9a7ff489b4db0b7e7747ae6abaf3502f21
https://jenkins.ubuntu.com/server/job/cloud-init-ci/824/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/824/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

LGTM

review: Approve

[Joshua Powers (powersj)]

LGTM thanks

Tried using ec2 + tree_run

review: Approve

[do3meli (d-info-e)]

i tested the run_tree call locally with ubuntu 16 + lxd and it seems to work again with this patch.

as this seems to be approved now can it be merged to prod to allow easier integration testing on local machines?

thanks

[Scott Moser (smoser)]

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=2f5d4ceb

diff --git a/tests/cloud_tests/bddeb.py b/tests/cloud_tests/bddeb.py
index a6d5069..b9cfcfa 100644
--- a/tests/cloud_tests/bddeb.py
+++ b/tests/cloud_tests/bddeb.py
@@ -16,7 +16,7 @@ pre_reqs = ['devscripts', 'equivs', 'git', 'tar']
 
 def _out(cmd_res):
     """Get clean output from cmd result."""
-    return cmd_res[0].strip()
+    return cmd_res[0].decode("utf-8").strip()
 
 
 def build_deb(args, instance):
diff --git a/tests/cloud_tests/platforms/platforms.py b/tests/cloud_tests/platforms/platforms.py
index 1542b3b..abbfebb 100644
--- a/tests/cloud_tests/platforms/platforms.py
+++ b/tests/cloud_tests/platforms/platforms.py
@@ -2,12 +2,15 @@
 
 """Base platform class."""
 import os
+import shutil
 
 from simplestreams import filters, mirrors
 from simplestreams import util as s_util
 
 from cloudinit import util as c_util
 
+from tests.cloud_tests import util
+
 
 class Platform(object):
     """Base class for platforms."""
@@ -17,7 +20,14 @@ class Platform(object):
     def __init__(self, config):
         """Set up platform."""
         self.config = config
-        self._generate_ssh_keys(config['data_dir'])
+        self.tmpdir = util.mkdtemp()
+        if 'data_dir' in config:
+            self.data_dir = config['data_dir']
+        else:
+            self.data_dir = os.path.join(self.tmpdir, "data_dir")
+            os.mkdir(self.data_dir)
+
+        self._generate_ssh_keys(self.data_dir)
 
     def get_image(self, img_conf):
         """Get image using specified image configuration.
@@ -29,7 +39,7 @@ class Platform(object):
 
     def destroy(self):
         """Clean up platform data."""
-        pass
+        shutil.rmtree(self.tmpdir)
 
     def _generate_ssh_keys(self, data_dir):
         """Generate SSH keys to be used with image."""
diff --git a/tests/cloud_tests/util.py b/tests/cloud_tests/util.py
index 6ff285e..3dd4996 100644
--- a/tests/cloud_tests/util.py
+++ b/tests/cloud_tests/util.py
@@ -460,6 +460,10 @@ class PlatformError(IOError):
         IOError.__init__(self, message)
 
 
+def mkdtemp(prefix='cloud_test_data'):
+    return tempfile.mkdtemp(prefix=prefix)
+
+
 class TempDir(object):
     """Configurable temporary directory like tempfile.TemporaryDirectory."""
 
@@ -480,7 +484,7 @@ class TempDir(object):
         @return_value: tempdir path
         """
         if not self.tmpdir:
-            self.tmpdir = tempfile.mkdtemp(prefix=self.prefix)
+            self.tmpdir = mkdtemp(prefix=self.prefix)
         LOG.debug('using tmpdir: %s', self.tmpdir)
         return self.tmpdir