Merge ~wesley-wiedenmeier/cloud-init:integration-testing-invocation-cleanup into cloud-init:master

[Wesley Wiedenmeier]

Integration Testing: provide commands to run tests from current tree

 - Add 'bddeb' command to build a deb from the current working tree
   cleanly in a container, so deps don't have to be installed on host
 - Add 'tree_collect' and 'tree_run' commands, to build deb from
   current working tree and call collect or run with deb
 - Fix bug in instance.pull_file()

[Joshua Powers (powersj)]

LGTM with two comments:

1) Had to increase the timeout as 120 seconds was not enough. I bumped to 600.
2) the tox target "cittest_run" actually does a "tree_run" In order to not be confused between run versus tree_run I think this should really be "cittest_tree_run"

Tested by running the following on my local system with pylxd 2.1.3:

# lint
$ tox
# runs with options
$ python3 -m tests.cloud_tests run -v -n trusty
$ python3 -m tests.cloud_tests run -v -n xenial
$ python3 -m tests.cloud_tests run -v -n yakkety --repo 'deb http://archive.ubuntu.com/ubuntu/ yakkety main'
$ python3 -m tests.cloud_tests run -v -n xenial --deb cloud-init_0.7.9-0ubuntu1_all.deb
# new tox based runs + tree_run
$ tox -e citest_run
$ tox -e citest -- tree_run -n zesty -t tests/cloud_tests/configs/modules/write_files.yaml
$ python3 -m tests.cloud_tests tree_run -v -n zesty -t tests/cloud_tests/configs/modules/timezone.yaml

lxc list was clean afterwards as expected. I believe I only missed a ppa test.

The only warning message I got other than those for pylxd's 2.2 deprecation was:
/home/powersj/Work/repos/cloud-init-wesley/tests/cloud_tests/collect.py:48: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

review: Approve

[Wesley Wiedenmeier (wesley-wiedenmeier)]

Thanks for reviewing

> 1) Had to increase the timeout as 120 seconds was not enough. I bumped to 600.
I'll go ahead and increase the timout a bit, or possibly add the option for the call to snapshot.launch() in bddeb to specify a longer timeout that normal, since the bddeb instance should be expected to take longer to boot than other instances. It would also work to not use cloud-init to install devscripts, and just apt-get it during the build process, that may be cleaner, and would allow finer grained logging messages.

> 2) the tox target "cittest_run" actually does a "tree_run" In order to not be
> confused between run versus tree_run I think this should really be
> "cittest_tree_run"
Good point, citest_run = tree_run may be confusing, I'll get the name switched over.

> lxc list was clean afterwards as expected. I believe I only missed a ppa test.
I think that is plenty of testing as the changes shouldn't have affected anything other than bddeb and adding the new commands. I'll run through with --ppa once to be sure though.

> The only warning message I got other than those for pylxd's 2.2 deprecation
> was:
> /home/powersj/Work/repos/cloud-init-wesley/tests/cloud_tests/collect.py:48:
> DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
I always forget whether its LOG.warn or LOG.warning that is depricated, got it wrong here, good catch. I'll fix here and grep through to check if its wrong anywhere else.

I should have all of that done at some point tonight, should be ready for merge tomorrow

[Wesley Wiedenmeier (wesley-wiedenmeier)]

Timeouts will no longer be an issue for bddeb as dep installation is now handled after boot. I think this should be ready to merge, I've tested again with tree_collect, tree_run, and just with a normal 'run --ppa'

[Joshua Powers (powersj)]

LGTM

Thanks for the changes! Ran the following successfully with no changes to timeouts:

$ tox
$ tox -e citest_tree_run
$ tox -e citest -- tree_run -n zesty -t tests/cloud_tests/configs/modules/write_files.yaml
$ python3 -m tests.cloud_tests tree_run -v -n zesty -t tests/cloud_tests/configs/modules/timezone.yaml

review: Approve

[Wesley Wiedenmeier (wesley-wiedenmeier)]

This merge proposal is no longer required, as everything here is included in the main one at:
https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/308218

[Wesley Wiedenmeier (wesley-wiedenmeier)]

Since there is additional work that can be done to improve the build system, this branch and this merge proposal will be used for development. Below is some of the discussion on bddeb for reference.

> >> * creating the tarball..
> >> I think what we want to do here is:
> >> a.) make-tarball on "this side"
> >> b.) modify bddeb to be able to just create a debian/ dir
> >> either
> >> i.) have it create the orig tarball and a debian.tar.gz
> >> into a provided (empty) output dir.
> >> ii.) have it just create the debian.tar.gz and we create
> >> the tarball example of doing that
> >> http://paste.ubuntu.com/23870964/
> >> c.) move the tarball and debian dir over, extract tarball,
> >> extract debian/
> >> d.) mk-build-deps --install .... debian/control
> >> e.) dpkg-buildpackage
> >>
> >> doing this should mean we do not have to 'install additional
> >> build deps' but rather just have mk-build-deps do the right
> >> thing.
>
> >I had been trying to avoid modifying anything outside cloud_tests,
> >but adding the --debian-tar-only option in bddeb definitely does
> >simplify this. This may be a useful feature in general for debugging
> >cloud-init builds that do not work or doing custom deb builds, as
> >going through the template system can be inconvenient.
>
> >This also avoids having to go through a source repo in case the
> >current version of cloud-init in tree has different deps from the
> >one in the source repo.
>
> > Using dpkg-buildpackage instead of debuild may be quicker too.
>
> Yeah, it makes the build much simpler in the container. I'm tempted
> to further improve bddeb to list what packages need installing even,
> then we could drop the mk-build-deps and equivs ... but that is not
> necessary now. The complexity in that comes really from the obnoxious
> 'python3-pyflakes | pyflakes (<< 1.1.0-2)'
>
> in package/debian/control.in

Yeah, I kinda like the idea of bddeb to handle build deps entirely. That may
make things simpler for users who want to build as well since there have been
lots of questions on irc about how to build packages. We should probably add
more documentation on cloud-inits build system at some point as well.

> >> * when building with dpkg-buildpackage or debuild, good to
> >> allow setting DEB_BUILD_OPTIONS=nocheck (which will not run pep8 and
> >> nosetests)
>
> >Nice, I didn't know that was an option. Lets us avoid the additional
> >build deps step easily, since that was mostly to avoid failing because
> >testing stuff wasn't there.
>
> There is info at https://bugs.launchpad.net/cloud-init/+bug/1652329
> on what the plan here is... ultimately, package build wont run pep8, but
> would still run nosetests and useful to be able to not run them.

I do like the idea of dropping pep8 in the build, since the actual source used
for the builds whill have already been checked.

> >> * i think the above path means we do not depend on deb-src lines.
>
> >The switch back to ubuntu daily will handle that, but this also
> >avoids going through source repo at all which can be nice if a local
> >change adds a new build dep.
>
> And simply use of mk-build-deps would too..
> I know i'm kind of over-doing this, but i really lik...

[Wesley Wiedenmeier (wesley-wiedenmeier)]

This branch should mostly be ready to merge in now. There is additonal work to be done on getting the cc_apt_configure issue resolved in debian jessie, and the hostnamectl issue resolved on centos70, but that can wait, as these are most likely not issues caused by the test suite. Support for package management on centos still needs to be switched to dnf, but since centos is blocked on the hostnamectl issue, that can wait as well.

The cleanup to system_ready_scripts has been done, and the ubuntu release config is back to using the ubuntu daily images. Ubuntu releases xenial -> zesty have been run with a couple of test configs successfully, both installing from ppa:cloud-init-dev/daily and from --deb with a deb of 0.7.9, and trusty has been run successfully with the version of cloud-init it ships with on modules/final_message. Additional testing of this merge may be good, but the tests currently being run by jenkins should work.

[Wesley Wiedenmeier (wesley-wiedenmeier)]

This branch is not yet ready to merge. The above comment was meant for the main merge proposal at https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/308218

The build system and the bddeb functionality for cloud-tests can be improved further.

[Wesley Wiedenmeier (wesley-wiedenmeier)]

This branch should be ready to merge into trunk now. There were some additional improvements to the build system that had been discussed, but it may be best to work on those later. If the build system is improved in the future, then this code can easily be updated as well.

The tree_run command that this branch provides is useful enough on its own that this is worth merging in on its own, although this is a lower priority than either the main integration-testing update or the distro feature flags branch that is being worked on right now.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:ac847078d33be75a7b74390e13f9b3144f75f604
https://jenkins.ubuntu.com/server/job/cloud-init-ci/119/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/119
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/119
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/119
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/119
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/119

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/119/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:e63726f671ac88aabf83992e05a187a05d8c7431
https://jenkins.ubuntu.com/server/job/cloud-init-ci/133/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/133
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/133
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/133
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/133
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/133

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/133/rebuild

review: Approve (continuous-integration)

51fa870... by Wesley Wiedenmeier on 2017-03-19

Integration Testing: Updated documentation

 - Added documentation on the tree_run and tree_collect commands.
 - Also made minor documentation updates for additional useful
   features in the test suite.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:51fa870b721f0c43f570342c7799d783c1d65a5e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/137/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/137
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/137
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/137
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/137
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/137

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/137/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

I'm going to mark this 'work in progress'. Josh has integrated this branch and others into his merge proposal at https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324136 . We do plan to pull that all soon.

Thanks for your work, Wesley!

[Scott Moser (smoser)]

Hi.
I've marked this 'merged' as I think it is now in trunk under 76d58265e34851b78e952a7f275340863c90a9f5.
If you disagree, please feel free to re-open.

diff --git a/doc/rtd/topics/tests.rst b/doc/rtd/topics/tests.rst
index 0663811..a088e64 100644
--- a/doc/rtd/topics/tests.rst
+++ b/doc/rtd/topics/tests.rst
@@ -47,20 +47,23 @@ The test configuration is a YAML file such as *ntp_server.yaml* below:
         cat /etc/ntp.conf | grep '^server'
 
 
-There are two keys, 1 required and 1 optional, in the YAML file:
+There are several keys, 1 required and some optional, in the YAML file:
 
 1. The required key is ``cloud_config``. This should be a string of valid
    YAML that is exactly what would normally be placed in a cloud-config file,
    including the cloud-config header. This essentially sets up the scenario
    under test.
 
-2. The optional key is ``collect_scripts``. This key has one or more
+2. One optional key is ``collect_scripts``. This key has one or more
    sub-keys containing strings of arbitrary commands to execute (e.g.
    ```cat /var/log/cloud-config-output.log```). In the example above the
    output of dpkg is captured, grep for ntp, and the number of lines
    reported. The name of the sub-key is important. The sub-key is used by
    the verification script to recall the output of the commands ran.
 
+3. The optinal key ``enabled`` enables or disables the test case. By defaul
+   the test case will be enabled.
+
 Default Collect Scripts
 -----------------------
 
@@ -109,15 +112,25 @@ Here is a breakdown of the unit test file:
 
 * The import statement allows access to the output files.
 
-* The class can be named anything, but must import the ``base.CloudTestCase``
+* The class can be named anything, but must import the ``base.CloudTestCase``,
+  either directly or via another test class.
 
 * There can be 1 to N number of functions with any name, however only
   tests starting with ``test_*`` will be executed.
 
+* There can be 1 to N number of classes in a test module, however only classes
+  inheriting from ``base.CloudTestCase`` will be loaded.
+
 * Output from the commands can be accessed via
   ``self.get_data_file('key')`` where key is the sub-key of
   ``collect_scripts`` above.
 
+* The cloud config that the test ran with can be accessed via
+  ``self.cloud_config``, or any entry from the cloud config can be accessed via
+  ``self.get_config_entry('key')``.
+
+* See the base ``CloudTestCase`` for additional helper functions.
+
 Layout
 ======
 
@@ -144,6 +157,23 @@ The sub-folders of bugs, examples, main, and modules help organize the
 tests. View the README.md in each to understand in more detail each
 directory.
 
+Test Creation Helper
+====================
+
+The integration testing suite has a built in helper to aid in test development.
+Its help can be invoked via ``python3 -m tests.cloud_tests create --help``. It
+can create a template testcase config file with user data passed in from the
+command line, as well as a template testcase verifier module.
+
+The following would create a testcase named ``example`` under the ``modules``
+category with the given description, and cloud config data read in from
+``/tmp/user_data``.
+
+.. code-block:: bash
+
+    $ python3 -m tests.cloud_tests create modules/example \
+        -d "a simple example test case" -c "$(< /tmp/user_data)"
+
 
 Development Checklist
 =====================
@@ -171,7 +201,7 @@ Development Checklist
 Execution
 =========
 
-Executing tests has three options:
+Executing tests has several options:
 
 * ``run`` an alias to run both ``collect`` and ``verify``
 
@@ -182,6 +212,10 @@ Executing tests has three options:
 * ``verify`` given a directory of test data, run the Python unit tests on
   it to generate results.
 
+* ``tree_run`` and ``tree_collect`` are helpers to run ``run`` or ``collect``
+  respectively using a deb built from the copy of cloud-init in the current
+  working tree.
+
 Run
 ---
 The first example will provide a complete end-to-end run of data
@@ -238,6 +272,29 @@ without the more lengthy collect process. This can be done by running:
 The above command will run the verify scripts on the data discovered in
 `/tmp/collection`.
 
+TreeRun and TreeCollect
+-----------------------
+
+If working on a cloud-init feature or resolving a bug, it may be useful to run
+the current copy of cloud-init in the integration testing environment. The
+integration testing suite can automatically build a deb based on the current
+working tree of cloud-init and run the test suite using this deb.
+
+The ``tree_run`` and ``tree_collect`` commands take the same arguments as the
+``run`` and ``collect`` commands. These commands will build a deb and write it
+into a tempfile, then start the test suite and pass that deb in. To build a
+deb only, and not run the test suite, the ``bddeb`` command can be used.
+
+Note that code in the cloud-init working tree that has not been committed
+when the cloud-init deb is built will still be included. To build a cloud-init
+deb from or use the ``tree_run`` command using a copy of cloud-init located in
+a different directory, use the option ``--cloud-init /path/to/cloud-init``.
+
+.. code-block:: bash
+
+    $ python3 -m tests.cloud_tests tree_run -n xenial -n zesty \
+        -t modules/final_message -t modules/write_files -v -r /tmp/result.yaml
+
 Run via tox
 -----------
 In order to avoid the need for dependencies and ease the setup and
@@ -256,11 +313,13 @@ arguments.
 Architecture
 ============
 
-The following outlines the process flow during a complete end-to-end LXD-backed test.
+The following outlines the process flow during a complete end-to-end
+LXD-backed test.
 
 1. Configuration
     * The back end and specific OS releases are verified as supported
-    * The test or tests that need to be run are determined either by directory or by individual yaml
+    * The test or tests that need to be run are determined either by
+      directory or by individual yaml
 
 2. Image Creation
     * Acquire the daily LXD image
@@ -285,5 +344,5 @@ The following outlines the process flow during a complete end-to-end LXD-backed 
 
 5. Results
     * If any failures were detected the test suite returns a failure
-
-
+    * Results can be dumped in yaml format to a specified file using the
+      ``-r <result_file_name>.yaml`` option
diff --git a/tests/cloud_tests/__init__.py b/tests/cloud_tests/__init__.py
index 099c357..7959bd9 100644
--- a/tests/cloud_tests/__init__.py
+++ b/tests/cloud_tests/__init__.py
@@ -6,6 +6,7 @@ import os
 BASE_DIR = os.path.dirname(os.path.abspath(__file__))
 TESTCASES_DIR = os.path.join(BASE_DIR, 'testcases')
 TEST_CONF_DIR = os.path.join(BASE_DIR, 'configs')
+TREE_BASE = os.sep.join(BASE_DIR.split(os.sep)[:-2])
 
 
 def _initialize_logging():
diff --git a/tests/cloud_tests/__main__.py b/tests/cloud_tests/__main__.py
index ef7d187..74f29f6 100644
--- a/tests/cloud_tests/__main__.py
+++ b/tests/cloud_tests/__main__.py
@@ -2,11 +2,9 @@
 
 import argparse
 import logging
-import shutil
 import sys
-import tempfile
 
-from tests.cloud_tests import (args, collect, manage, verify)
+from tests.cloud_tests import args, bddeb, collect, manage, run_funcs, verify
 from tests.cloud_tests import LOG
 
 
@@ -22,28 +20,6 @@ def configure_log(args):
     LOG.setLevel(level)
 
 
-def run(args):
-    """
-    run full test suite
-    """
-    failed = 0
-    args.data_dir = tempfile.mkdtemp(prefix='cloud_test_data_')
-    LOG.debug('using tmpdir %s', args.data_dir)
-    try:
-        failed += collect.collect(args)
-        failed += verify.verify(args)
-    except Exception:
-        failed += 1
-        raise
-    finally:
-        # TODO: make this configurable via environ or cmdline
-        if failed:
-            LOG.warn('some tests failed, leaving data in %s', args.data_dir)
-        else:
-            shutil.rmtree(args.data_dir)
-    return failed
-
-
 def main():
     """
     entry point for cloud test suite
@@ -80,9 +56,12 @@ def main():
     # run handler
     LOG.debug('running with args: %s\n', parsed)
     return {
+        'bddeb': bddeb.bddeb,
         'collect': collect.collect,
         'create': manage.create,
-        'run': run,
+        'run': run_funcs.run,
+        'tree_collect': run_funcs.tree_collect,
+        'tree_run': run_funcs.tree_run,
         'verify': verify.verify,
     }[parsed.subcmd](parsed)
 
diff --git a/tests/cloud_tests/args.py b/tests/cloud_tests/args.py
index b68cc98..d051737 100644
--- a/tests/cloud_tests/args.py
+++ b/tests/cloud_tests/args.py
@@ -3,9 +3,24 @@
 import os
 
 from tests.cloud_tests import config, util
-from tests.cloud_tests import LOG
+from tests.cloud_tests import LOG, TREE_BASE
 
 ARG_SETS = {
+    'BDDEB': (
+        (('--bddeb-args',),
+         {'help': 'args to pass through to bddeb',
+          'action': 'store', 'default': None, 'required': False}),
+        (('--build-os',),
+         {'help': 'OS to use as build system (default is xenial)',
+          'action': 'store', 'choices': config.list_enabled_distros(),
+          'default': 'xenial', 'required': False}),
+        (('--build-platform',),
+         {'help': 'platform to use for build system (default is lxd)',
+          'action': 'store', 'choices': config.list_enabled_platforms(),
+          'default': 'lxd', 'required': False}),
+        (('--cloud-init',),
+         {'help': 'path to base of cloud-init tree', 'metavar': 'DIR',
+          'action': 'store', 'required': False, 'default': TREE_BASE}),),
     'COLLECT': (
         (('-p', '--platform'),
          {'help': 'platform(s) to run tests on', 'metavar': 'PLATFORM',
@@ -42,6 +57,10 @@ ARG_SETS = {
         (('-d', '--data-dir'),
          {'help': 'directory to store test data in',
           'action': 'store', 'metavar': 'DIR', 'required': True}),),
+    'OUTPUT_DEB': (
+        (('--deb',),
+         {'help': 'path to write output deb to', 'metavar': 'FILE',
+          'action': 'store', 'required': True}),),
     'RESULT': (
         (('-r', '--result'),
          {'help': 'file to write results to',
@@ -66,10 +85,16 @@ ARG_SETS = {
 }
 
 SUBCMDS = {
+    'bddeb': ('build cloud-init deb from tree',
+              ('BDDEB', 'OUTPUT_DEB', 'INTERFACE')),
     'collect': ('collect test data',
                 ('COLLECT', 'INTERFACE', 'OUTPUT', 'RESULT', 'SETUP')),
     'create': ('create new test case', ('CREATE', 'INTERFACE')),
     'run': ('run test suite', ('COLLECT', 'INTERFACE', 'RESULT', 'SETUP')),
+    'tree_collect': ('collect using current working tree',
+                     ('BDDEB', 'COLLECT', 'INTERFACE', 'OUTPUT', 'RESULT')),
+    'tree_run': ('run using current working tree',
+                 ('BDDEB', 'COLLECT', 'INTERFACE', 'RESULT')),
     'verify': ('verify test data', ('INTERFACE', 'OUTPUT', 'RESULT')),
 }
 
@@ -81,6 +106,20 @@ def _empty_normalizer(args):
     return args
 
 
+def normalize_bddeb_args(args):
+    """
+    normalize BDDEB arguments
+    args: parsed args
+    return_value: updated args, or None if errors encountered
+    """
+    # make sure cloud-init dir is accessible
+    if not (args.cloud_init and os.path.isdir(args.cloud_init)):
+        LOG.error('invalid cloud-init tree path')
+        return None
+
+    return args
+
+
 def normalize_create_args(args):
     """
     normalize CREATE arguments
@@ -185,6 +224,17 @@ def normalize_output_args(args):
     return args
 
 
+def normalize_output_deb_args(args):
+    """
+    normalize OUTPUT_DEB arguments
+    args: parsed args
+    return_value: updated args, or None if erros occurred
+    """
+    # make sure to use abspath for deb
+    args.deb = os.path.abspath(args.deb)
+    return args
+
+
 def normalize_setup_args(args):
     """
     normalize SETUP arguments
@@ -210,10 +260,12 @@ def normalize_setup_args(args):
 
 
 NORMALIZERS = {
+    'BDDEB': normalize_bddeb_args,
     'COLLECT': normalize_collect_args,
     'CREATE': normalize_create_args,
     'INTERFACE': _empty_normalizer,
     'OUTPUT': normalize_output_args,
+    'OUTPUT_DEB': normalize_output_deb_args,
     'RESULT': _empty_normalizer,
     'SETUP': normalize_setup_args,
 }
diff --git a/tests/cloud_tests/bddeb.py b/tests/cloud_tests/bddeb.py
new file mode 100644
index 0000000..3b79cd2
--- /dev/null
+++ b/tests/cloud_tests/bddeb.py
@@ -0,0 +1,124 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+from tests.cloud_tests import (config, LOG)
+from tests.cloud_tests import (platforms, images, snapshots, instances)
+from tests.cloud_tests.stage import (PlatformComponent, run_stage, run_single)
+
+from cloudinit import util as c_util
+
+from functools import partial
+import os
+
+build_deps = ['devscripts', 'equivs', 'git', 'tar']
+
+
+def _out(cmd_res):
+    """
+    get clean output from cmd result
+    """
+    return cmd_res[0].strip()
+
+
+def build_deb(args, instance):
+    """
+    build deb on system and copy out to location at args.deb
+    args: cmdline arguments
+    return_value: tuple of results and fail count
+    """
+    # update remote system package list and install build deps
+    LOG.debug('installing build deps')
+    pkgs = ' '.join(build_deps)
+    cmd = 'apt-get update && apt-get install --yes {}'.format(pkgs)
+    instance.execute(['/bin/sh', '-c', cmd])
+    instance.execute(['mk-build-deps', '--install', '-t',
+                      'apt-get --no-install-recommends --yes', 'cloud-init'])
+
+    # local tmpfile that must be deleted
+    local_tarball = _out(c_util.subp(['mktemp'], capture=True))
+
+    try:
+        # paths to use in remote system
+        remote_tarball = _out(instance.execute(['mktemp']))
+        extract_dir = _out(instance.execute(['mktemp', '--directory']))
+        bddeb_path = os.path.join(extract_dir, 'packages', 'bddeb')
+        output_link = '/cloud-init_all.deb'
+        git_env = {'GIT_DIR': os.path.join(extract_dir, '.git'),
+                   'GIT_WORK_TREE': extract_dir}
+
+        # create a tarball of cloud init tree and copy to remote system
+        LOG.debug('creating tarball of cloud-init at: %s', local_tarball)
+        c_util.subp(['tar', 'cf', local_tarball, '--owner', 'root',
+                     '--group', 'root', '-C', args.cloud_init, '.'])
+        LOG.debug('copying to remote system at: %s', remote_tarball)
+        instance.push_file(local_tarball, remote_tarball)
+
+        # extract tarball in remote system and commit anything uncommitted
+        LOG.debug('extracting tarball in remote system at: %s', extract_dir)
+        instance.execute(['tar', 'xf', remote_tarball, '-C', extract_dir])
+        instance.execute(['git', 'commit', '-a', '-m', 'tmp', '--allow-empty'],
+                         env=git_env)
+
+        # build the deb, ignoring missing deps (flake8)
+        LOG.debug('building deb in remote system at: %s', output_link)
+        bddeb_args = args.bddeb_args.split() if args.bddeb_args else []
+        instance.execute([bddeb_path, '-d'] + bddeb_args, env=git_env)
+
+        # copy the deb back to the host system
+        LOG.debug('copying built deb to host at: %s', args.deb)
+        instance.pull_file(output_link, args.deb)
+
+    finally:
+        os.remove(local_tarball)
+
+
+def setup_build(args):
+    """
+    set build system up then run build
+    args: cmdline arguments
+    return_value: tuple of results and fail count
+    """
+    res = ({}, 1)
+
+    # set up platform
+    LOG.info('setting up platform: %s', args.build_platform)
+    platform_config = config.load_platform_config(args.build_platform)
+    platform_call = partial(platforms.get_platform, args.build_platform,
+                            platform_config)
+    with PlatformComponent(platform_call) as platform:
+
+        # set up image
+        LOG.info('acquiring image for os: %s', args.build_os)
+        img_conf = config.load_os_config(args.build_os)
+        image_call = partial(images.get_image, platform, img_conf)
+        with PlatformComponent(image_call) as image:
+
+            # set up snapshot
+            snapshot_call = partial(snapshots.get_snapshot, image)
+            with PlatformComponent(snapshot_call) as snapshot:
+
+                # create instance with cloud-config to set it up
+                LOG.info('creating instance to build deb in')
+                empty_cloud_config = "#cloud-config\n{}"
+                instance_call = partial(
+                    instances.get_instance, snapshot, empty_cloud_config,
+                    use_desc='build cloud-init deb')
+                with PlatformComponent(instance_call) as instance:
+
+                    # build the deb
+                    res = run_single('build deb on system',
+                                     partial(build_deb, args, instance))
+
+    return res
+
+
+def bddeb(args):
+    """
+    entry point for build deb
+    args: cmdline arguments
+    return_value: fail count
+    """
+    LOG.info('preparing to build cloud-init deb')
+    (res, failed) = run_stage('build deb', [partial(setup_build, args)])
+    return failed
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/base.py b/tests/cloud_tests/instances/base.py
index 9559d28..1782c1d 100644
--- a/tests/cloud_tests/instances/base.py
+++ b/tests/cloud_tests/instances/base.py
@@ -51,7 +51,7 @@ class Instance(object):
         copy file at 'remote_path', from instance to 'local_path'
         """
         with open(local_path, 'wb') as fp:
-            fp.write(self.read_data(remote_path), encode=True)
+            fp.write(self.read_data(remote_path))
 
     def push_file(self, local_path, remote_path):
         """
diff --git a/tests/cloud_tests/run_funcs.py b/tests/cloud_tests/run_funcs.py
new file mode 100644
index 0000000..683a3f6
--- /dev/null
+++ b/tests/cloud_tests/run_funcs.py
@@ -0,0 +1,65 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+from tests.cloud_tests import bddeb, collect, util, verify
+
+import os
+
+
+def tree_collect(args):
+    """
+    collect data using deb build from current tree
+    args: cmdline args
+    return_value: fail count
+    """
+    failed = 0
+
+    with util.TempDir(args) as tmpdir:
+        args.deb = os.path.join(tmpdir, 'cloud-init.deb')
+        try:
+            failed += bddeb.bddeb(args)
+            failed += collect.collect(args)
+        except Exception:
+            failed += 1
+            raise
+
+    return failed
+
+
+def tree_run(args):
+    """
+    run test suite using deb build from current tree
+    args: cmdline args
+    return_value: fail count
+    """
+    failed = 0
+
+    with util.TempDir(args) as tmpdir:
+        args.deb = os.path.join(tmpdir, 'cloud-init.deb')
+        try:
+            failed += bddeb.bddeb(args)
+            failed += run(args)
+        except Exception:
+            failed += 1
+            raise
+
+    return failed
+
+
+def run(args):
+    """
+    run test suite
+    """
+    failed = 0
+
+    with util.TempDir(args) as tmpdir:
+        args.data_dir = tmpdir
+        try:
+            failed += collect.collect(args)
+            failed += verify.verify(args)
+        except Exception:
+            failed += 1
+            raise
+
+    return failed
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/util.py b/tests/cloud_tests/util.py
index 64a8667..18f54b4 100644
--- a/tests/cloud_tests/util.py
+++ b/tests/cloud_tests/util.py
@@ -3,6 +3,7 @@
 import glob
 import os
 import random
+import shutil
 import string
 import tempfile
 import yaml
@@ -160,4 +161,36 @@ def write_file(*args, **kwargs):
     """
     c_util.write_file(*args, **kwargs)
 
+
+class TempDir(object):
+    """
+    temporary directory like tempfile.TemporaryDirectory, but configurable
+    """
+
+    def __init__(self, args):
+        """
+        setup and store args
+        args: cmdline arguments
+        """
+        self.args = args
+        self.tmpdir = None
+
+    def __enter__(self):
+        """
+        create tempdir
+        return_value: tempdir path
+        """
+        self.tmpdir = tempfile.mkdtemp(prefix='cloud_test_')
+        LOG.debug('using tmpdir: %s', self.tmpdir)
+        return self.tmpdir
+
+    def __exit__(self, etype, value, trace):
+        """
+        destroy tempdir if no errors occurred
+        """
+        if etype:
+            LOG.warn('erros occurred, leaving data in %s', self.tmpdir)
+        else:
+            shutil.rmtree(self.tmpdir)
+
 # vi: ts=4 expandtab