Merge ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master

[Wesley Wiedenmeier]

integration-testing updates

 - enable additional distros
 - allow use of either pylxd 2.1.3 or 2.2.3
   - detected at runtime during instance.execute()
   - when using 2.1.3, the test suite will not have proper
     error handling during setup phase, as 2.1.3 does not
     return command exit codes. this is adequate for use on
     jenkins until the fix for 2.2 is merged in upstream
 - refactor image setup code
   - improve error handling throughout
   - change behavior of --upgrade setup_image to only
     upgrade cloud-init
   - add --upgrade-all setup_image flag to upgrade all
     packages on system
   - clean up detection of cloud-init version
   - output cloud-init version at end of setup_image
   - use 'yum install' rather than 'yum update' in
     setup_image.upgrade in case cloud-init not installed
   - update help in setup_image args
 - clean up image config
   - new image config format allows finer control over
     platform specific image config, with less verbosity
   - clean up handling of image config throughout platform
   - added additional releases and distros
   - re-ordered releases.yaml in order of release date, and
     added EOL comments for future reference
   - allow image config to override setup options with
     'setup_overrides'
   - allow image config to specify preserving base images
     for lxd
   - allow using images from images.linuxcontainers.org for
     releases other than ubuntu
 - improve waiting for system to boot
   - separated out determining if the system has booted from
     determining if cloud-init has finished
   - do not wait for cloud-init during setup phases, as it
     may not be installed
   - use 'systemctl is-system-running' or 'runlevel'
     depending on init system
   - poll for system ready inside the image rather than
     through instance.execute() to make pylxd too many open
     files error take longer to strike
   - clean up code for handling waiting
 - update lxd image metadata and templates if necessary
   - many lxd images that do not come with cloud-init
     installed do not have the necessary templates to write
     a nocloud seed into the image
   - if necessary, export image, extract metadata, add new
     template scripts, and import as new image
 - add new tox environment to run tests
   - new versions of pylxd only available through pip on
     xenial, best to avoid using pip outside a venv
   - 'citest' provides an entry points into the cli,
     allowing use of any test commands, using pylxd 2.2.3

[Wesley Wiedenmeier (wesley-wiedenmeier)]

There are 2 outstanding issues with releases that are now enabled:

Debian Jessie encounteres errors with cc_apt_configure, even with no 'apt'
config keys presented. This does not cause cloud-init to completely fail;
however, the tests fail as they all check that cloud-init stages did not
encounter any errors. I suspect there may be a bug in cloud-init here, as
it is failing on a supported system even with default config. It's worth
noting that on daily builds of Stretch, everything works properly, and
everything works on Sid, so this only affects Wheezy.

Errors:
  2017-01-05 00:15:47,016 - util.py[WARNING]: Running apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_apt_configure.pyc'>) failed
 "('apt-configure', NotFound(u\"cannot find 'security'\",))"

All CentOS releases, Debian Wheezy and Ubuntu Precise fail while pylxd
attempts to create a websocket to communicate with the image in execute().
Since the failures are consistant and always on older releases (counting
centos 7 as an old release, as most versions in it are fairly old), I
suspect that this has something to do with library versions in target.

The actual error message is:
  [Errno 24] Too many open files

I will mark this is 'needs review' once I have fixed these issues or found
better explainations.

[Wesley Wiedenmeier (wesley-wiedenmeier)]

Switching to polling in the image rather than with repeated calls to pylxd's execute() seems to have fixed the too many open files issue, although once the test suite is made to run in parallel this may become an issue again. Centos tests are not running smoothly yet because of what looks like a bug in cc_set_hostname, but there is no longer an issue with the test infrastructure on centos and old releases of other distros.

[Wesley Wiedenmeier (wesley-wiedenmeier)]

It looks like the remaining centos70 issue is not an error in the test suite or in cloud-init; its a systemd bug in the centos images from images.linuxcontainers.org. I haven't tested on official centos images yet, so its possible either that it is a bug in centos, in centos/lxd integration, or in the centos lxd image build.

Setting the system hostname using hostnamectl fails with:
-[laptop1]- -[~]-
-[08:20 PM]-lxc launch images:centos/7/default
Creating picked-snapper
Starting picked-snapper
-[laptop1]- -[~]-
-[08:21 PM]-lxc exec picked-snapper bash
[root@picked-snapper ~]# hostnamectl set-hostname "test-hostname"
Could not set property: Connection timed out

If 'preserve_hostname: true' is added to the cloud-config for a test that should work on centos the test will pass.

Once I figure out whether the bug with hostnamectl is in the image itself or in all of centos I will get a bug filed. I don't think that further work on the test suite should be done to resolve this, as modifying the test configs as a workaround does not seem right here.

[Wesley Wiedenmeier (wesley-wiedenmeier)]

This is blocked on pylxd:
https://github.com/lxc/pylxd/issues/209

A fix for the pylxd bug is available at:
https://github.com/lxc/pylxd/pull/211

[Wesley Wiedenmeier (wesley-wiedenmeier)]

This should be ready to merge and includes all changes from:
https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/314496

[Joshua Powers (powersj)]

@wesley, per the commit message you enabled additional distros. It looks like wheezy, jessie and centros66, and centos70. Do these require the newer version of pylxd for running? Should they work as expected? Trying to understand how to test these.

[Wesley Wiedenmeier (wesley-wiedenmeier)]

All distros will work on both pylxd 2.1.3 and pylxd 2.2, the only problem running with 2.1.3 is the lack of error handling, which makes debugging setup_image difficult when it isn't working.

Not all of the new distros work correctly, centos66 will fail due to not having cloud-init in its repos, although I have gotten a run to go through by installing a rpm + a couple deps. It probably shouldn't be used for any sort of testing. I'll go ahead and disable it.

Debian wheezy is not enabled, as it is very old. I could probably get it working but it wasn't a priority for testing.

Debian stretch works, at least for single tests. I haven't run through a full test yet, although I suspect that modules/apt_configure_sources_ppa will fail. I haven't run into any issue with it in terms of distro and platform support, so the only issues should be cloud-init features that are not supported on debian.

Debian jessie is enabled, but does not work due to what I believe is a bug in cc_apt_configure which I haven't had time to resolve yet. Other than that bug, it should work.

Centos70 does not work because of a bug in hostnamectl on the centos images from images.linuxcontainers.org. I have an example of the bug in a comment a few above this one. I haven't had time to look into that bug any more, but the test suite and distro do work together properly there, as adding 'preserve_hostname: true' to a test config and running that test seems to work for most simple test cases (modules/final_message, modules/debug_enable...)

Due to feature incompatibility between ubuntu/debian and centos, and lack of support for modules/cc_apt_configure_sources_ppa outside of ubuntu, distro feature groups are probably going to need to be added soon. For the time being, tests which make sense to work on these distos can be run manually for testing, or perhaps a list could be put together for jenkins. I think it makes the most sense to just get this merged in soon and deal with distro feature flags and platform feature flags in a new merge proposal, since most distros other than ubuntu are blocked by bugs right now.

The supported distro/release matrix has been updated at: goo.gl/q78sY8

[Joshua Powers (powersj)]

LGTM, with two minor comments below. Reusing images has really sped up execution and printing the cloud-init version is great. I ran through the following tests:

$ tox
$ tox -e citest -- run -v -n zesty -t tests/cloud_tests/configs/modules/write_files.yaml
$ tox -e citest_new_pylxd -- run -v -n zesty -t tests/cloud_tests/configs/modules/write_files.yaml
$ tox -e citest_tree_run
$ python3 -m tests.cloud_tests run -v -n zesty -t tests/cloud_tests/configs/modules/write_files.yaml
$ python3 -m tests.cloud_tests run -v -n stretch -t tests/cloud_tests/configs/modules/write_files.yaml
$ python3 -m tests.cloud_tests run -v -n jessie -t tests/cloud_tests/configs/modules/write_files.yaml

Comment 1:
The warning messages about using pylxd < 2.2 [1]. There are far too many messages right now. I see the message in pylxd.py as it checks for the exit code; can we suppress the warning until 2.2 is stable or even consider changing it to show up when the user is using less than 2.1.3?

Comment 2:
Running on jessie I get a failure running no_stages_errors [2] about security. Is it trying to add the jessie security repo?

[1] https://paste.ubuntu.com/23817502/
[2] https://paste.ubuntu.com/23817710/

review: Approve

[Wesley Wiedenmeier (wesley-wiedenmeier)]

> LGTM, with two minor comments below. Reusing images has really sped up
> execution and printing the cloud-init version is great.

Thanks. Yeah, re-using the images should help quite a bit with runtime.

> Comment 1:
> The warning messages about using pylxd < 2.2 [1]. There are far too many
> messages right now. I see the message in pylxd.py as it checks for the exit
> code; can we suppress the warning until 2.2 is stable or even consider
> changing it to show up when the user is using less than 2.1.3?

I'll go ahead and comment the message out, we can always put it back later.
Having the message display each execute() does make things harder to read

> Comment 2:
> Running on jessie I get a failure running no_stages_errors [2] about security.
> Is it trying to add the jessie security repo?

Yeah, jessie fails right now due to cc_apt_configure trying to find the security
repo in sources. I'm not 100% sure why its looking for it, as it won't be there.
I think this is likely a bug that needs to be filed, I just haven't had time to
find a way to reproduce the issue outside of cloud tests yet. I'll try to get
that done later this week.

[Scott Moser (smoser)]

lots of comments...
Thanks for all the work here, we need to try to do things more iteratively. I know we're still bringing this all up, but there is just so much here that its almost impossible to review.

General:
 * please lets try to do smaller merge proposals, doing one set of things
   at a time.
 * full chart of enabled/disabled at: http://goo.gl/q78sY8
   this is a private url, not suitable for a commit message
   possibly we just move this into doc in cloud-init?
 * where is the upstream bug that prevents 2.2 from working correctly?

 * i think we can live without 'alias: t'.
 * apparently we should be using dnf rather than yum in any recent RH distro
 * "switch to using images.linuxcontainers.org from using ubuntu daily images"
    why ?? we really need to be testing Ubuntu images, not linuxcontainers.org
    I like using them for other distros where we have nothing else, but
    ubuntu should test primarily on ubuntu images.

 * system_ready_script can be simplified by being a command, passed
   directly to 'execute'.
     system_ready_script: ['test', '-e', '/run/cloud-init/result.json']
   that saves writing the file and executing it on the other side, and
   also then assuming 'bash'.
 * "Pass $HOME through in tox citest envs for lxc client use"
   message doesn't make sense. where are we using the lxc client? just
   for developer debug ?
 * "Disable centos66 test"
   why ?

tox.ini:
 * multiple citest_* are fine for now, but i really do not want them
   going forward.

Easily droppable commits / things that can be done separartely:
- "Use LOG.warning instead of deprectated LOG.warn"
   why? This is just not necessary in this merge proposal.
- "Change behavior of upgrade in setup_image"

tests/cloud_tests/bddeb.py
 * 7cd4a3b5 source_archive: can we either
   a.) just assume the deb-src lines are present
   b.) hueristically grab one... i'd like to not use us.archive.com if
       the image is otherwise using archive.ubuntu.com as that will
       avoid any proxy in the path. this is kind of tough actually..

 * creating the tarball..
   I think what we want to do here is:
     a.) make-tarball on "this side"
     b.) modify bddeb to be able to just create a debian/ dir
         either
         i.) have it create the orig tarball and a debian.tar.gz into a
             provided (empty) output dir.
         ii.) have it just create the debian.tar.gz and we create the tarball
              example of doing that http://paste.ubuntu.com/23870964/
     c.) move the tarball and debian dir over, extract tarball, extract debian/
     d.) mk-build-deps --install .... debian/control
     e.) dpkg-buildpackage

   doing this should mean we do not have to 'install additional build deps'
   but rather just have mk-build-deps do the right thing.

 * when building with dpkg-buildpackage or debuild, good to
   allow setting DEB_BUILD_OPTIONS=nocheck (which will not run pep8 and
   nosetests)

 * i think we can drop 'git' due to above, 'tar' is essential so you
   do not have to declare it.
   lets do the install of devscripts and equivs with --no-install-recommends
   and also eatmydata everywhere

 * i think the above path means we do no...

[Wesley Wiedenmeier (wesley-wiedenmeier)]

> lots of comments...
> Thanks for all the work here, we need to try to do things more
> iteratively. I know we're still bringing this all up, but there
> is just so much here that its almost impossible to review.

Yeah this was definitely too large a merge, we just got stuck with
lots of changes that depend on each other. This replaces a pretty
huge chunk of the current codebase as most of that was just
temporary to get some tests running while the main part of this
got finished.

> General:
> * please lets try to do smaller merge proposals, doing one set
> of things at a time.

Definitely, this should be the last major merge, since all of the
pieces that had to be rewritten are finished. Next merge should just
be the new platform.

> * full chart of enabled/disabled at: http://goo.gl/q78sY8
> this is a private url, not suitable for a commit message
> possibly we just move this into doc in cloud-init?

I'll go ahead and pull this link out of the commit message. I have
been meaning to do some doc updates, I'll add a link to the
spreadsheet and make it public when I put together the merge for
that.

> * where is the upstream bug that prevents 2.2 from working
> correctly?

It was at https://github.com/lxc/pylxd/issues/209, but the fix has
been pulled in now. I'm going to set up a ppa to build a version with
the fix for devel use. However, for now using 2.1.3 works for
jenkins, the only disadvantage there is that if anything goes wrong
during setup it will fail silently. This version of the code will run
with either 2.1.3 or 2.2, so when the fix is released all we have to
do to switch is change the version in the tox env.

> * i think we can live without 'alias: t'.

The 'alias' key is used to reference the image, on the ubuntu daily
sstream server the aliases I found were p, t, x, y, z. It's now been
switched over to ubuntu/trusty/default with the switch to images from
images.linuxcontainers.org.

> * apparently we should be using dnf rather than yum in any recent
> RH distro

Oh, I didn't know that dnf was officially supported outside Fedora.
It shouldn't take much to switch that back over, just gotta update
the setup image functions.

> * "switch to using images.linuxcontainers.org from using ubuntu
> daily images" why ?? we really need to be testing Ubuntu
> images, not linuxcontainers.org I like using them for other
> distros where we have nothing else, but ubuntu should test
> primarily on ubuntu images.

I had mainly done it for consistancy across the different distros and
because linuxcontainers images download way faster than ubuntu daily
images (at least for me) but its just a config change to switch back
to the ubuntu daily repo for ubuntu images. I can switch that back
tonight.

> * system_ready_script can be simplified by being a command,
> passed directly to 'execute'.
> system_ready_script: ['test', '-e',
> '/run/cloud-init/result.json']
> that saves writing the file and executing it on the other
> side, and also then assuming 'bash'.

That makes sense, although it does limit how complex the scripts can
get. Its probably best to keep them simple though...

[Scott Moser (smoser)]

>> * full chart of enabled/disabled at: http://goo.gl/q78sY8
>> this is a private url, not suitable for a commit message
>> possibly we just move this into doc in cloud-init?
>
>I'll go ahead and pull this link out of the commit message. I have
>been meaning to do some doc updates, I'll add a link to the
>spreadsheet and make it public when I put together the merge for
>that.

Great.

>> * where is the upstream bug that prevents 2.2 from working
>> correctly?

> It was at https://github.com/lxc/pylxd/issues/209, but the fix has

Thanks, please mention it in the code.

>> * i think we can live without 'alias: t'.

> The 'alias' key is used to reference the image, on the ubuntu daily
> sstream server the aliases I found were p, t, x, y, z. It's now been switched
> over to ubuntu/trusty/default with the switch to images from
> images.linuxcontainers.org.

The ubuntu/daily streams also have other more explicit alias:
  lxc launch ubuntu-daily:xenial
  lxc launch ubuntu-daily:16.04

>> * apparently we should be using dnf rather than yum in any recent
>> RH distro

> Oh, I didn't know that dnf was officially supported outside Fedora.
> It shouldn't take much to switch that back over, just gotta update
> the setup image functions.

I only know because cloud-init had a MP from a RH developer asking for it (see
commit log)
 https://bugs.launchpad.net/cloud-init/+bug/1647118

>> * system_ready_script can be simplified by being a command,
>> passed directly to 'execute'.
>> system_ready_script: ['test', '-e',
>> '/run/cloud-init/result.json']
>> that saves writing the file and executing it on the other
>> side, and also then assuming 'bash'.

>That makes sense, although it does limit how complex the scripts can
>get. Its probably best to keep them simple though.

If we needed, we could allow the config to provide stdin also, but with
the above, the only limit on complexity that i can see is the command
line length limit, which shouldn't be too big of a problem
  http://stackoverflow.com/questions/6846263/maximum-length-of-command-line-argument-that-can-be-passed-to-sqlplus-from-lin

xargs --show-limits:
  POSIX upper limit on argument length (this system): 2091520

:)

>> * "Pass $HOME through in tox citest envs for lxc client use"
>> message doesn't make sense. where are we using the lxc client? just
>> for developer debug ?

> It is used for

Sentance fragment ^ ?

>> * "Disable centos66 test"
>> why ?

> Its such an old release that pretty much nothing is going to be
> backported there, testing on it doesn't really make sense IMO. I
> think the one time I tried running centos 6 tests they failed, but I
> don't recall why. I can get tests going there in the future if
> they're needed.

Well, if they were working, then it makes sense to run it, as a stable
platform likely means regression/change-of-behavior are the fault of
cloud-init, not the OS.

>> Easily droppable commits / things that can be done separartely:
>> - "Use LOG.warning instead of deprectated LOG.warn"
>> why? This is just not necessary in this merge proposal.
>> - "Change behavior of upgrade in setup_image"

>Th...

[Wesley Wiedenmeier (wesley-wiedenmeier)]

> >> * where is the upstream bug that prevents 2.2 from working
> >> correctly?
>
> > It was at https://github.com/lxc/pylxd/issues/209, but the fix has
>
> Thanks, please mention it in the code.

Sure, I'll add a comment in there.

> >> * i think we can live without 'alias: t'.
>
> > The 'alias' key is used to reference the image, on the ubuntu daily
> > sstream server the aliases I found were p, t, x, y, z. It's now been
> switched
> > over to ubuntu/trusty/default with the switch to images from
> > images.linuxcontainers.org.
>
> The ubuntu/daily streams also have other more explicit alias:
> lxc launch ubuntu-daily:xenial
> lxc launch ubuntu-daily:16.04

Oh, nice. I'll switch to using those.

> >> * apparently we should be using dnf rather than yum in any recent
> >> RH distro
>
> > Oh, I didn't know that dnf was officially supported outside Fedora.
> > It shouldn't take much to switch that back over, just gotta update
> > the setup image functions.
>
> I only know because cloud-init had a MP from a RH developer asking
> for it (see commit log)
> https://bugs.launchpad.net/cloud-init/+bug/1647118

It may be good to keep both the yum path in place and add a config
directive to tell the platform which to use, just in case things
switch around more.

> >> * system_ready_script can be simplified by being a command,
> >> passed directly to 'execute'.
> >> system_ready_script: ['test', '-e',
> >> '/run/cloud-init/result.json']
> >> that saves writing the file and executing it on the other
> >> side, and also then assuming 'bash'.
>
> >That makes sense, although it does limit how complex the scripts can
> >get. Its probably best to keep them simple though.
>
> If we needed, we could allow the config to provide stdin also, but with
> the above, the only limit on complexity that i can see is the command
> line length limit, which shouldn't be too big of a problem
> http://stackoverflow.com/questions/6846263/maximum-length-of-command-line-
> argument-that-can-be-passed-to-sqlplus-from-lin
>
> xargs --show-limits:
> POSIX upper limit on argument length (this system): 2091520
>
> :)

Haha yeah, we're definitely not in danger of hitting that limit.

> >> * "Pass $HOME through in tox citest envs for lxc client use"
> >> message doesn't make sense. where are we using the lxc client? just
> >> for developer debug ?
>
> > It is used for
>
> Sentance fragment ^ ?

Whoops, I must have forgotten to finish that. It is used for exporting
lxd images that need to have their metadata modified to write the
cloud seed into the target on. Pylxd does have an export function for
images, but it does the export as a tmpfile with handle, not as a split
tarball in the filesystem like we need. The lxc client handles that
nicely, but it needs $HOME set so it can locate its config file.

> >> * "Disable centos66 test"
> >> why ?
>
> > Its such an old release that pretty much nothing is going to be
> > backported there, testing on it doesn't really make sense IMO. I
> > think the one time I tried running centos 6 tests they failed, but I
> > don't recall why. I can get tests going there in th...

[Wesley Wiedenmeier (wesley-wiedenmeier)]

This branch should mostly be ready to merge in now. There is additonal work to be done on getting the cc_apt_configure issue resolved in debian jessie, and the hostnamectl issue resolved on centos70, but that can wait, as these are most likely not issues caused by the test suite. Support for package management on centos still needs to be switched to dnf, but since centos is blocked on the hostnamectl issue, that can wait as well.

The cleanup to system_ready_scripts has been done, and the ubuntu release config is back to using the ubuntu daily images. Ubuntu releases xenial -> zesty have been run with a couple of test configs successfully, both installing from ppa:cloud-init-dev/daily and from --deb with a deb of 0.7.9, and trusty has been run successfully with the version of cloud-init it ships with on modules/final_message. Additional testing of this merge may be good, but the tests currently being run by jenkins should work.

Note: I commented on the wrong MP a couple days ago, this was meant to be here

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:b7d483f8dce139c688cd8e28b4195768365e7741
https://jenkins.ubuntu.com/server/job/cloud-init-ci/76/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/76
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/76
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/76
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-amd64/76
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/76

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/76/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:bc332ca6b4d28e73efcd7adeb1854358a1395afa
https://jenkins.ubuntu.com/server/job/cloud-init-ci/93/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/93
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/93
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/93
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/93
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/93

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/93/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:c6e896b627e594444d7ce8684dcc190d4adc018a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/104/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/104/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/104/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/104/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/104/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/104/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/104/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:1c3b07a8634f5b219b41f138ac016bfaccff4042
https://jenkins.ubuntu.com/server/job/cloud-init-ci/105/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/105
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/105
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/105
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/105
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/105

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/105/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:cffc558b6ef2d98a5bdacf98be15f6946042fa68
https://jenkins.ubuntu.com/server/job/cloud-init-ci/106/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/106
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/106
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/106
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/106
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/106

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/106/rebuild

review: Approve (continuous-integration)

0bb1f96... by Wesley Wiedenmeier on 2017-03-12

Integration Testing: Update platform command execution and util

 - Add util.InTargetExecuteError, from cloudinit.util.ProcessExecutionError:
   - raise error if command has non-zero exit by default
   - allow non-zero exit codes if ignore_errors=True is set on execute calls
   - add description for in-target commands that is output in the event of a
     command error to simplify tracking down errors
 - Update calls to instance.execute() and instance.run_script()
 - Update image calls to instance execute and run
 - Update docstrings where required
 - Fix handling of os family mappings in util

4ebe02d... by Wesley Wiedenmeier on 2017-03-12

Integration Testing: Update setup_image and fix issues with package handling

 - Change behavior of upgrade in setup_image:
   - The --upgrade flag only upgrades or installs cloud-init
   - The --upgrade-full flag has been added to do a full system upgrade
 - Clean up detection of installed cloud-init version and output after setup
 - Use yum install rather than update for setup_image.upgrade to handle rpm
   based images that don't have cloud-init installed
 - Improve handling of deb installation, use 'apt-get instll -f' if a dep was
   missing

5952da7... by Wesley Wiedenmeier on 2017-03-12

Integration Testing: Clean up and improve instance boot and execute

 - Improve instance.run_script():
   - return full (out, err, exit) tuple instead of just the stdout,
     allowing additional error checking
   - clean up tempfile after script has been run
   - generate tempfile names using 'mktemp' to avoid possible name conflicts
 - Refactor code to wait for the system to boot
   - use 'system_ready_script' from image config rather than hardcoded script
   - poll for in the instance rather than on the host to avoid excessive calls
     to execute()
   - check for the system to boot and cloud-init to complete separately,
     allowing images with cloud-init not installed to be booted by setup_image
   - execute ready script commands directly in loop, instead of writing tempfile
   - use .execute() instead of .run_script() to avoid file writes

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:bee10bde56a37eafcb61ebaa1f8db7143c35de37
https://jenkins.ubuntu.com/server/job/cloud-init-ci/107/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/107
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/107
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/107
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/107
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/107

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/107/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:9e3596dd6f481e42095b32f972acd6309e12ccb0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/108/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/108
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/108
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/108
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/108
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/108

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/108/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:f28a848f23abce1c2630d63b4ad8879e884c4fdb
https://jenkins.ubuntu.com/server/job/cloud-init-ci/111/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/111
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/111
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/111
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/111
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/111

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/111/rebuild

review: Approve (continuous-integration)

[Wesley Wiedenmeier (wesley-wiedenmeier)]

I've squashed the commit history for this branch together into a patch series, so it should be easier to review commit by commit now.

c931e06... by Wesley Wiedenmeier on 2017-03-15

Integration Testing: remove citest_old_pylxd tox environment

 - pylxd 2.1.3 does not allow error checking during setup as it does nto return
   command exit code
 - remove the tox env for running integration tests using pylxd 2.1.3 and keep
   default environment using 2.2.3

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:1c55c8bb2a965cd95f33b1b464554048385eabda
https://jenkins.ubuntu.com/server/job/cloud-init-ci/117/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/117
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/117
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/117
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/117
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/117

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/117/rebuild

review: Approve (continuous-integration)

7918b5c... by Wesley Wiedenmeier on 2017-03-15

Integration Testing: improve code to allow additional error codes in execute

 - Old behavior was to allow execute to ignore any return code from an
   in-target command with ignore_errors
 - Replace this with a list of acceptable return codes similar to util.subp
   which allows better control over command execution

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:b9f673fcc73a818cf30c963cd088f4dee7e34c49
https://jenkins.ubuntu.com/server/job/cloud-init-ci/118/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/118
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/118
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/118
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/118
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/118

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/118/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:7918b5cf3fdbfe8ca854f9c78c5c05c9ab2d1e64
https://jenkins.ubuntu.com/server/job/cloud-init-ci/134/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/134
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/134
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/134
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/134
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/134

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/134/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

I'm going to mark this 'work in progress'. Josh has integrated this branch and others into his merge proposal at https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/324136 . We do plan to pull that all soon.

Thanks for your work, Wesley!

[Scott Moser (smoser)]

Hi.
I've marked this 'merged' as I think it is now in trunk under 76d58265e34851b78e952a7f275340863c90a9f5.
If you disagree, please feel free to re-open.

diff --git a/tests/cloud_tests/args.py b/tests/cloud_tests/args.py
index b68cc98..a96714c 100644
--- a/tests/cloud_tests/args.py
+++ b/tests/cloud_tests/args.py
@@ -9,11 +9,11 @@ ARG_SETS = {
     'COLLECT': (
         (('-p', '--platform'),
          {'help': 'platform(s) to run tests on', 'metavar': 'PLATFORM',
-          'action': 'append', 'choices': config.list_enabled_platforms(),
+          'action': 'append', 'choices': config.ENABLED_PLATFORMS,
           'default': []}),
         (('-n', '--os-name'),
          {'help': 'the name(s) of the OS(s) to test', 'metavar': 'NAME',
-          'action': 'append', 'choices': config.list_enabled_distros(),
+          'action': 'append', 'choices': config.ENABLED_DISTROS,
           'default': []}),
         (('-t', '--test-config'),
          {'help': 'test config file(s) to use', 'metavar': 'FILE',
@@ -61,8 +61,12 @@ ARG_SETS = {
          {'help': 'ppa to enable (implies -u)', 'metavar': 'NAME',
           'action': 'store'}),
         (('-u', '--upgrade'),
-         {'help': 'upgrade before starting tests', 'action': 'store_true',
-          'default': False}),),
+         {'help': 'upgrade or install cloud-init from repo',
+          'action': 'store_true', 'default': False}),
+        (('--upgrade-full',),
+         {'help': 'do full system upgrade from repo (implies -u)',
+          'action': 'store_true', 'default': False}),),
+
 }
 
 SUBCMDS = {
@@ -121,15 +125,15 @@ def normalize_collect_args(args):
     """
     # platform should default to all supported
     if len(args.platform) == 0:
-        args.platform = config.list_enabled_platforms()
+        args.platform = config.ENABLED_PLATFORMS
     args.platform = util.sorted_unique(args.platform)
 
     # os name should default to all enabled
     # if os name is provided ensure that all provided are supported
     if len(args.os_name) == 0:
-        args.os_name = config.list_enabled_distros()
+        args.os_name = config.ENABLED_DISTROS
     else:
-        supported = config.list_enabled_distros()
+        supported = config.ENABLED_DISTROS
         invalid = [os_name for os_name in args.os_name
                    if os_name not in supported]
         if len(invalid) != 0:
diff --git a/tests/cloud_tests/collect.py b/tests/cloud_tests/collect.py
index 68b47d7..1d77d1c 100644
--- a/tests/cloud_tests/collect.py
+++ b/tests/cloud_tests/collect.py
@@ -18,8 +18,10 @@ def collect_script(instance, base_dir, script, script_name):
     return_value: None, may raise errors
     """
     LOG.debug('running collect script: %s', script_name)
-    util.write_file(os.path.join(base_dir, script_name),
-                    instance.run_script(script))
+    (out, err, exit) = instance.run_script(
+        script, rcs=range(0, 256),
+        description='collect: {}'.format(script_name))
+    util.write_file(os.path.join(base_dir, script_name), out)
 
 
 def collect_test_data(args, snapshot, os_name, test_name):
@@ -39,9 +41,6 @@ def collect_test_data(args, snapshot, os_name, test_name):
     test_scripts = test_config['collect_scripts']
     test_output_dir = os.sep.join(
         (args.data_dir, snapshot.platform_name, os_name, test_name))
-    boot_timeout = (test_config.get('boot_timeout')
-                    if isinstance(test_config.get('boot_timeout'), int) else
-                    snapshot.config.get('timeout'))
 
     # if test is not enabled, skip and return 0 failures
     if not test_config.get('enabled', False):
@@ -56,7 +55,7 @@ def collect_test_data(args, snapshot, os_name, test_name):
     LOG.info('collecting test data for test: %s', test_name)
     with component as instance:
         start_call = partial(run_single, 'boot instance', partial(
-            instance.start, wait=True, wait_time=boot_timeout))
+            instance.start, wait=True, wait_for_cloud_init=True))
         collect_calls = [partial(run_single, 'script {}'.format(script_name),
                                  partial(collect_script, instance,
                                          test_output_dir, script, script_name))
@@ -100,10 +99,8 @@ def collect_image(args, platform, os_name):
     """
     res = ({}, 1)
 
-    os_config = config.load_os_config(os_name)
-    if not os_config.get('enabled'):
-        raise ValueError('OS {} not enabled'.format(os_name))
-
+    os_config = config.load_os_config(
+        platform.platform_name, os_name, require_enabled=True)
     component = PlatformComponent(
         partial(images.get_image, platform, os_config))
 
@@ -126,10 +123,8 @@ def collect_platform(args, platform_name):
     """
     res = ({}, 1)
 
-    platform_config = config.load_platform_config(platform_name)
-    if not platform_config.get('enabled'):
-        raise ValueError('Platform {} not enabled'.format(platform_name))
-
+    platform_config = config.load_platform_config(
+        platform_name, require_enabled=True)
     component = PlatformComponent(
         partial(platforms.get_platform, platform_name, platform_config))
 
diff --git a/tests/cloud_tests/config.py b/tests/cloud_tests/config.py
index f3a13c9..01cddf7 100644
--- a/tests/cloud_tests/config.py
+++ b/tests/cloud_tests/config.py
@@ -14,6 +14,20 @@ RELEASES_CONF = os.path.join(BASE_DIR, 'releases.yaml')
 TESTCASE_CONF = os.path.join(BASE_DIR, 'testcases.yaml')
 
 
+def get(base, key):
+    """
+    get config entry 'key' from base, ensuring is dictionary
+    """
+    return base[key] if key in base and base[key] is not None else {}
+
+
+def enabled(config):
+    """
+    test if config item is enabled
+    """
+    return isinstance(config, dict) and config.get('enabled', False)
+
+
 def path_to_name(path):
     """
     convert abs or rel path to test config to path under configs/
@@ -61,22 +75,39 @@ def merge_config(base, override):
     return res
 
 
-def load_platform_config(platform):
+def load_platform_config(platform_name, require_enabled=False):
     """
     load configuration for platform
+    platform_name: name of platform to retrieve config for
+    require_enabled: if true, raise error if 'enabled' not True
+    return_value: config dict
     """
     main_conf = c_util.read_conf(PLATFORM_CONF)
-    return merge_config(main_conf.get('default_platform_config'),
-                        main_conf.get('platforms')[platform])
+    conf = merge_config(main_conf['default_platform_config'],
+                        main_conf['platforms'][platform_name])
+    if require_enabled and not enabled(conf):
+        raise ValueError('Platform is not enabled')
+    return conf
 
 
-def load_os_config(os_name):
+def load_os_config(platform_name, os_name, require_enabled=False):
     """
     load configuration for os
+    platform_name: platform name to load os config for
+    os_name: name of os to retrieve config for
+    require_enabled: if true, raise error if 'enabled' not True
+    return_value: config dict
     """
     main_conf = c_util.read_conf(RELEASES_CONF)
-    return merge_config(main_conf.get('default_release_config'),
-                        main_conf.get('releases')[os_name])
+    default = main_conf['default_release_config']
+    image = main_conf['releases'][os_name]
+    conf = merge_config(merge_config(get(default, 'default'),
+                                     get(default, platform_name)),
+                        merge_config(get(image, 'default'),
+                                     get(image, platform_name)))
+    if require_enabled and not enabled(conf):
+        raise ValueError('OS is not enabled')
+    return conf
 
 
 def load_test_config(path):
@@ -91,16 +122,22 @@ def list_enabled_platforms():
     """
     list all platforms enabled for testing
     """
-    platforms = c_util.read_conf(PLATFORM_CONF).get('platforms')
-    return [k for k, v in platforms.items() if v.get('enabled')]
+    platforms = get(c_util.read_conf(PLATFORM_CONF), 'platforms')
+    return [k for k, v in platforms.items() if enabled(v)]
 
 
-def list_enabled_distros():
+def list_enabled_distros(platforms):
     """
-    list all distros enabled for testing
+    list all distros enabled for testing on specified platforms
     """
-    releases = c_util.read_conf(RELEASES_CONF).get('releases')
-    return [k for k, v in releases.items() if v.get('enabled')]
+
+    def platform_has_enabled(config):
+        return any(enabled(merge_config(get(config, 'default'),
+                                        get(config, platform)))
+                   for platform in platforms)
+
+    releases = get(c_util.read_conf(RELEASES_CONF), 'releases')
+    return [k for k, v in releases.items() if platform_has_enabled(v)]
 
 
 def list_test_configs():
@@ -110,4 +147,8 @@ def list_test_configs():
     return [os.path.abspath(f) for f in
             glob.glob(os.sep.join((TEST_CONF_DIR, '*', '*.yaml')))]
 
+
+ENABLED_PLATFORMS = list_enabled_platforms()
+ENABLED_DISTROS = list_enabled_distros(ENABLED_PLATFORMS)
+
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/images/base.py b/tests/cloud_tests/images/base.py
index 394b11f..e61a928 100644
--- a/tests/cloud_tests/images/base.py
+++ b/tests/cloud_tests/images/base.py
@@ -7,13 +7,14 @@ class Image(object):
     """
     platform_name = None
 
-    def __init__(self, name, config, platform):
+    def __init__(self, platform, config):
         """
-        setup
+        Set up image
+        platform: platform object
+        config: image configuration
         """
-        self.name = name
-        self.config = config
         self.platform = platform
+        self.config = config
 
     def __str__(self):
         """
@@ -28,10 +29,16 @@ class Image(object):
         """
         raise NotImplementedError
 
-    # FIXME: instead of having execute and push_file and other instance methods
-    #        here which pass through to a hidden instance, it might be better
-    #        to expose an instance that the image can be modified through
-    def execute(self, command, stdin=None, stdout=None, stderr=None, env={}):
+    @property
+    def setup_overrides(self):
+        """
+        setup options that need to be overridden for the image
+        return_value: dictionary to update args with
+        """
+        # NOTE: more sophisticated options may be requied at some point
+        return self.config.get('setup_overrides', {})
+
+    def execute(self, *args, **kwargs):
         """
         execute command in image, modifying image
         """
@@ -43,7 +50,7 @@ class Image(object):
         """
         raise NotImplementedError
 
-    def run_script(self, script):
+    def run_script(self, *args, **kwargs):
         """
         run script in image, modifying image
         return_value: script output
diff --git a/tests/cloud_tests/images/lxd.py b/tests/cloud_tests/images/lxd.py
index 7a41614..1fdb91e 100644
--- a/tests/cloud_tests/images/lxd.py
+++ b/tests/cloud_tests/images/lxd.py
@@ -2,6 +2,10 @@
 
 from tests.cloud_tests.images import base
 from tests.cloud_tests.snapshots import lxd as lxd_snapshot
+from tests.cloud_tests import util
+
+import os
+import shutil
 
 
 class LXDImage(base.Image):
@@ -10,27 +14,43 @@ class LXDImage(base.Image):
     """
     platform_name = "lxd"
 
-    def __init__(self, name, config, platform, pylxd_image):
+    def __init__(self, platform, config, pylxd_image):
         """
-        setup
+        Set up image
+        platform: platform object
+        config: image configuration
         """
-        self.platform = platform
-        self._pylxd_image = pylxd_image
+        self.modified = False
         self._instance = None
-        super(LXDImage, self).__init__(name, config, platform)
+        self._pylxd_image = None
+        self.pylxd_image = pylxd_image
+        super(LXDImage, self).__init__(platform, config)
 
     @property
     def pylxd_image(self):
-        self._pylxd_image.sync()
+        if self._pylxd_image:
+            self._pylxd_image.sync()
         return self._pylxd_image
 
+    @pylxd_image.setter
+    def pylxd_image(self, pylxd_image):
+        if self._instance:
+            self._instance.destroy()
+            self._instance = None
+        if (self._pylxd_image and
+                (self._pylxd_image is not pylxd_image) and
+                (not self.config.get('cache_base_image') or self.modified)):
+            self._pylxd_image.delete(wait=True)
+        self.modified = False
+        self._pylxd_image = pylxd_image
+
     @property
     def instance(self):
         if not self._instance:
             self._instance = self.platform.launch_container(
-                image=self.pylxd_image.fingerprint,
-                image_desc=str(self), use_desc='image-modification')
-        self._instance.start(wait=True, wait_time=self.config.get('timeout'))
+                self.properties, self.config, use_desc='image-modification',
+                image_desc=str(self), image=self.pylxd_image.fingerprint)
+        self._instance.start()
         return self._instance
 
     @property
@@ -46,6 +66,78 @@ class LXDImage(base.Image):
             'release': properties.get('release'),
         }
 
+    def export_image(self, output_dir):
+        """
+        export image from lxd image store to (split) tarball on disk
+        output_dir: dir to store tarballs in
+        return_value: tuple of path to metadata tarball and rootfs tarball
+        """
+        # pylxd's image export feature doesn't do split exports, so use cmdline
+        util.subp(['lxc', 'image', 'export', self.pylxd_image.fingerprint,
+                   output_dir], capture=True)
+        tarballs = [p for p in os.listdir(output_dir) if p.endswith('tar.xz')]
+        metadata = os.path.join(
+            output_dir, next(p for p in tarballs if p.startswith('meta-')))
+        rootfs = os.path.join(
+            output_dir, next(p for p in tarballs if not p.startswith('meta-')))
+        return (metadata, rootfs)
+
+    def import_image(self, metadata, rootfs):
+        """
+        import image to lxd image store from (split) tarball on disk
+        note, this will replace and delete the current pylxd_image
+        metadata: metadata tarball
+        rootfs: rootfs tarball
+        return_value: imported image fingerprint
+        """
+        alias = util.gen_instance_name(
+            image_desc=str(self), use_desc='update-metadata')
+        util.subp(['lxc', 'image', 'import', metadata, rootfs,
+                   '--alias', alias], capture=True)
+        self.pylxd_image = self.platform.query_image_by_alias(alias)
+        return self.pylxd_image.fingerprint
+
+    def update_templates(self, template_config, template_data):
+        """
+        update the image's template configuration
+        note, this will replace and delete the current pylxd_image
+        template_config: config overrides for template portion of metadata
+        template_data: template data to place into templates/
+        """
+        # set up tmp files
+        export_dir = util.tmpdir()
+        extract_dir = util.tmpdir()
+        new_metadata = os.path.join(export_dir, 'new-meta.tar.xz')
+        metadata_yaml = os.path.join(extract_dir, 'metadata.yaml')
+        template_dir = os.path.join(extract_dir, 'templates')
+
+        try:
+            # extract old data
+            (metadata, rootfs) = self.export_image(export_dir)
+            shutil.unpack_archive(metadata, extract_dir)
+
+            # update metadata
+            metadata = util.read_conf(metadata_yaml)
+            templates = metadata.get('templates', {})
+            templates.update(template_config)
+            metadata['templates'] = templates
+            util.yaml_dump(metadata, metadata_yaml)
+
+            # write out template files
+            for name, content in template_data.items():
+                path = os.path.join(template_dir, name)
+                util.write_file(path, content)
+
+            # store new data, mark new image as modified
+            util.flat_tar(new_metadata, extract_dir)
+            self.import_image(new_metadata, rootfs)
+            self.modified = True
+
+        finally:
+            # remove tmpfiles
+            shutil.rmtree(export_dir)
+            shutil.rmtree(extract_dir)
+
     def execute(self, *args, **kwargs):
         """
         execute command in image, modifying image
@@ -58,12 +150,12 @@ class LXDImage(base.Image):
         """
         return self.instance.push_file(local_path, remote_path)
 
-    def run_script(self, script):
+    def run_script(self, *args, **kwargs):
         """
         run script in image, modifying image
         return_value: script output
         """
-        return self.instance.run_script(script)
+        return self.instance.run_script(*args, **kwargs)
 
     def snapshot(self):
         """
@@ -71,22 +163,20 @@ class LXDImage(base.Image):
         """
         # clone current instance, start and freeze clone
         instance = self.platform.launch_container(
-            container=self.instance.name, image_desc=str(self),
-            use_desc='snapshot')
-        instance.start(wait=True, wait_time=self.config.get('timeout'))
+            self.properties, self.config, container=self.instance.name,
+            image_desc=str(self), use_desc='snapshot')
+        instance.start()
         if self.config.get('boot_clean_script'):
             instance.run_script(self.config.get('boot_clean_script'))
         instance.freeze()
         return lxd_snapshot.LXDSnapshot(
-            self.properties, self.config, self.platform, instance)
+            self.platform, self.properties, self.config, instance)
 
     def destroy(self):
         """
         clean up data associated with image
         """
-        if self._instance:
-            self._instance.destroy()
-        self.pylxd_image.delete(wait=True)
+        self.pylxd_image = None
         super(LXDImage, self).destroy()
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/base.py b/tests/cloud_tests/instances/base.py
index 9559d28..9bcba6a 100644
--- a/tests/cloud_tests/instances/base.py
+++ b/tests/cloud_tests/instances/base.py
@@ -1,8 +1,5 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
-import os
-import uuid
-
 
 class Instance(object):
     """
@@ -10,26 +7,37 @@ class Instance(object):
     """
     platform_name = None
 
-    def __init__(self, name):
+    def __init__(self, platform, name, properties, config):
         """
-        setup
+        Set up instance
+        platform: platform object
+        name: hostname of instance
+        properties: image properties
+        config: image config
         """
+        self.platform = platform
         self.name = name
+        self.properties = properties
+        self.config = config
 
-    def execute(self, command, stdin=None, stdout=None, stderr=None, env={}):
+    def execute(self, command, stdout=None, stderr=None, env={},
+                rcs=None, description=None):
         """
+        Execute command in instance, recording output, error and exit code.
+        Assumes functional networking and execution as root with the
+        target filesystem being available at /.
+
         command: the command to execute as root inside the image
-        stdin, stderr, stdout: file handles
+        stdout, stderr: file handles to write output and error to
         env: environment variables
-
-        Execute assumes functional networking and execution as root with the
-        target filesystem being available at /.
+        rcs: allowed return codes from command
+        description: purpose of command
 
         return_value: tuple containing stdout data, stderr data, exit code
         """
         raise NotImplementedError
 
-    def read_data(self, remote_path, encode=False):
+    def read_data(self, remote_path, decode=False):
         """
         read_data from instance filesystem
         remote_path: path in instance
@@ -49,6 +57,8 @@ class Instance(object):
     def pull_file(self, remote_path, local_path):
         """
         copy file at 'remote_path', from instance to 'local_path'
+        remote_path: path on remote instance
+        local_path: path on local instance
         """
         with open(local_path, 'wb') as fp:
             fp.write(self.read_data(remote_path), encode=True)
@@ -56,18 +66,34 @@ class Instance(object):
     def push_file(self, local_path, remote_path):
         """
         copy file at 'local_path' to instance at 'remote_path'
+        local_path: path on local instance
+        remote_path: path on remote instance
         """
         with open(local_path, 'rb') as fp:
             self.write_data(remote_path, fp.read())
 
-    def run_script(self, script):
+    def run_script(self, script, rcs=None, description=None):
         """
         run script in target and return stdout
+        script: script contents
+        rcs: allowed return codes from script
+        description: purpose of script
+        return_value: stdout from script
         """
-        script_path = os.path.join('/tmp', str(uuid.uuid1()))
-        self.write_data(script_path, script)
-        (out, err, exit_code) = self.execute(['/bin/bash', script_path])
-        return out
+        script_path = self.tmpfile()
+        try:
+            self.write_data(script_path, script)
+            return self.execute(
+                ['/bin/bash', script_path], rcs=rcs, description=description)
+        finally:
+            self.execute(['rm', script_path], rcs=rcs)
+
+    def tmpfile(self):
+        """
+        get a tmp file in the target
+        return_value: path to new file in target
+        """
+        return self.execute(['mktemp'])[0].strip()
 
     def console_log(self):
         """
@@ -87,7 +113,7 @@ class Instance(object):
         """
         raise NotImplementedError
 
-    def start(self, wait=True):
+    def start(self, wait=True, wait_for_cloud_init=False):
         """
         start instance
         """
@@ -99,22 +125,32 @@ class Instance(object):
         """
         pass
 
-    def _wait_for_cloud_init(self, wait_time):
+    def _wait_for_system(self, wait_for_cloud_init):
         """
         wait until system has fully booted and cloud-init has finished
+        wait_time: maximum time to wait
+        return_value: None, may raise OSError if wait_time exceeded
         """
-        if not wait_time:
-            return
 
-        found_msg = 'found'
-        cmd = ('for ((i=0;i<{wait};i++)); do [ -f "{file}" ] && '
-               '{{ echo "{msg}";break; }} || sleep 1; done').format(
-            file='/run/cloud-init/result.json',
-            wait=wait_time, msg=found_msg)
+        def clean_test(test):
+            """
+            clean formatting for system ready test testcase
+            """
+            return ' '.join(l for l in test.strip().splitlines()
+                            if not l.lstrip().startswith('#'))
+
+        time = self.config['boot_timeout']
+        tests = [self.config['system_ready_script']]
+        if wait_for_cloud_init:
+            tests.append(self.config['cloud_init_ready_script'])
+
+        formatted_tests = ' && '.join(clean_test(t) for t in tests)
+        test_cmd = ('for ((i=0;i<{time};i++)); do {test} && exit 0; sleep 1; '
+                    'done; exit 1;').format(time=time, test=formatted_tests)
+        cmd = ['/bin/bash', '-c', test_cmd]
+
+        if self.execute(cmd, rcs=(0, 1))[-1] != 0:
+            raise OSError('timeout: after {}s system not started'.format(time))
 
-        (out, err, exit) = self.execute(['/bin/bash', '-c', cmd])
-        if out.strip() != found_msg:
-            raise OSError('timeout: after {}s, cloud-init has not started'
-                          .format(wait_time))
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/lxd.py b/tests/cloud_tests/instances/lxd.py
index f0aa121..a90c88d 100644
--- a/tests/cloud_tests/instances/lxd.py
+++ b/tests/cloud_tests/instances/lxd.py
@@ -1,6 +1,7 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
 from tests.cloud_tests.instances import base
+from tests.cloud_tests import util
 
 
 class LXDInstance(base.Instance):
@@ -9,41 +10,66 @@ class LXDInstance(base.Instance):
     """
     platform_name = "lxd"
 
-    def __init__(self, name, platform, pylxd_container):
+    def __init__(self, platform, name, properties, config, pylxd_container):
         """
-        setup
+        Set up instance
+        platform: platform object
+        name: hostname of instance
+        properties: image properties
+        config: image config
         """
-        self.platform = platform
         self._pylxd_container = pylxd_container
-        super(LXDInstance, self).__init__(name)
+        super(LXDInstance, self).__init__(platform, name, properties, config)
 
     @property
     def pylxd_container(self):
         self._pylxd_container.sync()
         return self._pylxd_container
 
-    def execute(self, command, stdin=None, stdout=None, stderr=None, env={}):
+    def execute(self, command, stdout=None, stderr=None, env={},
+                rcs=None, description=None):
         """
+        Execute command in instance, recording output, error and exit code.
+        Assumes functional networking and execution as root with the
+        target filesystem being available at /.
+
         command: the command to execute as root inside the image
-        stdin, stderr, stdout: file handles
+        stdout, stderr: file handles to write output and error to
         env: environment variables
-
-        Execute assumes functional networking and execution as root with the
-        target filesystem being available at /.
+        rcs: allowed return codes from command
+        description: purpose of command
 
         return_value: tuple containing stdout data, stderr data, exit code
         """
-        # TODO: the pylxd api handler for container.execute needs to be
-        #       extended to properly pass in stdin
-        # TODO: the pylxd api handler for container.execute needs to be
-        #       extended to get the return code, for now just use 0
+        # ensure instance is running and execute the command
         self.start()
-        if stdin:
-            raise NotImplementedError
         res = self.pylxd_container.execute(command, environment=env)
-        for (f, data) in (i for i in zip((stdout, stderr), res) if i[0]):
-            f.write(data)
-        return res + (0,)
+
+        # get out, exit and err from pylxd return
+        if hasattr(res, 'exit_code'):
+            # pylxd 2.2 returns ContainerExecuteResult, named tuple of
+            # (exit_code, out, err)
+            (exit, out, err) = res
+        else:
+            # pylxd 2.1.3 and earlier only return out and err, no exit
+            # LOG.warning('using pylxd version < 2.2')
+            (out, err) = res
+            exit = 0
+
+        # write data to file descriptors if needed
+        if stdout:
+            stdout.write(out)
+        if stderr:
+            stderr.write(err)
+
+        # if the command exited with a code not allowed in rcs, then fail
+        if exit not in (rcs if rcs else (0,)):
+            error_desc = ('Failed command to: {}'.format(description)
+                          if description else None)
+            raise util.InTargetExecuteError(
+                out, err, exit, command, self.name, error_desc)
+
+        return (out, err, exit)
 
     def read_data(self, remote_path, decode=False):
         """
@@ -83,14 +109,14 @@ class LXDInstance(base.Instance):
         if self.pylxd_container.status != 'Stopped':
             self.pylxd_container.stop(wait=wait)
 
-    def start(self, wait=True, wait_time=None):
+    def start(self, wait=True, wait_for_cloud_init=False):
         """
         start instance
         """
         if self.pylxd_container.status != 'Running':
             self.pylxd_container.start(wait=wait)
-            if wait and isinstance(wait_time, int):
-                self._wait_for_cloud_init(wait_time)
+            if wait:
+                self._wait_for_system(wait_for_cloud_init)
 
     def freeze(self):
         """
diff --git a/tests/cloud_tests/platforms.yaml b/tests/cloud_tests/platforms.yaml
index 5972b32..b91834a 100644
--- a/tests/cloud_tests/platforms.yaml
+++ b/tests/cloud_tests/platforms.yaml
@@ -10,7 +10,55 @@ default_platform_config:
 platforms:
     lxd:
         enabled: true
-        get_image_timeout: 600
+        # overrides for image templates
+        template_overrides:
+            /var/lib/cloud/seed/nocloud-net/meta-data:
+                when:
+                    - create
+                    - copy
+                template: cloud-init-meta.tpl
+            /var/lib/cloud/seed/nocloud-net/network-config:
+                when:
+                    - create
+                    - copy
+                template: cloud-init-network.tpl
+            /var/lib/cloud/seed/nocloud-net/user-data:
+                when:
+                    - create
+                    - copy
+                template: cloud-init-user.tpl
+                properties:
+                    default: |
+                        #cloud-config
+                        {}
+            /var/lib/cloud/seed/nocloud-net/vendor-data:
+                when:
+                    - create
+                    - copy
+                template: cloud-init-vendor.tpl
+                properties:
+                    default: |
+                        #cloud-config
+                        {}
+        # overrides image template files
+        template_files:
+            cloud-init-meta.tpl: |
+                #cloud-config
+                instance-id: {{ container.name }}
+                local-hostname: {{ container.name }}
+                {{ config_get("user.meta-data", "") }}
+            cloud-init-network.tpl: |
+                {% if config_get("user.network-config", "") == "" %}version: 1
+                config:
+                    - type: physical
+                      name: eth0
+                      subnets:
+                          - type: {% if config_get("user.network_mode", "") == "link-local" %}manual{% else %}dhcp{% endif %}
+                            control: auto{% else %}{{ config_get("user.network-config", "") }}{% endif %}
+            cloud-init-user.tpl: |
+                {{ config_get("user.user-data", properties.default) }}
+            cloud-init-vendor.tpl: |
+                {{ config_get("user.vendor-data", properties.default) }}
     ec2: {}
     azure: {}
 
diff --git a/tests/cloud_tests/platforms/base.py b/tests/cloud_tests/platforms/base.py
index 615e2e0..2b6e514 100644
--- a/tests/cloud_tests/platforms/base.py
+++ b/tests/cloud_tests/platforms/base.py
@@ -15,17 +15,7 @@ class Platform(object):
 
     def get_image(self, img_conf):
         """
-        Get image using 'img_conf', where img_conf is a dict containing all
-        image configuration parameters
-
-        in this dict there must be a 'platform_ident' key containing
-        configuration for identifying each image on a per platform basis
-
-        see implementations for get_image() for details about the contents
-        of the platform's config entry
-
-        note: see 'releases' main_config.yaml for example entries
-
+        get image using specified image configuration
         img_conf: configuration for image
         return_value: cloud_tests.images instance
         """
@@ -37,17 +27,4 @@ class Platform(object):
         """
         pass
 
-    def _extract_img_platform_config(self, img_conf):
-        """
-        extract platform configuration for current platform from img_conf
-        """
-        platform_ident = img_conf.get('platform_ident')
-        if not platform_ident:
-            raise ValueError('invalid img_conf, missing \'platform_ident\'')
-        ident = platform_ident.get(self.platform_name)
-        if not ident:
-            raise ValueError('img_conf: {} missing config for platform {}'
-                             .format(img_conf, self.platform_name))
-        return ident
-
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/platforms/lxd.py b/tests/cloud_tests/platforms/lxd.py
index 847cc54..82e2710 100644
--- a/tests/cloud_tests/platforms/lxd.py
+++ b/tests/cloud_tests/platforms/lxd.py
@@ -27,28 +27,30 @@ class LXDPlatform(base.Platform):
 
     def get_image(self, img_conf):
         """
-        Get image
-        img_conf: dict containing config for image. platform_ident must have:
-            alias: alias to use for simplestreams server
-            sstreams_server: simplestreams server to use, or None for default
+        get image using specified image configuration
+        img_conf: configuration for image
         return_value: cloud_tests.images instance
         """
-        lxd_conf = self._extract_img_platform_config(img_conf)
-        image = self.client.images.create_from_simplestreams(
-            lxd_conf.get('sstreams_server', DEFAULT_SSTREAMS_SERVER),
-            lxd_conf['alias'])
-        return lxd_image.LXDImage(
-            image.properties['description'], img_conf, self, image)
+        pylxd_image = self.client.images.create_from_simplestreams(
+            img_conf.get('sstreams_server', DEFAULT_SSTREAMS_SERVER),
+            img_conf['alias'])
+        image = lxd_image.LXDImage(self, img_conf, pylxd_image)
+        if img_conf.get('override_templates', False):
+            image.update_templates(self.config.get('template_overrides', {}),
+                                   self.config.get('template_files', {}))
+        return image
 
-    def launch_container(self, image=None, container=None, ephemeral=False,
-                         config=None, block=True,
+    def launch_container(self, properties, config, image=None, container=None,
+                         ephemeral=False, container_config=None, block=True,
                          image_desc=None, use_desc=None):
         """
         launch a container
+        properties: image properties
+        config: image configuration
         image: image fingerprint to launch from
         container: container to copy
         ephemeral: delete image after first shutdown
-        config: config options for instance as dict
+        container_config: config options for instance as dict
         block: wait until container created
         image_desc: description of image being launched
         use_desc: description of container's use
@@ -61,11 +63,13 @@ class LXDPlatform(base.Platform):
                                            use_desc=use_desc,
                                            used_list=self.list_containers()),
             'ephemeral': bool(ephemeral),
-            'config': config if isinstance(config, dict) else {},
+            'config': (container_config
+                       if isinstance(container_config, dict) else {}),
             'source': ({'type': 'image', 'fingerprint': image} if image else
                        {'type': 'copy', 'source': container})
         }, wait=block)
-        return lxd_instance.LXDInstance(container.name, self, container)
+        return lxd_instance.LXDInstance(self, container.name, properties,
+                                        config, container)
 
     def container_exists(self, container_name):
         """
@@ -88,6 +92,14 @@ class LXDPlatform(base.Platform):
         """
         return [container.name for container in self.client.containers.all()]
 
+    def query_image_by_alias(self, alias):
+        """
+        get image by alias in local image store
+        alias: alias of image
+        return_value: pylxd image (not cloud_tests.images instance)
+        """
+        return self.client.images.get_by_alias(alias)
+
     def destroy(self):
         """
         Clean up platform data
diff --git a/tests/cloud_tests/releases.yaml b/tests/cloud_tests/releases.yaml
index 3ffa68f..b18b095 100644
--- a/tests/cloud_tests/releases.yaml
+++ b/tests/cloud_tests/releases.yaml
@@ -1,79 +1,145 @@
 # ============================= Release Config ================================
 default_release_config:
-    # all are disabled by default
-    enabled: false
-    # timeout for booting image and running cloud init
-    timeout: 120
-    # platform_ident values for the image, with data to identify the image
-    # on that platform. see platforms.base for more information
-    platform_ident: {}
-    # a script to run after a boot that is used to modify an image, before
-    # making a snapshot of the image. may be useful for removing data left
-    # behind from cloud-init booting, such as logs, to ensure that data from
-    # snapshot.launch() will not include a cloud-init.log from a boot used to
-    # create the snapshot, if cloud-init has not run
-    boot_clean_script: |
-        #!/bin/bash
-        rm -rf /var/log/cloud-init.log /var/log/cloud-init-output.log \
-            /var/lib/cloud/ /run/cloud-init/ /var/log/syslog
+    # global default configuration options
+    default:
+        # all are disabled by default
+        enabled: false
+        # timeout for booting image and running cloud init
+        boot_timeout: 120
+        # a script to run after a boot that is used to modify an image, before
+        # making a snapshot of the image. may be useful for removing data left
+        # behind from cloud-init booting, such as logs, to ensure that data
+        # from snapshot.launch() will not include a cloud-init.log from a boot
+        # used to create the snapshot, if cloud-init has not run
+        boot_clean_script: |
+            #!/bin/bash
+            rm -rf /var/log/cloud-init.log /var/log/cloud-init-output.log \
+                /var/lib/cloud/ /run/cloud-init/ /var/log/syslog
+        # test script to determine if system is booted fully
+        system_ready_script: |
+            # permit running or degraded state as both indicate complete boot
+            [ $(systemctl is-system-running) = 'running' -o
+              $(systemctl is-system-running) = 'degraded' ]
+        # test script to determine if cloud-init has finished
+        cloud_init_ready_script: |
+            [ -f '/run/cloud-init/result.json' ]
+
+    # lxd specific default configuration options
+    lxd:
+        # default sstreams server to use for lxd image retrieval
+        sstreams_server: https://us.images.linuxcontainers.org:8443
+        # keep base image, avoids downloading again next run
+        cache_base_image: true
+        # lxd images from linuxcontainers.org do not have the nocloud seed
+        # templates in place, so the image metadata must be modified
+        override_templates: true
+        # arg overrides to set image up
+        setup_overrides:
+            # lxd images from linuxcontainers.org do not come with
+            # cloud-init, so must pull cloud-init in from repo using
+            # setup_image.upgrade
+            upgrade: true
 
 releases:
-    trusty:
-        enabled: true
-        platform_ident:
-            lxd:
-                # if sstreams_server is omitted, default is used, defined in
-                # tests.cloud_tests.platforms.lxd.DEFAULT_SSTREAMS_SERVER as:
-                # sstreams_server: https://us.images.linuxcontainers.org:8443
-                #alias: ubuntu/trusty/default
-                alias: t
-                sstreams_server: https://cloud-images.ubuntu.com/daily
-    xenial:
-        enabled: true
-        platform_ident:
-            lxd:
-                #alias: ubuntu/xenial/default
-                alias: x
-                sstreams_server: https://cloud-images.ubuntu.com/daily
-    yakkety:
-        enabled: true
-        platform_ident:
-            lxd:
-                #alias: ubuntu/yakkety/default
-                alias: y
-                sstreams_server: https://cloud-images.ubuntu.com/daily
+    # UBUNTU =================================================================
     zesty:
-        enabled: true
-        platform_ident:
-            lxd:
-                #alias: ubuntu/zesty/default
-                alias: z
-                sstreams_server: https://cloud-images.ubuntu.com/daily
-    jessie:
-        platform_ident:
-            lxd:
-                alias: debian/jessie/default
+        # EOL: Jan 2018
+        default:
+            enabled: true
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: zesty
+            setup_overrides: null
+            override_templates: false
+    yakkety:
+        # EOL: Jul 2017
+        default:
+            enabled: true
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: yakkety
+            setup_overrides: null
+            override_templates: false
+    xenial:
+        # EOL: Apr 2021
+        default:
+            enabled: true
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: xenial
+            setup_overrides: null
+            override_templates: false
+    trusty:
+        # EOL: Apr 2019
+        default:
+            enabled: true
+            system_ready_script: |
+                #!/bin/bash
+                # upstart based, so use old style runlevels
+                [ $(runlevel | awk '{print $2}') = '2' ]
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: trusty
+            setup_overrides: null
+            override_templates: false
+    precise:
+        # EOL: Apr 2017
+        default:
+            # still supported but not relevant for development, not enabled
+            # tests should still work though unless they use newer features
+            enabled: false
+            system_ready_script: |
+                #!/bin/bash
+                # upstart based, so use old style runlevels
+                [ $(runlevel | awk '{print $2}') = '2' ]
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: precise
+            setup_overrides: null
+            override_templates: false
+    # DEBIAN =================================================================
     sid:
-        platform_ident:
-            lxd:
-                alias: debian/sid/default
+        # EOL: N/A
+        default:
+            # tests should work on sid, however it is not always stable
+            enabled: false
+        lxd:
+            alias: debian/sid/default
     stretch:
-        platform_ident:
-            lxd:
-                alias: debian/stretch/default
+        # EOL: Not yet released
+        default:
+            enabled: true
+        lxd:
+            alias: debian/stretch/default
+    jessie:
+        # EOL: Jun 2020
+        default:
+            enabled: true
+        lxd:
+            alias: debian/jessie/default
     wheezy:
-        platform_ident:
-            lxd:
-                alias: debian/wheezy/default
+        # EOL: May 2018 (Apr 2016 - end of full updates)
+        default:
+            # this is old enough that it is no longer relevant for development
+            enabled: false
+        lxd:
+            alias: debian/wheezy/default
+    # CENTOS =================================================================
     centos70:
-        timeout: 180
-        platform_ident:
-            lxd:
-                alias: centos/7/default
+        # EOL: Jun 2024 (2020 - end of full updates)
+        default:
+            enabled: true
+        lxd:
+            alias: centos/7/default
     centos66:
-        timeout: 180
-        platform_ident:
-            lxd:
-                alias: centos/6/default
+        # EOL: Nov 2020
+        default:
+            enabled: true
+            # still supported, but only bugfixes after may 2017
+            system_ready_script: |
+                #!/bin/bash
+                [ $(runlevel | awk '{print $2}') = '3' ]
+        lxd:
+            alias: centos/6/default
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/setup_image.py b/tests/cloud_tests/setup_image.py
index 5d6c638..1b74ceb 100644
--- a/tests/cloud_tests/setup_image.py
+++ b/tests/cloud_tests/setup_image.py
@@ -7,6 +7,30 @@ from functools import partial
 import os
 
 
+def installed_version(image, package, ensure_installed=True):
+    """
+    get installed version of package
+    image: cloud_tests.images instance to operate on
+    package: name of package
+    ensure_installed: raise error if not installed
+    return_value: cloud-init version string
+    """
+    # get right cmd for os family
+    os_family = util.get_os_family(image.properties['os'])
+    if os_family == 'debian':
+        cmd = ['dpkg-query', '-W', "--showformat='${Version}'", package]
+    elif os_family == 'redhat':
+        cmd = ['rpm', '-q', '--queryformat', "'%{VERSION}'", package]
+    else:
+        raise NotImplementedError
+
+    # query version
+    msg = 'query version for package: {}'.format(package)
+    (out, err, exit) = image.execute(
+        cmd, description=msg, rcs=(0,) if ensure_installed else range(0, 256))
+    return out.strip()
+
+
 def install_deb(args, image):
     """
     install deb into image
@@ -21,20 +45,18 @@ def install_deb(args, image):
                                   'family: {}'.format(args.deb, os_family))
 
     # install deb
-    LOG.debug('installing deb: %s into target', args.deb)
+    msg = 'install deb: "{}" into target'.format(args.deb)
+    LOG.debug(msg)
     remote_path = os.path.join('/tmp', os.path.basename(args.deb))
     image.push_file(args.deb, remote_path)
-    (out, err, exit) = image.execute(['dpkg', '-i', remote_path])
-    if exit != 0:
-        raise OSError('failed install deb: {}\n\tstdout: {}\n\tstderr: {}'
-                      .format(args.deb, out, err))
+    cmd = 'dpkg -i {} || apt-get install --yes -f'.format(remote_path)
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
 
     # check installed deb version matches package
     fmt = ['-W', "--showformat='${Version}'"]
     (out, err, exit) = image.execute(['dpkg-deb'] + fmt + [remote_path])
     expected_version = out.strip()
-    (out, err, exit) = image.execute(['dpkg-query'] + fmt + ['cloud-init'])
-    found_version = out.strip()
+    found_version = installed_version(image, 'cloud-init')
     if expected_version != found_version:
         raise OSError('install deb version "{}" does not match expected "{}"'
                       .format(found_version, expected_version))
@@ -52,24 +74,21 @@ def install_rpm(args, image):
     """
     # ensure system is compatible with package format
     os_family = util.get_os_family(image.properties['os'])
-    if os_family not in ['redhat', 'sles']:
+    if os_family != 'redhat':
         raise NotImplementedError('install rpm: {} not supported on os '
                                   'family: {}'.format(args.rpm, os_family))
 
     # install rpm
-    LOG.debug('installing rpm: %s into target', args.rpm)
+    msg = 'install rpm: "{}" into target'.format(args.rpm)
+    LOG.debug(msg)
     remote_path = os.path.join('/tmp', os.path.basename(args.rpm))
     image.push_file(args.rpm, remote_path)
-    (out, err, exit) = image.execute(['rpm', '-U', remote_path])
-    if exit != 0:
-        raise OSError('failed to install rpm: {}\n\tstdout: {}\n\tstderr: {}'
-                      .format(args.rpm, out, err))
+    image.execute(['rpm', '-U', remote_path], description=msg)
 
     fmt = ['--queryformat', '"%{VERSION}"']
     (out, err, exit) = image.execute(['rpm', '-q'] + fmt + [remote_path])
     expected_version = out.strip()
-    (out, err, exit) = image.execute(['rpm', '-q'] + fmt + ['cloud-init'])
-    found_version = out.strip()
+    found_version = installed_version(image, 'cloud-init')
     if expected_version != found_version:
         raise OSError('install rpm version "{}" does not match expected "{}"'
                       .format(found_version, expected_version))
@@ -80,13 +99,34 @@ def install_rpm(args, image):
 
 def upgrade(args, image):
     """
-    run the system's upgrade command
+    upgrade or install cloud-init from repo
+    args: cmdline arguments
+    image: cloud_tests.images instance to operate on
+    return_value: None, may raise errors
+    """
+    # determine command for os_family
+    os_family = util.get_os_family(image.properties['os'])
+    if os_family == 'debian':
+        cmd = 'apt-get update && apt-get install cloud-init --yes'
+    elif os_family == 'redhat':
+        cmd = 'yum install cloud-init --assumeyes'
+    else:
+        raise NotImplementedError
+
+    # upgrade cloud-init
+    msg = 'upgrading cloud-init'
+    LOG.debug(msg)
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
+
+
+def upgrade_full(args, image):
+    """
+    run the system's full upgrade command
     args: cmdline arguments
     image: cloud_tests.images instance to operate on
     return_value: None, may raise errors
     """
     # determine appropriate upgrade command for os_family
-    # TODO: maybe use cloudinit.distros for this?
     os_family = util.get_os_family(image.properties['os'])
     if os_family == 'debian':
         cmd = 'apt-get update && apt-get upgrade --yes'
@@ -97,11 +137,9 @@ def upgrade(args, image):
                                   'from family: {}'.format(os_family))
 
     # upgrade system
-    LOG.debug('upgrading system')
-    (out, err, exit) = image.execute(['/bin/sh', '-c', cmd])
-    if exit != 0:
-        raise OSError('failed to upgrade system\n\tstdout: {}\n\tstderr:{}'
-                      .format(out, err))
+    msg = 'full system upgrade'
+    LOG.debug(msg)
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
 
 
 def run_script(args, image):
@@ -111,9 +149,9 @@ def run_script(args, image):
     image: cloud_tests.images instance to operate on
     return_value: None, may raise errors
     """
-    # TODO: get exit status back from script and add error handling here
-    LOG.debug('running setup image script in target image')
-    image.run_script(args.script)
+    msg = 'run setup image script in target image'
+    LOG.debug(msg)
+    image.run_script(args.script, description=msg)
 
 
 def enable_ppa(args, image):
@@ -124,17 +162,15 @@ def enable_ppa(args, image):
     return_value: None, may raise errors
     """
     # ppa only supported on ubuntu (maybe debian?)
-    if image.properties['os'] != 'ubuntu':
+    if image.properties['os'].lower() != 'ubuntu':
         raise NotImplementedError('enabling a ppa is only available on ubuntu')
 
     # add ppa with add-apt-repository and update
     ppa = 'ppa:{}'.format(args.ppa)
-    LOG.debug('enabling %s', ppa)
+    msg = 'enable ppa: "{}" in target'.format(ppa)
+    LOG.debug(msg)
     cmd = 'add-apt-repository --yes {} && apt-get update'.format(ppa)
-    (out, err, exit) = image.execute(['/bin/sh', '-c', cmd])
-    if exit != 0:
-        raise OSError('enable ppa for {} failed\n\tstdout: {}\n\tstderr: {}'
-                      .format(ppa, out, err))
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
 
 
 def enable_repo(args, image):
@@ -155,11 +191,9 @@ def enable_repo(args, image):
         raise NotImplementedError('enable repo command not configured for '
                                   'distro from family: {}'.format(os_family))
 
-    LOG.debug('enabling repo: "%s"', args.repo)
-    (out, err, exit) = image.execute(['/bin/sh', '-c', cmd])
-    if exit != 0:
-        raise OSError('enable repo {} failed\n\tstdout: {}\n\tstderr: {}'
-                      .format(args.repo, out, err))
+    msg = 'enable repo: "{}" in target'.format(args.repo)
+    LOG.debug(msg)
+    image.execute(['/bin/sh', '-c', cmd], description=msg)
 
 
 def setup_image(args, image):
@@ -169,6 +203,11 @@ def setup_image(args, image):
     image: cloud_tests.image instance to operate on
     return_value: tuple of results and fail count
     """
+    # update the args if necessary for this image
+    overrides = image.setup_overrides
+    LOG.debug('updating args for setup with: %s', overrides)
+    args = util.update_args(args, overrides, preserve_old=True)
+
     # mapping of setup cmdline arg name to setup function
     # represented as a tuple rather than a dict or odict as lookup by name not
     # needed, and order is important as --script and --upgrade go at the end
@@ -179,17 +218,19 @@ def setup_image(args, image):
         ('repo', enable_repo, 'setup func for --repo, enable repo'),
         ('ppa', enable_ppa, 'setup func for --ppa, enable ppa'),
         ('script', run_script, 'setup func for --script, run script'),
-        ('upgrade', upgrade, 'setup func for --upgrade, upgrade pkgs'),
+        ('upgrade', upgrade, 'setup func for --upgrade, upgrade cloud-init'),
+        ('upgrade-full', upgrade_full, 'setup func for --upgrade-full'),
     )
 
     # determine which setup functions needed
     calls = [partial(stage.run_single, desc, partial(func, args, image))
              for name, func, desc in handlers if getattr(args, name, None)]
 
-    image_name = 'image: distro={}, release={}'.format(
-        image.properties['os'], image.properties['release'])
-    LOG.info('setting up %s', image_name)
-    return stage.run_stage('set up for {}'.format(image_name), calls,
-                           continue_after_error=False)
+    LOG.info('setting up %s', image)
+    res = stage.run_stage(
+        'set up for {}'.format(image), calls, continue_after_error=False)
+    LOG.debug('after setup complete, installed cloud-init version is: %s',
+              installed_version(image, 'cloud-init'))
+    return res
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/snapshots/base.py b/tests/cloud_tests/snapshots/base.py
index d715f03..e9b721e 100644
--- a/tests/cloud_tests/snapshots/base.py
+++ b/tests/cloud_tests/snapshots/base.py
@@ -7,10 +7,11 @@ class Snapshot(object):
     """
     platform_name = None
 
-    def __init__(self, properties, config):
+    def __init__(self, platform, properties, config):
         """
         Set up snapshot
         """
+        self.platform = platform
         self.properties = properties
         self.config = config
 
diff --git a/tests/cloud_tests/snapshots/lxd.py b/tests/cloud_tests/snapshots/lxd.py
index eabbce3..d2a123a 100644
--- a/tests/cloud_tests/snapshots/lxd.py
+++ b/tests/cloud_tests/snapshots/lxd.py
@@ -9,13 +9,12 @@ class LXDSnapshot(base.Snapshot):
     """
     platform_name = "lxd"
 
-    def __init__(self, properties, config, platform, pylxd_frozen_instance):
+    def __init__(self, platform, properties, config, pylxd_frozen_instance):
         """
         Set up snapshot
         """
-        self.platform = platform
         self.pylxd_frozen_instance = pylxd_frozen_instance
-        super(LXDSnapshot, self).__init__(properties, config)
+        super(LXDSnapshot, self).__init__(platform, properties, config)
 
     def launch(self, user_data, meta_data=None, block=True, start=True,
                use_desc=None):
@@ -34,10 +33,11 @@ class LXDSnapshot(base.Snapshot):
         if meta_data:
             inst_config['user.meta-data'] = meta_data
         instance = self.platform.launch_container(
-            container=self.pylxd_frozen_instance.name, config=inst_config,
-            block=block, image_desc=str(self), use_desc=use_desc)
+            self.properties, self.config, block=block, image_desc=str(self),
+            container=self.pylxd_frozen_instance.name, use_desc=use_desc,
+            container_config=inst_config)
         if start:
-            instance.start(wait=True, wait_time=self.config.get('timeout'))
+            instance.start()
         return instance
 
     def destroy(self):
diff --git a/tests/cloud_tests/util.py b/tests/cloud_tests/util.py
index 64a8667..b47862e 100644
--- a/tests/cloud_tests/util.py
+++ b/tests/cloud_tests/util.py
@@ -1,5 +1,6 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
+import copy
 import glob
 import os
 import random
@@ -7,10 +8,18 @@ import string
 import tempfile
 import yaml
 
-from cloudinit.distros import OSFAMILIES
 from cloudinit import util as c_util
 from tests.cloud_tests import LOG
 
+OS_FAMILY_MAPPING = {
+    'debian': ['debian', 'ubuntu'],
+    'redhat': ['centos', 'rhel', 'fedora'],
+    'gentoo': ['gentoo'],
+    'freebsd': ['freebsd'],
+    'suse': ['sles'],
+    'arch': ['arch'],
+}
+
 
 def list_test_data(data_dir):
     """
@@ -68,7 +77,7 @@ def gen_instance_name(prefix='cloud-test', image_desc=None, use_desc=None,
         """
         filter bad characters out of elem and trim to length
         """
-        elem = elem[:max_len] if elem else unknown
+        elem = elem.lower()[:max_len] if elem else unknown
         return ''.join(c if c in valid else delim for c in elem)
 
     return next(name for name in
@@ -88,7 +97,8 @@ def get_os_family(os_name):
     """
     get os family type for os_name
     """
-    return next((k for k, v in OSFAMILIES.items() if os_name in v), None)
+    return next((k for k, v in OS_FAMILY_MAPPING.items()
+                 if os_name.lower() in v), None)
 
 
 def current_verbosity():
@@ -158,6 +168,81 @@ def write_file(*args, **kwargs):
     """
     write a file using cloudinit.util.write_file
     """
-    c_util.write_file(*args, **kwargs)
+    return c_util.write_file(*args, **kwargs)
+
+
+def read_conf(*args, **kwargs):
+    """
+    read configuration using cloudinit.util.read_conf
+    """
+    return c_util.read_conf(*args, **kwargs)
+
+
+def subp(*args, **kwargs):
+    """
+    execute a command on the system shell using cloudinit.util.subp
+    """
+    return c_util.subp(*args, **kwargs)
+
+
+def tmpdir(prefix='cloud_test_util_'):
+    return tempfile.mkdtemp(prefix=prefix)
+
+
+def rel_files(basedir):
+    """
+    list of files under directory by relative path, not including directories
+    return_value: list or relative paths
+    """
+    basedir = os.path.normpath(basedir)
+    return [path[len(basedir) + 1:] for path in
+            glob.glob(os.path.join(basedir, '**'), recursive=True)
+            if not os.path.isdir(path)]
+
+
+def flat_tar(output, basedir, owner='root', group='root'):
+    """
+    create a flat tar archive (no leading ./) from basedir
+    output: output tar file to write
+    basedir: base directory for archive
+    owner: owner of archive files
+    group: group archive files belong to
+    return_value: none
+    """
+    c_util.subp(['tar', 'cf', output, '--owner', owner, '--group', group,
+                 '-C', basedir] + rel_files(basedir), capture=True)
+
+
+def update_args(args, updates, preserve_old=True):
+    """
+    update cmdline arguments from a dictionary
+    args: cmdline arguments
+    updates: dictionary of {arg_name: new_value} mappings
+    preserve_old: if true, create a deep copy of args before updating
+    return_value: updated cmdline arguments, as new object if preserve_old=True
+    """
+    args = copy.deepcopy(args) if preserve_old else args
+    if updates:
+        vars(args).update(updates)
+    return args
+
+
+class InTargetExecuteError(c_util.ProcessExecutionError):
+    """
+    Error type for in target commands that fail
+    """
+    default_desc = 'Unexpected error while running command in target instance'
+
+    def __init__(self, stdout, stderr, exit_code, cmd, instance,
+                 description=None):
+        """
+        init error and parent error class
+        """
+        if isinstance(cmd, (tuple, list)):
+            cmd = ' '.join(cmd)
+        super(InTargetExecuteError, self).__init__(
+            stdout=stdout, stderr=stderr, exit_code=exit_code, cmd=cmd,
+            reason="Instance: {}".format(instance),
+            description=description if description else self.default_desc)
 
 # vi: ts=4 expandtab
diff --git a/tox.ini b/tox.ini
index f016f20..d8e99e0 100644
--- a/tox.ini
+++ b/tox.ini
@@ -93,4 +93,4 @@ basepython = python3
 commands = {envpython} -m tests.cloud_tests {posargs}
 passenv = HOME
 deps =
-    pylxd==2.1.3
+    pylxd==2.2.3