Merge lp:~bcsaller/pyjuju/lxc-lib into lp:pyjuju

[Benjamin Saller]

This branch creates a LXCContainer which wraps the lxc cli tools in a way that easy to integrate with twisted.

The associated tests are skipped by default unless

TEST_LXC=1

is in the shell environment.

Other items of note:
- `ensemble/lib/lxc/data/lxc-ensemble` must be copied to `/usr/lib/lxc/templates/`
- container creation requires root. The tests will ask for `sudo` permissions to function.

[Gustavo Niemeyer (niemeyer)]

I'm lacking a bit of context on the content of the branch. We debated this in Austin,
and I was told that we were reusing existing tools for maintaining containers, rather
than rolling our own. This branch includes a 600+ line shell script to maintain
containers. There's some misunderstanding we should catch up on.

review: Needs Information

[Kapil Thangavelu (hazmat)]

This no longer needs a fork of the upstream lxc template, i talked to ben and it looks like its good for another review.

[Gustavo Niemeyer (niemeyer)]

Quite a few details, but it looks very nice overall.V

+1 with these sorted.

[1]

=== added file 'ensemble/lib/lxc/data/ensemble-create'

Nicely organized script, thanks.

[2]

+ ENSEMBLE_SOURCE="ppa:ensemble/ppa"
+ elif [ $ENSEMBLE_ORIGIN = "lp" ]; then
+ echo "Using Ensemble Branch $ensemble_source"

s/$ensemble_source/$ENSEMBLE_SOURCE/

Environment variables are case sensitive.

Also, let's s/lp/branch/ please. Seems more clear and PPAs
are also in Launchpad.

As a minor, s/Branch/branch/

[3]

+ elif [ $ENSEMBLE_ORIGIN = "distro" ]; then
+ echo "Using Ensemble distribution packages"
+ fi

Should error if $ENSEMBLE_ORIGIN contains garbage.

[4]

+ ENSEMBLE_SOURCE="ppa:ensemble/ppa"
(...)
+ apt-add-repository $ENSEMBLE_SOURCE

Seems unnecessary to use the variable if the script is
hardcoding the behavior. Just use the value itself in
the second line.

[5]

- env_defaults = environment.get_serialization_data()
- options.placement = env_defaults[environment.name].get(
- "placement", UNASSIGNED_POLICY)
+ options.placement = environment.placement

Nice.

[6]

+BASE_PATH = os.path.normpath(
+ os.path.abspath(
+ os.path.join(os.path.dirname(__file__), "data")))

s/BASE_PATH/DATA_PATH/?

Also, abspath calls normpath internally.

[7]

+class LXCError(EnsembleError):
+ """Indicates a low level error with an LXC container"""
+ pass

/pass/d

[8]

+ p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ env=dict(os.environ))
+ r = p.wait()
+ if r != 0:
+ # read the stdout/err streams and show the user
+ print >>sys.stderr, p.stderr.read()
+ raise LXCError(p.stdout.read())

This logic is suboptimal in a few ways:

- The stderr and stdout will be separated out, which means depending on
  the output it will be hard to interpret what the error was about.

- In case stdout or stderr get enough data to fill the kernel buffer,
  the p.wait() will hang completely and never return.

- The argument of env provided is the default behavior if not provided.

- The stderr printed has no context.

To fix all of these, use something along these lines:

    p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    output = p.communicate()[0]
    if p.wait() != 0:
        raise LXCError(output)

Note the argument of stderr (STDOUT, not PIPE). This will make them share
the same file descriptor and intersperse the output as it happens, fixing
the first point. communicate() avoids the second problem.

[9]

+ "-f", config_file,
+ "--",
+ "-r", release]

Weird.. what's the -- about for lxc-create? Can you please add a comment
in the code?

[10]

+ return _cmd(args)
+
+
+def _lxc_start(container_name, console_log=None):

As a suggestion only, I'd remove the two spaces between functions. I know
about PEP8, but this is one of those cases where it feels detrimental to
readability.

[11]

+ subprocess.Popen(["sudo", "lxc-stop", "-n",
+ container_name],
(...)
+ output = subprocess.Popen(["lxc-ls"],
+ ...

review: Approve

[Kapil Thangavelu (hazmat)]

Gustavo has done a great review on details, some additional high level things...

[0]

While building on top of lxc-lib, i ended up needing two additional functionalities, listing all containers with a given prefix, and finding out if a given container is running. The current usage of the running attribute is broken for usages where the container was not started by the same instance of an lxccontainer class.

i used this implementation to satisfy both
http://bazaar.launchpad.net/~hazmat/ensemble/local-provider/view/head:/ensemble/providers/lxc/container.py

I'm fine with this branch being merged without, i can incorporate this fix directly into lib/lxc this in a subsequent local-dev branch.

[1] Pep8 nitpicks

http://paste.ubuntu.com/682624/

[2] Re gustavo's _cmd comments, it would be nice if this was made public or pushed into a separate process module inside of ensemble/lib.. I'm currently using subprocess.check_output/subprocess.check_call very regularly, which are bit suboptimal in light of gustavo's comments.

cheers,

Kapil

review: Approve

[Kapil Thangavelu (hazmat)]

also the script should be using ENSEMBLE_ZOOKEEPER instead of ZOOKEEPER_ADDRESS

[lp:~bcsaller/pyjuju/lxc-lib]

316. By Benjamin Saller on 2011-09-08

review changes and test fixes

317. By Benjamin Saller on 2011-09-08

allow _cmd to return (returncode, output) for debugging

318. By Benjamin Saller on 2011-09-08

use _cmd everywhere

319. By Benjamin Saller on 2011-09-08

pep8

320. By Benjamin Saller on 2011-09-08

s/ZOOKEEPER_ADDRESS/ENSEMBLE_ZOOKEEPER/

[Benjamin Saller (bcsaller)]

Responded to these point and Kapils later one. All changes have been
addressed. Re [22] the tests are not run as root, they ask for
permissions the first time they need them. When not run as root they
require interactivity but they don't run by default in the test suite
(as they can be quite expensive timewise and do need perms).

On Sat, Sep 3, 2011 at 9:25 AM, Gustavo Niemeyer <email address hidden> wrote:
> Review: Approve
> Quite a few details, but it looks very nice overall.V
>
> +1 with these sorted.
>
>
> [1]
>
> === added file 'ensemble/lib/lxc/data/ensemble-create'
>
> Nicely organized script, thanks.
>
> [2]
>
> +        ENSEMBLE_SOURCE="ppa:ensemble/ppa"
> +    elif [ $ENSEMBLE_ORIGIN = "lp" ]; then
> +        echo "Using Ensemble Branch $ensemble_source"
>
> s/$ensemble_source/$ENSEMBLE_SOURCE/
>
> Environment variables are case sensitive.
>
> Also, let's s/lp/branch/ please. Seems more clear and PPAs
> are also in Launchpad.
>
> As a minor, s/Branch/branch/
>
> [3]
>
> +    elif [ $ENSEMBLE_ORIGIN = "distro" ]; then
> +        echo "Using Ensemble distribution packages"
> +    fi
>
> Should error if $ENSEMBLE_ORIGIN contains garbage.
>
> [4]
>
> +        ENSEMBLE_SOURCE="ppa:ensemble/ppa"
> (...)
> +        apt-add-repository $ENSEMBLE_SOURCE
>
> Seems unnecessary to use the variable if the script is
> hardcoding the behavior.  Just use the value itself in
> the second line.
>
> [5]
>
> -        env_defaults = environment.get_serialization_data()
> -        options.placement = env_defaults[environment.name].get(
> -            "placement", UNASSIGNED_POLICY)
> +        options.placement = environment.placement
>
> Nice.
>
> [6]
>
> +BASE_PATH = os.path.normpath(
> +    os.path.abspath(
> +        os.path.join(os.path.dirname(__file__), "data")))
>
> s/BASE_PATH/DATA_PATH/?
>
> Also, abspath calls normpath internally.
>
> [7]
>
> +class LXCError(EnsembleError):
> +    """Indicates a low level error with an LXC container"""
> +    pass
>
> /pass/d
>
> [8]
>
> +    p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
> +                         env=dict(os.environ))
> +    r = p.wait()
> +    if  r != 0:
> +        # read the stdout/err streams and show the user
> +        print >>sys.stderr, p.stderr.read()
> +        raise LXCError(p.stdout.read())
>
> This logic is suboptimal in a few ways:
>
> - The stderr and stdout will be separated out, which means depending on
>  the output it will be hard to interpret what the error was about.
>
> - In case stdout or stderr get enough data to fill the kernel buffer,
>  the p.wait() will hang completely and never return.
>
> - The argument of env provided is the default behavior if not provided.
>
> - The stderr printed has no context.
>
> To fix all of these, use something along these lines:
>
>    p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>    output = p.communicate()[0]
>    if p.wait() != 0:
>        raise LXCError(output)
>
> Note the argument of stderr (STDOUT, not PIPE). This will make them share
> the same file descriptor and intersperse the output as it happens, fixing
> the first point. communicate() avoids the second proble...

=== modified file 'ensemble/control/add_unit.py'
--- ensemble/control/add_unit.py	2011-08-08 17:25:27 +0000
+++ ensemble/control/add_unit.py	2011-09-08 23:54:45 +0000
@@ -28,9 +28,7 @@
     """Add a new service unit."""
     environment = get_environment(options)
     if not options.placement:
-        env_defaults = environment.get_serialization_data()
-        options.placement = env_defaults[environment.name].get(
-            "placement", UNASSIGNED_POLICY)
+        options.placement = environment.placement
 
     return add_unit(
         options.environments,

=== modified file 'ensemble/control/deploy.py'
--- ensemble/control/deploy.py	2011-08-17 18:42:35 +0000
+++ ensemble/control/deploy.py	2011-09-08 23:54:45 +0000
@@ -56,9 +56,7 @@
     environment = get_environment(options)
 
     if not options.placement:
-        env_defaults = environment.get_serialization_data()
-        options.placement = env_defaults[environment.name].get(
-            "placement", UNASSIGNED_POLICY)
+        options.placement = environment.placement
 
     return deploy(
         options.environments,

=== modified file 'ensemble/environment/environment.py'
--- ensemble/environment/environment.py	2011-08-09 01:41:35 +0000
+++ ensemble/environment/environment.py	2011-09-08 23:54:45 +0000
@@ -40,5 +40,10 @@
 
     @property
     def placement(self):
-        """Unit placement policy name or None"""
+        """The name of the default placement policy.
+
+        If the environment doesn't have a default unit placement
+        policy None is returned
+        """
         return self._environment_config.get("placement")
+

=== added directory 'ensemble/lib/lxc'
=== added file 'ensemble/lib/lxc/__init__.py'
--- ensemble/lib/lxc/__init__.py	1970-01-01 00:00:00 +0000
+++ ensemble/lib/lxc/__init__.py	2011-09-08 23:54:45 +0000
@@ -0,0 +1,231 @@
+import os
+import pipes
+import subprocess
+import sys
+import tempfile
+
+from twisted.internet.defer import inlineCallbacks, returnValue
+from twisted.internet.threads import deferToThread
+
+from ensemble.errors import EnsembleError
+
+DATA_PATH = os.path.abspath(
+    os.path.join(os.path.dirname(__file__), "data"))
+
+
+class LXCError(EnsembleError):
+    """Indicates a low level error with an LXC container"""
+
+
+def _cmd(args):
+    p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+    stdout_data, _ = p.communicate()
+    r = p.returncode
+    if  r != 0:
+        # read the stdout/err streams and show the user
+        print >>sys.stderr, stdout_data
+        raise LXCError(stdout_data)
+    return (r, stdout_data)
+
+
+# Wrapped lxc cli primitives
+def _lxc_create(
+    container_name, template="ubuntu", config_file=None, release="oneiric"):
+    # the -- argument indicates the last parameters are passed
+    # to the template and not handled by lxc-create
+    args = ["sudo", "lxc-create",
+            "-n", container_name,
+            "-t", template,
+            "-f", config_file,
+            "--",
+            "-r", release]
+    return _cmd(args)
+
+
+def _lxc_start(container_name, console_log=None):
+    args = ["sudo", "lxc-start", "--daemon", "-n", container_name]
+    if console_log:
+        args.append("-c")
+        args.append(console_log)
+    return _cmd(args)
+
+
+def _lxc_stop(container_name):
+    _cmd(["sudo", "lxc-stop", "-n", container_name])
+
+
+def _lxc_destroy(container_name):
+    return _cmd(["sudo", "lxc-destroy", "-n", container_name])
+
+
+def _lxc_ls():
+    _, output = _cmd(["lxc-ls"])
+    output = output.replace("\n", " ")
+    return set([c for c in output.split(" ") if c])
+
+
+def _lxc_wait(container_name, state="RUNNING"):
+    """Wait for container to be in a given state RUNNING|STOPPED."""
+
+    def wait(container_name):
+        rc, _ = _cmd(["sudo", "lxc-wait",
+                   "-n", container_name,
+                   "-s", state])
+        return rc == 0
+
+    return deferToThread(wait, container_name)
+
+
+def _customize_container(customize_script, container_root):
+    if not os.path.isdir(container_root):
+        raise LXCError("Expect container root directory: %s" %
+                       container_root)
+    # write the container scripts into the container
+    fd, in_path = tempfile.mkstemp(prefix=os.path.basename(customize_script),
+                                   dir=os.path.join(container_root, "tmp"))
+
+    os.write(fd, open(customize_script, "r").read())
+    os.close(fd)
+    os.chmod(in_path, 0755)
+
+    args = ["sudo", "chroot", container_root,
+            os.path.join("/tmp", os.path.basename(in_path))]
+    return _cmd(args)
+
+
+def validate_path(pathname):
+    if not os.access(pathname, os.R_OK):
+        raise LXCError("Invalid or unreadable file: %s" % pathname)
+
+
+@inlineCallbacks
+def get_containers(prefix):
+    """Return a dictionary of containers key names to runtime boolean value.
+
+    :param prefix: Optionally specify a prefix that the container should
+    match any returned containers.
+    """
+    _, output = yield deferToThread(_cmd, ["lxc-ls"])
+
+    containers = {}
+    for i in filter(None, output.split("\n")):
+        if i in containers:
+            containers[i] = True
+        else:
+            containers[i] = False
+
+    if prefix:
+        remove = [k for k in containers.keys() if not
+                  k.startswith(prefix)]
+        map(containers.pop, remove)
+
+    returnValue(containers)
+
+
+class LXCContainer(object):
+    def __init__(self,
+                 container_name, configuration=None, network_name="virbr0",
+                 customize_script=None):
+        """Create an LXCContainer
+
+        :param container_name: should be unique within the system
+
+        :param configuration: dict of values which are written into
+        the containers /etc/ensemble/ensemble.conf file and expected
+        to be consumed by the customize script
+
+        :param network_name: name of network link
+
+        :param customize_script: script used inside container to make
+        it ensemble ready
+        """
+
+        self.container_name = container_name
+
+        if customize_script is None:
+            customize_script = os.path.join(DATA_PATH,
+                                            "ensemble-create")
+        self.customize_script = customize_script
+        validate_path(self.customize_script)
+
+        self.configuration = configuration
+        self.network_name = network_name
+        self.running = False
+
+    @property
+    def rootfs(self):
+        return "/var/lib/lxc/%s/rootfs/" % self.container_name
+
+    def _p(self, path):
+        if path[0] == "/":
+            path = path[1:]
+        return os.path.join(self.rootfs, path)
+
+    def _make_lxc_config(self, network_name):
+        lxc_config = os.path.join(DATA_PATH, "lxc.conf")
+        with open(lxc_config, "r") as fh:
+            template = fh.read()
+            fd, output_fn = tempfile.mkstemp(suffix=".conf")
+            output_config = open(output_fn, "w")
+            output_config.write(template % {"network_name": network_name})
+            output_config.close()
+
+            validate_path(output_fn)
+            return output_fn
+
+    @inlineCallbacks
+    def _customize_container(self):
+        config_dir = self._p("etc/ensemble")
+        if not os.path.exists(config_dir):
+            _cmd(["sudo", "mkdir", config_dir])
+
+        if not self.configuration:
+            return
+
+        _, fn = tempfile.mkstemp()
+        with open(fn, "w") as fp:
+            for name, value in self.configuration.items():
+                fp.write("%s=%s\n" % (name.upper(), pipes.quote(value)))
+
+        _cmd(["sudo", "mv", fn,
+              self._p("etc/ensemble/ensemble.conf")])
+
+        yield deferToThread(
+            _customize_container, self.customize_script, self.rootfs)
+
+    @inlineCallbacks
+    def execute(self, args):
+        if not isinstance(args, (list, tuple)):
+            args = [args, ]
+
+        args = ["sudo", "chroot", self.rootfs] + args
+        yield deferToThread(_cmd, args)
+
+    @inlineCallbacks
+    def create(self):
+        # open the template file and create a new temp processed
+        # version
+        lxc_config = self._make_lxc_config(self.network_name)
+        yield deferToThread(_lxc_create, self.container_name,
+                            config_file=lxc_config)
+        yield self._customize_container()
+        os.unlink(lxc_config)
+
+    @inlineCallbacks
+    def run(self):
+        if not os.path.exists(self.rootfs):
+            yield self.create()
+
+        yield deferToThread(_lxc_start, self.container_name)
+        self.running = yield _lxc_wait(self.container_name, "RUNNING")
+
+    @inlineCallbacks
+    def stop(self):
+        yield deferToThread(_lxc_stop, self.container_name)
+        yield _lxc_wait(self.container_name, "STOPPED")
+        self.running = False
+
+    @inlineCallbacks
+    def destroy(self):
+        yield self.stop()
+        yield deferToThread(_lxc_destroy, self.container_name)

=== added directory 'ensemble/lib/lxc/data'
=== added file 'ensemble/lib/lxc/data/ensemble-create'
--- ensemble/lib/lxc/data/ensemble-create	1970-01-01 00:00:00 +0000
+++ ensemble/lib/lxc/data/ensemble-create	2011-09-08 23:54:45 +0000
@@ -0,0 +1,63 @@
+#!bin/bash
+set -x
+
+setup_ensemble()
+{
+    if [ ! -e /var/lib/ensemble ]; then
+        mkdir /var/lib/ensemble
+    fi
+
+    if [ ! -e /var/log/ensemble ]; then
+        mkdir /var/log/ensemble
+    fi
+
+    if [ $ENSEMBLE_ORIGIN = "ppa" ]; then
+        echo "Using Ensemble PPA for container"
+    elif [ $ENSEMBLE_ORIGIN = "branch" ]; then
+        echo "Using Ensemble Branch $ENSEMBLE_SOURCE"
+    elif [ $ENSEMBLE_ORIGIN = "distro" ]; then
+        echo "Using Ensemble distribution packages"
+    else
+        echo "Unknown Ensemble origin policy $ENSEMBLE_ORIGIN: expected [ppa|branch|disto]"
+        exit 1
+    fi
+
+    echo "Setting up ensemble in container"
+    apt-get install --force-yes -y bzr tmux
+
+    if [ $ENSEMBLE_ORIGIN = "ppa" ]; then
+        # the echo forces an enter to get around the interactive
+        # prompt in apt-add-repository
+        echo y | apt-add-repository ppa:ensemble/ppa
+        apt-get update
+        apt-get install --force-yes -y ensemble python-txzookeeper
+    elif [ $ENSEMBLE_ORIGIN = "branch" ]; then
+        apt-get install --force-yes -y python-txzookeeper
+        mkdir /usr/lib/ensemble
+        bzr branch $ENSEMBLE_SOURCE /usr/lib/ensemble/ensemble
+        bash -c "cd /usr/lib/ensemble/ensemble && sudo python setup.py develop"
+    elif [ $ENSEMBLE_ORIGIN = "distro" ]; then
+        apt-get install --force-yes -y ensemble python-txzookeeper
+    fi
+
+}
+
+# load configuration
+source /etc/ensemble/ensemble.conf
+
+if [ -z "$ENSEMBLE_ORIGIN" ]; then
+    ENSEMBLE_ORIGIN=distro
+fi
+
+if [ -z "$ENSEMBLE_ZOOKEEPER" ]; then
+    echo "'ENSEMBLE_ZOOKEEPER' is required"
+    exit 1
+fi
+
+if [ "$(id -u)" != "0" ]; then
+    echo "This script should be run as 'root'"
+    exit 1
+fi
+
+
+setup_ensemble
\ No newline at end of file

=== added file 'ensemble/lib/lxc/data/lxc.conf'
--- ensemble/lib/lxc/data/lxc.conf	1970-01-01 00:00:00 +0000
+++ ensemble/lib/lxc/data/lxc.conf	2011-09-08 23:54:45 +0000
@@ -0,0 +1,3 @@
+lxc.network.type=veth
+lxc.network.link=%(network_name)s
+lxc.network.flags=up

=== added directory 'ensemble/lib/lxc/tests'
=== added file 'ensemble/lib/lxc/tests/__init__.py'
=== added file 'ensemble/lib/lxc/tests/test_lxc.py'
--- ensemble/lib/lxc/tests/test_lxc.py	1970-01-01 00:00:00 +0000
+++ ensemble/lib/lxc/tests/test_lxc.py	2011-09-08 23:54:45 +0000
@@ -0,0 +1,151 @@
+import os
+import tempfile
+
+from twisted.internet.defer import inlineCallbacks
+from twisted.internet.threads import deferToThread
+
+from ensemble.lib.lxc import (_lxc_start, _lxc_stop, _lxc_create,
+                              _lxc_wait, _lxc_ls, _lxc_destroy,
+                              LXCContainer, get_containers)
+from ensemble.lib.testing import TestCase, get_test_zookeeper_address
+
+
+def run_lxc_tests():
+    if os.environ.get("TEST_LXC"):
+        return None
+    return "TEST_LXC=1 to include lxc tests"
+
+
+DATA_PATH = os.path.abspath(
+    os.path.join(os.path.dirname(__file__), "..", "data"))
+
+
+DEFAULT_CONTAINER = "lxc_test"
+
+
+class LXCTest(TestCase):
+    timeout = 240
+    skip = run_lxc_tests()
+
+    def setUp(self):
+        self.config = self.make_config()
+
+        @self.addCleanup
+        def remove_config():
+            if os.path.exists(self.config):
+                os.unlink(self.config)
+
+    def make_config(self, network_name="virbr0"):
+        lxc_config = os.path.join(DATA_PATH, "lxc.conf")
+        template = open(lxc_config, "r").read()
+
+        fd, output_fn = tempfile.mkstemp(suffix=".conf")
+        output_config = open(output_fn, "w")
+        output_config.write(template % {"network_name": network_name})
+        output_config.close()
+
+        return output_fn
+
+    def cleanContainer(self, container_name):
+        if os.path.exists("/var/lib/lxc/%s" % container_name):
+            _lxc_stop(container_name)
+            _lxc_destroy(container_name)
+
+    def test_lxc_create(self):
+        self.addCleanup(self.cleanContainer, DEFAULT_CONTAINER)
+
+        _lxc_create(DEFAULT_CONTAINER, config_file=self.config)
+
+        # verify we can find the container
+        output = _lxc_ls()
+        self.assertIn(DEFAULT_CONTAINER, output)
+
+        # remove and verify the container was removed
+        _lxc_destroy(DEFAULT_CONTAINER)
+        output = _lxc_ls()
+        self.assertNotIn(DEFAULT_CONTAINER, output)
+
+    def test_lxc_start(self):
+        self.addCleanup(self.cleanContainer, DEFAULT_CONTAINER)
+
+        _lxc_create(DEFAULT_CONTAINER, config_file=self.config)
+
+        _lxc_start(DEFAULT_CONTAINER)
+        _lxc_stop(DEFAULT_CONTAINER)
+
+    @inlineCallbacks
+    def test_lxc_deferred(self):
+        self.addCleanup(self.cleanContainer, DEFAULT_CONTAINER)
+        yield deferToThread(_lxc_create, DEFAULT_CONTAINER, config_file=self.config)
+        yield deferToThread(_lxc_start, DEFAULT_CONTAINER)
+
+    @inlineCallbacks
+    def test_lxc_container(self):
+        self.addCleanup(self.cleanContainer, DEFAULT_CONTAINER)
+
+        c = LXCContainer(DEFAULT_CONTAINER,
+                         dict(ensemble_zookeeper=get_test_zookeeper_address(),
+                              ensemble_origin="distro"))
+        self.assertFalse(c.running)
+        yield c.run()
+        self.assertTrue(c.running)
+
+        output = _lxc_ls()
+        self.assertIn(DEFAULT_CONTAINER, output)
+
+        # verify we have a path into the container
+        self.assertTrue(os.path.exists(c.rootfs))
+
+        # expect a processed config file above the rootfs
+        # -- this requires root to read
+        #network = open(os.path.join(c.rootfs, "..", "config"), "r").read()
+        #self.assertIn("lxc.network.link=virbr0", network)
+
+        # check that some of the customize modifications were made
+        self.assertTrue(os.path.exists(os.path.join(
+                    c.rootfs, "etc", "ensemble")))
+        self.assertTrue(os.path.exists(os.path.join(
+                    c.rootfs, "usr", "bin", "ensemble")))
+
+        # verify that we have shell variables in rootfs/etc/ensemble.conf
+        config = open(os.path.join(c.rootfs, "etc", "ensemble", "ensemble.conf"), "r").read()
+        self.assertIn("ENSEMBLE_ZOOKEEPER=%s" % get_test_zookeeper_address(), config)
+
+        # verify that we are in containers
+        containers = yield get_containers(None)
+        self.assertEqual(containers[DEFAULT_CONTAINER], True)
+
+        # tear it down
+        yield c.destroy()
+        self.assertFalse(c.running)
+
+        containers = yield get_containers(None)
+        self.assertNotIn(DEFAULT_CONTAINER, containers)
+
+        # and its gone
+        output = _lxc_ls()
+        self.assertNotIn(DEFAULT_CONTAINER, output)
+
+    @inlineCallbacks
+    def test_lxc_wait(self):
+        self.addCleanup(self.cleanContainer, DEFAULT_CONTAINER)
+
+        _lxc_create(DEFAULT_CONTAINER, config_file=self.config)
+
+        _lxc_start(DEFAULT_CONTAINER)
+
+        def waitForState(result):
+            self.assertEqual(result, True)
+
+        d = _lxc_wait(DEFAULT_CONTAINER, "RUNNING")
+        d.addCallback(waitForState)
+        yield d
+
+        _lxc_stop(DEFAULT_CONTAINER)
+        yield _lxc_wait(DEFAULT_CONTAINER, "STOPPED")
+        _lxc_destroy(DEFAULT_CONTAINER)
+
+
+
+
+