Merge lp:~themue/juju-core/021-deployer-lxc-context into lp:~juju/juju-core/trunk

[Frank Mueller]

lxc: adding lxc context for local provider

The LxcContext is the second context beside the currently used
SimpleContext to be used by the Deployer to deploy and recall
units. It uses a small subset of the LXC commands by calling
the binaries/scripts. Currently the new context isn't used. This
has to be done by cmd/jujud/agent.go when creating a new Deployer
instance.

https://codereview.appspot.com/9336045/

[William Reade (fwereade)]

NOT LGTM -- I would be prepared to let the unit-specific stuff slide,
but there's quite a lot to address here.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go
File worker/deployer/lxc/context.go (right):

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode26
worker/deployer/lxc/context.go:26: environSeries string
An environment does not have a series, other than default-series, which
is not relevant.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode39
worker/deployer/lxc/context.go:39: // initDir specifies the directory
used by upstart on the local system.
If LXC autoconf works, we don't need upstart on the local system. We
*do* need it relative to a container's root though.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode45
worker/deployer/lxc/context.go:45: dataDir string
Why not const? We're completely in control here.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode53
worker/deployer/lxc/context.go:53: logDir string
Similarly, let's just make that const. No need to ever swap it out that
I can see.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode57
worker/deployer/lxc/context.go:57: lxcContainerDir string
Hmm, I think I saw serge explaining that we can use whatever dir we
want. If we can, I'd really like us to just put them directly inside the
deploying agent's data dir... possible?

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode67
worker/deployer/lxc/context.go:67: // entity specified in info, that
deploys unit agents as upstart jobs in
FWIW, this can't be unit-specific any more. All containers are
"machines" and get their own machine agents.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode77
worker/deployer/lxc/context.go:77: hostDataDir = "/var/lib/juju"
I don't think we should default this. We should always have the data dir
available, right?

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode104
worker/deployer/lxc/context.go:104: if err :=
ctx.createContainer(containerName); err != nil {
Shouldn't we trash it and try again? Failure partway through the first
attempt should not make a container undeployable on subsequent
attempts...

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode127
worker/deployer/lxc/context.go:127: }
If we fail here, the unit'll keep running forever.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode146
worker/deployer/lxc/context.go:146: if groups[2] != ctx.deployerTag {
This is broken unless we also take into account environ name. But I'm
not sure we should care about environ names...

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode167
worker/deployer/lxc/context.go:167: svc.InitDir = ctx.initDir
Why are we using the host upstart dir? ISTM that this will cause us to
start containers that don't run agents, and start agents outside
containers.

https://codereview.appspot.com/9336045/diff/1/work...

[Frank Mueller (themue)]

Closed.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go
File worker/deployer/lxc/context.go (right):

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode26
worker/deployer/lxc/context.go:26: environSeries string
On 2013/05/17 06:46:27, fwereade wrote:
> An environment does not have a series, other than default-series,
which is not
> relevant.

Renamed in defaultSeries, it's needed for the lxc container creation.

PyJuju gets this value by reading $JUJU_ENV, which we don't have so far.
But could also be a solution.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode39
worker/deployer/lxc/context.go:39: // initDir specifies the directory
used by upstart on the local system.
On 2013/05/17 06:46:27, fwereade wrote:
> If LXC autoconf works, we don't need upstart on the local system. We
*do* need
> it relative to a container's root though.

Wrong comment, but removed it due to wrong concept.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode45
worker/deployer/lxc/context.go:45: dataDir string
On 2013/05/17 06:46:27, fwereade wrote:
> Why not const? We're completely in control here.

Done.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode53
worker/deployer/lxc/context.go:53: logDir string
On 2013/05/17 06:46:27, fwereade wrote:
> Similarly, let's just make that const. No need to ever swap it out
that I can
> see.

Done.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode57
worker/deployer/lxc/context.go:57: lxcContainerDir string
On 2013/05/17 06:46:27, fwereade wrote:
> Hmm, I think I saw serge explaining that we can use whatever dir we
want. If we
> can, I'd really like us to just put them directly inside the deploying
agent's
> data dir... possible?

Seems to be possible by generating the directory in the container
configuration. Still one question left, will check.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode77
worker/deployer/lxc/context.go:77: hostDataDir = "/var/lib/juju"
On 2013/05/17 06:46:27, fwereade wrote:
> I don't think we should default this. We should always have the data
dir
> available, right?

Done.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode104
worker/deployer/lxc/context.go:104: if err :=
ctx.createContainer(containerName); err != nil {
On 2013/05/17 06:46:27, fwereade wrote:
> Shouldn't we trash it and try again? Failure partway through the first
attempt
> should not make a container undeployable on subsequent attempts...

Done.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode127
worker/deployer/lxc/context.go:127: }
On 2013/05/17 06:46:27, fwereade wrote:
> If we fail here, the unit'll keep running forever.

Done.

https://codereview.appspot.com/9336045/diff/1/worker/deployer/lxc/context.go#newcode167
worker/deployer/lxc/context.go:167: svc.InitDir = ctx.initDir
On 2013/05/17 06:46:27, fwereade wrote:
> Why are we using the host upstart dir? ISTM that this will cause us to
start
> containers that don't run age...

Unmerged revisions

972. By Frank Mueller on 2013-05-09

lxc: merged trunk

971. By Frank Mueller on 2013-05-09

lxc: refactored into own package; first network code

970. By Frank Mueller on 2013-05-08

lxc: improved context

969. By Frank Mueller on 2013-03-11

deployer: merged trunk and done review changes

968. By Frank Mueller on 2013-03-08

deployer/lxc: merged after review changes

967. By Frank Mueller on 2013-03-08

deployer/lxc: changes after reviews

966. By Frank Mueller on 2013-03-08

deployer: merged trunk and renamed manager to context

965. By Frank Mueller on 2013-03-08

deployer: added LXC manager

964. By Francesco Banconi on 2013-03-06

Add a CharmInfo API command.

Add a CharmInfo API command to get information
about a charm, given its URL. It is needed by the
Juju GUI, modeled upon the similar one in pyJuju.

R=dimitern, rog
CC=
https://codereview.appspot.com/7444052

963. By Martin Packman on 2013-03-06

Correct url for openstack info on metadata service

The location on the metadata service we were using to retrieve the openstack
specific json file was misnamed and lacked the 'openstack' path component,
which was causing the legacy id to be used in all cases. That results in a
suicidal state server, which also hints we need to worry more about the
other cases when we may run into non-canonical server ids.

R=dfc, jameinel
CC=
https://codereview.appspot.com/7527043

=== modified file 'worker/deployer/export_test.go'
--- worker/deployer/export_test.go	2013-03-29 14:54:20 +0000
+++ worker/deployer/export_test.go	2013-05-09 23:22:32 +0000
@@ -1,5 +1,9 @@
 package deployer
 
+import (
+	"launchpad.net/juju-core/version"
+)
+
 type fakeAddresser struct{}
 
 func (*fakeAddresser) Addresses() []string {
@@ -17,3 +21,19 @@
 		syslogConfigDir: syslogConfigDir,
 	}
 }
+
+func NewTestLxcContext(deployerTag, initDir, hostDataDir, lxcContainerDir, lxcAutoDir string) *LxcContext {
+	return &LxcContext{
+		environName:     "lxc-test",
+		environSeries:   version.Current.Series,
+		dataDir:         "/var/lib/juju",
+		hostDataDir:     hostDataDir,
+		addresser:       &fakeAddresser{},
+		caCert:          []byte("test-cert"),
+		deployerTag:     deployerTag,
+		initDir:         initDir,
+		logDir:          "/var/log/juju",
+		lxcContainerDir: lxcContainerDir,
+		lxcAutoDir:      lxcAutoDir,
+	}
+}

=== added directory 'worker/deployer/lxc'
=== added file 'worker/deployer/lxc/context.go'
--- worker/deployer/lxc/context.go	1970-01-01 00:00:00 +0000
+++ worker/deployer/lxc/context.go	2013-05-09 23:22:32 +0000
@@ -0,0 +1,355 @@
+package lxc
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+
+	"launchpad.net/juju-core/environs/agent"
+	"launchpad.net/juju-core/log"
+	"launchpad.net/juju-core/state"
+	"launchpad.net/juju-core/upstart"
+	"launchpad.net/juju-core/version"
+	"launchpad.net/juju-core/worker/deployer"
+)
+
+// LxcContext stores details of the LXC environment on the local system.
+type LxcContext struct {
+	// environName identifies the environment the context is running in.
+	environName string
+
+	// environSeries identifies the series of the environment.
+	environSeries string
+
+	// addresser is used to get the current state server addresses at the time
+	// the given unit is deployed.
+	addresser deployer.Addresser
+
+	// caCert holds the CA certificate that will be used
+	// to validate the state server's certificate, in PEM format.
+	caCert []byte
+
+	// deployerTag identifies the agent on whose behalf this context is running.
+	deployerTag string
+
+	// initDir specifies the directory used by upstart on the local system.
+	// It is typically set to "/etc/init".
+	initDir string
+
+	// dataDir specifies the directory used by juju to store its state. It
+	// is typically set to "/var/lib/juju".
+	dataDir string
+
+	// hostDataDir specifies the directory used by juju on the host system to 
+	// store its tools. It is typically set to "/var/lib/juju".
+	hostDataDir string
+
+	// logDir specifies the directory to which installed units will write
+	// their log files. It is typically set to "/var/log/juju".
+	logDir string
+
+	// lxcContainerDir specifies the base directory for all containers. It
+	// is typically set to "/var/lib/lxc".
+	lxcContainerDir string
+
+	// lxcAutoDir specifies the directory for the links to confguration files
+	// needed for autostarting containers. It is typically set to "/etc/lxc/auto".
+	lxcAutoDir string
+}
+
+var _ deployer.Context = (*LxcContext)(nil)
+
+// NewLxcContext returns a new LXC context, acting on behalf of the
+// entity specified in info, that deploys unit agents as upstart jobs in
+// "/etc/init" logging to "/var/log/juju". Paths to which agents and tools
+// are installed are relative to dataDir; if dataDir or hostDataDir are empty, 
+// they will be set to "/var/lib/juju".
+func NewLxcContext(environName, environSeries, dataDir, hostDataDir string, caCert []byte,
+	deployerTag string, addresser deployer.Addresser) *LxcContext {
+	if dataDir == "" {
+		dataDir = "/var/lib/juju"
+	}
+	if hostDataDir == "" {
+		hostDataDir = "/var/lib/juju"
+	}
+	return &LxcContext{
+		environName:     environName,
+		environSeries:   environSeries,
+		dataDir:         dataDir,
+		hostDataDir:     hostDataDir,
+		addresser:       addresser,
+		caCert:          caCert,
+		deployerTag:     deployerTag,
+		initDir:         "/etc/init",
+		logDir:          "/var/log/juju",
+		lxcContainerDir: "/var/lib/lxc",
+		lxcAutoDir:      "/etc/lxc/auto",
+	}
+}
+
+// DeployUnit creates a container for a unit and installs the required 
+// upstart job in it.
+func (ctx *LxcContext) DeployUnit(unitName, initialPassword string) error {
+	if ctx.configLinkExists(unitName) {
+		return fmt.Errorf("unit %q is already deployed", unitName)
+	}
+
+	// Create container and initialize it.
+	containerName := ctx.containerName(unitName)
+	unitTag := state.UnitTag(unitName)
+	if err := ctx.createContainer(containerName); err != nil {
+		return err
+	}
+	if err := ctx.initContainer(containerName, unitName, unitTag, initialPassword); err != nil {
+		if e := ctx.destroyContainer(containerName); e != nil {
+			return fmt.Errorf("cannot destroy container after init error '%v': %v", err, e)
+		}
+		return err
+	}
+	if err := ctx.createConfigLink(unitName); err != nil {
+		return fmt.Errorf("cannot create auto start link: %v", err)
+	}
+	return startContainer(containerName)
+}
+
+// RecallUnit destroys the container and removes the link in the
+// /etc/lxc/auto folder.
+func (ctx *LxcContext) RecallUnit(unitName string) error {
+	if !ctx.configLinkExists(unitName) {
+		return fmt.Errorf("unit %q is not deployed", unitName)
+	}
+	if err := ctx.destroyConfigLink(unitName); err != nil {
+		return err
+	}
+
+	// Destroy container.
+	containerName := ctx.containerName(unitName)
+	return ctx.destroyContainer(containerName)
+}
+
+var autoRe = regexp.MustCompile("^juju-([a-z0-9-]+):([a-z0-9-]+):unit-([a-z0-9-]+)-([0-9]+)\\.conf$")
+
+// DeployedUnits returns which units are deployed right now.
+func (ctx *LxcContext) DeployedUnits() ([]string, error) {
+	fis, err := ioutil.ReadDir(ctx.lxcAutoDir)
+	if err != nil {
+		return nil, err
+	}
+	var installed []string
+	for _, fi := range fis {
+		groups := autoRe.FindStringSubmatch(fi.Name())
+		if len(groups) == 5 {
+			if groups[2] != ctx.deployerTag {
+				continue
+			}
+			unitName := groups[3] + "/" + groups[4]
+			if !state.IsUnitName(unitName) {
+				continue
+			}
+			installed = append(installed, unitName)
+		}
+	}
+	return installed, nil
+}
+
+// upstartService returns an upstart.Service corresponding to the specified
+// unit. Its name is badged according to the entity responsible for the
+// context, so as to distinguish its own jobs from those installed by other
+// means.
+func (ctx *LxcContext) upstartService(unitName string) *upstart.Service {
+	unitTag := state.UnitTag(unitName)
+	svcName := "jujud-" + ctx.deployerTag + ":" + unitTag
+	svc := upstart.NewService(svcName)
+	svc.InitDir = ctx.initDir
+	return svc
+}
+
+// createContainer creates a new LXC container with the given name.
+func (ctx *LxcContext) createContainer(containerName string) error {
+	if ctx.containerExists(containerName) {
+		return fmt.Errorf("container %q is already created", containerName)
+	}
+	return createContainer(containerName, ctx.environSeries)
+}
+
+// initContainer creates the needed directories inside the container
+// and copies the tools.
+func (ctx *LxcContext) initContainer(containerName, unitName, unitTag, initialPassword string) error {
+	hostToolsDir := agent.SharedToolsDir(ctx.hostDataDir, version.Current)
+	toolsDir := ctx.containerDir(containerName, agent.ToolsDir(ctx.dataDir, unitTag))
+	err := os.MkdirAll(toolsDir, 0777)
+	if err != nil {
+		return err
+	}
+	defer removeOnErr(&err, toolsDir)
+
+	agentsDir := ctx.containerDir(containerName, ctx.dataDir, "agents", unitTag)
+	err = os.MkdirAll(agentsDir, 0777)
+	if err != nil {
+		return err
+	}
+	defer removeOnErr(&err, agentsDir)
+
+	logDir := ctx.containerDir(containerName, ctx.logDir)
+	err = os.MkdirAll(ctx.containerDir(containerName, ctx.logDir), 0777)
+	if err != nil {
+		return err
+	}
+	defer removeOnErr(&err, logDir)
+
+	if err = copyTools(hostToolsDir, toolsDir); err != nil {
+		return err
+	}
+	if _, err = agent.ChangeAgentTools(ctx.hostDataDir, unitTag, version.Current); err != nil {
+		return err
+	}
+
+	// Prepare the agent's configuration data.
+	info := state.Info{
+		Addrs:  ctx.addresser.Addresses(),
+		Tag:    unitTag,
+		CACert: ctx.caCert,
+	}
+	confDataDir := ctx.containerDir(containerName, ctx.dataDir)
+	conf := &agent.Conf{
+		DataDir:     confDataDir,
+		OldPassword: initialPassword,
+		StateInfo:   &info,
+	}
+	conf.StateInfo.Tag = unitTag
+	conf.StateInfo.Password = ""
+	if err = conf.Write(); err != nil {
+		return err
+	}
+	defer removeOnErr(&err, conf.Dir())
+
+	// Install an upstart job that runs the unit agent.
+	jujudPath := filepath.Join(toolsDir, "jujud")
+	logPath := filepath.Join(ctx.logDir, unitTag+".log")
+	cmd := strings.Join([]string{
+		jujudPath, "unit",
+		"--data-dir", ctx.dataDir,
+		"--unit-name", unitName,
+		"--debug", // TODO: propagate debug state sensibly
+	}, " ")
+	svc := ctx.upstartService(unitName)
+	uconf := &upstart.Conf{
+		Service: *svc,
+		Desc:    "juju unit agent for " + unitName,
+		Cmd:     cmd,
+		Out:     logPath,
+	}
+	return uconf.Install()
+}
+
+// destroyContainer stops and removes the container.
+func (ctx *LxcContext) destroyContainer(containerName string) error {
+	if !ctx.containerExists(containerName) {
+		return fmt.Errorf("container %q does not exist", containerName)
+	}
+	if err := stopContainer(containerName); err != nil {
+		return err
+	}
+	if err := waitContainerStopped(containerName); err != nil {
+		return err
+	}
+	return destroyContainer(containerName)
+}
+
+// containerName returns a unique name for the container and its upstart 
+// script based on the unit name.
+func (ctx *LxcContext) containerName(unitName string) string {
+	unitTag := state.UnitTag(unitName)
+	return "juju-" + ctx.environName + ":" + ctx.deployerTag + ":" + unitTag
+}
+
+// configLinkExists checks if the configuration link for an automatic start exists.
+func (ctx *LxcContext) configLinkExists(unitName string) bool {
+	containerName := ctx.containerName(unitName)
+	autoConfigPath := filepath.Join(ctx.lxcAutoDir, containerName+".conf")
+	containerConfigPath := filepath.Join(ctx.lxcContainerDir, containerName, "config")
+	targetPath, err := os.Readlink(autoConfigPath)
+	return err == nil && targetPath == containerConfigPath
+}
+
+// createConfigLink links from a container config to the LXC auto-start folder.
+func (ctx *LxcContext) createConfigLink(unitName string) error {
+	containerName := ctx.containerName(unitName)
+	autoConfigPath := filepath.Join(ctx.lxcAutoDir, containerName+".conf")
+	containerConfigPath := filepath.Join(ctx.lxcContainerDir, containerName, "config")
+	return os.Symlink(containerConfigPath, autoConfigPath)
+}
+
+// destroyConfigLink removes the link from a container config to the LXC auto-start folder.
+func (ctx *LxcContext) destroyConfigLink(unitName string) error {
+	containerName := ctx.containerName(unitName)
+	autoConfigPath := filepath.Join(ctx.lxcAutoDir, containerName+".conf")
+	return os.Remove(autoConfigPath)
+}
+
+// containerDir returns the full qualified name of a directory
+// inside the container.
+func (ctx *LxcContext) containerDir(containerName string, dirs ...string) string {
+	rootfs := []string{ctx.lxcContainerDir, containerName, "rootfs"}
+	return filepath.Join(append(rootfs, dirs...)...)
+}
+
+// containerExists checks if the container is created.
+func (ctx *LxcContext) containerExists(containerName string) bool {
+	fi, err := os.Stat(ctx.containerDir(containerName))
+	if err != nil {
+		if os.IsNotExist(err) {
+			return false
+		}
+		panic(err)
+	}
+	return fi.IsDir()
+}
+
+// copyTools copies the hosts juju tools into the container.
+func copyTools(hostToolsDir, containerToolsDir string) error {
+	hostLen := len(hostToolsDir)
+	copyTool := func(hostPath string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+		containerPath := filepath.Join(containerToolsDir, hostPath[hostLen:])
+		if info.IsDir() {
+			// Create directory.
+			if err := os.MkdirAll(containerPath, info.Mode().Perm()); err != nil {
+				return err
+			}
+		} else {
+			// Copy file.
+			sourceFile, err := os.Open(hostPath)
+			if err != nil {
+				return err
+			}
+			defer sourceFile.Close()
+			destFile, err := os.Create(containerPath)
+			if err != nil {
+				return err
+			}
+			defer destFile.Close()
+			if _, err := io.Copy(destFile, sourceFile); err != nil {
+				return err
+			}
+			return destFile.Chmod(info.Mode())
+		}
+		return nil
+	}
+	return filepath.Walk(hostToolsDir, copyTool)
+}
+
+// removeOnErr removes a path in case of an error.
+func removeOnErr(err *error, path string) {
+	if *err != nil {
+		if e := os.Remove(path); e != nil {
+			log.Warningf("installer: cannot remove %q: %v", path, e)
+		}
+	}
+}

=== added file 'worker/deployer/lxc/export_test.go'
--- worker/deployer/lxc/export_test.go	1970-01-01 00:00:00 +0000
+++ worker/deployer/lxc/export_test.go	2013-05-09 23:22:32 +0000
@@ -0,0 +1,27 @@
+package lxc
+
+import (
+	"launchpad.net/juju-core/version"
+)
+
+type fakeAddresser struct{}
+
+func (*fakeAddresser) Addresses() []string {
+	return []string{"s1:123", "s2:123"}
+}
+
+func NewTestLxcContext(deployerTag, initDir, hostDataDir, lxcContainerDir, lxcAutoDir string) *LxcContext {
+	return &LxcContext{
+		environName:     "lxc-test",
+		environSeries:   version.Current.Series,
+		dataDir:         "/var/lib/juju",
+		hostDataDir:     hostDataDir,
+		addresser:       &fakeAddresser{},
+		caCert:          []byte("test-cert"),
+		deployerTag:     deployerTag,
+		initDir:         initDir,
+		logDir:          "/var/log/juju",
+		lxcContainerDir: lxcContainerDir,
+		lxcAutoDir:      lxcAutoDir,
+	}
+}

=== added file 'worker/deployer/lxc/lxc.go'
--- worker/deployer/lxc/lxc.go	1970-01-01 00:00:00 +0000
+++ worker/deployer/lxc/lxc.go	2013-05-09 23:22:32 +0000
@@ -0,0 +1,213 @@
+package lxc
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os/exec"
+	"regexp"
+	"strings"
+)
+
+const (
+	defaultAddr   = "10.0.3.1"
+	defaultBridge = "lxcbr0"
+)
+
+var (
+	confPath string         = "/etc/default/lxc"
+	addrRE   *regexp.Regexp = regexp.MustCompile(`\n\s*LXC_ADDR="(\d+.\d+.\d+.\d+)"`)
+	bridgeRE *regexp.Regexp = regexp.MustCompile(`\n\s*LXC_BRIDGE="(\w+)"`)
+)
+
+// Error reports the failure of a LXC command.
+type Error struct {
+	Name   string
+	Err    error
+	Output []string
+}
+
+func (e Error) Error() string {
+	if e.Output == nil {
+		return fmt.Sprintf("error executing %q: %v", e.Name, e.Err)
+	}
+	if len(e.Output) == 1 {
+		return fmt.Sprintf("error executing %q: %v", e.Name, e.Output[0])
+	}
+	return fmt.Sprintf("error executing %q: %s", e.Name, strings.Join(e.Output, "; "))
+}
+
+// createContainer creates a new ubuntu cloud container.
+func createContainer(containerName, series string) error {
+	hostId := strings.Replace(containerName, ":", "-", -1)
+	_, err := run("lxc-create",
+		"-n", containerName,
+		"-t", "ubuntu-cloud",
+		"--",
+		"-d",
+		"-i", hostId,
+		"-r", series)
+	return err
+}
+
+// startContainer starts a container.
+func startContainer(containerName string) error {
+	_, err := run("lxc-start", "-n", containerName)
+	return err
+}
+
+// stopContainer stops a container.
+func stopContainer(containerName string) error {
+	_, err := run("lxc-stop", "-n", containerName)
+	return err
+}
+
+// waitContainerStopped waits until a container stopped.
+func waitContainerStopped(containerName string) error {
+	_, err := run("lxc-wait", "-n", containerName, "-s", "STOPPED")
+	return err
+}
+
+// destroyContainer destroys a container.
+func destroyContainer(containerName string) error {
+	_, err := run("lxc-destroy", "-n", containerName)
+	return err
+}
+
+// startNetwork starts the lxc network subsystem.
+func startNetwork() error {
+	goal, _, err := networkStatus()
+	if err != nil {
+		return err
+	}
+	if goal != "start" {
+		_, err := run("start", "lxc-net")
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// stopNetwork stops the lxc network subsystem.
+func stopNetwork() error {
+	goal, _, err := networkStatus()
+	if err != nil {
+		return err
+	}
+	if goal != "stop" {
+		_, err := run("stop", "lxc-net")
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// isNotworkRunning checks if the lxc network subsystem
+// is running.
+func isNetworkRunning() (bool, error) {
+	_, status, err := networkStatus()
+	if err != nil {
+		return false, err
+	}
+	return status == "running", nil
+}
+
+// networkAttributes returns the lxc network attributes: 
+// starting IP address and bridge name.
+func networkAttributes() (addr, bridge string, err error) {
+	config, err := readConf()
+	if err != nil {
+		return "", "", err
+	}
+	addr = config["address"]
+	if addr == "" {
+		addr = defaultAddr
+	}
+	bridge = config["bridge"]
+	if bridge == "" {
+		bridge = defaultBridge
+	}
+	return addr, bridge, nil
+}
+
+// networkStatus returns the status of the lxc network subsystem.
+func networkStatus() (goal, status string, err error) {
+	output, err := run("status", "lxc-net")
+	if err != nil {
+		return "", "", err
+	}
+	fields := strings.Fields(output)
+	if len(fields) != 2 {
+		return "", "", fmt.Errorf("unexpected status output: %q", output)
+	}
+	fields = strings.Split(fields[1], "/")
+	if len(fields) != 2 {
+		return "", "", fmt.Errorf("unexpected status output: %q", output)
+	}
+	return fields[0], fields[1], nil
+}
+
+// run executes the passed command and returns the out.
+func run(name string, args ...string) (string, error) {
+	cmd := exec.Command(name, args...)
+	// LXC tools do not use stdout and stderr in a predictable
+	// way; based on experimentation, the most convenient
+	// solution is to combine them and leave the client to
+	// determine sanity as best it can.
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		return "", runError(name, err, out)
+	}
+	return string(out), nil
+}
+
+// runError creates an error if run fails.
+func runError(name string, err error, out []byte) error {
+	e := &Error{name, err, nil}
+	for _, l := range strings.Split(string(out), "\n") {
+		if strings.HasPrefix(l, name+": ") {
+			// LXC tools do not always print their output with
+			// the command name as prefix. The name is part of
+			// the error struct, so stip it from the output if
+			// printed.
+			l = l[len(name)+2:]
+		}
+		if l != "" {
+			e.Output = append(e.Output, l)
+		}
+	}
+	return e
+}
+
+// keyValues retrieves key/value pairs out of a command out.
+func keyValues(raw string, sep string) map[string]string {
+	kv := map[string]string{}
+	lines := strings.Split(raw, "\n")
+	for _, line := range lines {
+		parts := strings.SplitN(line, sep, 2)
+		if len(parts) == 2 {
+			kv[parts[0]] = strings.TrimSpace(parts[1])
+		}
+	}
+	return kv
+}
+
+// readConf reads the LXC network address and bridge interface
+// out of the configuration file /etc/default/lxc.
+func readConf() (map[string]string, error) {
+	conf := make(map[string]string)
+	confData, err := ioutil.ReadFile(confPath)
+	if err != nil {
+		return nil, err
+	}
+	fetchValue := func(field string, re *regexp.Regexp) {
+		groups := re.FindStringSubmatch(string(confData))
+		if len(groups) == 2 {
+			conf[field] = groups[1]
+		}
+	}
+	fetchValue("address", addrRE)
+	fetchValue("bridge", bridgeRE)
+	return conf, nil
+}

=== added file 'worker/deployer/lxc/lxc_test.go'
--- worker/deployer/lxc/lxc_test.go	1970-01-01 00:00:00 +0000
+++ worker/deployer/lxc/lxc_test.go	2013-05-09 23:22:32 +0000
@@ -0,0 +1,241 @@
+package lxc_test
+
+import (
+	"bytes"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	stdtesting "testing"
+	"text/template"
+
+	. "launchpad.net/gocheck"
+	"launchpad.net/juju-core/environs/agent"
+	"launchpad.net/juju-core/state"
+	coretesting "launchpad.net/juju-core/testing"
+	"launchpad.net/juju-core/version"
+	"launchpad.net/juju-core/worker/deployer/lxc"
+)
+
+func TestPackage(t *stdtesting.T) {
+	coretesting.MgoTestPackage(t)
+}
+
+type LxcContextSuite struct {
+	LxcToolsFixture
+}
+
+var _ = Suite(&LxcContextSuite{})
+
+func (s *LxcContextSuite) SetUpTest(c *C) {
+	s.LxcToolsFixture.SetUp(c)
+}
+
+func (s *LxcContextSuite) TearDownTest(c *C) {
+	s.LxcToolsFixture.TearDown(c)
+}
+
+func (s *LxcContextSuite) TestDeployRecall(c *C) {
+	ctx0 := s.getContext(c, "test-entity-0")
+	units, err := ctx0.DeployedUnits()
+	c.Assert(err, IsNil)
+	c.Assert(units, HasLen, 0)
+	s.assertUpstartCount(c, 0)
+
+	err = ctx0.DeployUnit("foo/123", "some-password")
+	c.Assert(err, IsNil)
+	units, err = ctx0.DeployedUnits()
+	c.Assert(err, IsNil)
+	c.Assert(units, DeepEquals, []string{"foo/123"})
+	s.assertUpstartCount(c, 1)
+	s.checkUnitInstalled(c, "test-entity-0", "foo/123", "some-password")
+
+	err = ctx0.RecallUnit("foo/123")
+	c.Assert(err, IsNil)
+	units, err = ctx0.DeployedUnits()
+	c.Assert(err, IsNil)
+	c.Assert(units, HasLen, 0)
+	s.assertUpstartCount(c, 0)
+	s.checkUnitRemoved(c, "test-entity-0", "foo/123")
+}
+
+func (s *LxcContextSuite) TestIndependentManagers(c *C) {
+	ctx0 := s.getContext(c, "test-entity-0")
+	err := ctx0.DeployUnit("foo/123", "some-password")
+	c.Assert(err, IsNil)
+
+	ctx1 := s.getContext(c, "test-entity-1")
+	units, err := ctx1.DeployedUnits()
+	c.Assert(err, IsNil)
+	c.Assert(units, HasLen, 0)
+
+	err = ctx1.RecallUnit("foo/123")
+	c.Assert(err, ErrorMatches, `unit "foo/123" is not deployed`)
+	s.checkUnitInstalled(c, "test-entity-0", "foo/123", "some-password")
+
+	units, err = ctx0.DeployedUnits()
+	c.Assert(err, IsNil)
+	c.Assert(units, DeepEquals, []string{"foo/123"})
+	s.assertUpstartCount(c, 1)
+	s.checkUnitInstalled(c, "test-entity-0", "foo/123", "some-password")
+}
+
+type LxcToolsFixture struct {
+	origPath        string
+	hostDataDir     string
+	initDir         string
+	logDir          string
+	lxcContainerDir string
+	lxcAutoDir      string
+	binDir          string
+}
+
+var fakeJujud = "#!/bin/bash\n# fake-jujud\nexit 0\n"
+
+func (fix *LxcToolsFixture) SetUp(c *C) {
+	// Create some needed temporary directory and manipulate the path
+	// for the fake binaries.
+	fix.initDir = c.MkDir()
+	fix.logDir = c.MkDir()
+	fix.hostDataDir = c.MkDir()
+	fix.lxcContainerDir = c.MkDir()
+	fix.lxcAutoDir = c.MkDir()
+	fix.binDir = c.MkDir()
+	fix.origPath = os.Getenv("PATH")
+	os.Setenv("PATH", fix.binDir+":"+fix.origPath)
+	// Create the faked tools and binaries.
+	toolsDir := agent.SharedToolsDir(fix.hostDataDir, version.Current)
+	err := os.MkdirAll(toolsDir, 0755)
+	c.Assert(err, IsNil)
+	jujudPath := filepath.Join(toolsDir, "jujud")
+	err = ioutil.WriteFile(jujudPath, []byte(fakeJujud), 0755)
+	c.Assert(err, IsNil)
+	urlPath := filepath.Join(toolsDir, "downloaded-url.txt")
+	err = ioutil.WriteFile(urlPath, []byte("http://example.com/tools"), 0644)
+	c.Assert(err, IsNil)
+	fix.makeLxcBins(c)
+	fix.makeBin(c, "status", `echo "blah stop/waiting"`)
+	fix.makeBin(c, "stopped-status", `echo "blah stop/waiting"`)
+	fix.makeBin(c, "started-status", `echo "blah start/running, process 666"`)
+	fix.makeBin(c, "start", "cp $(which started-status) $(which status)")
+	fix.makeBin(c, "stop", "cp $(which stopped-status) $(which status)")
+}
+
+func (fix *LxcToolsFixture) TearDown(c *C) {
+	os.Setenv("PATH", fix.origPath)
+}
+
+var LxcBins = []struct {
+	Name string
+	Code string
+}{
+	{
+		Name: "lxc-create",
+		Code: `#!/bin/bash
+# lxc-create
+mkdir -p {{.LxcContainerDir}}/$2/rootfs
+touch {{.LxcContainerDir}}/$2/config
+exit 0`,
+	}, {
+		Name: "lxc-start",
+		Code: `#!/bin/bash
+# lxc-start
+exit 0`,
+	}, {
+		Name: "lxc-stop",
+		Code: `#!/bin/bash
+# lxc-stop
+exit 0`,
+	}, {
+		Name: "lxc-wait",
+		Code: `#!/bin/bash
+# lxc-wait
+exit 0`,
+	}, {
+		Name: "lxc-destroy",
+		Code: `#!/bin/bash
+# lxc-destroy
+rm -R {{.LxcContainerDir}}/$2
+exit 0`,
+	},
+}
+
+func (fix *LxcToolsFixture) makeLxcBins(c *C) {
+	conf := struct {
+		LxcContainerDir string
+		LxcAutoDir      string
+	}{
+		LxcContainerDir: fix.lxcContainerDir,
+		LxcAutoDir:      fix.lxcAutoDir,
+	}
+	for _, lxcBin := range LxcBins {
+		var buf bytes.Buffer
+		binT := template.Must(template.New("").Parse(lxcBin.Code))
+		err := binT.Execute(&buf, conf)
+		c.Assert(err, IsNil)
+		binPath := filepath.Join(fix.binDir, lxcBin.Name)
+		err = ioutil.WriteFile(binPath, buf.Bytes(), 0755)
+		c.Assert(err, IsNil)
+	}
+}
+
+func (fix *LxcToolsFixture) makeBin(c *C, name, script string) {
+	path := filepath.Join(fix.binDir, name)
+	err := ioutil.WriteFile(path, []byte("#!/bin/bash\n"+script), 0755)
+	c.Assert(err, IsNil)
+}
+
+func (fix *LxcToolsFixture) assertUpstartCount(c *C, count int) {
+	fis, err := ioutil.ReadDir(fix.lxcAutoDir)
+	c.Assert(err, IsNil)
+	c.Assert(fis, HasLen, count)
+}
+
+func (fix *LxcToolsFixture) getContext(c *C, deployerTag string) *lxc.LxcContext {
+	return lxc.NewTestLxcContext(deployerTag, fix.initDir, fix.hostDataDir, fix.lxcContainerDir, fix.lxcAutoDir)
+}
+
+func (fix *LxcToolsFixture) checkUnitInstalled(c *C, dname, uname, password string) {
+	unitTag := state.UnitTag(uname)
+	lxcContainerName := "juju-lxc-test:" + dname + ":" + unitTag
+	lxcContainerPath := filepath.Join(fix.lxcContainerDir, lxcContainerName)
+	lxcConfName := fmt.Sprintf("%s.conf", lxcContainerName)
+	lxcConfPath := filepath.Join(fix.lxcAutoDir, lxcConfName)
+
+	_, err := os.Stat(lxcContainerPath)
+	c.Assert(err, IsNil)
+	_, err = os.Stat(lxcConfPath)
+	c.Assert(err, IsNil)
+
+	dataDir := filepath.Join(fix.lxcContainerDir, lxcContainerName, "rootfs/var/lib/juju")
+	conf, err := agent.ReadConf(dataDir, unitTag)
+	c.Assert(err, IsNil)
+	c.Assert(conf, DeepEquals, &agent.Conf{
+		DataDir:     dataDir,
+		OldPassword: password,
+		StateInfo: &state.Info{
+			Addrs:  []string{"s1:123", "s2:123"},
+			CACert: []byte("test-cert"),
+			Tag:    unitTag,
+		},
+	})
+
+	toolsDir := agent.ToolsDir(dataDir, unitTag)
+	jujudPath := filepath.Join(toolsDir, "jujud")
+	jujudData, err := ioutil.ReadFile(jujudPath)
+	c.Assert(err, IsNil)
+	c.Assert(string(jujudData), Equals, fakeJujud)
+}
+
+func (fix *LxcToolsFixture) checkUnitRemoved(c *C, ename, uname string) {
+	unitTag := state.UnitTag(uname)
+	lxcContainerName := "juju-lxc-test:" + ename + ":" + unitTag
+	lxcContainerPath := filepath.Join(fix.lxcContainerDir, lxcContainerName)
+	lxcConfName := fmt.Sprintf("%s.conf", lxcContainerName)
+	lxcConfPath := filepath.Join(fix.lxcAutoDir, lxcConfName)
+
+	_, err := os.Stat(lxcContainerPath)
+	c.Assert(os.IsNotExist(err), Equals, true)
+	_, err = os.Stat(lxcConfPath)
+	c.Assert(os.IsNotExist(err), Equals, true)
+}