Merge lp:~axwalk/juju-core/destroy-environment-api into lp:~go-bot/juju-core/trunk

[Andrew Wilkins]

Implement DestroyEnvironment API

We introduce a new DestroyEnvironment client API
method, which:
 - checks that there are no manual non-manager machines in state
 - sets the environment to Dying, scheduling
   cleanups for services and preventing addition
   of new services and machines.
 - destroys non-manager instances

"juju destroy-environment" will call this new method,
ignoring the "no such method" error returned by old
API servers. If the API method succeeds, or it doesn't
exist, the provider.Destroy() method is called.

https://codereview.appspot.com/41280043/

Implement DestroyEnvironment API

We introduce a new DestroyEnvironment client API
method, which:
 - checks that there are no manual non-manager machines in state
 - sets the environment to Dying, scheduling
   cleanups for services and preventing addition
   of new services and machines.
 - destroys non-manager instances

"juju destroy-environment" will call this new method,
ignoring the "no such method" error returned by old
API servers. If the API method succeeds, or it doesn't
exist, the provider.Destroy() method is called.

https://codereview.appspot.com/41280043/

[Andrew Wilkins (axwalk)]

Reviewers: mp+198698_code.launchpad.net,

Message:
Please take a look.

Description:
Implement DestroyEnvironment API

We introduce a new DestroyEnvironment client API
method, which:
  - checks that there are no manual non-manager machines in state
  - sets the environment to Dying, scheduling
    cleanups for services and preventing addition
    of new services and machines.
  - destroys non-manager instances

"juju destroy-environment" will call this new method,
ignoring the "no such method" error returned by old
API servers. If the API method succeeds, or it doesn't
exist, the provider.Destroy() method is called.

https://code.launchpad.net/~axwalk/juju-core/destroy-environment-api/+merge/198698

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/41280043/

Affected files (+454, -132 lines):
   A [revision details]
   M cmd/juju/cmd_test.go
   M cmd/juju/destroyenvironment.go
   A cmd/juju/destroyenvironment_test.go
   M provider/dummy/environs.go
   M state/api/client.go
   A state/apiserver/client/destroy.go
   A state/apiserver/client/destroy_test.go

[William Reade (fwereade)]

LGTM with more-eager instance-killing, or a good reason not to, plus the
other trivials below

https://codereview.appspot.com/41280043/diff/1/state/apiserver/client/destroy.go
File state/apiserver/client/destroy.go (right):

https://codereview.appspot.com/41280043/diff/1/state/apiserver/client/destroy.go#newcode54
state/apiserver/client/destroy.go:54: if err :=
checkManualMachines(machines); err != nil {
Hmm, can we consider the ordering of the check-and-abort vs the
stop-all-instances? I feel like it's probably best to kill what we can,
and then return an error like "can't stop state server yet, manual
machines x, y, z are still using it, please destroy-machine".

Also... it crosses my mind that the environ provisioner might take some
time to get the memo about the environment dying. Do we have any way to
block racy *provisioning* of recently-added machines? That can be a
followup, though.

https://codereview.appspot.com/41280043/diff/1/state/apiserver/client/destroy.go#newcode110
state/apiserver/client/destroy.go:110: if inst == nil {
log the failed instances?

https://codereview.appspot.com/41280043/diff/1/state/apiserver/client/destroy.go#newcode145
state/apiserver/client/destroy.go:145: return
strings.HasPrefix(string(iid), "manual:")
Yeah, use the nonce -- if you need to make this a method on machine, so
be it.

https://codereview.appspot.com/41280043/

[Andrew Wilkins (axwalk)]

Please take a look.

https://codereview.appspot.com/41280043/diff/1/state/apiserver/client/destroy.go
File state/apiserver/client/destroy.go (right):

https://codereview.appspot.com/41280043/diff/1/state/apiserver/client/destroy.go#newcode54
state/apiserver/client/destroy.go:54: if err :=
checkManualMachines(machines); err != nil {
On 2013/12/18 12:52:20, fwereade wrote:
> Hmm, can we consider the ordering of the check-and-abort vs the
> stop-all-instances? I feel like it's probably best to kill what we
can, and then
> return an error like "can't stop state server yet, manual machines x,
y, z are
> still using it, please destroy-machine".

Moved the checkManualMachines after destroyInstances, and updated
destroyInstances to ignore manual machines.

> Also... it crosses my mind that the environ provisioner might take
some time to
> get the memo about the environment dying. Do we have any way to block
racy
> *provisioning* of recently-added machines? That can be a followup,
though.

I don't think we have any way of doing that yet.

https://codereview.appspot.com/41280043/diff/1/state/apiserver/client/destroy.go#newcode110
state/apiserver/client/destroy.go:110: if inst == nil {
On 2013/12/18 12:52:20, fwereade wrote:
> log the failed instances?

Done.

https://codereview.appspot.com/41280043/diff/1/state/apiserver/client/destroy.go#newcode145
state/apiserver/client/destroy.go:145: return
strings.HasPrefix(string(iid), "manual:")
On 2013/12/18 12:52:20, fwereade wrote:
> Yeah, use the nonce -- if you need to make this a method on machine,
so be it.

Done.

https://codereview.appspot.com/41280043/

[Andrew Wilkins (axwalk)]

Please take a look.

https://codereview.appspot.com/41280043/

[Andrew Wilkins (axwalk)]

On 2013/12/19 14:25:38, axw wrote:
> Please take a look.

FYI, I added a --force flag to destroy-environment which skips the API
call. This is necessary for destroying broken environments, where the
API server is inaccessible or otherwise broken. There's also a drive-by
fix to add-machine, which was returning spurious errors due to nil
interface assignment.

https://codereview.appspot.com/41280043/

[Andrew Wilkins (axwalk)]

On 2013/12/19 14:25:38, axw wrote:
> Please take a look.

FYI, I added a --force flag to destroy-environment which skips the API
call. This is necessary for destroying broken environments, where the
API server is inaccessible or otherwise broken. There's also a drive-by
fix to add-machine, which was returning spurious errors due to nil
interface assignment.

https://codereview.appspot.com/41280043/

=== modified file 'cmd/juju/addmachine.go'
--- cmd/juju/addmachine.go	2013-12-17 18:21:26 +0000
+++ cmd/juju/addmachine.go	2013-12-19 14:21:12 +0000
@@ -175,7 +175,11 @@
 	} else {
 		// Currently, only one machine is added, but in future there may be several added in one call.
 		machineInfo := results[0]
-		machineId, err = machineInfo.Machine, machineInfo.Error
+		var machineErr *params.Error
+		machineId, machineErr = machineInfo.Machine, machineInfo.Error
+		if machineErr != nil {
+			err = machineErr
+		}
 	}
 	if err != nil {
 		return err

=== modified file 'cmd/juju/cmd_test.go'
--- cmd/juju/cmd_test.go	2013-11-18 12:17:58 +0000
+++ cmd/juju/cmd_test.go	2013-12-19 14:21:12 +0000
@@ -4,7 +4,6 @@
 package main
 
 import (
-	"bytes"
 	"io"
 	"io/ioutil"
 	"os"
@@ -13,10 +12,6 @@
 	gc "launchpad.net/gocheck"
 
 	"launchpad.net/juju-core/cmd"
-	"launchpad.net/juju-core/environs"
-	"launchpad.net/juju-core/environs/configstore"
-	"launchpad.net/juju-core/errors"
-	"launchpad.net/juju-core/instance"
 	"launchpad.net/juju-core/juju/osenv"
 	"launchpad.net/juju-core/juju/testing"
 	"launchpad.net/juju-core/provider/dummy"
@@ -158,127 +153,6 @@
 	return
 }
 
-func (*CmdSuite) TestDestroyEnvironmentCommand(c *gc.C) {
-	// Prepare the environment so we can destroy it.
-	store, err := configstore.Default()
-	c.Assert(err, gc.IsNil)
-	_, err = environs.PrepareFromName("", store)
-	c.Assert(err, gc.IsNil)
-
-	// Verify that the environment information exists.
-	_, err = store.ReadInfo("peckham")
-	c.Assert(err, gc.IsNil)
-
-	// check environment is mandatory
-	opc, errc := runCommand(nullContext(), new(DestroyEnvironmentCommand))
-	c.Check(<-errc, gc.Equals, NoEnvironmentError)
-
-	// normal destroy
-	opc, errc = runCommand(nullContext(), new(DestroyEnvironmentCommand), "peckham", "--yes")
-	c.Check(<-errc, gc.IsNil)
-	c.Check((<-opc).(dummy.OpDestroy).Env, gc.Equals, "peckham")
-
-	// Verify that the environment information has been removed.
-	_, err = store.ReadInfo("peckham")
-	c.Assert(err, jc.Satisfies, errors.IsNotFoundError)
-
-	_, err = environs.PrepareFromName("brokenenv", store)
-	c.Assert(err, gc.IsNil)
-
-	// destroy with broken environment
-	opc, errc = runCommand(nullContext(), new(DestroyEnvironmentCommand), "brokenenv", "--yes")
-	c.Check(<-opc, gc.IsNil)
-	c.Check(<-errc, gc.ErrorMatches, "dummy.Destroy is broken")
-	c.Check(<-opc, gc.IsNil)
-}
-
-func (*CmdSuite) TestDestroyEnvironmentCommandConfirmationFlag(c *gc.C) {
-	com := new(DestroyEnvironmentCommand)
-	c.Check(coretesting.InitCommand(com, []string{"peckham"}), gc.IsNil)
-	c.Check(com.assumeYes, gc.Equals, false)
-
-	com = new(DestroyEnvironmentCommand)
-	c.Check(coretesting.InitCommand(com, []string{"peckham", "-y"}), gc.IsNil)
-	c.Check(com.assumeYes, gc.Equals, true)
-
-	com = new(DestroyEnvironmentCommand)
-	c.Check(coretesting.InitCommand(com, []string{"peckham", "--yes"}), gc.IsNil)
-	c.Check(com.assumeYes, gc.Equals, true)
-}
-
-func (*CmdSuite) TestDestroyEnvironmentCommandConfirmation(c *gc.C) {
-	var stdin, stdout bytes.Buffer
-	ctx := cmd.DefaultContext()
-	ctx.Stdout = &stdout
-	ctx.Stdin = &stdin
-
-	store, err := configstore.Default()
-	c.Assert(err, gc.IsNil)
-	// Prepare the environment so we can destroy it.
-	env, err := environs.PrepareFromName("", store)
-	c.Assert(err, gc.IsNil)
-
-	assertEnvironNotDestroyed(c, env, store)
-
-	// Ensure confirmation is requested if "-y" is not specified.
-	stdin.WriteString("n")
-	opc, errc := runCommand(ctx, new(DestroyEnvironmentCommand), "peckham")
-	c.Check(<-errc, gc.ErrorMatches, "Environment destruction aborted")
-	c.Check(<-opc, gc.IsNil)
-	c.Check(stdout.String(), gc.Matches, "WARNING!.*peckham.*\\(type: dummy\\)(.|\n)*")
-	assertEnvironNotDestroyed(c, env, store)
-
-	// EOF on stdin: equivalent to answering no.
-	stdin.Reset()
-	stdout.Reset()
-	opc, errc = runCommand(ctx, new(DestroyEnvironmentCommand), "peckham")
-	c.Check(<-opc, gc.IsNil)
-	c.Check(<-errc, gc.ErrorMatches, "Environment destruction aborted")
-	assertEnvironNotDestroyed(c, env, store)
-
-	// "--yes" passed: no confirmation request.
-	stdin.Reset()
-	stdout.Reset()
-	opc, errc = runCommand(ctx, new(DestroyEnvironmentCommand), "peckham", "--yes")
-	c.Check(<-errc, gc.IsNil)
-	c.Check((<-opc).(dummy.OpDestroy).Env, gc.Equals, "peckham")
-	c.Check(stdout.String(), gc.Equals, "")
-	assertEnvironDestroyed(c, env, store)
-
-	// Any of casing of "y" and "yes" will confirm.
-	for _, answer := range []string{"y", "Y", "yes", "YES"} {
-		// Prepare the environment so we can destroy it.
-		_, err := environs.PrepareFromName("", store)
-		c.Assert(err, gc.IsNil)
-
-		stdin.Reset()
-		stdout.Reset()
-		stdin.WriteString(answer)
-		opc, errc = runCommand(ctx, new(DestroyEnvironmentCommand), "peckham")
-		c.Check(<-errc, gc.IsNil)
-		c.Check((<-opc).(dummy.OpDestroy).Env, gc.Equals, "peckham")
-		c.Check(stdout.String(), gc.Matches, "WARNING!.*peckham.*\\(type: dummy\\)(.|\n)*")
-		assertEnvironDestroyed(c, env, store)
-	}
-}
-
-func assertEnvironDestroyed(c *gc.C, env environs.Environ, store configstore.Storage) {
-	_, err := store.ReadInfo(env.Name())
-	c.Assert(err, jc.Satisfies, errors.IsNotFoundError)
-
-	_, err = env.Instances([]instance.Id{"invalid"})
-	c.Assert(err, gc.ErrorMatches, "environment has been destroyed")
-}
-
-func assertEnvironNotDestroyed(c *gc.C, env environs.Environ, store configstore.Storage) {
-	info, err := store.ReadInfo(env.Name())
-	c.Assert(err, gc.IsNil)
-	c.Assert(info.Initialized(), jc.IsTrue)
-
-	_, err = environs.NewFromName(env.Name(), store)
-	c.Assert(err, gc.IsNil)
-}
-
 var deployTests = []struct {
 	args []string
 	com  *DeployCommand

=== modified file 'cmd/juju/destroyenvironment.go'
--- cmd/juju/destroyenvironment.go	2013-11-19 06:43:27 +0000
+++ cmd/juju/destroyenvironment.go	2013-12-19 14:21:12 +0000
@@ -15,6 +15,9 @@
 	"launchpad.net/juju-core/cmd"
 	"launchpad.net/juju-core/environs"
 	"launchpad.net/juju-core/environs/configstore"
+	"launchpad.net/juju-core/juju"
+	"launchpad.net/juju-core/state/api"
+	"launchpad.net/juju-core/state/api/params"
 )
 
 var NoEnvironmentError = errors.New("no environment specified")
@@ -24,6 +27,7 @@
 	cmd.CommandBase
 	envName   string
 	assumeYes bool
+	force     bool
 }
 
 func (c *DestroyEnvironmentCommand) Info() *cmd.Info {
@@ -37,6 +41,7 @@
 func (c *DestroyEnvironmentCommand) SetFlags(f *gnuflag.FlagSet) {
 	f.BoolVar(&c.assumeYes, "y", false, "Do not ask for confirmation")
 	f.BoolVar(&c.assumeYes, "yes", false, "")
+	f.BoolVar(&c.force, "force", false, "Forcefully destroy the environment, directly through the environment provider")
 }
 
 func (c *DestroyEnvironmentCommand) Run(ctx *cmd.Context) error {
@@ -62,11 +67,20 @@
 			return errors.New("Environment destruction aborted")
 		}
 	}
-
-	// TODO(axw) 2013-08-30 bug 1218688
-	// destroy manually provisioned machines, or otherwise
-	// block destroy-environment until all manually provisioned
-	// machines have been manually "destroyed".
+	// If --force is supplied, then don't attempt to use the API.
+	// This is necessary to destroy broken environments, where the
+	// API server is inaccessible or faulty.
+	if !c.force {
+		conn, err := juju.NewAPIConn(environ, api.DefaultDialOpts())
+		if err != nil {
+			return err
+		}
+		defer conn.Close()
+		err = conn.State.Client().DestroyEnvironment()
+		if err != nil && !params.IsCodeNotImplemented(err) {
+			return fmt.Errorf("destroying environment: %v", err)
+		}
+	}
 	return environs.Destroy(environ, store)
 }
 

=== added file 'cmd/juju/destroyenvironment_test.go'
--- cmd/juju/destroyenvironment_test.go	1970-01-01 00:00:00 +0000
+++ cmd/juju/destroyenvironment_test.go	2013-12-19 14:21:12 +0000
@@ -0,0 +1,160 @@
+// Copyright 2012, 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package main
+
+import (
+	"bytes"
+
+	gc "launchpad.net/gocheck"
+
+	"launchpad.net/juju-core/cmd"
+	"launchpad.net/juju-core/environs"
+	"launchpad.net/juju-core/environs/configstore"
+	"launchpad.net/juju-core/errors"
+	"launchpad.net/juju-core/instance"
+	"launchpad.net/juju-core/juju/testing"
+	"launchpad.net/juju-core/provider/dummy"
+	coretesting "launchpad.net/juju-core/testing"
+	jc "launchpad.net/juju-core/testing/checkers"
+)
+
+type destroyEnvSuite struct {
+	testing.JujuConnSuite
+}
+
+var _ = gc.Suite(&destroyEnvSuite{})
+
+func (s *destroyEnvSuite) TestDestroyEnvironmentCommand(c *gc.C) {
+	// Prepare the environment so we can destroy it.
+	_, err := environs.PrepareFromName("dummyenv", s.ConfigStore)
+	c.Assert(err, gc.IsNil)
+
+	// check environment is mandatory
+	opc, errc := runCommand(nullContext(), new(DestroyEnvironmentCommand))
+	c.Check(<-errc, gc.Equals, NoEnvironmentError)
+
+	// normal destroy
+	opc, errc = runCommand(nullContext(), new(DestroyEnvironmentCommand), "dummyenv", "--yes")
+	c.Check(<-errc, gc.IsNil)
+	c.Check((<-opc).(dummy.OpDestroy).Env, gc.Equals, "dummyenv")
+
+	// Verify that the environment information has been removed.
+	_, err = s.ConfigStore.ReadInfo("dummyenv")
+	c.Assert(err, jc.Satisfies, errors.IsNotFoundError)
+}
+
+func (s *destroyEnvSuite) TestDestroyEnvironmentCommandBroken(c *gc.C) {
+	oldinfo, err := s.ConfigStore.ReadInfo("dummyenv")
+	c.Assert(err, gc.IsNil)
+	bootstrapConfig := oldinfo.BootstrapConfig()
+	apiEndpoint := oldinfo.APIEndpoint()
+	apiCredentials := oldinfo.APICredentials()
+	err = oldinfo.Destroy()
+	c.Assert(err, gc.IsNil)
+	newinfo, err := s.ConfigStore.CreateInfo("dummyenv")
+	c.Assert(err, gc.IsNil)
+
+	bootstrapConfig["broken"] = "Destroy"
+	newinfo.SetBootstrapConfig(bootstrapConfig)
+	newinfo.SetAPIEndpoint(apiEndpoint)
+	newinfo.SetAPICredentials(apiCredentials)
+	err = newinfo.Write()
+	c.Assert(err, gc.IsNil)
+
+	// Prepare the environment so we can destroy it.
+	_, err = environs.PrepareFromName("dummyenv", s.ConfigStore)
+	c.Assert(err, gc.IsNil)
+
+	// destroy with broken environment
+	opc, errc := runCommand(nullContext(), new(DestroyEnvironmentCommand), "dummyenv", "--yes")
+	c.Check(<-opc, gc.IsNil)
+	c.Check(<-errc, gc.ErrorMatches, "dummy.Destroy is broken")
+	c.Check(<-opc, gc.IsNil)
+}
+
+func (*destroyEnvSuite) TestDestroyEnvironmentCommandConfirmationFlag(c *gc.C) {
+	com := new(DestroyEnvironmentCommand)
+	c.Check(coretesting.InitCommand(com, []string{"dummyenv"}), gc.IsNil)
+	c.Check(com.assumeYes, gc.Equals, false)
+
+	com = new(DestroyEnvironmentCommand)
+	c.Check(coretesting.InitCommand(com, []string{"dummyenv", "-y"}), gc.IsNil)
+	c.Check(com.assumeYes, gc.Equals, true)
+
+	com = new(DestroyEnvironmentCommand)
+	c.Check(coretesting.InitCommand(com, []string{"dummyenv", "--yes"}), gc.IsNil)
+	c.Check(com.assumeYes, gc.Equals, true)
+}
+
+func (s *destroyEnvSuite) TestDestroyEnvironmentCommandConfirmation(c *gc.C) {
+	var stdin, stdout bytes.Buffer
+	ctx := cmd.DefaultContext()
+	ctx.Stdout = &stdout
+	ctx.Stdin = &stdin
+
+	// Prepare the environment so we can destroy it.
+	env, err := environs.PrepareFromName("dummyenv", s.ConfigStore)
+	c.Assert(err, gc.IsNil)
+
+	assertEnvironNotDestroyed(c, env, s.ConfigStore)
+
+	// Ensure confirmation is requested if "-y" is not specified.
+	stdin.WriteString("n")
+	opc, errc := runCommand(ctx, new(DestroyEnvironmentCommand), "dummyenv")
+	c.Check(<-errc, gc.ErrorMatches, "Environment destruction aborted")
+	c.Check(<-opc, gc.IsNil)
+	c.Check(stdout.String(), gc.Matches, "WARNING!.*dummyenv.*\\(type: dummy\\)(.|\n)*")
+	assertEnvironNotDestroyed(c, env, s.ConfigStore)
+
+	// EOF on stdin: equivalent to answering no.
+	stdin.Reset()
+	stdout.Reset()
+	opc, errc = runCommand(ctx, new(DestroyEnvironmentCommand), "dummyenv")
+	c.Check(<-opc, gc.IsNil)
+	c.Check(<-errc, gc.ErrorMatches, "Environment destruction aborted")
+	assertEnvironNotDestroyed(c, env, s.ConfigStore)
+
+	// "--yes" passed: no confirmation request.
+	stdin.Reset()
+	stdout.Reset()
+	opc, errc = runCommand(ctx, new(DestroyEnvironmentCommand), "dummyenv", "--yes")
+	c.Check(<-errc, gc.IsNil)
+	c.Check((<-opc).(dummy.OpDestroy).Env, gc.Equals, "dummyenv")
+	c.Check(stdout.String(), gc.Equals, "")
+	assertEnvironDestroyed(c, env, s.ConfigStore)
+
+	// Any of casing of "y" and "yes" will confirm.
+	for _, answer := range []string{"y", "Y", "yes", "YES"} {
+		// Prepare the environment so we can destroy it.
+		s.Reset(c)
+		env, err := environs.PrepareFromName("dummyenv", s.ConfigStore)
+		c.Assert(err, gc.IsNil)
+
+		stdin.Reset()
+		stdout.Reset()
+		stdin.WriteString(answer)
+		opc, errc = runCommand(ctx, new(DestroyEnvironmentCommand), "dummyenv")
+		c.Check(<-errc, gc.IsNil)
+		c.Check((<-opc).(dummy.OpDestroy).Env, gc.Equals, "dummyenv")
+		c.Check(stdout.String(), gc.Matches, "WARNING!.*dummyenv.*\\(type: dummy\\)(.|\n)*")
+		assertEnvironDestroyed(c, env, s.ConfigStore)
+	}
+}
+
+func assertEnvironDestroyed(c *gc.C, env environs.Environ, store configstore.Storage) {
+	_, err := store.ReadInfo(env.Name())
+	c.Assert(err, jc.Satisfies, errors.IsNotFoundError)
+
+	_, err = env.Instances([]instance.Id{"invalid"})
+	c.Assert(err, gc.ErrorMatches, "environment has been destroyed")
+}
+
+func assertEnvironNotDestroyed(c *gc.C, env environs.Environ, store configstore.Storage) {
+	info, err := store.ReadInfo(env.Name())
+	c.Assert(err, gc.IsNil)
+	c.Assert(info.Initialized(), jc.IsTrue)
+
+	_, err = environs.NewFromName(env.Name(), store)
+	c.Assert(err, gc.IsNil)
+}

=== modified file 'provider/dummy/environs.go'
--- provider/dummy/environs.go	2013-12-16 09:53:26 +0000
+++ provider/dummy/environs.go	2013-12-19 14:21:12 +0000
@@ -786,8 +786,10 @@
 		if inst == nil {
 			err = environs.ErrPartialInstances
 			notFound++
+			insts = append(insts, nil)
+		} else {
+			insts = append(insts, inst)
 		}
-		insts = append(insts, inst)
 	}
 	if notFound == len(ids) {
 		return nil, environs.ErrNoInstances

=== modified file 'state/api/client.go'
--- state/api/client.go	2013-12-19 08:46:10 +0000
+++ state/api/client.go	2013-12-19 14:21:12 +0000
@@ -357,6 +357,14 @@
 	return c.st.Call("Client", "", "SetEnvironAgentVersion", args, nil)
 }
 
+// DestroyEnvironment puts the environment into a "dying" state,
+// and removes all non-manager machine instances. DestroyEnvironment
+// will fail if there are any manually-provisioned non-manager machines
+// in state.
+func (c *Client) DestroyEnvironment() error {
+	return c.st.Call("Client", "", "DestroyEnvironment", nil, nil)
+}
+
 // AddLocalCharm prepares the given charm with a local: schema in its
 // URL, and uploads it via the API server, returning the assigned
 // charm URL. If the API server does not support charm uploads, an

=== added file 'state/apiserver/client/destroy.go'
--- state/apiserver/client/destroy.go	1970-01-01 00:00:00 +0000
+++ state/apiserver/client/destroy.go	2013-12-19 14:21:12 +0000
@@ -0,0 +1,153 @@
+// Copyright 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package client
+
+import (
+	"fmt"
+	"strings"
+
+	"launchpad.net/juju-core/environs"
+	"launchpad.net/juju-core/instance"
+	"launchpad.net/juju-core/state"
+)
+
+// DestroyEnvironment destroys all services and non-manager machine
+// instances in the environment.
+func (c *Client) DestroyEnvironment() error {
+	// TODO(axw) 2013-08-30 bug 1218688
+	//
+	// There's a race here: a client might add a manual machine
+	// after another client checks. Destroy-environment will
+	// still fail, but the environment will be in a state where
+	// entities can only be destroyed. In the future we should
+	// introduce a method of preventing Environment.Destroy()
+	// from succeeding if machines have been added.
+
+	// First, check for manual machines. We bail out if there are any,
+	// to stop the user from prematurely hobbling the environment.
+	machines, err := c.api.state.AllMachines()
+	if err != nil {
+		return err
+	}
+	if err := checkManualMachines(machines); err != nil {
+		return err
+	}
+
+	// Set the environment to Dying, to lock out new machines and services.
+	// Environment.Destroy() also schedules a cleanup for existing services.
+	// Afterwards, refresh the machines in case any were added between the
+	// first check and the Environment.Destroy().
+	env, err := c.api.state.Environment()
+	if err != nil {
+		return err
+	}
+	if err = env.Destroy(); err != nil {
+		return err
+	}
+	machines, err = c.api.state.AllMachines()
+	if err != nil {
+		return err
+	}
+
+	// We must destroy instances server-side to support hosted Juju,
+	// as there's no CLI to fall back on. In that case, we only ever
+	// destroy non-state machines; we leave destroying state servers
+	// in non-hosted environments to the CLI, as otherwise the API
+	// server may get cut off.
+	if err := destroyInstances(c.api.state, machines); err != nil {
+		return err
+	}
+
+	// Make sure once again that there are no manually provisioned
+	// non-manager machines. This caters for the race between the
+	// first check and the Environment.Destroy().
+	if err := checkManualMachines(machines); err != nil {
+		return err
+	}
+
+	// Return to the caller. If it's the CLI, it will finish up
+	// by calling the provider's Destroy method, which will
+	// destroy the state servers, any straggler instances, and
+	// other provider-specific resources.
+	return nil
+}
+
+// destroyInstances directly destroys all non-manager,
+// non-manual machine instances.
+func destroyInstances(st *state.State, machines []*state.Machine) error {
+	var ids []instance.Id
+	for _, m := range machines {
+		if m.IsManager() {
+			continue
+		}
+		manual, err := m.IsManual()
+		if manual {
+			continue
+		} else if err != nil {
+			return err
+		}
+		id, err := m.InstanceId()
+		if err != nil {
+			continue
+		}
+		ids = append(ids, id)
+	}
+	if len(ids) == 0 {
+		return nil
+	}
+	envcfg, err := st.EnvironConfig()
+	if err != nil {
+		return err
+	}
+	env, err := environs.New(envcfg)
+	if err != nil {
+		return err
+	}
+	// TODO(axw) 2013-12-12 #1260171
+	// Modify InstanceBroker.StopInstances to take
+	// a slice of IDs rather than Instances.
+	instances, err := env.Instances(ids)
+	switch err {
+	case nil:
+	default:
+		return err
+	case environs.ErrNoInstances:
+		return nil
+	case environs.ErrPartialInstances:
+		var nonNilInstances []instance.Instance
+		for i, inst := range instances {
+			if inst == nil {
+				logger.Warningf("unknown instance ID: %v", ids[i])
+				continue
+			}
+			nonNilInstances = append(nonNilInstances, inst)
+		}
+		instances = nonNilInstances
+	}
+	return env.StopInstances(instances)
+}
+
+// checkManualMachines checks if any of the machines in the slice were
+// manually provisioned, and are non-manager machines. These machines
+// must (currently) by manually destroyed via destroy-machine before
+// destroy-environment can successfully complete.
+func checkManualMachines(machines []*state.Machine) error {
+	var ids []string
+	for _, m := range machines {
+		if m.IsManager() {
+			continue
+		}
+		manual, err := m.IsManual()
+		if err != nil {
+			return err
+		}
+		if manual {
+			ids = append(ids, m.Id())
+		}
+	}
+	if len(ids) > 0 {
+		return fmt.Errorf("manually provisioned machines must first be destroyed with `juju destroy-machine %s`", strings.Join(ids, " "))
+	}
+	return nil
+}

=== added file 'state/apiserver/client/destroy_test.go'
--- state/apiserver/client/destroy_test.go	1970-01-01 00:00:00 +0000
+++ state/apiserver/client/destroy_test.go	2013-12-19 14:21:12 +0000
@@ -0,0 +1,123 @@
+// Copyright 2012, 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package client_test
+
+import (
+	"fmt"
+
+	gc "launchpad.net/gocheck"
+
+	"launchpad.net/juju-core/environs"
+	coreerrors "launchpad.net/juju-core/errors"
+	"launchpad.net/juju-core/instance"
+	"launchpad.net/juju-core/juju/testing"
+	"launchpad.net/juju-core/state"
+	jc "launchpad.net/juju-core/testing/checkers"
+)
+
+type destroyEnvironmentSuite struct {
+	baseSuite
+}
+
+var _ = gc.Suite(&destroyEnvironmentSuite{})
+
+// setUpManual adds "manually provisioned" machines to state:
+// one manager machine, and one non-manager.
+func (s *destroyEnvironmentSuite) setUpManual(c *gc.C) (m0, m1 *state.Machine) {
+	m0, err := s.State.AddMachine("precise", state.JobManageEnviron, state.JobManageState)
+	c.Assert(err, gc.IsNil)
+	err = m0.SetProvisioned(instance.Id("manual:0"), "manual:0:fake_nonce", nil)
+	c.Assert(err, gc.IsNil)
+	m1, err = s.State.AddMachine("precise", state.JobHostUnits)
+	c.Assert(err, gc.IsNil)
+	err = m1.SetProvisioned(instance.Id("manual:1"), "manual:1:fake_nonce", nil)
+	c.Assert(err, gc.IsNil)
+	return m0, m1
+}
+
+// setUpInstances adds machines to state backed by instances:
+// one manager machine, and one non-manager.
+func (s *destroyEnvironmentSuite) setUpInstances(c *gc.C) (m0, m1 *state.Machine) {
+	m0, err := s.State.AddMachine("precise", state.JobManageEnviron, state.JobManageState)
+	c.Assert(err, gc.IsNil)
+	inst, _ := testing.AssertStartInstance(c, s.APIConn.Environ, m0.Id())
+	err = m0.SetProvisioned(inst.Id(), "fake_nonce", nil)
+	c.Assert(err, gc.IsNil)
+	m1, err = s.State.AddMachine("precise", state.JobHostUnits)
+	c.Assert(err, gc.IsNil)
+	inst, _ = testing.AssertStartInstance(c, s.APIConn.Environ, m1.Id())
+	err = m1.SetProvisioned(inst.Id(), "fake_nonce", nil)
+	c.Assert(err, gc.IsNil)
+	return m0, m1
+}
+
+func (s *destroyEnvironmentSuite) TestDestroyEnvironmentManual(c *gc.C) {
+	s.setUpScenario(c)
+	_, nonManager := s.setUpManual(c)
+
+	// If there are any non-manager manual machines in state, DestroyEnvironment will
+	// error. It will not set the Dying flag on the environment.
+	err := s.APIState.Client().DestroyEnvironment()
+	c.Assert(err, gc.ErrorMatches, fmt.Sprintf("manually provisioned machines must first be destroyed with `juju destroy-machine %s`", nonManager.Id()))
+	env, err := s.State.Environment()
+	c.Assert(err, gc.IsNil)
+	c.Assert(env.Life(), gc.Equals, state.Alive)
+
+	// If we remove the non-manager machine, it should pass.
+	// Manager machines will remain.
+	err = nonManager.EnsureDead()
+	c.Assert(err, gc.IsNil)
+	err = nonManager.Remove()
+	c.Assert(err, gc.IsNil)
+	err = s.APIState.Client().DestroyEnvironment()
+	c.Assert(err, gc.IsNil)
+	err = env.Refresh()
+	c.Assert(err, gc.IsNil)
+	c.Assert(env.Life(), gc.Equals, state.Dying)
+}
+
+func (s *destroyEnvironmentSuite) TestDestroyEnvironment(c *gc.C) {
+	s.setUpScenario(c)
+	manager, nonManager := s.setUpInstances(c)
+	managerId, _ := manager.InstanceId()
+	nonManagerId, _ := nonManager.InstanceId()
+
+	instances, err := s.APIConn.Environ.Instances([]instance.Id{managerId, nonManagerId})
+	c.Assert(err, gc.IsNil)
+	for _, inst := range instances {
+		c.Assert(inst, gc.NotNil)
+	}
+
+	services, err := s.State.AllServices()
+	c.Assert(err, gc.IsNil)
+
+	err = s.APIState.Client().DestroyEnvironment()
+	c.Assert(err, gc.IsNil)
+
+	// After DestroyEnvironment returns, we should have:
+	//   - all non-manager instances stopped
+	instances, err = s.APIConn.Environ.Instances([]instance.Id{managerId, nonManagerId})
+	c.Assert(err, gc.Equals, environs.ErrPartialInstances)
+	c.Assert(instances[0], gc.NotNil)
+	c.Assert(instances[1], gc.IsNil)
+	//   - all services in state are Dying or Dead (or removed altogether),
+	//     after running the state Cleanups.
+	needsCleanup, err := s.State.NeedsCleanup()
+	c.Assert(err, gc.IsNil)
+	c.Assert(needsCleanup, jc.IsTrue)
+	err = s.State.Cleanup()
+	c.Assert(err, gc.IsNil)
+	for _, s := range services {
+		err = s.Refresh()
+		if err != nil {
+			c.Assert(err, jc.Satisfies, coreerrors.IsNotFoundError)
+		} else {
+			c.Assert(s.Life(), gc.Not(gc.Equals), state.Alive)
+		}
+	}
+	//   - environment is Dying
+	env, err := s.State.Environment()
+	c.Assert(err, gc.IsNil)
+	c.Assert(env.Life(), gc.Equals, state.Dying)
+}

=== modified file 'state/machine.go'
--- state/machine.go	2013-12-04 15:34:47 +0000
+++ state/machine.go	2013-12-19 14:21:12 +0000
@@ -206,6 +206,26 @@
 	return false
 }
 
+// IsManual returns true if the machine was manually provisioned.
+func (m *Machine) IsManual() (bool, error) {
+	// Apart from the bootstrap machine, manually provisioned
+	// machines have a nonce prefixed with "manual:". This is
+	// unique to manual provisioning.
+	if strings.HasPrefix(m.doc.Nonce, "manual:") {
+		return true, nil
+	}
+	// The bootstrap machine uses BootstrapNonce, so in that
+	// case we need to check if its provider type is "null".
+	if m.doc.Id == "0" {
+		cfg, err := m.st.EnvironConfig()
+		if err != nil {
+			return false, err
+		}
+		return cfg.Type() == "null", nil
+	}
+	return false, nil
+}
+
 // AgentTools returns the tools that the agent is currently running.
 // It returns an error that satisfies IsNotFound if the tools have not yet been set.
 func (m *Machine) AgentTools() (*tools.Tools, error) {

=== modified file 'state/machine_test.go'
--- state/machine_test.go	2013-12-12 17:23:45 +0000
+++ state/machine_test.go	2013-12-19 14:21:12 +0000
@@ -90,6 +90,49 @@
 	}
 }
 
+func (s *MachineSuite) TestMachineIsManualBootstrap(c *gc.C) {
+	cfg, err := s.State.EnvironConfig()
+	c.Assert(err, gc.IsNil)
+	c.Assert(cfg.Type(), gc.Not(gc.Equals), "null")
+	c.Assert(s.machine.Id(), gc.Equals, "0")
+	manual, err := s.machine.IsManual()
+	c.Assert(err, gc.IsNil)
+	c.Assert(manual, jc.IsFalse)
+	newcfg, err := cfg.Apply(map[string]interface{}{"type": "null"})
+	c.Assert(err, gc.IsNil)
+	err = s.State.SetEnvironConfig(newcfg, cfg)
+	c.Assert(err, gc.IsNil)
+	manual, err = s.machine.IsManual()
+	c.Assert(err, gc.IsNil)
+	c.Assert(manual, jc.IsTrue)
+}
+
+func (s *MachineSuite) TestMachineIsManual(c *gc.C) {
+	tests := []struct {
+		instanceId instance.Id
+		nonce      string
+		isManual   bool
+	}{
+		{instanceId: "x", nonce: "y", isManual: false},
+		{instanceId: "manual:", nonce: "y", isManual: false},
+		{instanceId: "x", nonce: "manual:", isManual: true},
+		{instanceId: "x", nonce: "manual:y", isManual: true},
+		{instanceId: "x", nonce: "manual", isManual: false},
+	}
+	for _, test := range tests {
+		params := state.AddMachineParams{
+			Series:     "quantal",
+			Jobs:       []state.MachineJob{state.JobHostUnits},
+			InstanceId: test.instanceId,
+			Nonce:      test.nonce,
+		}
+		m, err := s.State.InjectMachine(&params)
+		c.Assert(err, gc.IsNil)
+		isManual, err := m.IsManual()
+		c.Assert(isManual, gc.Equals, test.isManual)
+	}
+}
+
 func (s *MachineSuite) TestLifeJobManageEnviron(c *gc.C) {
 	// A JobManageEnviron machine must never advance lifecycle.
 	m, err := s.State.AddMachine("quantal", state.JobManageEnviron)