Merge lp:~axwalk/juju-core/lp1246983-cli-api-removeagents into lp:~go-bot/juju-core/trunk

[Andrew Wilkins]

Migrate juju destroy-environment to use API

This change introduces a new client API
method, DestroyJuju. This method does the
following, server-side, in order:
 - marks the environment as Dying, preventing
   the addition of machines and services to
   state
 - ensures there are no non-manager manually-
   provisioned machines in state
 - marks all services as Dying, thus prevening
   units from being added
 - stops all non-manager machine instances
 - marks the environment as Dead

Machine agents watch, via the machiner API, for
changes to the environment's lifecycle, and
terminate themselves when the environment
becomes Dead. This is how we terminate manual
bootstrap nodes.

A followup will update agent termination logic
to uninstall mongo on state servers, and also
remove the data-dir/log-dir.

Fixes #1218688
Fixes #1246343

https://codereview.appspot.com/22870045/

[Andrew Wilkins (axwalk)]

Reviewers: mp+194801_code.launchpad.net,

Message:
Please take a look.

Description:
Migrate juju destroy-environment to use API

(This is a prototype, hence lacking tests.
Please review as such, and if it's in the
right direction I'll update with tests.)

This change introduces a new client API
method, DestroyJuju. This method (server-side)
marks the environment as Dying, destroys (and
waits on death of) all units, and then marks
the environment as Dead. When the environment
is marked as Dying, addition of units, services
and machines is prevented.

Machine agents watch, via the machiner API, for
changes to the environment's lifecycle, and
terminate themselves when the environment
becomes Dead.

A followup will update agent termination logic
to uninstall mongo on state servers, and also
remove the data-dir/log-dir.

Fixes #1218688
Fixes #1246343

https://code.launchpad.net/~axwalk/juju-core/lp1246983-cli-api-removeagents/+merge/194801

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/22870045/

Affected files (+425, -45 lines):
   A [revision details]
   M cmd/juju/destroyenvironment.go
   M cmd/jujud/machine.go
   M provider/null/environ.go
   M state/api/client.go
   A state/api/machiner/environ.go
   M state/api/machiner/machine.go
   M state/api/machiner/machiner.go
   M state/api/params/internal.go
   M state/api/params/params.go
   A state/apiserver/client/destroyjuju.go
   M state/apiserver/machine/machiner.go
   M state/environ.go
   M state/interface.go
   M state/service.go
   M state/service_test.go
   M state/state.go
   M state/watcher.go
   M worker/machiner/machiner.go

[John A Meinel (jameinel)]

some thougts. we did most of the discussion on IRC, I think.

https://codereview.appspot.com/22870045/diff/1/cmd/juju/destroyenvironment.go
File cmd/juju/destroyenvironment.go (right):

https://codereview.appspot.com/22870045/diff/1/cmd/juju/destroyenvironment.go#newcode83
cmd/juju/destroyenvironment.go:83: if err =
conn.State.Client().DestroyJuju(removeAgentsTimeout); err != nil {
I'm intrigued that you pass in the timeout.I was wondering if we need it
configable (I know my manual machines take at least 2min to shutdown
cleanly). At least we can expose it on the CLI if we find it is
necessary.

https://codereview.appspot.com/22870045/diff/1/state/api/machiner/machiner.go
File state/api/machiner/machiner.go (right):

https://codereview.appspot.com/22870045/diff/1/state/api/machiner/machiner.go#newcode58
state/api/machiner/machiner.go:58: func (st *State)
Environment(machineTag string) (*Environment, error) {
I don't quite understand why you take a machineTag for an environment.
Is this just a copy and paste that didn't finish renaming things?

But the MachineEnvironmentResult is still here, too.

If it *does* need a Machine, then I'm not sure why this is on State
rather than Machine. (though to be fair, this is Machine.State and not
state.State which is a personal source of much confusion).

https://codereview.appspot.com/22870045/diff/1/state/apiserver/client/destroyjuju.go
File state/apiserver/client/destroyjuju.go (right):

https://codereview.appspot.com/22870045/diff/1/state/apiserver/client/destroyjuju.go#newcode49
state/apiserver/client/destroyjuju.go:49: func destroyUnits(st
*state.State, attemptStrategy utils.AttemptStrategy) error {
It is surprising to me that destroyUnits would be triggered on the
Client side, rather than on the API server in response to the
env.Destroy request.

https://codereview.appspot.com/22870045/diff/1/state/apiserver/machine/machiner.go
File state/apiserver/machine/machiner.go (right):

https://codereview.appspot.com/22870045/diff/1/state/apiserver/machine/machiner.go#newcode61
state/apiserver/machine/machiner.go:61: // the corresponding machine's
environment.
I don't think we want to design multi-tenancy into the API today.

I'm willing to be corrected by someone else, but we should keep in mind
the future, but only build what we need right now. (Because we're likely
to get it wrong until we're actually building the next thing anyway.)

https://codereview.appspot.com/22870045/

[Andrew Wilkins (axwalk)]

So, based on conversations on IRC, I'm going to rework this. From
fwereade:

"(1) abort if manual machines exist (2) destroy services, set destroying
flag (3) directly StopInstances for non-managers in the API server (4)
environ.Destroy in the CLI"

Sounds fair, I'll do that.

A follow-up will deal with manual machines, and will do the two-phase
start/wait business with a watcher and timeout in between.

https://codereview.appspot.com/22870045/diff/1/cmd/juju/destroyenvironment.go
File cmd/juju/destroyenvironment.go (right):

https://codereview.appspot.com/22870045/diff/1/cmd/juju/destroyenvironment.go#newcode83
cmd/juju/destroyenvironment.go:83: if err =
conn.State.Client().DestroyJuju(removeAgentsTimeout); err != nil {
On 2013/11/12 08:39:26, jameinel wrote:
> I'm intrigued that you pass in the timeout.I was wondering if we need
it
> configable (I know my manual machines take at least 2min to shutdown
cleanly).
> At least we can expose it on the CLI if we find it is necessary.

rogpeppe has suggested we break the API into two (in the usual async
pattern): one that starts, one that finalises. The client can use the
all-watcher to observe machine/unit destruction, and provide feedback
along the way. Despite the added complexity, I'm in favour.

https://codereview.appspot.com/22870045/diff/1/state/api/machiner/machiner.go
File state/api/machiner/machiner.go (right):

https://codereview.appspot.com/22870045/diff/1/state/api/machiner/machiner.go#newcode58
state/api/machiner/machiner.go:58: func (st *State)
Environment(machineTag string) (*Environment, error) {
On 2013/11/12 08:39:26, jameinel wrote:
> I don't quite understand why you take a machineTag for an environment.
> Is this just a copy and paste that didn't finish renaming things?

> But the MachineEnvironmentResult is still here, too.

> If it *does* need a Machine, then I'm not sure why this is on State
rather than
> Machine. (though to be fair, this is Machine.State and not state.State
which is
> a personal source of much confusion).

It needs a Machine for when we have multi-tenancy. The machine-agent
doesn't know its environment name at the point of call, hence this. I'm
open to suggestions, I know it's a bit weird.

https://codereview.appspot.com/22870045/diff/1/state/apiserver/client/destroyjuju.go
File state/apiserver/client/destroyjuju.go (right):

https://codereview.appspot.com/22870045/diff/1/state/apiserver/client/destroyjuju.go#newcode49
state/apiserver/client/destroyjuju.go:49: func destroyUnits(st
*state.State, attemptStrategy utils.AttemptStrategy) error {
On 2013/11/12 08:39:26, jameinel wrote:
> It is surprising to me that destroyUnits would be triggered on the
Client side,
> rather than on the API server in response to the env.Destroy request.

This is happening on the API server side. This is just the "Client API",
as used by the CLI and GUI. Am I misunderstanding your comment?

https://codereview.appspot.com/22870045/diff/1/state/apiserver/machine/machiner.go
File state/apiserver/machine/machiner.go (right):

https://codereview.appspot.com/22870045/diff/1/state/apiserver/machine/machiner.go#newcode61
state/apiserver/machine/machiner.go:61: // the corresponding machin...

[Andrew Wilkins (axwalk)]

Please take a look.

https://codereview.appspot.com/22870045/

[William Reade (fwereade)]

Some good bits, some I'ma bit alarmed about. Can we do a single CL for
the state changes, please, and leave the API bits and additional workers
and so on out for the moment?

There are a couple of open questions in the state code, it's true, but
resolving and reproposing those will be the quickest way to get some of
this landed.

https://codereview.appspot.com/22870045/diff/1/state/api/machiner/machiner.go
File state/api/machiner/machiner.go (right):

https://codereview.appspot.com/22870045/diff/1/state/api/machiner/machiner.go#newcode58
state/api/machiner/machiner.go:58: func (st *State)
Environment(machineTag string) (*Environment, error) {
On 2013/11/12 10:03:08, axw wrote:
> On 2013/11/12 08:39:26, jameinel wrote:
> > I don't quite understand why you take a machineTag for an
environment.
> > Is this just a copy and paste that didn't finish renaming things?
> >
> > But the MachineEnvironmentResult is still here, too.
> >
> > If it *does* need a Machine, then I'm not sure why this is on State
rather
> than
> > Machine. (though to be fair, this is Machine.State and not
state.State which
> is
> > a personal source of much confusion).

> It needs a Machine for when we have multi-tenancy. The machine-agent
doesn't
> know its environment name at the point of call, hence this. I'm open
to
> suggestions, I know it's a bit weird.

I don't think it is necessary, actually, because we will surely know
what environment we're connected to -- and I don't think it'd work
*anyway*, because the machine-tag doesn't identify an environment
anyway.

https://codereview.appspot.com/22870045/diff/20001/state/api/machiner/environ.go
File state/api/machiner/environ.go (right):

https://codereview.appspot.com/22870045/diff/20001/state/api/machiner/environ.go#newcode39
state/api/machiner/environ.go:39: func (e *Environment) Watch()
(watcher.NotifyWatcher, error) {
Environment?

https://codereview.appspot.com/22870045/diff/20001/state/api/machiner/machiner.go
File state/api/machiner/machiner.go (right):

https://codereview.appspot.com/22870045/diff/20001/state/api/machiner/machiner.go#newcode65
state/api/machiner/machiner.go:65: tag: result.EnvironmentTag,
Whoa. I hope this doesn't include a name -- the moment we start messing
around with environment tags we need to un-screw-up this situation
(environment names are not globally unique -- any time we use it we must
be aware it's only an alias).

Regardless I don't think there's a justification for talking about more
than one environment at this level anyway.

https://codereview.appspot.com/22870045/diff/20001/state/apiserver/client/destroyjuju.go
File state/apiserver/client/destroyjuju.go (right):

https://codereview.appspot.com/22870045/diff/20001/state/apiserver/client/destroyjuju.go#newcode97
state/apiserver/client/destroyjuju.go:97: return
env.StopInstances(instances)
There's a wrinkle here in that some machines may be missing for random
stupid provider reasons, and those won't end up getting stopped. I'm
becoming convinced that it's really stupid to have StopInstances take
Instances instead of ids -- it means that we need to go through all the
hassle of finding out everything about one (ie hitting the provider...

[Andrew Wilkins (axwalk)]

I'll create a new CL for the state bits, as requested.

https://codereview.appspot.com/22870045/diff/20001/state/api/machiner/environ.go
File state/api/machiner/environ.go (right):

https://codereview.appspot.com/22870045/diff/20001/state/api/machiner/environ.go#newcode39
state/api/machiner/environ.go:39: func (e *Environment) Watch()
(watcher.NotifyWatcher, error) {
On 2013/11/18 14:43:37, william.reade wrote:
> Environment?

Are you talking about the comment? Fixed.

https://codereview.appspot.com/22870045/diff/20001/state/api/machiner/machiner.go
File state/api/machiner/machiner.go (right):

https://codereview.appspot.com/22870045/diff/20001/state/api/machiner/machiner.go#newcode65
state/api/machiner/machiner.go:65: tag: result.EnvironmentTag,
On 2013/11/18 14:43:37, william.reade wrote:
> Whoa. I hope this doesn't include a name -- the moment we start
messing around
> with environment tags we need to un-screw-up this situation
(environment names
> are not globally unique -- any time we use it we must be aware it's
only an
> alias).

> Regardless I don't think there's a justification for talking about
more than one
> environment at this level anyway.

I'm not sure what you mean by "talking about more than one environment".
That tag is just a handle so we can get a watcher, and is always
expected to be the same.

Do you have a preferred API here?

https://codereview.appspot.com/22870045/diff/20001/state/apiserver/client/destroyjuju.go
File state/apiserver/client/destroyjuju.go (right):

https://codereview.appspot.com/22870045/diff/20001/state/apiserver/client/destroyjuju.go#newcode97
state/apiserver/client/destroyjuju.go:97: return
env.StopInstances(instances)
On 2013/11/18 14:43:37, william.reade wrote:
> There's a wrinkle here in that some machines may be missing for random
stupid
> provider reasons, and those won't end up getting stopped. I'm becoming
convinced
> that it's really stupid to have StopInstances take Instances instead
of ids --
> it means that we need to go through all the hassle of finding out
everything
> about one (ie hitting the provider api, potentially triggering rate
limiting,
> etc) before we can just simply stop them.

Agreed.

https://codereview.appspot.com/22870045/diff/20001/state/apiserver/client/destroyjuju.go#newcode124
state/apiserver/client/destroyjuju.go:124: // Due to an import loop in
tests, we cannot import manual.
On 2013/11/18 14:43:37, william.reade wrote:
> In tests? I'm surprised by that.

The environs/manual tests import something that imports state/apiserver.
Thus, importing environs/manual here prevents running the
environs/manual tests.

https://codereview.appspot.com/22870045/diff/20001/state/apiserver/machine/machiner.go
File state/apiserver/machine/machiner.go (right):

https://codereview.appspot.com/22870045/diff/20001/state/apiserver/machine/machiner.go#newcode60
state/apiserver/machine/machiner.go:60: var result
params.MachineEnvironmentResult
On 2013/11/18 14:43:37, william.reade wrote:
> Why do we persist in naming things after the entities that call them?
This is
> surely an EnvironmentResult.

The idea here was "give me the environment which this machine belongs
to", not an arbitrary...

=== modified file 'cmd/juju/destroyenvironment.go'
--- cmd/juju/destroyenvironment.go	2013-10-17 17:30:56 +0000
+++ cmd/juju/destroyenvironment.go	2013-11-13 04:45:49 +0000
@@ -15,6 +15,8 @@
 	"launchpad.net/juju-core/cmd"
 	"launchpad.net/juju-core/environs"
 	"launchpad.net/juju-core/environs/configstore"
+	"launchpad.net/juju-core/juju"
+	"launchpad.net/juju-core/state/api"
 )
 
 var NoEnvironmentError = errors.New("no environment specified")
@@ -63,10 +65,16 @@
 		}
 	}
 
-	// TODO(axw) 2013-08-30 bug 1218688
-	// destroy manually provisioned machines, or otherwise
-	// block destroy-environment until all manually provisioned
-	// machines have been manually "destroyed".
+	// First, cleanly remove Juju from the environment.
+	conn, err := juju.NewAPIConn(environ, api.DefaultDialOpts())
+	if err != nil {
+		return err
+	}
+	defer conn.Close()
+	if err = conn.State.Client().DestroyJuju(); err != nil {
+		return fmt.Errorf("could not remove agents: %v", err)
+	}
+	// Finally, allow the provider to release environment resources.
 	return environs.Destroy(environ, store)
 }
 

=== modified file 'cmd/jujud/machine.go'
--- cmd/jujud/machine.go	2013-10-02 23:18:21 +0000
+++ cmd/jujud/machine.go	2013-11-13 04:45:49 +0000
@@ -167,6 +167,9 @@
 	runner.StartWorker("machiner", func() (worker.Worker, error) {
 		return machiner.NewMachiner(st.Machiner(), agentConfig), nil
 	})
+	runner.StartWorker("machine-environer", func() (worker.Worker, error) {
+		return machiner.NewMachineEnvironer(st.Machiner(), agentConfig), nil
+	})
 	runner.StartWorker("upgrader", func() (worker.Worker, error) {
 		return upgrader.NewUpgrader(st.Upgrader(), agentConfig), nil
 	})

=== modified file 'environs/manual/provisioner.go'
--- environs/manual/provisioner.go	2013-11-08 02:52:25 +0000
+++ environs/manual/provisioner.go	2013-11-13 04:45:49 +0000
@@ -183,7 +183,7 @@
 		CACert:   configParameters.CACert,
 	}
 	apiInfo := &api.Info{
-		Addrs:    configParameters.StateAddrs,
+		Addrs:    configParameters.APIAddrs,
 		Password: configParameters.Password,
 		Tag:      configParameters.Tag,
 		CACert:   configParameters.CACert,

=== modified file 'provider/dummy/environs.go'
--- provider/dummy/environs.go	2013-11-07 09:09:55 +0000
+++ provider/dummy/environs.go	2013-11-13 04:45:49 +0000
@@ -779,8 +779,10 @@
 		if inst == nil {
 			err = environs.ErrPartialInstances
 			notFound++
+			insts = append(insts, nil)
+		} else {
+			insts = append(insts, inst)
 		}
-		insts = append(insts, inst)
 	}
 	if notFound == len(ids) {
 		return nil, environs.ErrNoInstances

=== modified file 'provider/null/environ.go'
--- provider/null/environ.go	2013-10-24 00:20:59 +0000
+++ provider/null/environ.go	2013-11-13 04:45:49 +0000
@@ -181,7 +181,7 @@
 }
 
 func (e *nullEnviron) Destroy() error {
-	return errors.New("null provider destruction is not implemented yet")
+	return nil
 }
 
 func (e *nullEnviron) OpenPorts(ports []instance.Port) error {

=== modified file 'state/api/client.go'
--- state/api/client.go	2013-11-11 22:19:10 +0000
+++ state/api/client.go	2013-11-13 04:45:49 +0000
@@ -341,3 +341,9 @@
 	args := params.EnvironmentSet{Config: config}
 	return c.st.Call("Client", "", "EnvironmentSet", args, nil)
 }
+
+// DestroyJuju cleanly removes all Juju agents from the environment,
+// except for manually-provisioned non-manager machines.
+func (c *Client) DestroyJuju() error {
+	return c.st.Call("Client", "", "DestroyJuju", nil, nil)
+}

=== added file 'state/api/machiner/environ.go'
--- state/api/machiner/environ.go	1970-01-01 00:00:00 +0000
+++ state/api/machiner/environ.go	2013-11-13 04:45:49 +0000
@@ -0,0 +1,41 @@
+// Copyright 2012, 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package machiner
+
+import (
+	"launchpad.net/juju-core/state/api/params"
+	"launchpad.net/juju-core/state/api/watcher"
+)
+
+// Environment represents a juju environment as seen by a machiner worker.
+type Environment struct {
+	tag  string
+	life params.Life
+	st   *State
+}
+
+// Tag returns the environment's tag.
+func (e *Environment) Tag() string {
+	return e.tag
+}
+
+// Life returns the environment's lifecycle value.
+func (e *Environment) Life() params.Life {
+	return e.life
+}
+
+// Refresh updates the cached local copy of the environment's data.
+func (e *Environment) Refresh() error {
+	life, err := e.st.entityLife(e.tag)
+	if err != nil {
+		return err
+	}
+	e.life = life
+	return nil
+}
+
+// Watch returns a watcher for observing changes to the machine.
+func (e *Environment) Watch() (watcher.NotifyWatcher, error) {
+	return e.st.watch(e.tag)
+}

=== modified file 'state/api/machiner/machine.go'
--- state/api/machiner/machine.go	2013-09-17 18:18:44 +0000
+++ state/api/machiner/machine.go	2013-11-13 04:45:49 +0000
@@ -4,8 +4,6 @@
 package machiner
 
 import (
-	"fmt"
-
 	"launchpad.net/juju-core/state/api/params"
 	"launchpad.net/juju-core/state/api/watcher"
 )
@@ -29,7 +27,7 @@
 
 // Refresh updates the cached local copy of the machine's data.
 func (m *Machine) Refresh() error {
-	life, err := m.st.machineLife(m.tag)
+	life, err := m.st.entityLife(m.tag)
 	if err != nil {
 		return err
 	}
@@ -68,21 +66,5 @@
 
 // Watch returns a watcher for observing changes to the machine.
 func (m *Machine) Watch() (watcher.NotifyWatcher, error) {
-	var results params.NotifyWatchResults
-	args := params.Entities{
-		Entities: []params.Entity{{Tag: m.tag}},
-	}
-	err := m.st.caller.Call("Machiner", "", "Watch", args, &results)
-	if err != nil {
-		return nil, err
-	}
-	if len(results.Results) != 1 {
-		return nil, fmt.Errorf("expected one result, got %d", len(results.Results))
-	}
-	result := results.Results[0]
-	if result.Error != nil {
-		return nil, result.Error
-	}
-	w := watcher.NewNotifyWatcher(m.st.caller, result)
-	return w, nil
+	return m.st.watch(m.tag)
 }

=== modified file 'state/api/machiner/machiner.go'
--- state/api/machiner/machiner.go	2013-07-22 12:19:48 +0000
+++ state/api/machiner/machiner.go	2013-11-13 04:45:49 +0000
@@ -8,6 +8,7 @@
 
 	"launchpad.net/juju-core/state/api/common"
 	"launchpad.net/juju-core/state/api/params"
+	"launchpad.net/juju-core/state/api/watcher"
 )
 
 // State provides access to the Machiner API facade.
@@ -20,8 +21,8 @@
 	return &State{caller}
 }
 
-// machineLife requests the lifecycle of the given machine from the server.
-func (st *State) machineLife(tag string) (params.Life, error) {
+// entityLife requests the lifecycle of the given machine from the server.
+func (st *State) entityLife(tag string) (params.Life, error) {
 	var result params.LifeResults
 	args := params.Entities{
 		Entities: []params.Entity{{Tag: tag}},
@@ -41,7 +42,7 @@
 
 // Machine provides access to methods of a state.Machine through the facade.
 func (st *State) Machine(tag string) (*Machine, error) {
-	life, err := st.machineLife(tag)
+	life, err := st.entityLife(tag)
 	if err != nil {
 		return nil, err
 	}
@@ -51,3 +52,38 @@
 		st:   st,
 	}, nil
 }
+
+// Environment returns the Environment corresponding
+// to the machine with the given tag.
+func (st *State) Environment() (*Environment, error) {
+	var result params.MachineEnvironmentResult
+	err := st.caller.Call("Machiner", "", "Environment", nil, &result)
+	if err != nil {
+		return nil, err
+	}
+	return &Environment{
+		tag:  result.EnvironmentTag,
+		life: result.Life,
+		st:   st,
+	}, nil
+}
+
+// watch returns a watcher for observing changes to the given entity.
+func (st *State) watch(tag string) (watcher.NotifyWatcher, error) {
+	var results params.NotifyWatchResults
+	args := params.Entities{
+		Entities: []params.Entity{{Tag: tag}},
+	}
+	err := st.caller.Call("Machiner", "", "Watch", args, &results)
+	if err != nil {
+		return nil, err
+	}
+	if len(results.Results) != 1 {
+		return nil, fmt.Errorf("expected one result, got %d", len(results.Results))
+	}
+	result := results.Results[0]
+	if result.Error != nil {
+		return nil, result.Error
+	}
+	return watcher.NewNotifyWatcher(st.caller, result), nil
+}

=== modified file 'state/api/params/internal.go'
--- state/api/params/internal.go	2013-10-03 13:13:25 +0000
+++ state/api/params/internal.go	2013-11-13 04:45:49 +0000
@@ -490,3 +490,9 @@
 type RelationUnitsWatchResults struct {
 	Results []RelationUnitsWatchResult
 }
+
+// MachineEnvironment holds the results of the Machine.Environment API call.
+type MachineEnvironmentResult struct {
+	EnvironmentTag string
+	Life           Life
+}

=== added file 'state/apiserver/client/destroyjuju.go'
--- state/apiserver/client/destroyjuju.go	1970-01-01 00:00:00 +0000
+++ state/apiserver/client/destroyjuju.go	2013-11-13 04:45:49 +0000
@@ -0,0 +1,126 @@
+// Copyright 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package client
+
+import (
+	"fmt"
+	"strings"
+
+	"launchpad.net/juju-core/environs"
+	"launchpad.net/juju-core/instance"
+	"launchpad.net/juju-core/state"
+)
+
+// DestroyJuju cleanly removes all Juju agents from the environment.
+func (c *Client) DestroyJuju() error {
+	// First, set the environment to Dying, to lock out new machines,
+	// services and units.
+	env, err := c.api.state.Environment()
+	if err != nil {
+		return err
+	}
+	if err = env.Destroy(); err != nil {
+		return err
+	}
+	// Destroy all services, preventing addition of units.
+	services, err := c.api.state.AllServices()
+	for _, s := range services {
+		if err := s.Destroy(); err != nil {
+			return err
+		}
+	}
+	// Make sure there are no manually provisioned non-manager machines.
+	// Manually provisioned manager machines can self-destruct when
+	// environment goes to Dead.
+	machines, err := c.api.state.AllMachines()
+	if err != nil {
+		return err
+	}
+	if err := checkManualMachines(machines); err != nil {
+		return err
+	}
+	// We must destroy instances server-side to support hosted Juju,
+	// as there's no CLI to fall back on. In that case, we only ever
+	// destroy non-state machines; we leave destroying state servers
+	// in non-hosted environments to the CLI, as otherwise the API
+	// server may get cut off.
+	if err := destroyInstances(c.api.state, machines); err != nil {
+		return err
+	}
+	// Set the environment to Dead, which will cause all agents to
+	// terminate and uninstall themselves. This must be the last thing
+	// we do.
+	return env.EnsureDead()
+}
+
+// destroyInstances directly destroys all non-manager machine instances.
+func destroyInstances(st *state.State, machines []*state.Machine) error {
+	var ids []instance.Id
+	for _, m := range machines {
+		if m.IsStateServer() {
+			continue
+		}
+		id, err := m.InstanceId()
+		if err == nil {
+			ids = append(ids, id)
+		}
+	}
+	if len(ids) == 0 {
+		return nil
+	}
+	envcfg, err := st.EnvironConfig()
+	if err != nil {
+		return err
+	}
+	env, err := environs.New(envcfg)
+	if err != nil {
+		return err
+	}
+	instances, err := env.Instances(ids)
+	switch err {
+	case nil:
+	default:
+		return err
+	case environs.ErrNoInstances:
+		return nil
+	case environs.ErrPartialInstances:
+		var nonNilInstances []instance.Instance
+		for _, inst := range instances {
+			if inst == nil {
+				continue
+			}
+			nonNilInstances = append(nonNilInstances, inst)
+		}
+		instances = nonNilInstances
+	}
+	return env.StopInstances(instances)
+}
+
+// checkManualMachines checks if any of the machines in the slice were
+// manually provisioned, and are non-manager machines. These machines
+// must (currently) by manually destroyed via destroy-machine before
+// destroy-environment can successfully complete.
+func checkManualMachines(machines []*state.Machine) error {
+	var ids []string
+	for _, m := range machines {
+		if isManuallyProvisioned(m) && !m.IsStateServer() {
+			ids = append(ids, m.Id())
+		}
+	}
+	if len(ids) > 0 {
+		return fmt.Errorf("manually provisioned machines must first be destroyed with `juju destroy-machine %s`", strings.Join(ids, " "))
+	}
+	return nil
+}
+
+// isManuallyProvisioned returns true iff the the machine was
+// manually provisioned.
+func isManuallyProvisioned(m *state.Machine) bool {
+	iid, err := m.InstanceId()
+	if err != nil {
+		return false
+	}
+	// Due to an import loop in tests, we cannot import manual.
+	return strings.HasPrefix(string(iid), "manual:")
+}

=== added file 'state/apiserver/client/destroyjuju_test.go'
--- state/apiserver/client/destroyjuju_test.go	1970-01-01 00:00:00 +0000
+++ state/apiserver/client/destroyjuju_test.go	2013-11-13 04:45:49 +0000
@@ -0,0 +1,118 @@
+// Copyright 2012, 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package client_test
+
+import (
+	"fmt"
+
+	gc "launchpad.net/gocheck"
+
+	"launchpad.net/juju-core/environs"
+	coreerrors "launchpad.net/juju-core/errors"
+	"launchpad.net/juju-core/instance"
+	"launchpad.net/juju-core/juju/testing"
+	"launchpad.net/juju-core/state"
+	jc "launchpad.net/juju-core/testing/checkers"
+)
+
+type destroyJujuSuite struct {
+	baseSuite
+}
+
+var _ = gc.Suite(&destroyJujuSuite{})
+
+// setUpManual adds "manually provisioned" machines to state:
+// one manager machine, and one non-manager.
+func (s *destroyJujuSuite) setUpManual(c *gc.C) (m0, m1 *state.Machine) {
+	m0, err := s.State.AddMachine("precise", state.JobManageEnviron, state.JobManageState)
+	c.Assert(err, gc.IsNil)
+	err = m0.SetProvisioned(instance.Id("manual:0"), "fake_nonce", nil)
+	c.Assert(err, gc.IsNil)
+	m1, err = s.State.AddMachine("precise", state.JobHostUnits)
+	c.Assert(err, gc.IsNil)
+	err = m1.SetProvisioned(instance.Id("manual:1"), "fake_nonce", nil)
+	c.Assert(err, gc.IsNil)
+	return m0, m1
+}
+
+// setUpInstances adds machines to state backed by instances:
+// one manager machine, and one non-manager.
+func (s *destroyJujuSuite) setUpInstances(c *gc.C) (m0, m1 *state.Machine) {
+	m0, err := s.State.AddMachine("precise", state.JobManageEnviron, state.JobManageState)
+	c.Assert(err, gc.IsNil)
+	inst, _ := testing.AssertStartInstance(c, s.APIConn.Environ, m0.Id())
+	err = m0.SetProvisioned(inst.Id(), "fake_nonce", nil)
+	c.Assert(err, gc.IsNil)
+	m1, err = s.State.AddMachine("precise", state.JobHostUnits)
+	c.Assert(err, gc.IsNil)
+	inst, _ = testing.AssertStartInstance(c, s.APIConn.Environ, m1.Id())
+	err = m1.SetProvisioned(inst.Id(), "fake_nonce", nil)
+	c.Assert(err, gc.IsNil)
+	return m0, m1
+}
+
+func (s *destroyJujuSuite) TestDestroyJujuManual(c *gc.C) {
+	s.setUpScenario(c)
+	_, nonManager := s.setUpManual(c)
+
+	// If there are any non-manager manual machines in state, DestroyJuju will
+	// error. It *will* still set the Dying flag on the environment, though.
+	err := s.APIState.Client().DestroyJuju()
+	c.Assert(err, gc.ErrorMatches, fmt.Sprintf("manually provisioned machines must first be destroyed with `juju destroy-machine %s`", nonManager.Id()))
+	env, err := s.State.Environment()
+	c.Assert(err, gc.IsNil)
+	c.Assert(env.Life(), gc.Equals, state.Dying)
+
+	// If we remove the non-manager machine, it should pass. Manager machines
+	// will remain, but should tear themselves down (in a real agent) when
+	// they see that the environment is Dead.
+	err = nonManager.EnsureDead()
+	c.Assert(err, gc.IsNil)
+	err = nonManager.Remove()
+	c.Assert(err, gc.IsNil)
+	err = s.APIState.Client().DestroyJuju()
+	c.Assert(err, gc.IsNil)
+	err = env.Refresh()
+	c.Assert(err, gc.IsNil)
+	c.Assert(env.Life(), gc.Equals, state.Dead)
+}
+
+func (s *destroyJujuSuite) TestDestroyJuju(c *gc.C) {
+	s.setUpScenario(c)
+	manager, nonManager := s.setUpInstances(c)
+	managerId, _ := manager.InstanceId()
+	nonManagerId, _ := nonManager.InstanceId()
+
+	instances, err := s.APIConn.Environ.Instances([]instance.Id{managerId, nonManagerId})
+	c.Assert(err, gc.IsNil)
+	for _, inst := range instances {
+		c.Assert(inst, gc.NotNil)
+	}
+
+	services, err := s.State.AllServices()
+	c.Assert(err, gc.IsNil)
+
+	err = s.APIState.Client().DestroyJuju()
+	c.Assert(err, gc.IsNil)
+
+	// After DestroyJuju returns, we should have:
+	//   - all non-manager instances stopped
+	instances, err = s.APIConn.Environ.Instances([]instance.Id{managerId, nonManagerId})
+	c.Assert(err, gc.Equals, environs.ErrPartialInstances)
+	c.Assert(instances[0], gc.NotNil)
+	c.Assert(instances[1], gc.IsNil)
+	//   - all services in state are Dying or Dead (or removed altogether)
+	for _, s := range services {
+		err = s.Refresh()
+		if err != nil {
+			c.Assert(err, jc.Satisfies, coreerrors.IsNotFoundError)
+		} else {
+			c.Assert(s.Life(), gc.Not(gc.Equals), state.Alive)
+		}
+	}
+	//   - environment is Dead
+	env, err := s.State.Environment()
+	c.Assert(err, gc.IsNil)
+	c.Assert(env.Life(), gc.Equals, state.Dead)
+}

=== modified file 'state/apiserver/machine/machiner.go'
--- state/apiserver/machine/machiner.go	2013-08-08 17:38:41 +0000
+++ state/apiserver/machine/machiner.go	2013-11-13 04:45:49 +0000
@@ -6,7 +6,9 @@
 package machine
 
 import (
+	"launchpad.net/juju-core/names"
 	"launchpad.net/juju-core/state"
+	"launchpad.net/juju-core/state/api/params"
 	"launchpad.net/juju-core/state/apiserver/common"
 )
 
@@ -26,15 +28,41 @@
 	if !authorizer.AuthMachineAgent() {
 		return nil, common.ErrPerm
 	}
-	getCanRead := func() (common.AuthFunc, error) {
+	// getAuthEnviron returns an AuthFunc that permits anyone
+	// access to an environment tag.
+	getAuthEnviron := func() (common.AuthFunc, error) {
+		auth := func(tag string) bool {
+			kind, err := names.TagKind(tag)
+			if err != nil {
+				panic(err)
+			}
+			return kind == names.EnvironTagKind
+		}
+		return auth, nil
+	}
+	getAuthOwner := func() (common.AuthFunc, error) {
 		return authorizer.AuthOwner, nil
 	}
+	getCanRead := common.AuthEither(getAuthEnviron, getAuthOwner)
 	return &MachinerAPI{
 		LifeGetter:         common.NewLifeGetter(st, getCanRead),
-		StatusSetter:       common.NewStatusSetter(st, getCanRead),
-		DeadEnsurer:        common.NewDeadEnsurer(st, getCanRead),
+		StatusSetter:       common.NewStatusSetter(st, getAuthOwner),
+		DeadEnsurer:        common.NewDeadEnsurer(st, getAuthOwner),
 		AgentEntityWatcher: common.NewAgentEntityWatcher(st, resources, getCanRead),
 		st:                 st,
 		auth:               authorizer,
 	}, nil
 }
+
+// Environment returns the tag and life of the specified
+// machine's enclosing environment.
+func (m *MachinerAPI) Environment() (params.MachineEnvironmentResult, error) {
+	var result params.MachineEnvironmentResult
+	env, err := m.st.Environment()
+	if err != nil {
+		return result, err
+	}
+	result.EnvironmentTag = env.Tag()
+	result.Life = params.Life(env.Life().String())
+	return result, nil
+}

=== modified file 'state/environ.go'
--- state/environ.go	2013-07-29 15:15:41 +0000
+++ state/environ.go	2013-11-13 04:45:49 +0000
@@ -22,25 +22,21 @@
 	annotator
 }
 
+var _ Living = (*Environment)(nil)
+
 // environmentDoc represents the internal state of the environment in MongoDB.
 type environmentDoc struct {
 	UUID string `bson:"_id"`
 	Name string
+	Life Life `bson:-`
 }
 
 // Environment returns the environment entity.
 func (st *State) Environment() (*Environment, error) {
-	doc := environmentDoc{}
-	err := st.environments.Find(D{{"uuid", D{{"$ne", ""}}}}).One(&doc)
-	if err == mgo.ErrNotFound {
-		return nil, errors.NotFoundf("environment")
-	} else if err != nil {
+	env := &Environment{st: st}
+	if err := env.Refresh(); err != nil {
 		return nil, err
 	}
-	env := &Environment{
-		st:  st,
-		doc: doc,
-	}
 	env.annotator = annotator{
 		globalKey: env.globalKey(),
 		tag:       env.Tag(),
@@ -52,24 +48,70 @@
 // Tag returns a name identifying the environment.
 // The returned name will be different from other Tag values returned
 // by any other entities from the same state.
-func (e Environment) Tag() string {
+func (e *Environment) Tag() string {
 	return names.EnvironTag(e.doc.Name)
 }
 
 // UUID returns the universally unique identifier of the environment.
-func (e Environment) UUID() string {
+func (e *Environment) UUID() string {
 	return e.doc.UUID
 }
 
+// Life returns whether the environment is Alive, Dying or Dead.
+func (e *Environment) Life() Life {
+	return e.doc.Life
+}
+
 // globalKey returns the global database key for the environment.
 func (e *Environment) globalKey() string {
 	return environGlobalKey
 }
 
+func (e *Environment) Refresh() error {
+	err := e.st.environments.Find(D{{"uuid", D{{"$ne", ""}}}}).One(&e.doc)
+	if err == mgo.ErrNotFound {
+		return errors.NotFoundf("environment")
+	}
+	return err
+}
+
+func (e *Environment) Destroy() error {
+	return e.advanceLifecycle(Dying)
+}
+
+func (e *Environment) EnsureDead() error {
+	return e.advanceLifecycle(Dead)
+}
+
+// advanceLifecycle ensures that the environment's lifecycle is no earlier
+// than the supplied value.
+func (e *Environment) advanceLifecycle(life Life) error {
+	var assert D
+	if life != Dead {
+		assert = notDeadDoc
+	}
+	op := txn.Op{
+		C:      e.st.environments.Name,
+		Id:     e.doc.UUID,
+		Update: D{{"$set", D{{"life", life}}}},
+		Assert: assert,
+	}
+	err := e.st.runTransaction([]txn.Op{op})
+	switch err {
+	case nil:
+		e.doc.Life = life
+	case txn.ErrAborted:
+		// If the transaction aborted, the environment is already Dead.
+		e.doc.Life = Dead
+		err = nil
+	}
+	return err
+}
+
 // createEnvironmentOp returns the operation needed to create
 // an environment document with the given name and UUID.
 func createEnvironmentOp(st *State, name, uuid string) txn.Op {
-	doc := &environmentDoc{uuid, name}
+	doc := &environmentDoc{uuid, name, Alive}
 	return txn.Op{
 		C:      st.environments.Name,
 		Id:     uuid,
@@ -77,3 +119,13 @@
 		Insert: doc,
 	}
 }
+
+// assertAliveOp returns a read-only txn.Op that asserts
+// the environment is alive.
+func (e *Environment) assertAliveOp() txn.Op {
+	return txn.Op{
+		C:      e.st.environments.Name,
+		Id:     e.UUID(),
+		Assert: isAliveDoc,
+	}
+}

=== modified file 'state/interface.go'
--- state/interface.go	2013-10-02 13:14:49 +0000
+++ state/interface.go	2013-11-13 04:45:49 +0000
@@ -128,6 +128,7 @@
 	_ NotifyWatcherFactory = (*Machine)(nil)
 	_ NotifyWatcherFactory = (*Unit)(nil)
 	_ NotifyWatcherFactory = (*Service)(nil)
+	_ NotifyWatcherFactory = (*Environment)(nil)
 )
 
 // AgentEntity represents an entity that can

=== modified file 'state/state.go'
--- state/state.go	2013-11-11 22:19:10 +0000
+++ state/state.go	2013-11-13 04:45:49 +0000
@@ -355,12 +355,17 @@
 	sdoc := statusDoc{
 		Status: params.StatusPending,
 	}
+	env, err := st.Environment()
+	if err != nil {
+		return nil, nil, err
+	}
 	// Machine constraints do not use a container constraint value.
 	// Both provisioning and deployment constraints use the same constraints.Value struct
 	// so here we clear the container value. Provisioning ignores the container value but
 	// clearing it avoids potential confusion.
 	cons.Container = nil
 	ops := []txn.Op{
+		env.assertAliveOp(),
 		{
 			C:      st.machines.Name,
 			Id:     mdoc.Id,
@@ -477,10 +482,16 @@
 	}
 	ops = append(ops, machineOps...)
 
-	err = st.runTransaction(ops)
-	if err != nil {
+	if err = st.runTransaction(ops); err == txn.ErrAborted {
+		if env, err := st.Environment(); err != nil {
+			return nil, err
+		} else if env.Life() != Alive {
+			return nil, fmt.Errorf("environment is being destroyed")
+		}
+	} else if err != nil {
 		return nil, err
 	}
+
 	// Refresh to pick the txn-revno.
 	m = newMachine(st, mdoc)
 	if err = m.Refresh(); err != nil {
@@ -716,6 +727,10 @@
 	} else if exists {
 		return nil, fmt.Errorf("service already exists")
 	}
+	env, err := st.Environment()
+	if err != nil {
+		return nil, err
+	}
 	// Create the service addition operations.
 	peers := ch.Meta().Peers
 	svcDoc := &serviceDoc{
@@ -728,6 +743,7 @@
 	}
 	svc := newService(st, svcDoc)
 	ops := []txn.Op{
+		env.assertAliveOp(),
 		createConstraintsOp(st, svc.globalKey(), constraints.Value{}),
 		createSettingsOp(st, svc.settingsKey(), nil),
 		{
@@ -752,6 +768,11 @@
 	// because all the possible failed assertions imply that the service
 	// already exists.
 	if err := st.runTransaction(ops); err == txn.ErrAborted {
+		if err := env.Refresh(); err != nil {
+			return nil, err
+		} else if env.Life() != Alive {
+			return nil, fmt.Errorf("environment is being destroyed")
+		}
 		return nil, fmt.Errorf("service already exists")
 	} else if err != nil {
 		return nil, err

=== modified file 'state/state_test.go'
--- state/state_test.go	2013-11-08 16:15:10 +0000
+++ state/state_test.go	2013-11-13 04:45:49 +0000
@@ -206,6 +206,23 @@
 	check(m[1], "1", "blahblah", allJobs)
 }
 
+func (s *StateSuite) TestAddMachinesEnvironmentLife(c *gc.C) {
+	_, err := s.State.AddMachine("quantal", state.JobHostUnits)
+	c.Assert(err, gc.IsNil)
+	// Check that machines cannot be added if the environment is Dying.
+	env, err := s.State.Environment()
+	c.Assert(err, gc.IsNil)
+	err = env.Destroy()
+	c.Assert(err, gc.IsNil)
+	_, err = s.State.AddMachine("quantal", state.JobHostUnits)
+	c.Assert(err, gc.ErrorMatches, ".*environment is being destroyed")
+	// Same again for Dead.
+	err = env.EnsureDead()
+	c.Assert(err, gc.IsNil)
+	_, err = s.State.AddMachine("quantal", state.JobHostUnits)
+	c.Assert(err, gc.ErrorMatches, ".*environment is being destroyed")
+}
+
 func (s *StateSuite) TestAddMachineExtraConstraints(c *gc.C) {
 	err := s.State.SetEnvironConstraints(constraints.MustParse("mem=4G"))
 	c.Assert(err, gc.IsNil)
@@ -529,6 +546,25 @@
 	c.Assert(ch.URL(), gc.DeepEquals, charm.URL())
 }
 
+func (s *StateSuite) TestAddServiceEnvironmentLife(c *gc.C) {
+	charm := s.AddTestingCharm(c, "dummy")
+	_, err := s.State.AddService("s0", charm)
+	c.Assert(err, gc.IsNil)
+
+	// Check that machines cannot be added if the environment is Dying.
+	env, err := s.State.Environment()
+	c.Assert(err, gc.IsNil)
+	err = env.Destroy()
+	c.Assert(err, gc.IsNil)
+	_, err = s.State.AddService("s1", charm)
+	c.Assert(err, gc.ErrorMatches, ".*environment is being destroyed")
+	// Same again for Dead.
+	err = env.EnsureDead()
+	c.Assert(err, gc.IsNil)
+	_, err = s.State.AddService("s2", charm)
+	c.Assert(err, gc.ErrorMatches, ".*environment is being destroyed")
+}
+
 func (s *StateSuite) TestServiceNotFound(c *gc.C) {
 	_, err := s.State.Service("bummer")
 	c.Assert(err, gc.ErrorMatches, `service "bummer" not found`)

=== modified file 'state/watcher.go'
--- state/watcher.go	2013-09-11 13:48:43 +0000
+++ state/watcher.go	2013-11-13 04:45:49 +0000
@@ -1059,6 +1059,11 @@
 	return newEntityWatcher(u.st, u.st.units, u.doc.Name)
 }
 
+// Watch returns a watcher for observing changes to an environment.
+func (e *Environment) Watch() NotifyWatcher {
+	return newEntityWatcher(e.st, e.st.environments, e.doc.UUID)
+}
+
 // WatchForEnvironConfigChanges return a NotifyWatcher waiting for the Environ
 // Config to change. This differs from WatchEnvironConfig in that the watcher
 // is a NotifyWatcher that does not give content during Changes()

=== modified file 'worker/machiner/machiner.go'
--- worker/machiner/machiner.go	2013-09-17 18:18:44 +0000
+++ worker/machiner/machiner.go	2013-11-13 04:45:49 +0000
@@ -23,12 +23,26 @@
 	machine *machiner.Machine
 }
 
+// machineEnvironer is responsible for watching a machine agent's
+// environment lifecycle, and terminating the agent if it becomes
+// Dead.
+type machineEnvironer struct {
+	st         *machiner.State
+	machineTag string
+	environ    *machiner.Environment
+}
+
 // NewMachiner returns a Worker that will wait for the identified machine
 // to become Dying and make it Dead; or until the machine becomes Dead by
 // other means.
 func NewMachiner(st *machiner.State, agentConfig agent.Config) worker.Worker {
-	mr := &Machiner{st: st, tag: agentConfig.Tag()}
-	return worker.NewNotifyWorker(mr)
+	machiner := &Machiner{st: st, tag: agentConfig.Tag()}
+	return worker.NewNotifyWorker(machiner)
+}
+
+func NewMachineEnvironer(st *machiner.State, agentConfig agent.Config) worker.Worker {
+	machineEnvironer := &machineEnvironer{st: st, machineTag: agentConfig.Tag()}
+	return worker.NewNotifyWorker(machineEnvironer)
 }
 
 func isNotFoundOrUnauthorized(err error) bool {
@@ -80,3 +94,33 @@
 	// Nothing to do here.
 	return nil
 }
+
+func (me *machineEnvironer) SetUp() (watcher.NotifyWatcher, error) {
+	env, err := me.st.Environment()
+	if isNotFoundOrUnauthorized(err) {
+		return nil, worker.ErrTerminateAgent
+	} else if err != nil {
+		return nil, err
+	}
+	me.environ = env
+	return env.Watch()
+}
+
+func (me *machineEnvironer) Handle() error {
+	if err := me.environ.Refresh(); isNotFoundOrUnauthorized(err) {
+		return worker.ErrTerminateAgent
+	} else if err != nil {
+		return err
+	}
+	if me.environ.Life() == params.Dead {
+		logger.Infof("%q is dead, terminating agent", me.environ.Tag())
+		return worker.ErrTerminateAgent
+	}
+	logger.Debugf("%q is now %s", me.environ.Tag(), me.environ.Life())
+	return nil
+}
+
+func (me *machineEnvironer) TearDown() error {
+	// Nothing to do here.
+	return nil
+}