Merge lp:~jameinel/juju-core/gomaxprocs-apiserver-1290841 into lp:~go-bot/juju-core/trunk

[John A Meinel]

cmd/juju/machine: Enable GOMAXPROCS(NumCPU)

As part of my scale testing work, I noticed the API server is CPU bound
in many circumstances (logging in lots of machines at the same time).
This changes it so machine agents that manage the environment (i.e. the
API Server) changes the default for GOMAXPROCS to the number of CPUs.

cmd/juju/machine: Enable GOMAXPROCS(NumCPU)

As part of my scale testing work, I noticed the API server is CPU bound
in many circumstances (logging in lots of machines at the same time).
However, we know that our test suite doesn't play nicely if we have
GOMAXPROCS>1. (Something about it running multiple tests concurrently
and tests that mutate global state.)

To avoid this affecting lots of code, I did it as a two phase approach.
We only will set GOMAXPROCS if

  1) GOMAXPROCS isn't already set in the environment. Since that would
     mean the go runtime would have already done its thing.
  2) We are running in response to the jujuDMain() (so it won't trigger
     in normal test suite operations.)
  3) We are running JobManageEnviron, so this will only affect the API
     server, and not all the other machines. (Even though it should be
     fine, if we were CPU bound there, we probably don't want to
     saturate machines that are running real workloads.)

I added tests that things get called properly when they are (and
aren't) enabled, but I didn't add a test that jujuDMain enables it
properly. I couldn't see an obvious way to hook into that code, so I
filed bug #1293383. Though if someone has a good way to test it, I'll
just do so.

This also has a change to how the MachineAgent works, in that it now
has a channel to let us know once it has called all of the StartWorker
steps and is going into the Wait() step. It made it possible to test
that we *didn't* call UseMultipleCPUs() for agents that aren't the API
server.

https://codereview.appspot.com/76120044/

[John A Meinel (jameinel)]

Reviewers: mp+211261_code.launchpad.net,

Message:
Please take a look.

Description:
cmd/juju/machine: Enable GOMAXPROCS(NumCPU)

As part of my scale testing work, I noticed the API server is CPU bound
in many circumstances (logging in lots of machines at the same time).
However, we know that our test suite doesn't play nicely if we have
GOMAXPROCS>1. (Something about it running multiple tests concurrently
and tests that mutate global state.)

To avoid this affecting lots of code, I did it as a two phase approach.
We only will set GOMAXPROCS if

   1) GOMAXPROCS isn't already set in the environment. Since that would
      mean the go runtime would have already done its thing.
   2) We are running in response to the jujuDMain() (so it won't trigger
      in normal test suite operations.)
   3) We are running JobManageEnviron, so this will only affect the API
      server, and not all the other machines. (Even though it should be
      fine, if we were CPU bound there, we probably don't want to
      saturate machines that are running real workloads.)

I added tests that things get called properly when they are (and
aren't) enabled, but I didn't add a test that jujuDMain enables it
properly. I couldn't see an obvious way to hook into that code, so I
filed bug #1293383. Though if someone has a good way to test it, I'll
just do so.

This also has a change to how the MachineAgent works, in that it now
has a channel to let us know once it has called all of the StartWorker
steps and is going into the Wait() step. It made it possible to test
that we *didn't* call UseMultipleCPUs() for agents that aren't the API
server.

https://code.launchpad.net/~jameinel/juju-core/gomaxprocs-apiserver-1290841/+merge/211261

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/76120044/

Affected files (+223, -1 lines):
   A [revision details]
   M cmd/jujud/machine.go
   M cmd/jujud/machine_test.go
   M cmd/jujud/main.go
   A utils/export_test.go
   A utils/gomaxprocs.go
   A utils/gomaxprocs_test.go

[Roger Peppe (rogpeppe)]

This does not LGTM - I think it could be considerably simpler - a
suggestion below.

https://codereview.appspot.com/76120044/diff/1/utils/gomaxprocs.go
File utils/gomaxprocs.go (right):

https://codereview.appspot.com/76120044/diff/1/utils/gomaxprocs.go#newcode12
utils/gomaxprocs.go:12: var mu = sync.Mutex{}
This does not fill me with joy.
If our tests really do not play nicely with GOMAXPROCS>1,
(they *really* should, but I understand that they
might not) there's a much easier way to override it - just set
the environment variable.

However, for the jujud tests specifically, the tests
pass fine for me when jujud sets GOMAXPROCS.

I suggest that we add just implement the following
function in cmd/jujud
(I don't think it's generally useful enough to deserve a place in utils,
but it can go here if you think it's worth it):

func UseMultipleCPUs() {
    if os.Getenv("GOMAXPROCS") == "" {
        runtime.GOMAXPROCS(runtime.NumCPU())
    }
}

and call it from the same place it's called currently.
To test it, you could declare it as a variable and
replace it to ensure it's actually called.

If we really find that it makes the tests flaky,
the relevant tests can patch $GOMAXPROCS.

https://codereview.appspot.com/76120044/

[Dave Cheney (dave-cheney)]

On Mon, Mar 17, 2014 at 9:03 PM, Roger Peppe <email address hidden>wrote:

> This does not LGTM - I think it could be considerably simpler - a
> suggestion below.
>

I second rogers' objections, mainly on emotional grounds.

>
>
> https://codereview.appspot.com/76120044/diff/1/utils/gomaxprocs.go
> File utils/gomaxprocs.go (right):
>
>
> https://codereview.appspot.com/76120044/diff/1/utils/gomaxprocs.go#newcode12
> utils/gomaxprocs.go:12: var mu = sync.Mutex{}
> This does not fill me with joy.
> If our tests really do not play nicely with GOMAXPROCS>1,
> (they *really* should, but I understand that they
> might not) there's a much easier way to override it - just set
> the environment variable.
>
> However, for the jujud tests specifically, the tests
> pass fine for me when jujud sets GOMAXPROCS.
>
> I suggest that we add just implement the following
> function in cmd/jujud
> (I don't think it's generally useful enough to deserve a place in utils,
> but it can go here if you think it's worth it):
>
> func UseMultipleCPUs() {
> if os.Getenv("GOMAXPROCS") == "" {
> runtime.GOMAXPROCS(runtime.NumCPU())
> }
> }
>
>
This shoudn't be needed, if GOMAXPROCS is set in the environment then the
runtime already has initialised the value of runtime.GOMAXPROCS to that
value.

What this appears to be doing is _defaulting_ to runtime.NumCPU(), _unless_
runtime.GOMAXPROCS is set to a specific value, one can only assume the
value "1".

Really, this is not the Go way. I'm willing to accept arguments that jujud
needs this to effectively handle a large number of PBKDF2 handshakes, but
if so this should be done in cloud init or somethign else that can set the
GOMAXPROCS environment variable inside the upstart script ... we do already
have that information in the ec2 tables.`

>
> and call it from the same place it's called currently.
> To test it, you could declare it as a variable and
> replace it to ensure it's actually called.
>
> If we really find that it makes the tests flaky,
> the relevant tests can patch $GOMAXPROCS.
>
> https://codereview.appspot.com/76120044/
>
> --
>
> https://code.launchpad.net/~jameinel/juju-core/gomaxprocs-apiserver-1290841/+merge/211261
> You are subscribed to branch lp:juju-core.
>

[John A Meinel (jameinel)]

On 2014/03/17 10:02:15, rog wrote:
> This does not LGTM - I think it could be considerably simpler - a
suggestion
> below.

> https://codereview.appspot.com/76120044/diff/1/utils/gomaxprocs.go
> File utils/gomaxprocs.go (right):

https://codereview.appspot.com/76120044/diff/1/utils/gomaxprocs.go#newcode12
> utils/gomaxprocs.go:12: var mu = sync.Mutex{}
> This does not fill me with joy.
> If our tests really do not play nicely with GOMAXPROCS>1,
> (they *really* should, but I understand that they
> might not) there's a much easier way to override it - just set
> the environment variable.

> However, for the jujud tests specifically, the tests
> pass fine for me when jujud sets GOMAXPROCS.

> I suggest that we add just implement the following
> function in cmd/jujud
> (I don't think it's generally useful enough to deserve a place in
utils,
> but it can go here if you think it's worth it):

> func UseMultipleCPUs() {
> if os.Getenv("GOMAXPROCS") == "" {
> runtime.GOMAXPROCS(runtime.NumCPU())
> }
> }

> and call it from the same place it's called currently.
> To test it, you could declare it as a variable and
> replace it to ensure it's actually called.

> If we really find that it makes the tests flaky,
> the relevant tests can patch $GOMAXPROCS.

The new code is simplified to just always call GOMAXPROCS without the
Enable step.

https://codereview.appspot.com/76120044/

[John A Meinel (jameinel)]

Please take a look.

https://codereview.appspot.com/76120044/diff/1/utils/gomaxprocs.go
File utils/gomaxprocs.go (right):

https://codereview.appspot.com/76120044/diff/1/utils/gomaxprocs.go#newcode12
utils/gomaxprocs.go:12: var mu = sync.Mutex{}
On 2014/03/17 10:02:16, rog wrote:
> This does not fill me with joy.
> If our tests really do not play nicely with GOMAXPROCS>1,
> (they *really* should, but I understand that they
> might not) there's a much easier way to override it - just set
> the environment variable.

> However, for the jujud tests specifically, the tests
> pass fine for me when jujud sets GOMAXPROCS.

> I suggest that we add just implement the following
> function in cmd/jujud
> (I don't think it's generally useful enough to deserve a place in
utils,
> but it can go here if you think it's worth it):

> func UseMultipleCPUs() {
> if os.Getenv("GOMAXPROCS") == "" {
> runtime.GOMAXPROCS(runtime.NumCPU())
> }
> }

> and call it from the same place it's called currently.
> To test it, you could declare it as a variable and
> replace it to ensure it's actually called.

> If we really find that it makes the tests flaky,
> the relevant tests can patch $GOMAXPROCS.

Done.

https://codereview.appspot.com/76120044/

[John A Meinel (jameinel)]

Please take a look.

https://codereview.appspot.com/76120044/

[Roger Peppe (rogpeppe)]

I like that much better, thanks.
Some further simplification suggestions below.

https://codereview.appspot.com/76120044/diff/20001/cmd/jujud/main.go
File cmd/jujud/main.go (right):

https://codereview.appspot.com/76120044/diff/20001/cmd/jujud/main.go#newcode1
cmd/jujud/main.go:1: // Copyright 2012-2014 Canonical Ltd.
I think it's reasonable to leave the copyright dates alone. That's
certainly the explicit convention in Go core, and Juju de facto.

https://codereview.appspot.com/76120044/diff/40001/cmd/jujud/machine.go
File cmd/jujud/machine.go (right):

https://codereview.appspot.com/76120044/diff/40001/cmd/jujud/machine.go#newcode75
cmd/jujud/machine.go:75: workersStarted chan struct{}
Perhaps some documentation for this might be appropriate, given
that it's only here for tests.

Or a method:

// WorkersStarted returns a channel that's closed when
// the top level workers have been started. This is
// provided for testing purposes.
func (a *MachineAgent) WorkersStarted() <-chan struct{}
     return a.workersStarted
}

https://codereview.appspot.com/76120044/diff/40001/cmd/jujud/machine.go#newcode345
cmd/jujud/machine.go:345: useMultipleCPUs()
If you took logging out of the utils package, you could add a log
message here instead:

    logger.Debugf("GOMAXPROCS set to %d", runtime.GOMAXPROCS(0))

That's probably sufficient for diagnosing gomaxprocs-related problems.

https://codereview.appspot.com/76120044/diff/40001/utils/export_test.go
File utils/export_test.go (right):

https://codereview.appspot.com/76120044/diff/40001/utils/export_test.go#newcode9
utils/export_test.go:9: func OverrideCPUFuncs(maxprocsfunc func(int)
int, numCPUFunc func() int) func() {
Assuming we go with this, I'd do:

var (
     GoMaxProcs = &gomaxprocs
     NumCPU = &numcpu
)

then we can use PatchValue in the tests.

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go
File utils/gomaxprocs.go (right):

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go#newcode11
utils/gomaxprocs.go:11: var gomaxprocs = runtime.GOMAXPROCS
This seems a bit of overkill for such a function with such simple logic.
I'd just test that it does the right thing (even though it's not a full
test on a machine with a single cpu). It'll fail on any machine
with more than one CPU, which seems adequate to me. Is that overly
pragmatic?

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go#newcode16
utils/gomaxprocs.go:16: // variable if it is set.
// UseMultipleCPUs sets GOMAXPROCS to the number of CPU
// cores unless overridden by the GOMAXPROCS environment
// variable.

?

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go#newcode20
utils/gomaxprocs.go:20: logger.Debugf("GOMAXPROCS already set in
environment to %q, %d internally",
I'm not that keen on utils functions producing log messages (and since
they're only at Debug level, they can't really be that important).
Honestly,
I'd just do:

func UseMultipleCPUs() {
      if os.Getenv("GOMAXPROCS") == "" {
          runtime.GOMAXPROCS(runtime.NumCPU())
      }
}

and be done with it. Alternatively, if it lived in cmd/jujud then
logging might be more appropriate, as it can ...

[John A Meinel (jameinel)]

Please take a look.

https://codereview.appspot.com/76120044/diff/40001/cmd/jujud/machine.go
File cmd/jujud/machine.go (right):

https://codereview.appspot.com/76120044/diff/40001/cmd/jujud/machine.go#newcode75
cmd/jujud/machine.go:75: workersStarted chan struct{}
On 2014/03/19 10:50:28, rog wrote:
> Perhaps some documentation for this might be appropriate, given
> that it's only here for tests.

> Or a method:

> // WorkersStarted returns a channel that's closed when
> // the top level workers have been started. This is
> // provided for testing purposes.
> func (a *MachineAgent) WorkersStarted() <-chan struct{}
> return a.workersStarted
> }

Done.

https://codereview.appspot.com/76120044/diff/40001/cmd/jujud/machine.go#newcode345
cmd/jujud/machine.go:345: useMultipleCPUs()
On 2014/03/19 10:50:28, rog wrote:
> If you took logging out of the utils package, you could add a log
> message here instead:

> logger.Debugf("GOMAXPROCS set to %d", runtime.GOMAXPROCS(0))

> That's probably sufficient for diagnosing gomaxprocs-related problems.

That explicitly misses out on distinguishing between GOMAXPROCS was set
in the environment vs GOMAXPROCS was set from NumCPU(). Which (IMO) is
actually a useful thing to distinguish.

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go
File utils/gomaxprocs.go (right):

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go#newcode11
utils/gomaxprocs.go:11: var gomaxprocs = runtime.GOMAXPROCS
On 2014/03/19 10:50:28, rog wrote:
> This seems a bit of overkill for such a function with such simple
logic.
> I'd just test that it does the right thing (even though it's not a
full
> test on a machine with a single cpu). It'll fail on any machine
> with more than one CPU, which seems adequate to me. Is that overly
> pragmatic?

Unless someone (we?) start setting GOMAXPROCS in the environment for
regular running of tests, etc.

I find it particularly useful to be able to write tests that are
abstracted from the runtime.

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go#newcode16
utils/gomaxprocs.go:16: // variable if it is set.
On 2014/03/19 10:50:28, rog wrote:
> // UseMultipleCPUs sets GOMAXPROCS to the number of CPU
> // cores unless overridden by the GOMAXPROCS environment
> // variable.

> ?

Done.

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go#newcode20
utils/gomaxprocs.go:20: logger.Debugf("GOMAXPROCS already set in
environment to %q, %d internally",
On 2014/03/19 10:50:28, rog wrote:
> I'm not that keen on utils functions producing log messages (and since
they're
> only at Debug level, they can't really be that important). Honestly,
> I'd just do:

> func UseMultipleCPUs() {
> if os.Getenv("GOMAXPROCS") == "" {
> runtime.GOMAXPROCS(runtime.NumCPU())
> }
> }

> and be done with it. Alternatively, if it lived in cmd/jujud then
logging might
> be more appropriate, as it can be more location-specific there.

We already have logging in utils (I didn't have to create 'logger'), so
I'm pretty sure its fine as a pattern.

I do think it is useful to be logged, and I think any code that does
start changing G...

[Roger Peppe (rogpeppe)]

LGTM, your call.

https://codereview.appspot.com/76120044/diff/40001/cmd/jujud/machine.go
File cmd/jujud/machine.go (right):

https://codereview.appspot.com/76120044/diff/40001/cmd/jujud/machine.go#newcode345
cmd/jujud/machine.go:345: useMultipleCPUs()
On 2014/03/20 06:48:46, jameinel wrote:
> On 2014/03/19 10:50:28, rog wrote:
> > If you took logging out of the utils package, you could add a log
> > message here instead:
> >
> > logger.Debugf("GOMAXPROCS set to %d", runtime.GOMAXPROCS(0))
> >
> > That's probably sufficient for diagnosing gomaxprocs-related
problems.

> That explicitly misses out on distinguishing between GOMAXPROCS was
set in the
> environment vs GOMAXPROCS was set from NumCPU(). Which (IMO) is
actually a
> useful thing to distinguish.

I'm struggling to think of a scenario where we'd really care, but fair
enough.

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go
File utils/gomaxprocs.go (right):

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go#newcode11
utils/gomaxprocs.go:11: var gomaxprocs = runtime.GOMAXPROCS
> Unless someone (we?) start setting GOMAXPROCS in the environment for
> regular running of tests, etc.

It's trivial to patch the GOMAXPROCS environment variable in the test.

> I find it particularly useful to be able to write tests that are
> abstracted from the runtime.

Fair enough. I like seeing code that's as straightforwardly obvious as
possible, but I appreciate that it's a trade-off.

https://codereview.appspot.com/76120044/diff/40001/utils/gomaxprocs.go#newcode20
utils/gomaxprocs.go:20: logger.Debugf("GOMAXPROCS already set in
environment to %q, %d internally",
On 2014/03/20 06:48:46, jameinel wrote:
> On 2014/03/19 10:50:28, rog wrote:
> > I'm not that keen on utils functions producing log messages (and
since they're
> > only at Debug level, they can't really be that important). Honestly,
> > I'd just do:
> >
> > func UseMultipleCPUs() {
> > if os.Getenv("GOMAXPROCS") == "" {
> > runtime.GOMAXPROCS(runtime.NumCPU())
> > }
> > }
> >
> > and be done with it. Alternatively, if it lived in cmd/jujud then
logging
> might
> > be more appropriate, as it can be more location-specific there.

> We already have logging in utils (I didn't have to create 'logger'),
so I'm
> pretty sure its fine as a pattern.

> I do think it is useful to be logged, and I think any code that does
start
> changing GOMAXPROCS at runtime should be logging that fact. So I think
it is
> perfectly appropriate to log here.

There is only one other place in utils that logs messages, and that
place (GetAddressForInterface) is a mistake (it should return an
annotated
error rather than logging).

> Given we stopped doing the Enable vs set it, and set it always, we
could have
> just embedded it, but if we do grow more places that want to take
advantage of
> >1 CPU, I'd rather have that logic centralized.

FWIW, no non-main package should be changing GOMAXPROCS at all,
which is why IMHO it's a mistake to put it in utils at all. It might fit
reasonably inside cmd, I guess.

https://codereview.appspot.com/76120044/diff/60001/utils/gomaxprocs.go
File utils/gomaxprocs.go (right):

ht...

=== modified file 'cmd/jujud/machine.go'
--- cmd/jujud/machine.go	2014-03-18 09:21:53 +0000
+++ cmd/jujud/machine.go	2014-03-20 09:04:40 +0000
@@ -28,6 +28,7 @@
 	"launchpad.net/juju-core/state/apiserver"
 	"launchpad.net/juju-core/upgrades"
 	"launchpad.net/juju-core/upstart"
+	"launchpad.net/juju-core/utils"
 	"launchpad.net/juju-core/version"
 	"launchpad.net/juju-core/worker"
 	"launchpad.net/juju-core/worker/authenticationworker"
@@ -60,6 +61,8 @@
 
 var jujuRun = "/usr/local/bin/juju-run"
 
+var useMultipleCPUs = utils.UseMultipleCPUs
+
 // MachineAgent is a cmd.Command responsible for running a machine agent.
 type MachineAgent struct {
 	cmd.CommandBase
@@ -69,6 +72,7 @@
 	runner          worker.Runner
 	upgradeComplete chan struct{}
 	stateOpened     chan struct{}
+	workersStarted  chan struct{}
 	st              *state.State
 }
 
@@ -96,6 +100,7 @@
 	a.runner = newRunner(isFatal, moreImportant)
 	a.upgradeComplete = make(chan struct{})
 	a.stateOpened = make(chan struct{})
+	a.workersStarted = make(chan struct{})
 	return nil
 }
 
@@ -155,6 +160,8 @@
 	a.runner.StartWorker("termination", func() (worker.Worker, error) {
 		return terminationworker.NewWorker(), nil
 	})
+	// At this point, all workers will have been configured to start
+	close(a.workersStarted)
 	err := a.runner.Wait()
 	if err == worker.ErrTerminateAgent {
 		err = a.uninstallAgent()
@@ -335,6 +342,7 @@
 		case state.JobHostUnits:
 			// Implemented in APIWorker.
 		case state.JobManageEnviron:
+			useMultipleCPUs()
 			a.startWorkerAfterUpgrade(runner, "instancepoller", func() (worker.Worker, error) {
 				return instancepoller.NewWorker(st), nil
 			})
@@ -468,6 +476,13 @@
 	return a.Conf.config.WriteUpgradedToVersion(version.Current.Number)
 }
 
+// WorkersStarted returns a channel that's closed once all top level workers
+// have been started. This is provided for testing purposes.
+func (a *MachineAgent) WorkersStarted() <-chan struct{} {
+	return a.workersStarted
+
+}
+
 func (a *MachineAgent) Entity(st *state.State) (AgentState, error) {
 	m, err := st.Machine(a.MachineId)
 	if err != nil {

=== modified file 'cmd/jujud/machine_test.go'
--- cmd/jujud/machine_test.go	2014-03-19 21:23:05 +0000
+++ cmd/jujud/machine_test.go	2014-03-20 09:04:40 +0000
@@ -397,6 +397,48 @@
 
 }
 
+func (s *MachineSuite) TestManageEnvironCallsUseMultipleCPUs(c *gc.C) {
+	// If it has been enabled, the JobManageEnviron agent should call utils.UseMultipleCPUs
+	usefulVersion := version.Current
+	usefulVersion.Series = "quantal"
+	envtesting.AssertUploadFakeToolsVersions(c, s.Conn.Environ.Storage(), usefulVersion)
+	m, _, _ := s.primeAgent(c, version.Current, state.JobManageEnviron)
+	s.setFakeMachineAddresses(c, m)
+	calledChan := make(chan struct{}, 1)
+	s.PatchValue(&useMultipleCPUs, func() { calledChan <- struct{}{} })
+	// Now, start the agent, and observe that a JobManageEnviron agent
+	// calls UseMultipleCPUs
+	a := s.newAgent(c, m)
+	defer a.Stop()
+	go func() {
+		c.Check(a.Run(nil), gc.IsNil)
+	}()
+	// Wait for configuration to be finished
+	<-a.WorkersStarted()
+	select {
+	case <-calledChan:
+	case <-time.After(coretesting.LongWait):
+		c.Errorf("we failed to call UseMultipleCPUs()")
+	}
+	c.Check(a.Stop(), gc.IsNil)
+	// However, an agent that just JobHostUnits doesn't call UseMultipleCPUs
+	m2, _, _ := s.primeAgent(c, version.Current, state.JobHostUnits)
+	s.setFakeMachineAddresses(c, m2)
+	a2 := s.newAgent(c, m2)
+	defer a2.Stop()
+	go func() {
+		c.Check(a2.Run(nil), gc.IsNil)
+	}()
+	// Wait until all the workers have been started, and then kill everything
+	<-a2.workersStarted
+	c.Check(a2.Stop(), gc.IsNil)
+	select {
+	case <-calledChan:
+		c.Errorf("we should not have called UseMultipleCPUs()")
+	case <-time.After(coretesting.ShortWait):
+	}
+}
+
 func (s *MachineSuite) waitProvisioned(c *gc.C, unit *state.Unit) (*state.Machine, instance.Id) {
 	c.Logf("waiting for unit %q to be provisioned", unit)
 	machineId, err := unit.AssignedMachineId()

=== modified file 'cmd/jujud/main.go'
--- cmd/jujud/main.go	2014-03-05 19:41:34 +0000
+++ cmd/jujud/main.go	2014-03-20 09:04:40 +0000
@@ -1,4 +1,4 @@
-// Copyright 2012, 2013 Canonical Ltd.
+// Copyright 2012-2014 Canonical Ltd.
 // Licensed under the AGPLv3, see LICENCE file for details.
 
 package main

=== added file 'utils/export_test.go'
--- utils/export_test.go	1970-01-01 00:00:00 +0000
+++ utils/export_test.go	2014-03-20 09:04:40 +0000
@@ -0,0 +1,9 @@
+// Copyright 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package utils
+
+var (
+	GOMAXPROCS = &gomaxprocs
+	NumCPU     = &numCPU
+)

=== added file 'utils/gomaxprocs.go'
--- utils/gomaxprocs.go	1970-01-01 00:00:00 +0000
+++ utils/gomaxprocs.go	2014-03-20 09:04:40 +0000
@@ -0,0 +1,26 @@
+// Copyright 2014 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package utils
+
+import (
+	"os"
+	"runtime"
+)
+
+var gomaxprocs = runtime.GOMAXPROCS
+var numCPU = runtime.NumCPU
+
+// UseMultipleCPUs sets GOMAXPROCS to the number of CPU cores unless it has
+// already been overridden by the GOMAXPROCS environment variable.
+func UseMultipleCPUs() {
+	if envGOMAXPROCS := os.Getenv("GOMAXPROCS"); envGOMAXPROCS != "" {
+		n := gomaxprocs(0)
+		logger.Debugf("GOMAXPROCS already set in environment to %q, %d internally",
+			envGOMAXPROCS, n)
+		return
+	}
+	n := numCPU()
+	logger.Debugf("setting GOMAXPROCS to %d", n)
+	gomaxprocs(n)
+}

=== added file 'utils/gomaxprocs_test.go'
--- utils/gomaxprocs_test.go	1970-01-01 00:00:00 +0000
+++ utils/gomaxprocs_test.go	2014-03-20 09:04:40 +0000
@@ -0,0 +1,51 @@
+// Copyright 2014 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package utils_test
+
+import (
+	"os"
+
+	gc "launchpad.net/gocheck"
+
+	"launchpad.net/juju-core/testing/testbase"
+	"launchpad.net/juju-core/utils"
+)
+
+type gomaxprocsSuite struct {
+	testbase.LoggingSuite
+	setmaxprocs    chan int
+	numCPUResponse int
+	setMaxProcs    int
+}
+
+var _ = gc.Suite(&gomaxprocsSuite{})
+
+func (s *gomaxprocsSuite) SetUpTest(c *gc.C) {
+	s.LoggingSuite.SetUpTest(c)
+	// always stub out GOMAXPROCS so we don't actually change anything
+	s.numCPUResponse = 2
+	s.setMaxProcs = -1
+	maxProcsFunc := func(n int) int {
+		s.setMaxProcs = n
+		return 1
+	}
+	numCPUFunc := func() int { return s.numCPUResponse }
+	s.PatchValue(utils.GOMAXPROCS, maxProcsFunc)
+	s.PatchValue(utils.NumCPU, numCPUFunc)
+	s.PatchEnvironment("GOMAXPROCS", "")
+}
+
+func (s *gomaxprocsSuite) TestUseMultipleCPUsDoesNothingWhenGOMAXPROCSSet(c *gc.C) {
+	os.Setenv("GOMAXPROCS", "1")
+	utils.UseMultipleCPUs()
+	c.Check(s.setMaxProcs, gc.Equals, 0)
+}
+
+func (s *gomaxprocsSuite) TestUseMultipleCPUsWhenEnabled(c *gc.C) {
+	utils.UseMultipleCPUs()
+	c.Check(s.setMaxProcs, gc.Equals, 2)
+	s.numCPUResponse = 4
+	utils.UseMultipleCPUs()
+	c.Check(s.setMaxProcs, gc.Equals, 4)
+}