Merge lp:~axwalk/juju-core/lp1281071-rsyslog-tls into lp:~go-bot/juju-core/trunk

[Andrew Wilkins]

Implement rsyslog TLS support

This changes the rsyslog configuration
we generate to use TLS. We install the
rsyslog-gnutls package and generate a
new CA cert, server cert and key
specifically for rsyslog.

This completely changes the way rsyslog
configuration is managed. Now, instead
of writing at bootstrap time and having
an upgrade step, we have a worker that
writes the syslog config whenever syslog
parameters change. The state server will
generate certificates and propagate the
CA cert to other agents via environ config.

There are some other miscellaneous changes:
  - we now use reliable forwarding, as
    otherwise the machine agent and unit
    agent will restart rsyslog while
    log messages are buffered causing
    message loss
  - dedicated upgrades for rsyslog are
    redundant and removed. The new worker
    upgrades machine and unit agent rsyslog
    automatically.
  - syslog-port can now be changed, and must
    be changed to work around the privilege
    drop race in rsyslog 5.x (this is the
    sole motivation)
  - the ryslog config path is no longer
    populated into agent.conf, and we
    no longer use the existing value to
    perform cleanups. The worker is now
    responsible for cleaning up the config
    file on tear-down.
  - the local provider symlinks machine-0.log
    into /var/log/juju$namespace so that we
    do not need any configuration other than
    the existing namespace to determine log
    location

TODO(axw) tests in worker/rsyslog
TODO(axw) tests in state/api/rsyslog
TODO(axw) tests in state/apiserver/rsyslog

Fixes lp:1281071

https://codereview.appspot.com/68070044/

[Andrew Wilkins (axwalk)]

Reviewers: mp+207889_code.launchpad.net,

Message:
Please take a look.

Description:
Implement rsyslog TLS support

A new config attribute, syslog-tls, is introduced.
This is true for all new environments except for
the local provider, and false (rather, omitted) for
existing environments and any local provider
environments. As with syslog-port, this is immutable.

If syslog-tls is true, then rsyslog-gnutls is
installed and used by the rsyslog configuration that
we generate. The syslog configuration conditionally
uses the TLS transport.

We cannot automatically upgrade existing environments
without additional tooling, as existing server certs
are incompatible with GnuTLS (incorrect Key Usage).

Fixes lp:1281071

https://code.launchpad.net/~axwalk/juju-core/lp1281071-rsyslog-tls/+merge/207889

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/68070044/

Affected files (+334, -64 lines):
   A [revision details]
   M environs/cloudinit.go
   M environs/cloudinit/cloudinit.go
   M environs/cloudinit/cloudinit_test.go
   M environs/config/config.go
   M environs/config/config_test.go
   M log/syslog/config.go
   M log/syslog/config_test.go
   M log/syslog/testing/syslogconf.go
   M provider/local/environprovider.go
   M state/address.go
   M state/api/params/params.go
   M state/apiserver/deployer/deployer.go
   M state/apiserver/provisioner/provisioner.go
   M upgrades/rsyslogconf.go
   M upgrades/rsyslogconf_test.go
   M worker/deployer/export_test.go
   M worker/deployer/simple.go
   M worker/provisioner/kvm-broker.go
   M worker/provisioner/lxc-broker.go

[Tim Penhey (thumper)]

We already need to rewrite the syslog config files as part of the 1.18
upgrade process as they are all out of date and wrong, so this is
something we need to address anyway.

I'd like to see us move from UDP to TCP for the logging.

I'm a little surprised that I have never been hit by the inability for a
machine to listen on the current port.

https://codereview.appspot.com/68070044/diff/1/environs/cloudinit/cloudinit.go
File environs/cloudinit/cloudinit.go (right):

https://codereview.appspot.com/68070044/diff/1/environs/cloudinit/cloudinit.go#newcode434
environs/cloudinit/cloudinit.go:434: configRenderer.TLSCACertPath =
path.Join(cfg.LogDir, "ca.pem")
Do we really want to reuse the same cert? I strongly suggest that we
have different certs / keys for different jobs.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go
File log/syslog/config.go (right):

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode38
log/syslog/config.go:38: // File="{{logDir}}/all-machines.log"
Argh... you can't just change this.

rsyslog has an apparmor profile that restricts where it can write files.
  This was a source of many hours of searching.

Needs to be /var/log/** somewhere.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode64
log/syslog/config.go:64: $ModLoad imudp
I think we should actually be using TCP not UDP for log messages anyway.
  There is a built in store and forward mechanism.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode65
log/syslog/config.go:65: $UDPServerRun {{portNumber}}
Why had I not been hit by this port number problem?

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode198
log/syslog/config.go:198: addressPrefix = "@@"
If we switch to TCP for all, we can avoid this.

https://codereview.appspot.com/68070044/diff/1/log/syslog/testing/syslogconf.go
File log/syslog/testing/syslogconf.go (right):

https://codereview.appspot.com/68070044/diff/1/log/syslog/testing/syslogconf.go#newcode25
log/syslog/testing/syslogconf.go:25: $DefaultNetstreamDriverCAFile
/var/log/juju/ca.pem
Oh? so this isn't the same as the mongo cert?

https://codereview.appspot.com/68070044/

[Andrew Wilkins (axwalk)]

I'll start on a new worker, and do the certificate stuff at the same
time.

If we don't mind installing rsyslog-gnutls on machine-0 for local, we
can remove the syslog-tls config attr and make TLS mandatory.

https://codereview.appspot.com/68070044/diff/1/environs/cloudinit/cloudinit.go
File environs/cloudinit/cloudinit.go (right):

https://codereview.appspot.com/68070044/diff/1/environs/cloudinit/cloudinit.go#newcode434
environs/cloudinit/cloudinit.go:434: configRenderer.TLSCACertPath =
path.Join(cfg.LogDir, "ca.pem")
On 2014/02/24 21:48:29, thumper wrote:
> Do we really want to reuse the same cert? I strongly suggest that we
have
> different certs / keys for different jobs.

I will update the code to generate a new CA cert/key for rsyslog, and
use that to sign a new server cert. The CA cert will be populated into
environment config, which the rsyslog worker will watch for.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go
File log/syslog/config.go (right):

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode38
log/syslog/config.go:38: // File="{{logDir}}/all-machines.log"
On 2014/02/24 21:48:29, thumper wrote:
> Argh... you can't just change this.

I tested it and it works on both precise and trusty.

> rsyslog has an apparmor profile that restricts where it can write
files. This
> was a source of many hours of searching.

I know, and I did it knowingly. It is writing to /var/log/juju, on
account of the symlink.

> Needs to be /var/log/** somewhere.

If we hard-code it, then the SyslogConfig parameters ought to be
changed. Otherwise it looks like it's going to write to the configured
log-dir, but doesn't.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode64
log/syslog/config.go:64: $ModLoad imudp
On 2014/02/24 21:48:29, thumper wrote:
> I think we should actually be using TCP not UDP for log messages
anyway. There
> is a built in store and forward mechanism.

Probably not a bad idea, but let's please leave that until after TLS is
sorted.

https://codereview.appspot.com/68070044/diff/1/log/syslog/config.go#newcode65
log/syslog/config.go:65: $UDPServerRun {{portNumber}}
On 2014/02/24 21:48:29, thumper wrote:
> Why had I not been hit by this port number problem?

I don't know the specifics of the bug. All I know is that it only hit me
when I was using TLS, and not UDP.

https://codereview.appspot.com/68070044/diff/1/log/syslog/testing/syslogconf.go
File log/syslog/testing/syslogconf.go (right):

https://codereview.appspot.com/68070044/diff/1/log/syslog/testing/syslogconf.go#newcode25
log/syslog/testing/syslogconf.go:25: $DefaultNetstreamDriverCAFile
/var/log/juju/ca.pem
On 2014/02/24 21:48:29, thumper wrote:
> Oh? so this isn't the same as the mongo cert?

It is, it's just copied to another place to allow for using a different
one.

https://codereview.appspot.com/68070044/

[Andrew Wilkins (axwalk)]

Please take a look.

https://codereview.appspot.com/68070044/

[Andrew Wilkins (axwalk)]

On 2014/02/25 14:22:14, axw wrote:
> Please take a look.

This CL has jumped the shark, I guess, but I wanted to get it out there
to get feedback.
If the direction is good I will try to pare it back, keeping the
existing environs/cloudinit and related code, and remove all that in a
followup.

https://codereview.appspot.com/68070044/

[Andrew Wilkins (axwalk)]

This is *not* merged, lying bot! I created another branch and merged this in, removing all the bits I didn't care about. Bot merged the second branch and thought this one was merged too.

[John A Meinel (jameinel)]

This revision is in the history, though maybe it didn't merge like it looks here?

[Andrew Wilkins (axwalk)]

On 2014/02/25 14:27:20, axw wrote:
> On 2014/02/25 14:22:14, axw wrote:
> > Please take a look.

> This CL has jumped the shark, I guess, but I wanted to get it out
there to get
> feedback.
> If the direction is good I will try to pare it back, keeping the
existing
> environs/cloudinit and related code, and remove all that in a
followup.

This is superseded by https://codereview.appspot.com/68930045

https://codereview.appspot.com/68070044/

=== modified file 'agent/agent.go'
--- agent/agent.go	2014-02-18 05:43:06 +0000
+++ agent/agent.go	2014-02-25 14:03:09 +0000
@@ -30,7 +30,6 @@
 	StorageAddr      = "STORAGE_ADDR"
 	AgentServiceName = "AGENT_SERVICE_NAME"
 	MongoServiceName = "MONGO_SERVICE_NAME"
-	RsyslogConfPath  = "RSYSLOG_CONF_PATH"
 	BootstrapJobs    = "BOOTSTRAP_JOBS"
 )
 

=== modified file 'cmd/juju/environment_test.go'
--- cmd/juju/environment_test.go	2013-11-28 09:11:12 +0000
+++ cmd/juju/environment_test.go	2014-02-25 14:03:09 +0000
@@ -167,7 +167,6 @@
 	"firewall-mode": "global",
 	"state-port":    "1",
 	"api-port":      "666",
-	"syslog-port":   "42",
 }
 
 func (s *SetEnvironmentSuite) TestImmutableConfigValues(c *gc.C) {

=== modified file 'cmd/jujud/machine.go'
--- cmd/jujud/machine.go	2014-02-20 08:23:40 +0000
+++ cmd/jujud/machine.go	2014-02-25 14:03:09 +0000
@@ -18,7 +18,6 @@
 	"launchpad.net/juju-core/cmd"
 	"launchpad.net/juju-core/container/kvm"
 	"launchpad.net/juju-core/instance"
-	"launchpad.net/juju-core/log/syslog"
 	"launchpad.net/juju-core/names"
 	"launchpad.net/juju-core/provider"
 	"launchpad.net/juju-core/state"
@@ -42,6 +41,7 @@
 	"launchpad.net/juju-core/worker/minunitsworker"
 	"launchpad.net/juju-core/worker/provisioner"
 	"launchpad.net/juju-core/worker/resumer"
+	"launchpad.net/juju-core/worker/rsyslog"
 	"launchpad.net/juju-core/worker/terminationworker"
 	"launchpad.net/juju-core/worker/upgrader"
 )
@@ -65,6 +65,8 @@
 	Conf      AgentConf
 	MachineId string
 	runner    worker.Runner
+
+	rsyslogConfPath string
 }
 
 // Info returns usage information for the command.
@@ -157,6 +159,8 @@
 	return err
 }
 
+var newRsyslogConfigWorker = rsyslog.NewRsyslogConfigWorker
+
 // APIWorker returns a Worker that connects to the API and starts any
 // workers that need an API connection.
 //
@@ -174,6 +178,14 @@
 			break
 		}
 	}
+	rsyslogMode := rsyslog.RsyslogModeForwarding
+	for _, job := range entity.Jobs() {
+		if job == params.JobManageEnviron {
+			rsyslogMode = rsyslog.RsyslogModeAccumulate
+			break
+		}
+	}
+
 	runner := newRunner(connectionIsFatal(st), moreImportant)
 	runner.StartWorker("machiner", func() (worker.Worker, error) {
 		return machiner.NewMachiner(st.Machiner(), agentConfig), nil
@@ -187,6 +199,9 @@
 	runner.StartWorker("machineenvironmentworker", func() (worker.Worker, error) {
 		return machineenvironmentworker.NewMachineEnvironmentWorker(st.Environment(), agentConfig), nil
 	})
+	runner.StartWorker("rsyslog", func() (worker.Worker, error) {
+		return newRsyslogConfigWorker(st.Rsyslog(), agentConfig, rsyslogMode)
+	})
 
 	// If not a local provider bootstrap machine, start the worker to manage SSH keys.
 	providerType := agentConfig.Value(agent.ProviderType)
@@ -385,15 +400,6 @@
 			errors = append(errors, fmt.Errorf("cannot remove service %q: %v", agentServiceName, err))
 		}
 	}
-	// Remove the rsyslog conf file and restart rsyslogd.
-	if rsyslogConfPath := a.Conf.config.Value(agent.RsyslogConfPath); rsyslogConfPath != "" {
-		if err := os.Remove(rsyslogConfPath); err != nil {
-			errors = append(errors, err)
-		}
-		if err := syslog.Restart(); err != nil {
-			errors = append(errors, err)
-		}
-	}
 	// Remove the juju-run symlink.
 	if err := os.Remove(jujuRun); err != nil && !os.IsNotExist(err) {
 		errors = append(errors, err)

=== modified file 'cmd/jujud/machine_test.go'
--- cmd/jujud/machine_test.go	2014-02-19 02:25:30 +0000
+++ cmd/jujud/machine_test.go	2014-02-25 14:03:09 +0000
@@ -28,6 +28,7 @@
 	"launchpad.net/juju-core/state/api"
 	apideployer "launchpad.net/juju-core/state/api/deployer"
 	"launchpad.net/juju-core/state/api/params"
+	apirsyslog "launchpad.net/juju-core/state/api/rsyslog"
 	charmtesting "launchpad.net/juju-core/state/apiserver/charmrevisionupdater/testing"
 	statetesting "launchpad.net/juju-core/state/testing"
 	"launchpad.net/juju-core/state/watcher"
@@ -40,10 +41,12 @@
 	"launchpad.net/juju-core/utils/ssh"
 	sshtesting "launchpad.net/juju-core/utils/ssh/testing"
 	"launchpad.net/juju-core/version"
+	"launchpad.net/juju-core/worker"
 	"launchpad.net/juju-core/worker/authenticationworker"
 	"launchpad.net/juju-core/worker/deployer"
 	"launchpad.net/juju-core/worker/instancepoller"
 	"launchpad.net/juju-core/worker/machineenvironmentworker"
+	"launchpad.net/juju-core/worker/rsyslog"
 	"launchpad.net/juju-core/worker/upgrader"
 )
 
@@ -750,6 +753,30 @@
 	c.Assert(err, jc.Satisfies, os.IsNotExist)
 }
 
+func (s *MachineSuite) TestMachineAgentRsyslogManageEnviron(c *gc.C) {
+	s.testMachineAgentRsyslogConfigWorker(c, state.JobManageEnviron, rsyslog.RsyslogModeAccumulate)
+}
+
+func (s *MachineSuite) TestMachineAgentRsyslogHostUnits(c *gc.C) {
+	s.testMachineAgentRsyslogConfigWorker(c, state.JobHostUnits, rsyslog.RsyslogModeForwarding)
+}
+
+func (s *MachineSuite) testMachineAgentRsyslogConfigWorker(c *gc.C, job state.MachineJob, expectedMode rsyslog.RsyslogMode) {
+	created := make(chan rsyslog.RsyslogMode, 1)
+	s.PatchValue(&newRsyslogConfigWorker, func(_ *apirsyslog.State, _ agent.Config, mode rsyslog.RsyslogMode) (worker.Worker, error) {
+		created <- mode
+		return worker.NewRunner(isFatal, moreImportant), nil
+	})
+	s.assertJobWithAPI(c, job, func(conf agent.Config, st *api.State) {
+		select {
+		case <-time.After(testing.LongWait):
+			c.Fatalf("timeout while waiting for rsyslog worker to be created")
+		case mode := <-created:
+			c.Assert(mode, gc.Equals, expectedMode)
+		}
+	})
+}
+
 // MachineWithCharmsSuite provides infrastructure for tests which need to
 // work with charms.
 type MachineWithCharmsSuite struct {

=== modified file 'cmd/jujud/unit.go'
--- cmd/jujud/unit.go	2014-02-18 00:15:30 +0000
+++ cmd/jujud/unit.go	2014-02-25 14:03:09 +0000
@@ -15,6 +15,7 @@
 	"launchpad.net/juju-core/state"
 	"launchpad.net/juju-core/worker"
 	workerlogger "launchpad.net/juju-core/worker/logger"
+	"launchpad.net/juju-core/worker/rsyslog"
 	"launchpad.net/juju-core/worker/uniter"
 	"launchpad.net/juju-core/worker/upgrader"
 )
@@ -94,6 +95,9 @@
 	runner.StartWorker("uniter", func() (worker.Worker, error) {
 		return uniter.NewUniter(st.Uniter(), entity.Tag(), dataDir), nil
 	})
+	runner.StartWorker("rsyslog", func() (worker.Worker, error) {
+		return rsyslog.NewRsyslogConfigWorker(st.Rsyslog(), agentConfig, rsyslog.RsyslogModeForwarding)
+	})
 	return newCloseWorker(runner, st), nil
 }
 

=== modified file 'container/testing/common.go'
--- container/testing/common.go	2013-12-04 04:33:23 +0000
+++ container/testing/common.go	2014-02-25 14:03:09 +0000
@@ -21,7 +21,6 @@
 	stateInfo := jujutesting.FakeStateInfo(machineId)
 	apiInfo := jujutesting.FakeAPIInfo(machineId)
 	machineConfig := environs.NewMachineConfig(machineId, "fake-nonce", stateInfo, apiInfo)
-	machineConfig.SyslogPort = 2345
 	machineConfig.Tools = &tools.Tools{
 		Version: version.MustParseBinary("2.3.4-foo-bar"),
 		URL:     "http://tools.testing.invalid/2.3.4-foo-bar.tgz",

=== modified file 'environs/cloudinit.go'
--- environs/cloudinit.go	2014-02-13 05:22:01 +0000
+++ environs/cloudinit.go	2014-02-25 14:03:09 +0000
@@ -31,12 +31,6 @@
 // CloudInitOutputLog is the default cloud-init-output.log file path.
 const CloudInitOutputLog = "/var/log/cloud-init-output.log"
 
-// DefaultRsyslogConfPath is the default rsyslogd conf file path.
-const DefaultRsyslogConfPath = "/etc/rsyslog.d/25-juju.conf"
-
-// Override for testing.
-var RsyslogConfPath = DefaultRsyslogConfPath
-
 // MongoServiceName is the default Upstart service name for Mongo.
 const MongoServiceName = "juju-db"
 
@@ -50,7 +44,6 @@
 		DataDir:                 DataDir,
 		LogDir:                  LogDir,
 		CloudInitOutputLog:      CloudInitOutputLog,
-		RsyslogConfPath:         RsyslogConfPath,
 		MachineAgentServiceName: "jujud-" + names.MachineTag(machineID),
 		MongoServiceName:        MongoServiceName,
 
@@ -85,7 +78,6 @@
 func PopulateMachineConfig(mcfg *cloudinit.MachineConfig,
 	providerType, authorizedKeys string,
 	sslHostnameVerification bool,
-	syslogPort int,
 	proxy, aptProxy osenv.ProxySettings,
 ) error {
 	if authorizedKeys == "" {
@@ -98,7 +90,6 @@
 	mcfg.AgentEnvironment[agent.ProviderType] = providerType
 	mcfg.AgentEnvironment[agent.ContainerType] = string(mcfg.MachineContainerType)
 	mcfg.DisableSSLHostnameVerification = !sslHostnameVerification
-	mcfg.SyslogPort = syslogPort
 	mcfg.ProxySettings = proxy
 	mcfg.AptProxySettings = aptProxy
 	return nil
@@ -122,7 +113,6 @@
 		cfg.Type(),
 		cfg.AuthorizedKeys(),
 		cfg.SSLHostnameVerification(),
-		cfg.SyslogPort(),
 		cfg.ProxySettings(),
 		cfg.AptProxySettings(),
 	); err != nil {

=== modified file 'environs/cloudinit/cloudinit.go'
--- environs/cloudinit/cloudinit.go	2014-02-20 15:03:08 +0000
+++ environs/cloudinit/cloudinit.go	2014-02-25 14:03:09 +0000
@@ -20,7 +20,6 @@
 	"launchpad.net/juju-core/environs/config"
 	"launchpad.net/juju-core/instance"
 	"launchpad.net/juju-core/juju/osenv"
-	"launchpad.net/juju-core/log/syslog"
 	"launchpad.net/juju-core/names"
 	"launchpad.net/juju-core/state"
 	"launchpad.net/juju-core/state/api"
@@ -63,10 +62,6 @@
 	// if StateServer is true.
 	APIPort int
 
-	// SyslogPort specifies the port number that will be used when
-	// sending the log messages using rsyslog.
-	SyslogPort int
-
 	// StateInfo holds the means for the new instance to communicate with the
 	// juju state. Unless the new machine is running a state server (StateServer is
 	// set), there must be at least one state server address supplied.
@@ -95,10 +90,6 @@
 	// LogDir holds the directory that juju logs will be written to.
 	LogDir string
 
-	// RsyslogConfPath is the path to the rsyslogd conf file written
-	// for configuring distributed logging.
-	RsyslogConfPath string
-
 	// CloudInitOutputLog specifies the path to the output log for cloud-init.
 	// The directory containing the log file must already exist.
 	CloudInitOutputLog string
@@ -317,10 +308,6 @@
 		fmt.Sprintf("printf %%s %s > $bin/downloaded-tools.txt", shquote(string(toolsJson))),
 	)
 
-	if err := cfg.addLogging(c); err != nil {
-		return err
-	}
-
 	// We add the machine agent's configuration info
 	// before running bootstrap-state so that bootstrap-state
 	// has a chance to rerwrite it to change the password.
@@ -401,26 +388,6 @@
 	return cfg.addMachineAgentToBoot(c, machineTag, cfg.MachineId)
 }
 
-func (cfg *MachineConfig) addLogging(c *cloudinit.Config) error {
-	namespace := cfg.AgentEnvironment[agent.Namespace]
-	var configRenderer *syslog.SyslogConfig
-	if cfg.StateServer {
-		configRenderer = syslog.NewAccumulateConfig(
-			names.MachineTag(cfg.MachineId), cfg.SyslogPort, namespace)
-	} else {
-		configRenderer = syslog.NewForwardConfig(
-			names.MachineTag(cfg.MachineId), cfg.SyslogPort, namespace, cfg.stateHostAddrs())
-	}
-	configRenderer.LogDir = cfg.LogDir
-	content, err := configRenderer.Render()
-	if err != nil {
-		return err
-	}
-	c.AddFile(cfg.RsyslogConfPath, string(content), 0644)
-	c.AddRunCmd("restart rsyslog")
-	return nil
-}
-
 func (cfg *MachineConfig) dataFile(name string) string {
 	return path.Join(cfg.DataDir, name)
 }
@@ -464,7 +431,6 @@
 	if err != nil {
 		return nil, err
 	}
-	acfg.SetValue(agent.RsyslogConfPath, cfg.RsyslogConfPath)
 	acfg.SetValue(agent.AgentServiceName, cfg.MachineAgentServiceName)
 	if cfg.StateServer {
 		acfg.SetValue(agent.MongoServiceName, cfg.MongoServiceName)
@@ -656,9 +622,6 @@
 	if cfg.CloudInitOutputLog == "" {
 		return fmt.Errorf("missing cloud-init output log path")
 	}
-	if cfg.RsyslogConfPath == "" {
-		return fmt.Errorf("missing rsyslog.d conf path")
-	}
 	if cfg.Tools == nil {
 		return fmt.Errorf("missing tools")
 	}
@@ -674,9 +637,6 @@
 	if cfg.APIInfo == nil {
 		return fmt.Errorf("missing API info")
 	}
-	if cfg.SyslogPort == 0 {
-		return fmt.Errorf("missing syslog port")
-	}
 	if len(cfg.APIInfo.CACert) == 0 {
 		return fmt.Errorf("missing API CA certificate")
 	}

=== modified file 'environs/cloudinit/cloudinit_test.go'
--- environs/cloudinit/cloudinit_test.go	2014-02-21 17:32:31 +0000
+++ environs/cloudinit/cloudinit_test.go	2014-02-25 14:03:09 +0000
@@ -76,7 +76,6 @@
 			StateServerKey:  serverKey,
 			StatePort:       37017,
 			APIPort:         17070,
-			SyslogPort:      514,
 			MachineNonce:    "FAKE_NONCE",
 			StateInfo: &state.Info{
 				Password: "arble",
@@ -90,7 +89,6 @@
 			DataDir:                 environs.DataDir,
 			LogDir:                  environs.LogDir,
 			CloudInitOutputLog:      environs.CloudInitOutputLog,
-			RsyslogConfPath:         environs.RsyslogConfPath,
 			StateInfoURL:            "some-url",
 			SystemPrivateSSHKey:     "private rsa key",
 			MachineAgentServiceName: "jujud-machine-0",
@@ -116,9 +114,6 @@
 tar zxf \$bin/tools.tar.gz -C \$bin
 rm \$bin/tools\.tar\.gz && rm \$bin/juju1\.2\.3-precise-amd64\.sha256
 printf %s '{"version":"1\.2\.3-precise-amd64","url":"http://foo\.com/tools/releases/juju1\.2\.3-precise-amd64\.tgz","sha256":"1234","size":10}' > \$bin/downloaded-tools\.txt
-install -D -m 644 /dev/null '/etc/rsyslog\.d/25-juju\.conf'
-printf '%s\\n' '.*' > '/etc/rsyslog.d/25-juju.conf'
-restart rsyslog
 mkdir -p '/var/lib/juju/agents/machine-0'
 install -m 644 /dev/null '/var/lib/juju/agents/machine-0/format'
 printf '%s\\n' '.*' > '/var/lib/juju/agents/machine-0/format'
@@ -165,7 +160,6 @@
 			StateServerKey:  serverKey,
 			StatePort:       37017,
 			APIPort:         17070,
-			SyslogPort:      514,
 			MachineNonce:    "FAKE_NONCE",
 			StateInfo: &state.Info{
 				Password: "arble",
@@ -179,7 +173,6 @@
 			DataDir:                 environs.DataDir,
 			LogDir:                  environs.LogDir,
 			CloudInitOutputLog:      environs.CloudInitOutputLog,
-			RsyslogConfPath:         environs.RsyslogConfPath,
 			StateInfoURL:            "some-url",
 			SystemPrivateSSHKey:     "private rsa key",
 			MachineAgentServiceName: "jujud-machine-0",
@@ -207,7 +200,6 @@
 			DataDir:            environs.DataDir,
 			LogDir:             environs.LogDir,
 			CloudInitOutputLog: environs.CloudInitOutputLog,
-			RsyslogConfPath:    environs.RsyslogConfPath,
 			StateServer:        false,
 			Tools:              newSimpleTools("1.2.3-linux-amd64"),
 			MachineNonce:       "FAKE_NONCE",
@@ -223,7 +215,6 @@
 				Password: "bletch",
 				CACert:   []byte("CA CERT\n" + testing.CACert),
 			},
-			SyslogPort:              514,
 			MachineAgentServiceName: "jujud-machine-99",
 		},
 		expectScripts: `
@@ -244,9 +235,6 @@
 tar zxf \$bin/tools.tar.gz -C \$bin
 rm \$bin/tools\.tar\.gz && rm \$bin/juju1\.2\.3-linux-amd64\.sha256
 printf %s '{"version":"1\.2\.3-linux-amd64","url":"http://foo\.com/tools/releases/juju1\.2\.3-linux-amd64\.tgz","sha256":"1234","size":10}' > \$bin/downloaded-tools\.txt
-install -D -m 644 /dev/null '/etc/rsyslog\.d/25-juju\.conf'
-printf '%s\\n' '.*' > '/etc/rsyslog\.d/25-juju\.conf'
-restart rsyslog
 mkdir -p '/var/lib/juju/agents/machine-99'
 install -m 644 /dev/null '/var/lib/juju/agents/machine-99/format'
 printf '%s\\n' '.*' > '/var/lib/juju/agents/machine-99/format'
@@ -267,7 +255,6 @@
 			DataDir:              environs.DataDir,
 			LogDir:               environs.LogDir,
 			CloudInitOutputLog:   environs.CloudInitOutputLog,
-			RsyslogConfPath:      environs.RsyslogConfPath,
 			StateServer:          false,
 			Tools:                newSimpleTools("1.2.3-linux-amd64"),
 			MachineNonce:         "FAKE_NONCE",
@@ -283,13 +270,10 @@
 				Password: "bletch",
 				CACert:   []byte("CA CERT\n" + testing.CACert),
 			},
-			SyslogPort:              514,
 			MachineAgentServiceName: "jujud-machine-2-lxc-1",
 		},
 		inexactMatch: true,
 		expectScripts: `
-printf '%s\\n' '.*' > '/etc/rsyslog\.d/25-juju\.conf'
-restart rsyslog
 mkdir -p '/var/lib/juju/agents/machine-2-lxc-1'
 install -m 644 /dev/null '/var/lib/juju/agents/machine-2-lxc-1/format'
 printf '%s\\n' '.*' > '/var/lib/juju/agents/machine-2-lxc-1/format'
@@ -308,7 +292,6 @@
 			DataDir:            environs.DataDir,
 			LogDir:             environs.LogDir,
 			CloudInitOutputLog: environs.CloudInitOutputLog,
-			RsyslogConfPath:    environs.RsyslogConfPath,
 			StateServer:        false,
 			Tools:              newSimpleTools("1.2.3-linux-amd64"),
 			MachineNonce:       "FAKE_NONCE",
@@ -324,7 +307,6 @@
 				Password: "bletch",
 				CACert:   []byte("CA CERT\n" + testing.CACert),
 			},
-			SyslogPort:                     514,
 			DisableSSLHostnameVerification: true,
 			MachineAgentServiceName:        "jujud-machine-99",
 		},
@@ -345,7 +327,6 @@
 			StateServerKey:  serverKey,
 			StatePort:       37017,
 			APIPort:         17070,
-			SyslogPort:      514,
 			MachineNonce:    "FAKE_NONCE",
 			StateInfo: &state.Info{
 				Password: "arble",
@@ -358,7 +339,6 @@
 			DataDir:                 environs.DataDir,
 			LogDir:                  environs.LogDir,
 			CloudInitOutputLog:      environs.CloudInitOutputLog,
-			RsyslogConfPath:         environs.RsyslogConfPath,
 			StateInfoURL:            "some-url",
 			SystemPrivateSSHKey:     "private rsa key",
 			MachineAgentServiceName: "jujud-machine-0",
@@ -644,9 +624,6 @@
 	{"missing API info", func(cfg *cloudinit.MachineConfig) {
 		cfg.APIInfo = nil
 	}},
-	{"missing syslog port", func(cfg *cloudinit.MachineConfig) {
-		cfg.SyslogPort = 0
-	}},
 	{"missing state hosts", func(cfg *cloudinit.MachineConfig) {
 		cfg.StateServer = false
 		cfg.StateInfo = &state.Info{
@@ -696,9 +673,6 @@
 	{"missing cloud-init output log path", func(cfg *cloudinit.MachineConfig) {
 		cfg.CloudInitOutputLog = ""
 	}},
-	{"missing rsyslog.d conf path", func(cfg *cloudinit.MachineConfig) {
-		cfg.RsyslogConfPath = ""
-	}},
 	{"missing tools", func(cfg *cloudinit.MachineConfig) {
 		cfg.Tools = nil
 	}},
@@ -765,7 +739,6 @@
 		StateServerKey:   serverKey,
 		StatePort:        1234,
 		APIPort:          1235,
-		SyslogPort:       2345,
 		MachineId:        "99",
 		Tools:            newSimpleTools("9.9.9-linux-arble"),
 		AuthorizedKeys:   "sshkey1",
@@ -783,7 +756,6 @@
 		DataDir:                 environs.DataDir,
 		LogDir:                  environs.LogDir,
 		CloudInitOutputLog:      environs.CloudInitOutputLog,
-		RsyslogConfPath:         environs.RsyslogConfPath,
 		MachineNonce:            "FAKE_NONCE",
 		SystemPrivateSSHKey:     "private rsa key",
 		MachineAgentServiceName: "jujud-machine-99",
@@ -803,12 +775,20 @@
 	}
 }
 
-func (*cloudinitSuite) createMachineConfig(c *gc.C, environConfig *config.Config) *cloudinit.MachineConfig {
+func (*cloudinitSuite) createMachineConfig(c *gc.C, environConfig *config.Config, stateServer bool) *cloudinit.MachineConfig {
 	machineId := "42"
 	machineNonce := "fake-nonce"
 	stateInfo := jujutesting.FakeStateInfo(machineId)
 	apiInfo := jujutesting.FakeAPIInfo(machineId)
-	machineConfig := environs.NewMachineConfig(machineId, machineNonce, stateInfo, apiInfo)
+	var machineConfig *cloudinit.MachineConfig
+	if stateServer {
+		var err error
+		environConfig, err = environConfig.Apply(map[string]interface{}{"agent-version": "2.3.4"})
+		c.Assert(err, gc.IsNil)
+		machineConfig = environs.NewBootstrapMachineConfig("http://testing.invalid/provider-state", "invalid ssh key")
+	} else {
+		machineConfig = environs.NewMachineConfig(machineId, machineNonce, stateInfo, apiInfo)
+	}
 	machineConfig.Tools = &tools.Tools{
 		Version: version.MustParseBinary("2.3.4-foo-bar"),
 		URL:     "http://tools.testing.invalid/2.3.4-foo-bar.tgz",
@@ -820,7 +800,7 @@
 
 func (s *cloudinitSuite) TestAptProxyNotWrittenIfNotSet(c *gc.C) {
 	environConfig := minimalConfig(c)
-	machineCfg := s.createMachineConfig(c, environConfig)
+	machineCfg := s.createMachineConfig(c, environConfig, false)
 	cloudcfg := coreCloudinit.New()
 	err := cloudinit.Configure(machineCfg, cloudcfg)
 	c.Assert(err, gc.IsNil)
@@ -835,7 +815,7 @@
 		"apt-http-proxy": "http://user@10.0.0.1",
 	})
 	c.Assert(err, gc.IsNil)
-	machineCfg := s.createMachineConfig(c, environConfig)
+	machineCfg := s.createMachineConfig(c, environConfig, false)
 	cloudcfg := coreCloudinit.New()
 	err = cloudinit.Configure(machineCfg, cloudcfg)
 	c.Assert(err, gc.IsNil)
@@ -851,7 +831,7 @@
 		"http-proxy": "http://user@10.0.0.1",
 	})
 	c.Assert(err, gc.IsNil)
-	machineCfg := s.createMachineConfig(c, environConfig)
+	machineCfg := s.createMachineConfig(c, environConfig, false)
 	cloudcfg := coreCloudinit.New()
 	err = cloudinit.Configure(machineCfg, cloudcfg)
 	c.Assert(err, gc.IsNil)

=== modified file 'environs/cloudinit_test.go'
--- environs/cloudinit_test.go	2014-01-23 05:45:58 +0000
+++ environs/cloudinit_test.go	2014-02-25 14:03:09 +0000
@@ -63,7 +63,6 @@
 		StateInfo: &state.Info{Tag: "not touched"},
 		APIInfo:   &api.Info{Tag: "not touched"},
 		DisableSSLHostnameVerification: false,
-		SyslogPort:                     2345,
 	})
 }
 
@@ -71,7 +70,6 @@
 	attrs := dummySampleConfig().Merge(testing.Attrs{
 		"authorized-keys":           "we-are-the-keys",
 		"ssl-hostname-verification": false,
-		"syslog-port":               8888,
 	})
 	cfg, err := config.New(config.NoDefaults, attrs)
 	c.Assert(err, gc.IsNil)
@@ -90,7 +88,6 @@
 		StateInfo: &state.Info{Tag: "not touched"},
 		APIInfo:   &api.Info{Tag: "not touched"},
 		DisableSSLHostnameVerification: true,
-		SyslogPort:                     8888,
 	})
 }
 
@@ -178,11 +175,9 @@
 		DataDir:                 environs.DataDir,
 		LogDir:                  environs.LogDir,
 		CloudInitOutputLog:      environs.CloudInitOutputLog,
-		RsyslogConfPath:         environs.RsyslogConfPath,
 		Config:                  envConfig,
 		StatePort:               envConfig.StatePort(),
 		APIPort:                 envConfig.APIPort(),
-		SyslogPort:              envConfig.SyslogPort(),
 		StateServer:             stateServer,
 		AgentEnvironment:        map[string]string{agent.ProviderType: "dummy"},
 		AuthorizedKeys:          "wheredidileavemykeys",

=== modified file 'environs/config/config.go'
--- environs/config/config.go	2014-02-12 04:54:19 +0000
+++ environs/config/config.go	2014-02-25 14:03:09 +0000
@@ -43,9 +43,9 @@
 	// DefaultApiPort is the default port the API server is listening on.
 	DefaultAPIPort int = 17070
 
-	// DefaultSyslogPort is the default port that the syslog UDP listener is
+	// DefaultSyslogPort is the default port that the syslog UDP/TCP listener is
 	// listening on.
-	DefaultSyslogPort int = 514
+	DefaultSyslogPort int = 6514
 
 	// DefaultBootstrapSSHTimeout is the amount of time to wait
 	// contacting a state server, in seconds.
@@ -427,6 +427,16 @@
 	return c.mustInt("syslog-port")
 }
 
+// RsyslogCACert returns the certificate of the CA that signed the
+// rsyslog certificate, in PEM format, or nil if one hasn't been
+// generated yet.
+func (c *Config) RsyslogCACert() []byte {
+	if s, ok := c.defined["rsyslog-ca-cert"]; ok {
+		return []byte(s.(string))
+	}
+	return nil
+}
+
 // AuthorizedKeys returns the content for ssh's authorized_keys file.
 func (c *Config) AuthorizedKeys() string {
 	return c.mustString("authorized-keys")
@@ -668,6 +678,7 @@
 	"state-port":                schema.ForceInt(),
 	"api-port":                  schema.ForceInt(),
 	"syslog-port":               schema.ForceInt(),
+	"rsyslog-ca-cert":           schema.String(),
 	"logging-config":            schema.String(),
 	"charm-store-auth":          schema.String(),
 	"provisioner-safe-mode":     schema.Bool(),
@@ -711,6 +722,7 @@
 	"bootstrap-timeout":         schema.Omit,
 	"bootstrap-retry-delay":     schema.Omit,
 	"bootstrap-addresses-delay": schema.Omit,
+	"rsyslog-ca-cert":           schema.Omit,
 
 	// Deprecated fields, retain for backwards compatibility.
 	"tools-url": "",
@@ -742,6 +754,9 @@
 
 var defaults = allDefaults()
 
+// allDefaults returns a schema.Defaults that contains
+// defaults to be used when creating a new config with
+// UseDefaults.
 func allDefaults() schema.Defaults {
 	d := schema.Defaults{
 		"default-series":            DefaultSeries,
@@ -756,7 +771,9 @@
 		"bootstrap-addresses-delay": DefaultBootstrapSSHAddressesDelay,
 	}
 	for attr, val := range alwaysOptional {
-		d[attr] = val
+		if _, ok := d[attr]; !ok {
+			d[attr] = val
+		}
 	}
 	return d
 }
@@ -786,7 +803,6 @@
 	"firewall-mode",
 	"state-port",
 	"api-port",
-	"syslog-port",
 	"bootstrap-timeout",
 	"bootstrap-retry-delay",
 	"bootstrap-addresses-delay",

=== modified file 'environs/config/config_test.go'
--- environs/config/config_test.go	2014-02-12 04:54:19 +0000
+++ environs/config/config_test.go	2014-02-25 14:03:09 +0000
@@ -1072,11 +1072,6 @@
 	new:   testing.Attrs{"api-port": 42},
 	err:   `cannot change api-port from 17070 to 42`,
 }, {
-	about: "Cannot change the syslog-port",
-	old:   testing.Attrs{"syslog-port": 345},
-	new:   testing.Attrs{"syslog-port": 42},
-	err:   `cannot change syslog-port from 345 to 42`,
-}, {
 	about: "Can change the state-port from explicit-default to implicit-default",
 	old:   testing.Attrs{"state-port": config.DefaultStatePort},
 }, {
@@ -1096,10 +1091,6 @@
 	about: "Cannot change the api-port from implicit-default to different value",
 	new:   testing.Attrs{"api-port": 42},
 	err:   `cannot change api-port from 17070 to 42`,
-}, {
-	about: "Cannot change the syslog-port from implicit-default to different value",
-	new:   testing.Attrs{"syslog-port": 42},
-	err:   `cannot change syslog-port from 514 to 42`,
 }}
 
 func (*ConfigSuite) TestValidateChange(c *gc.C) {

=== modified file 'log/syslog/config.go'
--- log/syslog/config.go	2014-01-30 22:12:42 +0000
+++ log/syslog/config.go	2014-02-25 14:03:09 +0000
@@ -35,7 +35,7 @@
 //
 // if $syslogtag startswith "juju{{namespace}}-" then
 //   action(type="omfile"
-//          File="/var/log/juju{{namespace}}/all-machines.log"
+//          File="{{logDir}}/all-machines.log"
 //          Template="JujuLogFormat{{namespace}}"
 //          FileCreateMode="0644")
 // & stop
@@ -52,15 +52,21 @@
 $InputFileStateFile {{logfileName}}{{namespace}}
 $InputRunFileMonitor
 
-$ModLoad imudp
-$UDPServerRun {{portNumber}}
+$ModLoad imtcp
+$DefaultNetstreamDriver gtls
+$DefaultNetstreamDriverCAFile {{tlsCACertPath}}
+$DefaultNetstreamDriverCertFile {{tlsCertPath}}
+$DefaultNetstreamDriverKeyFile {{tlsKeyPath}}
+$InputTCPServerStreamDriverAuthMode anon
+$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
+$InputTCPServerRun {{portNumber}}
 
 # Messages received from remote rsyslog machines have messages prefixed with a space,
 # so add one in for local messages too if needed.
 $template JujuLogFormat{{namespace}},"%syslogtag:{{tagStart}}:$%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
 
 $FileCreateMode 0644
-:syslogtag, startswith, "juju{{namespace}}-" /var/log/juju{{namespace}}/all-machines.log;JujuLogFormat{{namespace}}
+:syslogtag, startswith, "juju{{namespace}}-" {{logDir}}/all-machines.log;JujuLogFormat{{namespace}}
 & ~
 $FileCreateMode 0640
 `
@@ -70,6 +76,12 @@
 const nodeRsyslogTemplate = `
 $ModLoad imfile
 
+# Enable reliable forwarding.
+$ActionQueueType LinkedList
+$ActionQueueFileName {{logfileName}}{{namespace}}
+$ActionResumeRetryCount -1
+$ActionQueueSaveOnShutdown on
+
 $InputFilePersistStateInterval 50
 $InputFilePollInterval 5
 $InputFileName {{logfilePath}}
@@ -77,12 +89,22 @@
 $InputFileStateFile {{logfileName}}{{namespace}}
 $InputRunFileMonitor
 
+$DefaultNetstreamDriver gtls
+$DefaultNetstreamDriverCAFile {{tlsCACertPath}}
+$ActionSendStreamDriverAuthMode anon
+$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
+
 $template LongTagForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%"
 
-:syslogtag, startswith, "juju{{namespace}}-" @{{bootstrapIP}}:{{portNumber}};LongTagForwardFormat
+:syslogtag, startswith, "juju{{namespace}}-" @@{{bootstrapIP}}:{{portNumber}};LongTagForwardFormat
 & ~
 `
 
+// nodeRsyslogTemplateTLSHeader is prepended to
+// nodeRsyslogTemplate if TLS is to be used.
+const nodeRsyslogTemplateTLSHeader = `
+`
+
 const defaultConfigDir = "/etc/rsyslog.d"
 
 // SyslogConfigRenderer instances are used to generate a rsyslog conf file.
@@ -104,7 +126,15 @@
 	LogFileName string
 	// the addresses of the state server to which messages should be forwarded.
 	StateServerAddresses []string
-	// the port number for the udp listener
+	// TLSCACertPath is the full path to the CA certificate that
+	// signed the rsyslog server's certificate. If this is non-empty,
+	// then TLS will be used.
+	TLSCACertPath string
+	// TLSCertPath is the full path to the server's certificate.
+	TLSCertPath string
+	// TLSKeyPath is the full path to the server's private key.
+	TLSKeyPath string
+	// the port number for the listener
 	Port int
 	// the directory for the logfiles
 	LogDir string
@@ -153,7 +183,6 @@
 
 // Render generates the rsyslog config.
 func (slConfig *SyslogConfig) Render() ([]byte, error) {
-
 	// TODO: for HA, we will want to send to all state server addresses (maybe).
 	var bootstrapIP = func() string {
 		addr := slConfig.StateServerAddresses[0]
@@ -167,13 +196,16 @@
 
 	t := template.New("")
 	t.Funcs(template.FuncMap{
-		"logfileName": func() string { return slConfig.LogFileName },
-		"bootstrapIP": bootstrapIP,
-		"logfilePath": logFilePath,
-		"portNumber":  func() int { return slConfig.Port },
-		"logDir":      func() string { return slConfig.LogDir },
-		"namespace":   func() string { return slConfig.Namespace },
-		"tagStart":    func() int { return tagOffset + len(slConfig.Namespace) },
+		"logfileName":   func() string { return slConfig.LogFileName },
+		"bootstrapIP":   bootstrapIP,
+		"logfilePath":   logFilePath,
+		"portNumber":    func() int { return slConfig.Port },
+		"logDir":        func() string { return slConfig.LogDir },
+		"namespace":     func() string { return slConfig.Namespace },
+		"tagStart":      func() int { return tagOffset + len(slConfig.Namespace) },
+		"tlsCACertPath": func() string { return slConfig.TLSCACertPath },
+		"tlsCertPath":   func() string { return slConfig.TLSCertPath },
+		"tlsKeyPath":    func() string { return slConfig.TLSKeyPath },
 	})
 
 	// Process the rsyslog config template and echo to the conf file.

=== modified file 'log/syslog/config_test.go'
--- log/syslog/config_test.go	2014-02-13 05:22:01 +0000
+++ log/syslog/config_test.go	2014-02-25 14:03:09 +0000
@@ -43,12 +43,18 @@
 
 func (s *SyslogConfigSuite) TestAccumulateConfigRender(c *gc.C) {
 	syslogConfigRenderer := syslog.NewAccumulateConfig("some-machine", 8888, "")
+	syslogConfigRenderer.TLSCACertPath = "/var/log/juju/ca.pem"
+	syslogConfigRenderer.TLSCertPath = "/var/log/juju/cert.pem"
+	syslogConfigRenderer.TLSKeyPath = "/var/log/juju/key.pem"
 	s.assertRsyslogConfigContents(
 		c, syslogConfigRenderer, syslogtesting.ExpectedAccumulateSyslogConf(c, "some-machine", "", 8888))
 }
 
 func (s *SyslogConfigSuite) TestAccumulateConfigWrite(c *gc.C) {
 	syslogConfigRenderer := syslog.NewAccumulateConfig("some-machine", 8888, "")
+	syslogConfigRenderer.TLSCACertPath = "/var/log/juju/ca.pem"
+	syslogConfigRenderer.TLSCertPath = "/var/log/juju/cert.pem"
+	syslogConfigRenderer.TLSKeyPath = "/var/log/juju/key.pem"
 	syslogConfigRenderer.ConfigDir = s.configDir
 	syslogConfigRenderer.ConfigFileName = "rsyslog.conf"
 	s.assertRsyslogConfigPath(c, syslogConfigRenderer)
@@ -61,24 +67,30 @@
 
 func (s *SyslogConfigSuite) TestAccumulateConfigRenderWithNamespace(c *gc.C) {
 	syslogConfigRenderer := syslog.NewAccumulateConfig("some-machine", 8888, "namespace")
+	syslogConfigRenderer.TLSCACertPath = "/var/log/juju/ca.pem"
+	syslogConfigRenderer.TLSCertPath = "/var/log/juju/cert.pem"
+	syslogConfigRenderer.TLSKeyPath = "/var/log/juju/key.pem"
 	s.assertRsyslogConfigContents(
 		c, syslogConfigRenderer, syslogtesting.ExpectedAccumulateSyslogConf(c, "some-machine", "namespace", 8888))
 }
 
 func (s *SyslogConfigSuite) TestForwardConfigRender(c *gc.C) {
 	syslogConfigRenderer := syslog.NewForwardConfig("some-machine", 999, "", []string{"server"})
+	syslogConfigRenderer.TLSCACertPath = "/var/log/juju/ca.pem"
 	s.assertRsyslogConfigContents(
 		c, syslogConfigRenderer, syslogtesting.ExpectedForwardSyslogConf(c, "some-machine", "", 999))
 }
 
 func (s *SyslogConfigSuite) TestForwardConfigRenderWithNamespace(c *gc.C) {
 	syslogConfigRenderer := syslog.NewForwardConfig("some-machine", 999, "namespace", []string{"server"})
+	syslogConfigRenderer.TLSCACertPath = "/var/log/juju/ca.pem"
 	s.assertRsyslogConfigContents(
 		c, syslogConfigRenderer, syslogtesting.ExpectedForwardSyslogConf(c, "some-machine", "namespace", 999))
 }
 
 func (s *SyslogConfigSuite) TestForwardConfigWrite(c *gc.C) {
 	syslogConfigRenderer := syslog.NewForwardConfig("some-machine", 999, "", []string{"server"})
+	syslogConfigRenderer.TLSCACertPath = "/var/log/juju/ca.pem"
 	syslogConfigRenderer.ConfigDir = s.configDir
 	syslogConfigRenderer.ConfigFileName = "rsyslog.conf"
 	s.assertRsyslogConfigPath(c, syslogConfigRenderer)

=== modified file 'log/syslog/testing/syslogconf.go'
--- log/syslog/testing/syslogconf.go	2014-02-13 05:41:33 +0000
+++ log/syslog/testing/syslogconf.go	2014-02-25 14:03:09 +0000
@@ -20,15 +20,21 @@
 $InputFileStateFile {{machine}}{{namespace}}
 $InputRunFileMonitor
 
-$ModLoad imudp
-$UDPServerRun {{port}}
+$ModLoad imtcp
+$DefaultNetstreamDriver gtls
+$DefaultNetstreamDriverCAFile /var/log/juju/ca.pem
+$DefaultNetstreamDriverCertFile /var/log/juju/cert.pem
+$DefaultNetstreamDriverKeyFile /var/log/juju/key.pem
+$InputTCPServerStreamDriverAuthMode anon
+$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
+$InputTCPServerRun {{port}}
 
 # Messages received from remote rsyslog machines have messages prefixed with a space,
 # so add one in for local messages too if needed.
 $template JujuLogFormat{{namespace}},"%syslogtag:{{offset}}:$%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
 
 $FileCreateMode 0644
-:syslogtag, startswith, "juju{{namespace}}-" /var/log/juju{{namespace}}/all-machines.log;JujuLogFormat{{namespace}}
+:syslogtag, startswith, "juju{{namespace}}-" /var/log/juju/all-machines.log;JujuLogFormat{{namespace}}
 & ~
 $FileCreateMode 0640
 `
@@ -55,6 +61,12 @@
 var expectedForwardSyslogConfTemplate = `
 $ModLoad imfile
 
+# Enable reliable forwarding.
+$ActionQueueType LinkedList
+$ActionQueueFileName {{machine}}{{namespace}}
+$ActionResumeRetryCount -1
+$ActionQueueSaveOnShutdown on
+
 $InputFilePersistStateInterval 50
 $InputFilePollInterval 5
 $InputFileName /var/log/juju/{{machine}}.log
@@ -62,9 +74,14 @@
 $InputFileStateFile {{machine}}{{namespace}}
 $InputRunFileMonitor
 
+$DefaultNetstreamDriver gtls
+$DefaultNetstreamDriverCAFile /var/log/juju/ca.pem
+$ActionSendStreamDriverAuthMode anon
+$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
+
 $template LongTagForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%"
 
-:syslogtag, startswith, "juju{{namespace}}-" @server:{{port}};LongTagForwardFormat
+:syslogtag, startswith, "juju{{namespace}}-" @@server:{{port}};LongTagForwardFormat
 & ~
 `
 

=== modified file 'provider/azure/customdata_test.go'
--- provider/azure/customdata_test.go	2014-01-23 05:21:21 +0000
+++ provider/azure/customdata_test.go	2014-02-25 14:03:09 +0000
@@ -33,7 +33,6 @@
 		DataDir:            environs.DataDir,
 		LogDir:             environs.LogDir,
 		CloudInitOutputLog: environs.CloudInitOutputLog,
-		RsyslogConfPath:    environs.RsyslogConfPath,
 		Tools:              &tools.Tools{URL: "file://" + c.MkDir()},
 		StateInfo: &state.Info{
 			CACert:   []byte(testing.CACert),
@@ -46,7 +45,6 @@
 			Addrs:  []string{"127.0.0.1:123"},
 			Tag:    names.MachineTag(machineID),
 		},
-		SyslogPort:              2345,
 		MachineAgentServiceName: "jujud-machine-0",
 	}
 }

=== modified file 'provider/local/config.go'
--- provider/local/config.go	2014-02-12 02:56:13 +0000
+++ provider/local/config.go	2014-02-25 14:03:09 +0000
@@ -107,7 +107,6 @@
 	for _, dirname := range []string{
 		c.storageDir(),
 		c.mongoDir(),
-		c.logDir(),
 	} {
 		logger.Tracef("creating directory %s", dirname)
 		if err := os.MkdirAll(dirname, 0755); err != nil {

=== modified file 'provider/local/environ.go'
--- provider/local/environ.go	2014-02-20 08:23:40 +0000
+++ provider/local/environ.go	2014-02-25 14:03:09 +0000
@@ -138,9 +138,8 @@
 	mcfg := environs.NewBootstrapMachineConfig(stateFileURL, privateKey)
 	mcfg.Tools = selectedTools[0]
 	mcfg.DataDir = env.config.rootDir()
-	mcfg.LogDir = env.config.logDir()
-	mcfg.RsyslogConfPath = env.rsyslogConfPath()
-	mcfg.CloudInitOutputLog = filepath.Join(mcfg.LogDir, "cloud-init-output.log")
+	mcfg.LogDir = fmt.Sprintf("/var/log/juju-%s", env.config.namespace())
+	mcfg.CloudInitOutputLog = filepath.Join(env.config.logDir(), "cloud-init-output.log")
 	mcfg.DisablePackageCommands = true
 	mcfg.MachineAgentServiceName = env.machineAgentServiceName()
 	mcfg.MongoServiceName = env.mongoServiceName()
@@ -162,10 +161,14 @@
 	// Also, we leave the old all-machines.log file in
 	// /var/log/juju-{{namespace}} until we start the environment again. So
 	// potentially remove it at the start of the cloud-init.
-	logfile := fmt.Sprintf("/var/log/juju-%s/all-machines.log", env.config.namespace())
+	os.RemoveAll(env.config.logDir())
+	os.MkdirAll(env.config.logDir(), 0755)
 	cloudcfg.AddScripts(
-		fmt.Sprintf("[ -f %s ] && rm %s", logfile, logfile),
-		fmt.Sprintf("ln -s %s %s/", logfile, env.config.logDir()))
+		fmt.Sprintf("rm -fr %s", mcfg.LogDir),
+		fmt.Sprintf("mkdir -p %s", mcfg.LogDir),
+		fmt.Sprintf("rm -f /var/spool/rsyslog/machine-0-%s", env.config.namespace()),
+		fmt.Sprintf("ln -s %s/all-machines.log %s/", mcfg.LogDir, env.config.logDir()),
+		fmt.Sprintf("ln -s %s/machine-0.log %s/", env.config.logDir(), mcfg.LogDir))
 	if err := cloudinit.ConfigureJuju(mcfg, cloudcfg); err != nil {
 		return err
 	}

=== modified file 'provider/local/environprovider.go'
--- provider/local/environprovider.go	2014-02-18 01:29:47 +0000
+++ provider/local/environprovider.go	2014-02-25 14:03:09 +0000
@@ -101,9 +101,10 @@
 	if err != nil {
 		return nil, err
 	}
+	attrs := make(map[string]interface{})
+
 	// If the user has specified no values for any of the three normal
 	// proxies, then look in the environment and set them.
-	attrs := make(map[string]interface{})
 	setIfNotBlank := func(key, value string) {
 		if value != "" {
 			attrs[key] = value
@@ -130,13 +131,11 @@
 		setIfNotBlank("apt-https-proxy", proxy.Https)
 		setIfNotBlank("apt-ftp-proxy", proxy.Ftp)
 	}
-	if len(attrs) > 0 {
-		cfg, err = cfg.Apply(attrs)
-		if err != nil {
-			return nil, err
-		}
-	}
 
+	cfg, err = cfg.Apply(attrs)
+	if err != nil {
+		return nil, err
+	}
 	return p.Open(cfg)
 }
 

=== modified file 'state/address.go'
--- state/address.go	2014-01-22 19:28:08 +0000
+++ state/address.go	2014-02-25 14:03:09 +0000
@@ -122,7 +122,6 @@
 type DeployerConnectionValues struct {
 	StateAddresses []string
 	APIAddresses   []string
-	SyslogPort     int
 }
 
 // DeployerConnectionInfo returns the address information necessary for the deployer.
@@ -139,6 +138,5 @@
 	return &DeployerConnectionValues{
 		StateAddresses: appendPort(addrs, config.StatePort()),
 		APIAddresses:   appendPort(addrs, config.APIPort()),
-		SyslogPort:     config.SyslogPort(),
 	}, nil
 }

=== modified file 'state/api/params/params.go'
--- state/api/params/params.go	2014-01-31 03:36:58 +0000
+++ state/api/params/params.go	2014-02-25 14:03:09 +0000
@@ -513,7 +513,6 @@
 	ProviderType            string
 	AuthorizedKeys          string
 	SSLHostnameVerification bool
-	SyslogPort              int
 	Proxy                   osenv.ProxySettings
 	AptProxy                osenv.ProxySettings
 }
@@ -562,10 +561,14 @@
 type DeployerConnectionValues struct {
 	StateAddresses []string
 	APIAddresses   []string
-	SyslogPort     int
 }
 
 // StatusParams holds parameters for the Status call.
 type StatusParams struct {
 	Patterns []string
 }
+
+// SetRsyslogCertParams holds parameters for the SetRsyslogCert call.
+type SetRsyslogCertParams struct {
+	CACert []byte
+}

=== modified file 'state/api/provisioner/provisioner_test.go'
--- state/api/provisioner/provisioner_test.go	2014-02-14 11:33:57 +0000
+++ state/api/provisioner/provisioner_test.go	2014-02-25 14:03:09 +0000
@@ -358,7 +358,6 @@
 	c.Assert(result.ProviderType, gc.Equals, "dummy")
 	c.Assert(result.AuthorizedKeys, gc.Equals, coretesting.FakeAuthKeys)
 	c.Assert(result.SSLHostnameVerification, jc.IsTrue)
-	c.Assert(result.SyslogPort, gc.Equals, 2345)
 }
 
 func (s *provisionerSuite) TestCACert(c *gc.C) {

=== added directory 'state/api/rsyslog'
=== added file 'state/api/rsyslog/package_test.go'
--- state/api/rsyslog/package_test.go	1970-01-01 00:00:00 +0000
+++ state/api/rsyslog/package_test.go	2014-02-25 14:03:09 +0000
@@ -0,0 +1,14 @@
+// Copyright 2014 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package rsyslog_test
+
+import (
+	stdtesting "testing"
+
+	"launchpad.net/juju-core/testing"
+)
+
+func TestAll(t *stdtesting.T) {
+	testing.MgoTestPackage(t)
+}

=== added file 'state/api/rsyslog/rsyslog.go'
--- state/api/rsyslog/rsyslog.go	1970-01-01 00:00:00 +0000
+++ state/api/rsyslog/rsyslog.go	2014-02-25 14:03:09 +0000
@@ -0,0 +1,44 @@
+// Copyright 2014 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package rsyslog
+
+import (
+	"launchpad.net/juju-core/state/api/base"
+	"launchpad.net/juju-core/state/api/common"
+	"launchpad.net/juju-core/state/api/params"
+)
+
+const rsyslogAPI = "Rsyslog"
+
+// State provides access to the Rsyslog API facade.
+type State struct {
+	*common.EnvironWatcher
+	caller base.Caller
+}
+
+// NewState creates a new client-side Rsyslog facade.
+func NewState(caller base.Caller) *State {
+	return &State{
+		EnvironWatcher: common.NewEnvironWatcher(rsyslogAPI, caller),
+		caller:         caller,
+	}
+}
+
+// SetRsyslogCert sets the rsyslog CA certificate,
+// which is used by clients to verify the server's
+// identity and establish a TLS session.
+func (st *State) SetRsyslogCert(caCert []byte) error {
+	var result params.ErrorResult
+	args := params.SetRsyslogCertParams{
+		CACert: caCert,
+	}
+	err := st.caller.Call(rsyslogAPI, "", "SetRsyslogCert", args, &result)
+	if err != nil {
+		return err
+	}
+	if result.Error != nil {
+		return result.Error
+	}
+	return nil
+}

=== added file 'state/api/rsyslog/rsyslog_test.go'
--- state/api/rsyslog/rsyslog_test.go	1970-01-01 00:00:00 +0000
+++ state/api/rsyslog/rsyslog_test.go	2014-02-25 14:03:09 +0000
@@ -0,0 +1,33 @@
+// Copyright 2014 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package rsyslog_test
+
+import (
+	gc "launchpad.net/gocheck"
+
+	jujutesting "launchpad.net/juju-core/juju/testing"
+	commontesting "launchpad.net/juju-core/state/api/common/testing"
+)
+
+type rsyslogSuite struct {
+	jujutesting.JujuConnSuite
+	*commontesting.EnvironWatcherTest
+}
+
+var _ = gc.Suite(&rsyslogSuite{})
+
+func (s *rsyslogSuite) SetUpTest(c *gc.C) {
+	s.JujuConnSuite.SetUpTest(c)
+
+	stateAPI, _ := s.OpenAPIAsNewMachine(c)
+	rsyslogAPI := stateAPI.Rsyslog()
+	c.Assert(rsyslogAPI, gc.NotNil)
+
+	s.EnvironWatcherTest = commontesting.NewEnvironWatcherTest(
+		rsyslogAPI,
+		s.State,
+		s.BackingState,
+		commontesting.NoSecrets,
+	)
+}

=== modified file 'state/api/state.go'
--- state/api/state.go	2014-02-18 00:15:30 +0000
+++ state/api/state.go	2014-02-25 14:03:09 +0000
@@ -14,6 +14,7 @@
 	"launchpad.net/juju-core/state/api/machiner"
 	"launchpad.net/juju-core/state/api/params"
 	"launchpad.net/juju-core/state/api/provisioner"
+	"launchpad.net/juju-core/state/api/rsyslog"
 	"launchpad.net/juju-core/state/api/uniter"
 	"launchpad.net/juju-core/state/api/upgrader"
 )
@@ -99,3 +100,8 @@
 func (st *State) CharmRevisionUpdater() *charmrevisionupdater.State {
 	return charmrevisionupdater.NewState(st)
 }
+
+// Rsyslog returns access to the Rsyslog API
+func (st *State) Rsyslog() *rsyslog.State {
+	return rsyslog.NewState(st)
+}

=== modified file 'state/apiserver/deployer/deployer.go'
--- state/apiserver/deployer/deployer.go	2014-01-21 15:54:31 +0000
+++ state/apiserver/deployer/deployer.go	2014-02-25 14:03:09 +0000
@@ -76,7 +76,6 @@
 		result = params.DeployerConnectionValues{
 			StateAddresses: info.StateAddresses,
 			APIAddresses:   info.APIAddresses,
-			SyslogPort:     info.SyslogPort,
 		}
 	}
 	return result, err

=== modified file 'state/apiserver/provisioner/provisioner.go'
--- state/apiserver/provisioner/provisioner.go	2014-01-31 03:36:58 +0000
+++ state/apiserver/provisioner/provisioner.go	2014-02-25 14:03:09 +0000
@@ -196,7 +196,6 @@
 	result.ProviderType = config.Type()
 	result.AuthorizedKeys = config.AuthorizedKeys()
 	result.SSLHostnameVerification = config.SSLHostnameVerification()
-	result.SyslogPort = config.SyslogPort()
 	result.Proxy = config.ProxySettings()
 	result.AptProxy = config.AptProxySettings()
 	return result, nil

=== modified file 'state/apiserver/provisioner/provisioner_test.go'
--- state/apiserver/provisioner/provisioner_test.go	2014-02-14 11:33:57 +0000
+++ state/apiserver/provisioner/provisioner_test.go	2014-02-25 14:03:09 +0000
@@ -700,7 +700,6 @@
 	c.Check(results.ProviderType, gc.Equals, "dummy")
 	c.Check(results.AuthorizedKeys, gc.Equals, coretesting.FakeAuthKeys)
 	c.Check(results.SSLHostnameVerification, jc.IsTrue)
-	c.Check(results.SyslogPort, gc.Equals, 2345)
 	c.Check(results.Proxy, gc.DeepEquals, expectedProxy)
 	c.Check(results.AptProxy, gc.DeepEquals, expectedProxy)
 }

=== modified file 'state/apiserver/root.go'
--- state/apiserver/root.go	2014-02-20 03:12:23 +0000
+++ state/apiserver/root.go	2014-02-25 14:03:09 +0000
@@ -24,6 +24,7 @@
 	loggerapi "launchpad.net/juju-core/state/apiserver/logger"
 	"launchpad.net/juju-core/state/apiserver/machine"
 	"launchpad.net/juju-core/state/apiserver/provisioner"
+	"launchpad.net/juju-core/state/apiserver/rsyslog"
 	"launchpad.net/juju-core/state/apiserver/uniter"
 	"launchpad.net/juju-core/state/apiserver/upgrader"
 	"launchpad.net/juju-core/state/multiwatcher"
@@ -182,6 +183,17 @@
 	return environment.NewEnvironmentAPI(r.srv.state, r.resources, r)
 }
 
+// Rsyslog returns an object that provides access to the Rsyslog API
+// facade. The id argument is reserved for future use and currently needs to
+// be empty.
+func (r *srvRoot) Rsyslog(id string) (*rsyslog.RsyslogAPI, error) {
+	if id != "" {
+		// Safeguard id for possible future use.
+		return nil, common.ErrBadId
+	}
+	return rsyslog.NewRsyslogAPI(r.srv.state, r.resources, r)
+}
+
 // Logger returns an object that provides access to the Logger API facade.
 // The id argument is reserved for future use and must be empty.
 func (r *srvRoot) Logger(id string) (*loggerapi.LoggerAPI, error) {

=== added directory 'state/apiserver/rsyslog'
=== added file 'state/apiserver/rsyslog/package_test.go'
--- state/apiserver/rsyslog/package_test.go	1970-01-01 00:00:00 +0000
+++ state/apiserver/rsyslog/package_test.go	2014-02-25 14:03:09 +0000
@@ -0,0 +1,14 @@
+// Copyright 2014 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package rsyslog_test
+
+import (
+	stdtesting "testing"
+
+	"launchpad.net/juju-core/testing"
+)
+
+func TestAll(t *stdtesting.T) {
+	testing.MgoTestPackage(t)
+}

=== added file 'state/apiserver/rsyslog/rsyslog.go'
--- state/apiserver/rsyslog/rsyslog.go	1970-01-01 00:00:00 +0000
+++ state/apiserver/rsyslog/rsyslog.go	2014-02-25 14:03:09 +0000
@@ -0,0 +1,56 @@
+// Copyright 2014 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package rsyslog
+
+import (
+	"launchpad.net/juju-core/cert"
+	"launchpad.net/juju-core/state"
+	"launchpad.net/juju-core/state/api/params"
+	"launchpad.net/juju-core/state/apiserver/common"
+)
+
+// RsyslogAPI implements the API used by the rsyslog worker.
+type RsyslogAPI struct {
+	*common.EnvironWatcher
+	st        *state.State
+	canModify bool
+}
+
+// NewRsyslogAPI creates a new instance of the Rsyslog API.
+func NewRsyslogAPI(st *state.State, resources *common.Resources, authorizer common.Authorizer) (*RsyslogAPI, error) {
+	// Can always watch for environ changes.
+	getCanWatch := common.AuthAlways(true)
+	// Does not get the secrets.
+	getCanReadSecrets := common.AuthAlways(false)
+	return &RsyslogAPI{
+		EnvironWatcher: common.NewEnvironWatcher(st, resources, getCanWatch, getCanReadSecrets),
+		st:             st,
+		canModify:      authorizer.AuthEnvironManager(),
+	}, nil
+}
+
+func (api *RsyslogAPI) SetRsyslogCert(args params.SetRsyslogCertParams) (params.ErrorResult, error) {
+	var result params.ErrorResult
+	if !api.canModify {
+		result.Error = common.ServerError(common.ErrBadCreds)
+		return result, nil
+	}
+	if _, err := cert.ParseCert(args.CACert); err != nil {
+		result.Error = common.ServerError(err)
+		return result, nil
+	}
+	old, err := api.st.EnvironConfig()
+	if err != nil {
+		return params.ErrorResult{}, err
+	}
+	cfg, err := old.Apply(map[string]interface{}{"rsyslog-ca-cert": string(args.CACert)})
+	if err != nil {
+		result.Error = common.ServerError(err)
+	} else {
+		if err := api.st.SetEnvironConfig(cfg, old); err != nil {
+			result.Error = common.ServerError(err)
+		}
+	}
+	return result, nil
+}

=== added file 'state/apiserver/rsyslog/rsyslog_test.go'
--- state/apiserver/rsyslog/rsyslog_test.go	1970-01-01 00:00:00 +0000
+++ state/apiserver/rsyslog/rsyslog_test.go	2014-02-25 14:03:09 +0000
@@ -0,0 +1,48 @@
+// Copyright 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package rsyslog_test
+
+import (
+	gc "launchpad.net/gocheck"
+
+	"launchpad.net/juju-core/juju/testing"
+	"launchpad.net/juju-core/state/apiserver/common"
+	commontesting "launchpad.net/juju-core/state/apiserver/common/testing"
+	"launchpad.net/juju-core/state/apiserver/rsyslog"
+	apiservertesting "launchpad.net/juju-core/state/apiserver/testing"
+)
+
+type rsyslogSuite struct {
+	testing.JujuConnSuite
+	*commontesting.EnvironWatcherTest
+	authorizer apiservertesting.FakeAuthorizer
+	resources  *common.Resources
+}
+
+var _ = gc.Suite(&rsyslogSuite{})
+
+func (s *rsyslogSuite) SetUpTest(c *gc.C) {
+	s.JujuConnSuite.SetUpTest(c)
+	s.authorizer = apiservertesting.FakeAuthorizer{
+		LoggedIn:       true,
+		EnvironManager: true,
+	}
+	s.resources = common.NewResources()
+	api, err := rsyslog.NewRsyslogAPI(s.State, s.resources, s.authorizer)
+	c.Assert(err, gc.IsNil)
+	s.EnvironWatcherTest = commontesting.NewEnvironWatcherTest(
+		api, s.State, s.resources, commontesting.NoSecrets)
+}
+
+func (s *rsyslogSuite) TestSetRsyslogCert(c *gc.C) {
+	st := s.APIState.Rsyslog()
+	err := st.SetRsyslogCert(nil)
+	// TODO(axw) finish me. this should fail due to being an invalid cert
+	c.Assert(err, gc.IsNil)
+}
+
+func (s *rsyslogSuite) TestSetRsyslogCertPerms(c *gc.C) {
+	// TODO(axw) SetRsyslogCert requires that the
+	// caller is a state server.
+}

=== modified file 'upgrades/export_test.go'
--- upgrades/export_test.go	2014-02-13 05:22:01 +0000
+++ upgrades/export_test.go	2014-02-25 14:03:09 +0000
@@ -10,6 +10,4 @@
 	// 118 upgrade functions
 	StepsFor118                          = stepsFor118
 	EnsureLockDirExistsAndUbuntuWritable = ensureLockDirExistsAndUbuntuWritable
-	UpgradeStateServerRsyslogConfig      = upgradeStateServerRsyslogConfig
-	UpgradeHostMachineRsyslogConfig      = upgradeHostMachineRsyslogConfig
 )

=== removed file 'upgrades/rsyslogconf.go'
--- upgrades/rsyslogconf.go	2014-02-14 00:48:23 +0000
+++ upgrades/rsyslogconf.go	1970-01-01 00:00:00 +0000
@@ -1,50 +0,0 @@
-// Copyright 2014 Canonical Ltd.
-// Licensed under the AGPLv3, see LICENCE file for details.
-
-package upgrades
-
-import (
-	"launchpad.net/juju-core/agent"
-	"launchpad.net/juju-core/environs"
-	"launchpad.net/juju-core/log/syslog"
-)
-
-func confParams(context Context) (machineTag, namespace string, port int, err error) {
-	namespace = context.AgentConfig().Value(agent.Namespace)
-	machineTag = context.AgentConfig().Tag()
-
-	environment := context.APIState().Environment()
-	config, err := environment.EnvironConfig()
-	if err != nil {
-		return "", "", 0, err
-	}
-	port = config.SyslogPort()
-	return machineTag, namespace, port, err
-}
-
-// upgradeStateServerRsyslogConfig upgrades a rsuslog config file on a state server.
-func upgradeStateServerRsyslogConfig(context Context) (err error) {
-	machineTag, namespace, syslogPort, err := confParams(context)
-	configRenderer := syslog.NewAccumulateConfig(machineTag, syslogPort, namespace)
-	data, err := configRenderer.Render()
-	if err != nil {
-		return nil
-	}
-	return WriteReplacementFile(environs.RsyslogConfPath, []byte(data), 0644)
-}
-
-// upgradeHostMachineRsyslogConfig upgrades a rsuslog config file on a host machine.
-func upgradeHostMachineRsyslogConfig(context Context) (err error) {
-	machineTag, namespace, syslogPort, err := confParams(context)
-	addr, err := context.AgentConfig().APIAddresses()
-	if err != nil {
-		return err
-	}
-
-	configRenderer := syslog.NewForwardConfig(machineTag, syslogPort, namespace, addr)
-	data, err := configRenderer.Render()
-	if err != nil {
-		return nil
-	}
-	return WriteReplacementFile(environs.RsyslogConfPath, []byte(data), 0644)
-}

=== removed file 'upgrades/rsyslogconf_test.go'
--- upgrades/rsyslogconf_test.go	2014-02-14 11:39:37 +0000
+++ upgrades/rsyslogconf_test.go	1970-01-01 00:00:00 +0000
@@ -1,72 +0,0 @@
-// Copyright 2014 Canonical Ltd.
-// Licensed under the AGPLv3, see LICENCE file for details.
-
-package upgrades_test
-
-import (
-	"io/ioutil"
-	"path/filepath"
-
-	gc "launchpad.net/gocheck"
-
-	"launchpad.net/juju-core/environs"
-	jujutesting "launchpad.net/juju-core/juju/testing"
-	syslogtesting "launchpad.net/juju-core/log/syslog/testing"
-	"launchpad.net/juju-core/state"
-	"launchpad.net/juju-core/upgrades"
-)
-
-type rsyslogSuite struct {
-	jujutesting.JujuConnSuite
-
-	syslogPath string
-	ctx        upgrades.Context
-}
-
-var _ = gc.Suite(&rsyslogSuite{})
-
-func (s *rsyslogSuite) SetUpTest(c *gc.C) {
-	s.JujuConnSuite.SetUpTest(c)
-
-	dir := c.MkDir()
-	s.syslogPath = filepath.Join(dir, "fakesyslog.conf")
-	s.PatchValue(&environs.RsyslogConfPath, s.syslogPath)
-
-	apiState, _ := s.OpenAPIAsNewMachine(c, state.JobManageEnviron)
-	s.ctx = &mockContext{
-		agentConfig: &mockAgentConfig{
-			tag:          "machine-tag",
-			namespace:    "namespace",
-			apiAddresses: []string{"server:1234"},
-		},
-		apiState: apiState,
-	}
-}
-
-func (s *rsyslogSuite) TestStateServerUpgrade(c *gc.C) {
-	err := upgrades.UpgradeStateServerRsyslogConfig(s.ctx)
-	c.Assert(err, gc.IsNil)
-
-	data, err := ioutil.ReadFile(s.syslogPath)
-	c.Assert(err, gc.IsNil)
-	c.Assert(string(data), gc.Equals, syslogtesting.ExpectedAccumulateSyslogConf(c, "machine-tag", "namespace", 2345))
-}
-
-func (s *rsyslogSuite) TestStateServerUpgradeIdempotent(c *gc.C) {
-	s.TestStateServerUpgrade(c)
-	s.TestStateServerUpgrade(c)
-}
-
-func (s *rsyslogSuite) TestHostMachineUpgrade(c *gc.C) {
-	err := upgrades.UpgradeHostMachineRsyslogConfig(s.ctx)
-	c.Assert(err, gc.IsNil)
-
-	data, err := ioutil.ReadFile(s.syslogPath)
-	c.Assert(err, gc.IsNil)
-	c.Assert(string(data), gc.Equals, syslogtesting.ExpectedForwardSyslogConf(c, "machine-tag", "namespace", 2345))
-}
-
-func (s *rsyslogSuite) TestHostServerUpgradeIdempotent(c *gc.C) {
-	s.TestHostMachineUpgrade(c)
-	s.TestHostMachineUpgrade(c)
-}

=== modified file 'upgrades/steps118.go'
--- upgrades/steps118.go	2014-02-16 23:35:10 +0000
+++ upgrades/steps118.go	2014-02-25 14:03:09 +0000
@@ -11,15 +11,5 @@
 			targets:     []Target{HostMachine},
 			run:         ensureLockDirExistsAndUbuntuWritable,
 		},
-		&upgradeStep{
-			description: "upgrade rsyslog config file on state server",
-			targets:     []Target{StateServer},
-			run:         upgradeStateServerRsyslogConfig,
-		},
-		&upgradeStep{
-			description: "upgrade rsyslog config file on host machine",
-			targets:     []Target{HostMachine},
-			run:         upgradeHostMachineRsyslogConfig,
-		},
 	}
 }

=== modified file 'upgrades/steps118_test.go'
--- upgrades/steps118_test.go	2014-02-13 05:22:01 +0000
+++ upgrades/steps118_test.go	2014-02-25 14:03:09 +0000
@@ -18,12 +18,10 @@
 
 var expectedSteps = []string{
 	"make $DATADIR/locks owned by ubuntu:ubuntu",
-	"upgrade rsyslog config file on state server",
-	"upgrade rsyslog config file on host machine",
 }
 
 func (s *steps118Suite) TestUpgradeOperationsContent(c *gc.C) {
 	upgradeSteps := upgrades.StepsFor118()
-	c.Assert(upgradeSteps, gc.HasLen, 3)
+	c.Assert(upgradeSteps, gc.HasLen, 1)
 	assertExpectedSteps(c, upgradeSteps, expectedSteps)
 }

=== modified file 'worker/deployer/export_test.go'
--- worker/deployer/export_test.go	2013-11-25 03:52:09 +0000
+++ worker/deployer/export_test.go	2014-02-25 14:03:09 +0000
@@ -12,9 +12,8 @@
 
 func (*fakeAPI) ConnectionInfo() (params.DeployerConnectionValues, error) {
 	return params.DeployerConnectionValues{
-		[]string{"s1:123", "s2:123"},
-		[]string{"a1:123", "a2:123"},
-		2345,
+		StateAddresses: []string{"s1:123", "s2:123"},
+		APIAddresses:   []string{"a1:123", "a2:123"},
 	}, nil
 }
 

=== modified file 'worker/deployer/simple.go'
--- worker/deployer/simple.go	2014-02-12 07:24:47 +0000
+++ worker/deployer/simple.go	2014-02-25 14:03:09 +0000
@@ -54,9 +54,6 @@
 	// will be written. It is set for testing and left empty for production, in
 	// which case the system default is used, typically /etc/rsyslog.d
 	syslogConfigDir string
-
-	// syslogConfigPath is the full path name of the syslog conf file.
-	syslogConfigPath string
 }
 
 var _ Context = (*SimpleContext)(nil)
@@ -100,6 +97,7 @@
 	logger.Debugf("API addresses: %q", result.APIAddresses)
 	containerType := ctx.agentConfig.Value(agent.ContainerType)
 	namespace := ctx.agentConfig.Value(agent.Namespace)
+	// TODO(axw) record rsyslog config dir.
 	conf, err := agent.NewAgentConfig(
 		agent.AgentConfigParams{
 			DataDir:  dataDir,
@@ -125,18 +123,6 @@
 
 	// Install an upstart job that runs the unit agent.
 	logPath := path.Join(ctx.logDir, tag+".log")
-	syslogConfigRenderer := syslog.NewForwardConfig(tag, result.SyslogPort, namespace, result.StateAddresses)
-	syslogConfigRenderer.ConfigDir = ctx.syslogConfigDir
-	syslogConfigRenderer.ConfigFileName = fmt.Sprintf("26-juju-%s.conf", tag)
-	if err := syslogConfigRenderer.Write(); err != nil {
-		return err
-	}
-	ctx.syslogConfigPath = syslogConfigRenderer.ConfigFilePath()
-	if err := syslog.Restart(); err != nil {
-		logger.Warningf("installer: cannot restart syslog daemon: %v", err)
-	}
-	defer removeOnErr(&err, ctx.syslogConfigPath)
-
 	cmd := strings.Join([]string{
 		path.Join(toolsDir, "jujud"), "unit",
 		"--data-dir", dataDir,
@@ -190,9 +176,6 @@
 	if err := os.RemoveAll(agentDir); err != nil {
 		return err
 	}
-	if err := os.Remove(ctx.syslogConfigPath); err != nil && !os.IsNotExist(err) {
-		logger.Warningf("installer: cannot remove %q: %v", ctx.syslogConfigPath, err)
-	}
 	// Defer this so a failure here does not impede the cleanup (as in tests).
 	defer func() {
 		if err := syslog.Restart(); err != nil {

=== modified file 'worker/deployer/simple_test.go'
--- worker/deployer/simple_test.go	2013-11-21 03:22:49 +0000
+++ worker/deployer/simple_test.go	2014-02-25 14:03:09 +0000
@@ -198,12 +198,11 @@
 	return deployer.NewTestSimpleContext(config, fix.initDir, fix.logDir, fix.syslogConfigDir)
 }
 
-func (fix *SimpleToolsFixture) paths(tag string) (confPath, agentDir, toolsDir, syslogConfPath string) {
+func (fix *SimpleToolsFixture) paths(tag string) (confPath, agentDir, toolsDir string) {
 	confName := fmt.Sprintf("jujud-%s.conf", tag)
 	confPath = filepath.Join(fix.initDir, confName)
 	agentDir = agent.Dir(fix.dataDir, tag)
 	toolsDir = tools.ToolsDir(fix.dataDir, tag)
-	syslogConfPath = filepath.Join(fix.syslogConfigDir, fmt.Sprintf("26-juju-%s.conf", tag))
 	return
 }
 
@@ -225,7 +224,7 @@
 
 func (fix *SimpleToolsFixture) checkUnitInstalled(c *gc.C, name, password string) {
 	tag := names.UnitTag(name)
-	uconfPath, _, toolsDir, syslogConfPath := fix.paths(tag)
+	uconfPath, _, toolsDir := fix.paths(tag)
 	uconfData, err := ioutil.ReadFile(uconfPath)
 	c.Assert(err, gc.IsNil)
 	uconf := string(uconfData)
@@ -261,20 +260,12 @@
 	jujudData, err := ioutil.ReadFile(jujudPath)
 	c.Assert(err, gc.IsNil)
 	c.Assert(string(jujudData), gc.Equals, fakeJujud)
-
-	syslogConfData, err := ioutil.ReadFile(syslogConfPath)
-	c.Assert(err, gc.IsNil)
-	parts := strings.SplitN(name, "/", 2)
-	unitTag := fmt.Sprintf("unit-%s-%s", parts[0], parts[1])
-	expectedSyslogConfReplaced := fmt.Sprintf(expectedSyslogConf, unitTag, unitTag, unitTag)
-	c.Assert(string(syslogConfData), gc.Equals, expectedSyslogConfReplaced)
-
 }
 
 func (fix *SimpleToolsFixture) checkUnitRemoved(c *gc.C, name string) {
 	tag := names.UnitTag(name)
-	confPath, agentDir, toolsDir, syslogConfPath := fix.paths(tag)
-	for _, path := range []string{confPath, agentDir, toolsDir, syslogConfPath} {
+	confPath, agentDir, toolsDir := fix.paths(tag)
+	for _, path := range []string{confPath, agentDir, toolsDir} {
 		_, err := ioutil.ReadFile(path)
 		if err == nil {
 			c.Log("Warning: %q not removed as expected", path)

=== modified file 'worker/provisioner/kvm-broker.go'
--- worker/provisioner/kvm-broker.go	2014-01-31 03:36:58 +0000
+++ worker/provisioner/kvm-broker.go	2014-02-25 14:03:09 +0000
@@ -84,7 +84,6 @@
 		config.ProviderType,
 		config.AuthorizedKeys,
 		config.SSLHostnameVerification,
-		config.SyslogPort,
 		config.Proxy,
 		config.AptProxy,
 	); err != nil {

=== modified file 'worker/provisioner/lxc-broker.go'
--- worker/provisioner/lxc-broker.go	2014-01-31 03:36:58 +0000
+++ worker/provisioner/lxc-broker.go	2014-02-25 14:03:09 +0000
@@ -74,7 +74,6 @@
 		config.ProviderType,
 		config.AuthorizedKeys,
 		config.SSLHostnameVerification,
-		config.SyslogPort,
 		config.Proxy,
 		config.AptProxy,
 	); err != nil {

=== modified file 'worker/provisioner/lxc-broker_test.go'
--- worker/provisioner/lxc-broker_test.go	2014-02-14 11:33:57 +0000
+++ worker/provisioner/lxc-broker_test.go	2014-02-25 14:03:09 +0000
@@ -285,6 +285,5 @@
 	return params.ContainerConfig{
 		ProviderType:            "fake",
 		AuthorizedKeys:          coretesting.FakeAuthKeys,
-		SSLHostnameVerification: true,
-		SyslogPort:              2345}, nil
+		SSLHostnameVerification: true}, nil
 }

=== added directory 'worker/rsyslog'
=== added file 'worker/rsyslog/worker.go'
--- worker/rsyslog/worker.go	1970-01-01 00:00:00 +0000
+++ worker/rsyslog/worker.go	2014-02-25 14:03:09 +0000
@@ -0,0 +1,245 @@
+// Copyright 2014 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package rsyslog
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"os/user"
+	"path/filepath"
+	"strconv"
+	"time"
+
+	"github.com/errgo/errgo"
+	"github.com/loggo/loggo"
+
+	"launchpad.net/juju-core/agent"
+	"launchpad.net/juju-core/cert"
+	"launchpad.net/juju-core/log/syslog"
+	"launchpad.net/juju-core/names"
+	apirsyslog "launchpad.net/juju-core/state/api/rsyslog"
+	"launchpad.net/juju-core/state/api/watcher"
+	"launchpad.net/juju-core/utils"
+	"launchpad.net/juju-core/worker"
+)
+
+var logger = loggo.GetLogger("juju.worker.rsyslog")
+
+const rsyslogConfDir = "/etc/rsyslog.d"
+
+// RsyslogMode describes how to configure rsyslog.
+type RsyslogMode int
+
+const (
+	RsyslogModeInvalid RsyslogMode = iota
+	// RsyslogModeForwarding is the mode in which
+	// rsyslog will be configured to forward logging
+	// to state servers.
+	RsyslogModeForwarding
+	// RsyslogModeAccumulate is the mode in which
+	// rsyslog will be configured to accumulate logging
+	// from other machines into an "all-machines.log".
+	RsyslogModeAccumulate
+)
+
+// RsyslogConfigHandler implements worker.NotifyWatchHandler, watching
+// environment configuration changes and generating new rsyslog
+// configuration.
+type RsyslogConfigHandler struct {
+	st                     *apirsyslog.State
+	mode                   RsyslogMode
+	syslogConfig           *syslog.SyslogConfig
+	rsyslogConfPath        string
+	rsyslogGnutlsInstalled bool
+}
+
+var _ worker.NotifyWatchHandler = (*RsyslogConfigHandler)(nil)
+
+// NewRsyslogConfigWorker returns a worker.Worker that watches
+// for config changes and updates rsyslog configuration based
+// on changes.
+// NewRsyslogConfigWorker also returns the path to the rsyslog
+// configuration file that is written, so that the agent may
+// remove it on shutdown.
+func NewRsyslogConfigWorker(st *apirsyslog.State, config agent.Config, mode RsyslogMode) (worker.Worker, error) {
+	namespace := config.Value(agent.Namespace)
+	var syslogConfig *syslog.SyslogConfig
+	if mode == RsyslogModeAccumulate {
+		syslogConfig = syslog.NewAccumulateConfig(config.Tag(), 0, namespace)
+	} else {
+		addr, err := config.APIAddresses()
+		if err != nil {
+			return nil, err
+		}
+		syslogConfig = syslog.NewForwardConfig(config.Tag(), 0, namespace, addr)
+	}
+
+	// Historically only machine-0 includes the namespace in the log
+	// dir/file; for backwards compatibility we continue the tradition.
+	if config.Tag() != "machine-0" {
+		namespace = ""
+	}
+	kind, err := names.TagKind(config.Tag())
+	if err != nil {
+		return nil, err
+	}
+	if kind == names.MachineTagKind {
+		if namespace == "" {
+			syslogConfig.ConfigFileName = "25-juju.conf"
+		} else {
+			syslogConfig.ConfigFileName = fmt.Sprintf("25-juju-%s.conf", namespace)
+		}
+	} else {
+		syslogConfig.ConfigFileName = fmt.Sprintf("26-juju-%s.conf", config.Tag())
+	}
+
+	if namespace != "" {
+		syslogConfig.LogDir += "-" + namespace
+	}
+	worker := worker.NewNotifyWorker(&RsyslogConfigHandler{
+		st:           st,
+		mode:         mode,
+		syslogConfig: syslogConfig,
+	})
+	return worker, nil
+}
+
+func (h *RsyslogConfigHandler) SetUp() (watcher.NotifyWatcher, error) {
+	if h.mode == RsyslogModeAccumulate {
+		if err := h.ensureCertificates(); err != nil {
+			return nil, errgo.Annotate(err, "failed to write rsyslog certificates")
+		}
+	}
+	return h.st.WatchForEnvironConfigChanges()
+}
+
+func (h *RsyslogConfigHandler) TearDown() error {
+	if err := os.Remove(h.syslogConfig.ConfigFilePath()); err == nil {
+		syslog.Restart()
+	}
+	return nil
+}
+
+func (h *RsyslogConfigHandler) Handle() error {
+	if !h.rsyslogGnutlsInstalled {
+		if err := utils.AptGetInstall("rsyslog-gnutls"); err != nil {
+			// apt-get may fail if another process has the lock,
+			// so keep we'll just exit and try again next time.
+			return errgo.Annotate(err, "cannot install rsyslog-gnutls")
+		}
+		h.rsyslogGnutlsInstalled = true
+	}
+	cfg, err := h.st.EnvironConfig()
+	if err != nil {
+		return errgo.Annotate(err, "cannot get environ config")
+	}
+	h.syslogConfig.Port = cfg.SyslogPort()
+	rsyslogCACert := cfg.RsyslogCACert()
+	if rsyslogCACert == nil {
+		return nil
+	}
+	if h.mode == RsyslogModeForwarding {
+		h.syslogConfig.TLSCACertPath = h.caCertPath()
+		if err := writeFileAtomic(h.syslogConfig.TLSCACertPath, rsyslogCACert, 0644, 0, 0); err != nil {
+			return err
+		}
+	}
+	data, err := h.syslogConfig.Render()
+	if err != nil {
+		return errgo.Annotate(err, "failed to render rsyslog configuration file")
+	}
+	if err := writeFileAtomic(h.syslogConfig.ConfigFilePath(), []byte(data), 0644, 0, 0); err != nil {
+		return errgo.Annotate(err, "failed to write rsyslog configuration file")
+	}
+	logger.Debugf("Reloading rsyslog configuration")
+	if err := syslog.Restart(); err != nil {
+		logger.Errorf("failed to reload rsyslog configuration")
+		return err
+	}
+	return nil
+}
+
+func (h *RsyslogConfigHandler) caCertPath() string {
+	return filepath.Join(h.syslogConfig.LogDir, "ca-cert.pem")
+}
+
+// ensureCertificates ensures that a CA certificate,
+// server certificate, and private key exist in the log
+// directory, and writes them if not. The CA certificate
+// is entered into the environment configuration to be
+// picked up by other agents.
+func (h *RsyslogConfigHandler) ensureCertificates() error {
+	// We write ca-cert.pem last, after propagating into state.
+	// If it's there, then there's nothing to do. Otherwise,
+	// start over.
+	caCertPEM := h.caCertPath()
+	if _, err := os.Stat(caCertPEM); err == nil {
+		return nil
+	}
+
+	// Files must be chowned to syslog:adm.
+	syslogUser, err := user.Lookup("syslog")
+	if err != nil {
+		return err
+	}
+	syslogUid, err := strconv.Atoi(syslogUser.Uid)
+	if err != nil {
+		return err
+	}
+	syslogGid, err := strconv.Atoi(syslogUser.Gid)
+	if err != nil {
+		return err
+	}
+
+	// Generate a new CA and server cert/key pairs.
+	// The CA key will be discarded after the server
+	// cert has been generated.
+	expiry := time.Now().UTC().AddDate(10, 0, 0)
+	caCertPEMBytes, caKeyPEMBytes, err := cert.NewCA("rsyslog", expiry)
+	if err != nil {
+		return err
+	}
+	rsyslogCertPEMBytes, rsyslogKeyPEMBytes, err := cert.NewServer(caCertPEMBytes, caKeyPEMBytes, expiry, nil)
+	if err != nil {
+		return err
+	}
+
+	// Update the environment config with the CA cert,
+	// so clients can configure rsyslog.
+	if err := h.st.SetRsyslogCert(caCertPEMBytes); err != nil {
+		return err
+	}
+
+	// Write the certificates and key. The CA certificate must be written last for idempotency.
+	h.syslogConfig.TLSCACertPath = caCertPEM
+	h.syslogConfig.TLSCertPath = filepath.Join(h.syslogConfig.LogDir, "rsyslog-cert.pem")
+	h.syslogConfig.TLSKeyPath = filepath.Join(h.syslogConfig.LogDir, "rsyslog-key.pem")
+	for _, pair := range []struct {
+		path string
+		data []byte
+	}{
+		{h.syslogConfig.TLSCertPath, rsyslogCertPEMBytes},
+		{h.syslogConfig.TLSKeyPath, rsyslogKeyPEMBytes},
+		{h.syslogConfig.TLSCACertPath, caCertPEMBytes},
+	} {
+		if err := writeFileAtomic(pair.path, pair.data, 0600, syslogUid, syslogGid); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func writeFileAtomic(path string, data []byte, mode os.FileMode, uid, gid int) error {
+	temp := path + ".temp"
+	if err := ioutil.WriteFile(temp, data, mode); err != nil {
+		return err
+	}
+	if uid != -1 {
+		if err := os.Chown(temp, uid, gid); err != nil {
+			return err
+		}
+	}
+	return os.Rename(temp, path)
+}