Merge lp:~smoser/pyjuju/cloud-init-output-log into lp:pyjuju

[Scott Moser]

log output from cloud-init and its subprocess to a file

By default cloud-init only outputs to its stdout and stderr.
In the case of EC2, that is to a console that you can get at
with get-console-output.

However, that is less than convienent as it is delayed by
4 minutes, and on other platforms, the console output isn't so easily
available.

This will just log stdout and stderr to /var/log/cloud-init-output.log

[Kapil Thangavelu (hazmat)]

Excerpts from Scott Moser's message of Wed Aug 31 21:18:33 UTC 2011:
> Scott Moser has proposed merging lp:~smoser/ensemble/cloud-init-output-log into lp:ensemble.
>
> Requested reviews:
> Ensemble Team (ensemble)
>
> For more details, see:
> https://code.launchpad.net/~smoser/ensemble/cloud-init-output-log/+merge/73596
>
> log output from cloud-init and its subprocess to a file
>
> By default cloud-init only outputs to its stdout and stderr.
> In the case of EC2, that is to a console that you can get at
> with get-console-output.
>
> However, that is less than convienent as it is delayed by
> 4 minutes, and on other platforms, the console output isn't so easily
> available.
>
> This will just log stdout and stderr to /var/log/cloud-init-output.log
>

fwiw, this will probably cause some minor breakage of the tests in the tree as
they check cloud init formatting exactly.

[Clint Byrum (clint-fewbar)]

This breaks the test suite, possibly because it is quite rigid about what it expects to be fed to cloud-init.

[ERROR]
Traceback (most recent call last):
  File "/«BUILDDIR»/ensemble-0.5+bzr336/ensemble/lib/mocker.py", line 161, in test_method_wrapper
    self.mocker.verify()
  File "/«BUILDDIR»/ensemble-0.5+bzr336/ensemble/lib/mocker.py", line 579, in verify
    raise AssertionError(os.linesep.join(message))
exceptions.AssertionError: [Mocker] Unmet expectations:

=> proxy_m.callRemote('modify_system', 'smith', 'ks_meta', MATCH(<function verify at 0x32cb5f0>), 'TOKEN')
 - Performed fewer times than expected.

ensemble.providers.orchestra.tests.test_bootstrap.OrchestraBootstrapTest.test_cannot_modify_machine
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/trial/unittest.py", line 402, in _eb
    raise self.failureException(output)
twisted.trial.unittest.FailTest:
Expected: (<class 'ensemble.errors.ProviderError'>,)
Got:
[Failure instance: Traceback: <type 'exceptions.AssertionError'>: [Mocker] Unmet expectation:

=> not equal:
 a = '\nseed_d=/var/lib/cloud/seed/nocloud-net\nmkdir -p "$seed_d"\ncat > "$seed_d"/meta-data <<"EOF"\ninstance-id: winston-uid\nEOF\n\ncat > "$seed_d"/user-data <<"EOF"\n#cloud-config\napt-update: true\napt-upgrade: true\napt_sources:\n- {source: \'ppa:ensemble/ppa\'}\nmachine-data: {ensemble-zookeeper-hosts: \'localhost:2181\', machine-id: \'0\'}\noutput: {all: \'| tee -a /var/log/cloud-init-output.log\'}\npackages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws,\n python-zookeeper, bzr, default-jre-headless, zookeeper, zookeeperd]\nruncmd: [export ENSEMBLE_INSTANCE_ID=winston-uid, sudo apt-get -y install ensemble,\n sudo mkdir -p /var/lib/ensemble, sudo mkdir -p /var/log/ensemble, \'ensemble-admin\n initialize --instance-id=$ENSEMBLE_INSTANCE_ID --admin-identity="admin:qRBXC1ubEEUqRL6wcBhgmc9xkaY="\',\n \'ENSEMBLE_MACHINE_ID=0 ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.machine\n -n --logfile=/var/log/ensemble/machine-agent.log --pidfile=/var/run/ensemble/machine-agent.pid\',\n \'ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log\n --pidfile=/var/run/ensemble/provision-agent.pid\']\nssh_authorized_keys: [this-is-a-public-key]\n\nEOF\n'
 b = '\nseed_d=/var/lib/cloud/seed/nocloud-net\nmkdir -p "$seed_d"\ncat > "$seed_d"/meta-data <<"EOF"\ninstance-id: winston-uid\nEOF\n\ncat > "$seed_d"/user-data <<"EOF"\n#cloud-config\napt-update: true\napt-upgrade: true\napt_sources:\n- {source: \'ppa:ensemble/ppa\'}\nmachine-data: {ensemble-zookeeper-hosts: \'localhost:2181\', machine-id: \'0\'}\npackages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws,\n python-zookeeper, bzr, default-jre-headless, zookeeper, zookeeperd]\nruncmd: [export ENSEMBLE_INSTANCE_ID=winston-uid, sudo apt-get -y install ensemble,\n sudo mkdir -p /var/lib/ensemble, sudo mkdir -p /var/log/ensemble, \'ensemble-admin\n initialize --instance-id=$ENSEMBLE_INSTANCE...

review: Needs Fixing

[lp:~smoser/pyjuju/cloud-init-output-log]

338. By Scott Moser on 2011-09-07

make tests pass

[Gustavo Niemeyer (niemeyer)]

The magic in there goes definitely beyond my knowledge of cloud-init,
but I trust your take on it, thanks Scott.

review: Approve

[Gustavo Niemeyer (niemeyer)]

Kapil has tested and it seems to work for real as well.

=== modified file 'ensemble/providers/common/utils.py'
--- ensemble/providers/common/utils.py	2011-08-23 09:59:59 +0000
+++ ensemble/providers/common/utils.py	2011-09-07 22:09:52 +0000
@@ -105,7 +105,8 @@
         "apt-update": True,
         "apt-upgrade": True,
         "ssh_authorized_keys": [authorized_keys],
-        "packages": []}
+        "packages": [],
+        "output": { "all": "| tee -a /var/log/cloud-init-output.log" }}
 
     if data:
         cloud_config["machine-data"] = data

=== modified file 'ensemble/providers/orchestra/tests/data/bootstrap_late_command'
--- ensemble/providers/orchestra/tests/data/bootstrap_late_command	2011-08-11 18:43:06 +0000
+++ ensemble/providers/orchestra/tests/data/bootstrap_late_command	2011-09-07 22:09:52 +0000
@@ -12,6 +12,7 @@
 apt_sources:
 - {source: 'ppa:ensemble/ppa'}
 machine-data: {ensemble-zookeeper-hosts: 'localhost:2181', machine-id: '0'}
+output: {all: '| tee -a /var/log/cloud-init-output.log'}
 packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws,
   python-zookeeper, bzr, default-jre-headless, zookeeper, zookeeperd]
 runcmd: [export ENSEMBLE_INSTANCE_ID=winston-uid, sudo apt-get -y install ensemble,

=== modified file 'ensemble/providers/orchestra/tests/data/launch_late_command'
--- ensemble/providers/orchestra/tests/data/launch_late_command	2011-08-17 11:12:29 +0000
+++ ensemble/providers/orchestra/tests/data/launch_late_command	2011-09-07 22:09:52 +0000
@@ -12,6 +12,7 @@
 apt_sources:
 - {source: 'ppa:ensemble/ppa'}
 machine-data: {ensemble-zookeeper-hosts: 'jennifer:2181', machine-id: '42'}
+output: {all: '| tee -a /var/log/cloud-init-output.log'}
 packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws,
   python-zookeeper]
 runcmd: [export ENSEMBLE_INSTANCE_ID=winston-uid, sudo apt-get -y install ensemble,