Merge lp:~jimbaker/juju-jitsu/juju-do into lp:juju-jitsu

[Jim Baker]

Adds a "juju do" subcommand. Use as follows:

$ juju do --help
usage: do [-h] [-e ENVIRONMENT] [--loglevel CRITICAL|ERROR|WARNING|INFO|DEBUG]
          [--verbose]
          SERVICE_UNIT COMMAND [ARG [ARG ...]]

Examples:

To introspect a relation setting on the db:0 relation for the mysql/0 service unit:

$ PASSWORD=$(juju do mysql/0 relation-get -r db:0 password)

Note that a relation id must be supplied when using relation hook commands. It might make sense to default this if there's only one relation; that could be done in a future branch.

Define a script refresh.sh like so:

  #!/bin/bash
  for relation_id in $(relation-ids db); do
      relation-set -r $relation_id ready=$1
  done

$ juju do --loglevel=DEBUG mysql/0 refresh.sh 32
2012-04-24 11:02:19,671 juju.ec2:WARNING txaws.client.ssl unavailable for SSL hostname verification
2012-04-24 11:02:19,671 juju.ec2:WARNING EC2 API calls encrypted but not authenticated
2012-04-24 11:02:19,671 juju.ec2:WARNING S3 API calls encrypted but not authenticated
2012-04-24 11:02:19,671 juju.ec2:WARNING Ubuntu Cloud Image lookups encrypted but not authenticated
2012-04-24 11:02:19,673 juju.common:INFO Connecting to environment...
2012-04-24 11:02:20,547 juju.common:DEBUG Connecting to environment using ec2-50-18-146-37.us-west-1.compute.amazonaws.com...
2012-04-24 11:02:20,548 juju.state.sshforward:DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="ec2-50-18-146-37.us-west-1.compute.amazonaws.com" remote_port="2181" local_port="60629".
2012-04-24 11:02:22,186 juju.common:DEBUG Environment is initialized.
2012-04-24 11:02:22,186 juju.common:INFO Connected to environment.
2012-04-24 11:02:22,592 juju-do-hook-output:DEBUG Cached relation hook contexts: ['db:0', 'db:1']
2012-04-24 11:02:23,352 juju.jitsu.do.unit-agent:DEBUG Setting relation db:0
2012-04-24 11:02:23,794 juju.jitsu.do.unit-agent:DEBUG Setting relation db:1
2012-04-24 11:02:23,875 juju-do-hook-output:DEBUG hook juju-do-hook-script exited, exit code 0.
2012-04-24 11:02:24,666 juju-do-hook-output:DEBUG Flushed values for hook 'juju-do-hook-script'
    Setting changed: 'ready'=u'32' (was '42') on 'db:0'
    Setting changed: 'ready'=u'32' (was '42') on 'db:1'

"juju do" can also run on a Juju machine as well. This can make it useful for running as part of a cron job, for example.

Still needs unit testing, but it should be suitable for merging against juju-jitsu at this time.

[Clint Byrum (clint-fewbar)]

Looks great Jim. I've not given it a line by line analysis, but I like the direction and the utility.

One thing, it should be documented as 'jitsu do' not 'juju do' since sub-commands are only added to juju if 'wrap-juju' is invoked.

Thats the only thing, otherwise, merge away!

[Jim Baker (jimbaker)]

I changed 'juju do' to 'jitsu do' accordingly in both the docs and the code (to keep it consistent with docs), then merged. Thanks for the review!

=== added directory 'sub-commands/aiki'
=== added file 'sub-commands/aiki/__init__.py'
--- sub-commands/aiki/__init__.py	1970-01-01 00:00:00 +0000
+++ sub-commands/aiki/__init__.py	2012-04-24 20:12:54 +0000
@@ -0,0 +1,1 @@
+"""Aiki is the blending with the attack. Aiki-jijitsu is one of the styles of jujitsu."""

=== added file 'sub-commands/aiki/introspect.py'
--- sub-commands/aiki/introspect.py	1970-01-01 00:00:00 +0000
+++ sub-commands/aiki/introspect.py	2012-04-24 20:12:54 +0000
@@ -0,0 +1,57 @@
+"""Reads /etc/init/juju-machine-agent.conf to determine Juju agent config"""
+
+from txzookeeper.client import ZookeeperClient
+
+import juju
+from juju.environment.config import EnvironmentsConfig
+
+
+def get_upstart_env(conf_path="/etc/init/juju-machine-agent.conf"):
+    # Find environment variables as follows
+    # env JUJU_MACHINE_ID="2"
+    # env JUJU_ZOOKEEPER="ip-10-171-74-191.us-west-1.compute.internal:2181"
+
+    env = {}
+    with open(conf_path) as conf:
+        for line in conf:
+            if line.startswith("env "):
+                kv = line[4:]
+                k, v = kv.split("=")
+                env[k] = v[1:-2] # remove quotes
+    return env
+
+
+def get_zk_client_connector(options):
+    # Running on a Juju machine or on an admin system? Distinguish by
+    # first checking for an ~/.juju/environments.yaml file; if not
+    # check the machine for an upstart conf file
+    env_config = EnvironmentsConfig()
+    try:
+        env_config.load()
+    except juju.errors.FileNotFound:
+        return get_juju_machine_zk_client_connector()
+
+    if options.environment is None:
+        environment = env_config.get_default()
+    else:
+        environment = env_config.get(options.environment)
+    if environment is None:
+        raise juju.errors.EnvironmentNotFound(
+            "Environment %r not configured in environments.yaml" % options.environment)
+    provider = environment.get_machine_provider()
+    
+    def connector():
+        # Note: returns a Deferred
+        return provider.connect()
+    
+    return connector
+
+
+def get_juju_machine_zk_client_connector():
+    env = get_upstart_env()
+    
+    # When principals for ACL support is added, this is the place to do it.
+    def connector():
+        return ZookeeperClient().connect(env["JUJU_ZOOKEEPER"])
+
+    return connector

=== added file 'sub-commands/aiki/invoker.py'
--- sub-commands/aiki/invoker.py	1970-01-01 00:00:00 +0000
+++ sub-commands/aiki/invoker.py	2012-04-24 20:12:54 +0000
@@ -0,0 +1,126 @@
+"""Creates a unit agent-like context for invoking hook commands.
+
+"""
+
+# XXX issues
+# still write some tests here
+
+
+import logging
+import os
+import pipes
+import stat
+import sys
+
+from twisted.internet.defer import inlineCallbacks
+
+from juju.hooks.invoker import Invoker
+from juju.hooks.protocol import UnitSettingsFactory
+from juju.state.hook import HookContext
+from juju.unit.lifecycle import HOOK_SOCKET_FILE
+
+
+class InvokerUnitAgent(object):
+    """Creates a unit-agent like context for running commands as hooks:
+
+      * Initializes a client state cache and domain socket
+      * Dynamically creates a wrapper script for the hook command
+      * Runs wrapper script
+    """
+    def __init__(self, client, charm_dir, log_file, error_file, options):
+        self.client = client
+        self.charm_dir = charm_dir
+        self.log_file = log_file
+        self.error_file = error_file
+        self.debug_file = sys.stderr
+        self.options = options
+        os.environ["JUJU_UNIT_NAME"] = self.unit_name = options.unit_name
+        self.socket_path = os.path.join(self.charm_dir, HOOK_SOCKET_FILE)
+        os.makedirs(os.path.join(self.charm_dir, "charm"))
+
+        self.context = None
+        self.invoker = None
+        self.server_listen()
+
+    @inlineCallbacks
+    def start(self):
+        """Build an Invoker for the execution of a hook."""
+
+        # Output from hook commands is relayed by logging. Capture it
+        # with this logger which is passed into the invoker. The
+        # "juju-do-hook-output" name of the logger is completely
+        # arbitrary, with this provision: it's not prefixed with
+        # "juju." because otherwise if the log level for "juju" is
+        # changed to above INFO, this capture doesn't actually occur!
+        # (Hope that's clear.)
+        logger = logging.getLogger("juju-do-hook-output")
+        logger.propagate = False
+        output_handler = logging.StreamHandler(self.log_file)
+        output_handler.setLevel(logging.INFO)
+        logger.addHandler(output_handler)
+
+        # Capture any errors separately
+        error_handler = logging.StreamHandler(self.error_file)
+        error_handler.setLevel(logging.ERROR)
+        logger.addHandler(error_handler)
+
+        # Also grab debug output from unit agent if loglevel is below INFO
+        if self.options.loglevel < logging.INFO:
+            debug_handler = logging.StreamHandler(self.debug_file)
+            debug_handler.setLevel(logging.DEBUG)
+            debug_handler.setFormatter(logging.Formatter("%(asctime)s %(name)s:%(levelname)s %(message)s"))
+            logger.addHandler(debug_handler)
+
+        self.context = HookContext(self.client, self.unit_name)
+        self.invoker = Invoker(
+            self.context, None, "client_id", self.socket_path,
+            self.charm_dir, logger)
+
+        # When working with relation hook commands, setting
+        # JUJU_REMOTE_UNIT to the same value as JUJU_UNIT_NAME
+        # provides a useful default. A relation must still be
+        # specified, however.
+        self.invoker.environment["JUJU_REMOTE_UNIT"] = self.unit_name
+        yield self.invoker.start()
+
+    def get_context(self, client_id):
+        """Required by the factory, always return this context"""
+        return self.context
+
+    def lookup_invoker(self, client_id):
+        """Required by the factory, always return this invoker"""
+        return self.invoker
+
+    def server_listen(self):
+        """Start Unix domain socket server for the constructed hook"""
+        from twisted.internet import reactor
+
+        self.server_factory = UnitSettingsFactory(
+            self.get_context, self.lookup_invoker,
+            logging.getLogger("juju.jitsu.do.unit-agent"))
+        self.server_socket = reactor.listenUNIX(
+            self.socket_path, self.server_factory)
+
+    def stop(self):
+        """Stop the process invocation."""
+        self.server_socket.stopListening()
+
+    def run_hook(self, command, args):
+        """Given `commmand` and any `args`, builds a script that can be invoked.
+
+        Both `command` and `args` are protected by shell quoting, as
+        necessary.
+
+        Returns a `Deferred` that can be yield upon to get the result,
+        an exit code. Note that this exit code is always 0 for hook
+        commands that complete, even if there's an error in their
+        execution.
+        """
+        hook_path = os.path.join(self.charm_dir, "juju-do-hook-script")
+        with open(hook_path, "w") as f:
+            f.writelines([
+                    "#!/bin/sh\n",
+                    pipes.quote(command), " ",
+                    " ".join(pipes.quote(arg) for arg in args), "\n"])
+        os.chmod(hook_path, stat.S_IEXEC | stat.S_IREAD)
+        return self.invoker(hook_path)

=== added file 'sub-commands/do'
--- sub-commands/do	1970-01-01 00:00:00 +0000
+++ sub-commands/do	2012-04-24 20:12:54 +0000
@@ -0,0 +1,203 @@
+#!/usr/bin/env python
+
+"""
+Examples:
+
+$ PASSWORD=$(./juju-do mysql/0 relation-get -r db:0 password)
+
+Note that a relation id must be supplied for relation hook commands.
+
+Define a script refresh.sh like so::
+
+  #!/bin/bash
+  for relation_id in $(relation-ids db); do
+      relation-set -r $relation_id ready=$1
+  done
+
+$ ./juju-do --loglevel=DEBUG mysql/0 test.sh 32
[phone]:02:19,671 juju.ec2:WARNING txaws.client.ssl unavailable for SSL hostname verification
[phone]:02:19,671 juju.ec2:WARNING EC2 API calls encrypted but not authenticated
[phone]:02:19,671 juju.ec2:WARNING S3 API calls encrypted but not authenticated
[phone]:02:19,671 juju.ec2:WARNING Ubuntu Cloud Image lookups encrypted but not authenticated
[phone]:02:19,673 juju.common:INFO Connecting to environment...
[phone]:02:20,547 juju.common:DEBUG Connecting to environment using ec2-50-18-146-37.us-west-1.compute.amazonaws.com...
[phone]:02:20,548 juju.state.sshforward:DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="ec2-50-18-146-37.us-west-1.compute.amazonaws.com" remote_port="2181" local_port="60629".
[phone]:02:22,186 juju.common:DEBUG Environment is initialized.
[phone]:02:22,186 juju.common:INFO Connected to environment.
[phone]:02:22,592 juju-do-hook-output:DEBUG Cached relation hook contexts: ['db:0', 'db:1']
[phone]:02:23,352 juju.jitsu.do.unit-agent:DEBUG Setting relation db:0
[phone]:02:23,794 juju.jitsu.do.unit-agent:DEBUG Setting relation db:1
[phone]:02:23,875 juju-do-hook-output:DEBUG hook juju-do-hook-script exited, exit code 0.
[phone]:02:24,666 juju-do-hook-output:DEBUG Flushed values for hook 'juju-do-hook-script'
+    Setting changed: 'ready'=u'32' (was '42') on 'db:0'
+    Setting changed: 'ready'=u'32' (was '42') on 'db:1'
+"""
+
+from StringIO import StringIO
+import argparse
+import zookeeper
+import logging
+import os
+import shutil
+import sys
+import tempfile
+import traceback
+
+from twisted.internet import reactor
+from twisted.internet.defer import inlineCallbacks
+
+from aiki.invoker import InvokerUnitAgent
+from aiki.introspect import get_zk_client_connector
+import juju
+
+
+def main():
+    loglevels=dict(
+            CRITICAL=logging.CRITICAL,
+            ERROR=logging.ERROR,
+            WARNING=logging.WARNING,
+            INFO=logging.INFO,
+            DEBUG=logging.DEBUG)
+
+    parser = argparse.ArgumentParser(
+        description="expose a service unit(s) port")
+    parser.add_argument(
+        "-e", "--environment", default=None,
+        help="Environment to act upon (otherwise uses default)")
+    parser.add_argument(
+        "--loglevel", default=None, choices=loglevels,
+        help="Log level",
+        metavar="CRITICAL|ERROR|WARNING|INFO|DEBUG")
+    parser.add_argument(
+        "--verbose", "-v", default=False, action="store_true",
+        help="Enable verbose logging")
+    parser.add_argument("unit_name", help="Local unit name for running command", metavar="SERVICE_UNIT")
+    parser.add_argument("command", help="Command", metavar="COMMAND")
+    parser.add_argument("arg", nargs="*", help="Optional args", metavar="ARG")
+
+    # Collect additional args to be passed through, such as arguments like -r db:0
+    options, extra = parser.parse_known_args()
+    extra.extend(options.arg)
+    options.extra = extra
+
+    # Prefix command with path if necessary
+    if os.path.exists(os.path.join(os.getcwd(), options.command)):
+        options.command = os.path.abspath(options.command)
+
+    # Set logging defaults with respect to verbose
+    options.loglevel = loglevels.get(options.loglevel)
+    if options.verbose:
+        if options.loglevel is None:
+            options.loglevel = logging.DEBUG
+    else:
+        # Avoid potentially voluminous ZK debug logging 
+        zookeeper.set_debug_level(0)
+        if options.loglevel is None:
+            options.loglevel = logging.WARNING
+
+    # Some extra complexity here in logging level setup vs a normal
+    # root logger; see the comments on the juju-do-hook-output logging
+    # for the motivation on why.
+    log_options = {
+        "level": min(options.loglevel, logging.INFO),
+        "format": "%(asctime)s %(name)s:%(levelname)s %(message)s"
+        }
+    logging.basicConfig(**log_options)
+    juju_root_log = logging.getLogger("juju")  # NB Cannot do this with the root logger!
+    juju_root_log.setLevel(logging.getLevelName(options.loglevel))
+
+    # Introspect to find the right client connector; this will be
+    # called, then yielded upon, to get a ZK client once async code is
+    # entered
+    try:
+        connector = get_zk_client_connector(options)
+    except Exception, e:
+        print >> sys.stderr, e
+        sys.exit(1)
+
+    # Setup temporary unit agent to invoke command + any args as a
+    # hook and run it
+    temp_dir = tempfile.mkdtemp(prefix="juju-do-")
+    result = [0]
+    reactor.callWhenRunning(
+        run_one, juju_do, options.verbose, result,
+        connector, temp_dir, options)
+    reactor.run()
+    shutil.rmtree(temp_dir)
+    sys.exit(result[0])
+
+
+@inlineCallbacks
+def run_one(func, verbose, result, *args, **kw):
+    try:
+        yield func(result, *args, **kw)
+    except Exception, e:
+        result[0] = 1
+        if verbose:
+            traceback.print_exc()  # Writes to stderr
+        else:
+            print >> sys.stderr, e
+    finally:
+        reactor.stop()
+
+
+@inlineCallbacks
+def juju_do(result, connector, temp_dir, options):
+    """Runs the hook in the reactor, as if it were a unit agent.
+
+    `result` is a list of one object that is used to communicate back
+    the exit code. Otherwise stdout/stderr are used for the unit agent
+    output and error text, if any.
+    """
+
+    # Most of this code is to handle the many, many ways that the hook
+    # can fail, while capturing useful exit codes and and error text
+    # (and omitting useless junk).
+
+    # Connect to ZK, either from admin machine or on a Juju machine
+    client = yield connector()
+
+    # Unit agent output/error
+    log_file = StringIO()
+    error_file = StringIO()
+
+    # And actually run the hook
+    invoker_ua = InvokerUnitAgent(client, temp_dir, log_file, error_file, options)
+    yield invoker_ua.start()
+    try:
+        result[0] = yield invoker_ua.run_hook(options.command, options.extra)
+    except juju.errors.CharmInvocationError, e:
+        # Get exit code from this exception, but pick up the actual
+        # error text from the error_file, since it is more detailed;
+        # see below
+        result[0] = e.exit_code
+    invoker_ua.stop()
+
+    # Write any error to stderr, otherwise write hook result to stdout
+    error = error_file.getvalue().rstrip()
+    if error:
+        error_printed = False
+        # Unless verbose, ignore the traceback produced by hook commands
+        if not options.verbose:
+            lines = error.split("\n")
+            if lines[0] == "Traceback (most recent call last):":
+                lines.pop(0)
+                print >> sys.stderr, lines[0]
+                error_printed = True
+            # Omit wrapper script from error
+            prefix = temp_dir + "/juju-do-hook-script: 2: "
+            if lines[0].startswith(prefix):
+                print >> sys.stderr, lines[0][len(prefix):]
+                error_printed = True
+        if not error_printed:
+            print >> sys.stderr, error
+        if result[0] == 0:
+            result[0] = 1  # Return a nonzero status code result
+    else:
+        value = log_file.getvalue().rstrip()
+        if value:
+            print value
+
+
+if __name__ == '__main__':
+    main()