Merge lp:~linaro-validation/lava-core/main into lp:lava-core

[Zygmunt Krynicki]

This branch adds the main module (as in "main" function in C). Please have a look at the output of $ lava-dev --help, if you can and suggest changes to the content displayed there.

NOTE: There is an unrelated code remove from lava.commands module that just need to wait a little longer but was not spotted before (in the demo -> trunk transplant that I'm doing). That code works, it just needs dependencies that come later.

  Add lava.core.main

  The .main module defines one, relatively big, lava.commands.Dispatcher
  subclass, LavaDispatcher, that is specialized for being the entry point
  to all of lava form the shell. There are a number of customizations as
  compared to the plain Dispatcher class.

  There is a custom argument parser, as produced by construct_parser(), that has
  a lot of additional options for debugging and logging. There is a set of
  methods that conditionally enable plain logging, verbose logging,
  exception/crash logging, and developer-debug logging. Those are all called from
  the private method _adjust_logging_level() (that is a part of the Command class
  API)

  There way commands are processed, via dispatch(), is customized to handle
  various exceptions, including CommandError which is considered normal (not an
  application crash) and KeyboardInterrupt (which, by default, is not producing a
  backtrace). This function also knows how to spawn pdb, the python debugger, for
  a post-mortem session, if requested.

  Lastly there is a helper method save_history_now() that exists there to work
  around scratch space going away by the time the command finishes (as typically
  commands manage their scratch space internally). This part could be changed so
  that commands get a scratch space from the Dispatcher (and thus extend the
  lifetime of temporary files long enough so that the dispatcher gets to save
  global history, including any attachments, just before discarding the top-level
  Scratch object)

[Michael Hudson-Doyle (mwhudson)]

Hi. This seems mostly fine. Two comments:

1) I don't think the help for -D is quite right: yes it enables DEBUG on the root logger, but the debug output only goes to ~/.cache/lava/logs/debug.log right? The way the help is written I think I expected it to be on stdout/stderr. I think the behaviour you have is better, can you update the description.

2) When I run lava-dev --help, I see this: http://people.linaro.org/~mwh/lava-dev-help.png -- all that epilog stuff is not what I want to see! Compare bzr --help or git --help. It's good information and should be accessible somewhere, but maybe it should be somewhere else? I don't know where. bzr has 'topics' -- bzr help topics -- maybe something like that?

[Michael Hudson-Doyle (mwhudson)]

Hi. This seems mostly fine. Two comments:

1) I don't think the help for -D is quite right: yes it enables DEBUG on the root logger, but the debug output only goes to ~/.cache/lava/logs/debug.log right? The way the help is written I think I expected it to be on stdout/stderr. I think the behaviour you have is better, can you update the description.

2) When I run lava-dev --help, I see this: http://people.linaro.org/~mwh/lava-dev-help.png -- all that epilog stuff is not what I want to see! Compare bzr --help or git --help. It's good information and should be accessible somewhere, but maybe it should be somewhere else? I don't know where. bzr has 'topics' -- bzr help topics -- maybe something like that?

[lp:~linaro-validation/lava-core/main]

8. By Zygmunt Krynicki on 2012-05-16

Infrastructure for history and logging

9. By Zygmunt Krynicki on 2012-05-18

Infrastructure for plugins

10. By Zygmunt Krynicki on 2012-05-21

Infrastructure for programming interfaces

11. By Zygmunt Krynicki on 2012-05-21

Infrastructure for command line tools

12. By Zygmunt Krynicki on 2012-05-22

Infrastructure for temporary files (scratch space)

13. By Zygmunt Krynicki on 2012-05-23

Add lava.core.main

The .main module defines one, relatively big, lava.commands.Dispatcher
subclass, LavaDispatcher, that is specialized for being the entry point
to all of lava form the shell. There are a number of customizations as
compared to the plain Dispatcher class.

There is a custom argument parser, as produced by construct_parser(), that has
a lot of additional options for debugging and logging. There is a set of
methods that conditionally enable plain logging, verbose logging,
exception/crash logging, and developer-debug logging. Those are all called from
the private method _adjust_logging_level() (that is a part of the Command class
API)

There way commands are processed, via dispatch(), is customized to handle
various exceptions, including CommandError which is considered normal (not an
application crash) and KeyboardInterrupt (which, by default, is not producing a
backtrace). This function also knows how to spawn pdb, the python debugger, for
a post-mortem session, if requested.

Lastly there is a helper method save_history_now() that exists there to work
around scratch space going away by the time the command finishes (as typically
commands manage their scratch space internally). This part could be changed so
that commands get a scratch space from the Dispatcher (and thus extend the
lifetime of temporary files long enough so that the dispatcher gets to save
global history, including any attachments, just before discarding the top-level
Scratch object)

[Zygmunt Krynicki (zyga)]

> Hi. This seems mostly fine. Two comments:
>
> 1) I don't think the help for -D is quite right: yes it enables DEBUG on the
> root logger, but the debug output only goes to ~/.cache/lava/logs/debug.log
> right? The way the help is written I think I expected it to be on
> stdout/stderr. I think the behaviour you have is better, can you update the
> description.

I gave it a thought but failed to come up with a better description. Please advise.

> 2) When I run lava-dev --help, I see this: http://people.linaro.org/~mwh/lava-
> dev-help.png -- all that epilog stuff is not what I want to see! Compare bzr
> --help or git --help. It's good information and should be accessible
> somewhere, but maybe it should be somewhere else? I don't know where. bzr
> has 'topics' -- bzr help topics -- maybe something like that?

Quite right. I've split most of the former `lava --help` to new `lava --about`> This made --help short and tidy.

[lp:~linaro-validation/lava-core/main]

14. By Michael Hudson-Doyle on 2012-05-23

add a description to the logging group, format epilog more carefully

=== added file 'lava/core/main.py'
--- lava/core/main.py	1970-01-01 00:00:00 +0000
+++ lava/core/main.py	2012-05-23 22:00:25 +0000
@@ -0,0 +1,346 @@
+# Copyright (C) 2010-2012 Linaro Limited
+#
+# Author: Zygmunt Krynicki <[e-mail]>
+#
+# This file is part of lava-core
+#
+# lava-core is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3
+# as published by the Free Software Foundation
+#
+# lava-core is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with lava-core.  If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import absolute_import, print_function
+
+"""
+lava.core.main
+==============
+
+Implementation of the `lava-dev` shell command.
+
+See the note in LavaDispatcher docstring about `lava-dev` vs `lava`
+"""
+
+import argparse
+import logging
+import os
+import pdb
+import sys
+
+from lava.core.commands import Dispatcher
+from lava.core.errors import CommandError
+from lava.core.utils  import Scratch
+from lava.core.history import history
+
+
+class AboutLavaAction(argparse.Action):
+    """
+    Argparse action that implements the `lava-dev --about` command
+    """
+
+    message = """
+About LAVA
+    LAVA, which stands for Linaro Automated Validation Architecture, is a
+    project started by Linaro tasked with build a collection of freely
+    available tools and equipment for test automation. LAVA is composed of many
+    extensible pieces that together help to plan, schedule, execute, track, and
+    run tests on virtual and physical devices, then finally to store and
+    visualize the results.
+
+    To learn more about LAVA please visit https://launchpad.net/lava-project/
+
+About Linaro
+    Linaro was established in June 2010 by founding members ARM, Freescale,
+    IBM, Samsung, ST-Ericsson and Texas instruments (TI). Members provide
+    engineering resources and funding. Linaro's goals are to deliver value to
+    its members through enabling their engineering teams to focus on
+    differentiation and product delivery, and to reduce time to market for
+    OEM/ODMs delivering open source based products using ARM technology.
+
+    To learn more about Linaro please visit http://www.linaro.org/about/
+
+Software development and hacking:
+    Source code is freely available and is managed by the Bazaar version
+    control system and the Launchpad (http://launchpad.net/) collaboration
+    platform. Note that while LAVA is located in more than one project you
+    should eventually be able to look at the lava-core project for most of
+    the code. The process of merging small lava projects into lava-core is
+    not complete yet, as of May 2012.
+
+    To get a copy of LAVA install bazaar and run the following command:
+
+        $ bzr branch lp:lava-core
+
+    We welcome all contributors, see HACKING to get started
+
+    Development is discussed on the [e-mail] mailing
+    list, you can subscribe by visiting http://lists.linaro.org/
+
+Support:
+    Report bugs at https://bugs.launchpad.net/lava-project/+filebug
+
+    Ask questions in your native language at
+    https://answers.launchpad.net/lava-project/+addquestion.
+
+    You can also ask for real-time support in the #linaro-lava channel on the
+    freenode IRC network. Keep in mind that although geographically distributed
+    we (LAVA developers) may be away at a given time. Note that developer time
+    is precious so please be considerate.
+
+Copyright and license:
+    Copyright (c) Linaro Limited 2010-2012
+
+    LAVA is free software, licensed under GLPv3, LGPLv3 and AGLPv3.
+    Check the license header in a specific file to know more.
+"""
+    message = message.lstrip()
+
+    def __init__(self,
+                 option_strings, dest, default=None, required=False,
+                 help=None):
+        super(AboutLavaAction, self).__init__(
+            option_strings=option_strings, dest=dest, default=default, nargs=0,
+            help=help)
+
+    def __call__(self, parser, namespace, values, option_string=None):
+        parser.exit(message=self.message)
+
+
+class LavaDispatcher(Dispatcher):
+    """
+    Dispatcher implementing the `lava-dev` shell command (note that
+    `lava-dev` shall be renamed to `lava` once lava-core replaces
+    current group of lava-* projects.
+
+    This dispatcher imports plugins from the pkg_resources namespace
+    `lava-dev.commands`. Additional plugins can be registered as either
+    :class:`lava.command.Command` or :class:`lava.command.CommandGroup`
+    sub-classes.
+    """
+
+    name = "lava-dev"
+    help = None
+    description = """
+This executable allows you to work with LAVA. LAVA commands form a tree, where
+you typically invoke sub or sub-sub commands to do actual things. Please
+inspect the list below for details.
+"""
+
+    epilog = """
+For information about LAVA and Linaro run: lava-dev --about"""
+
+    def __init__(self):
+        # Call this early so that we don't get logging.basicConfig
+        # being called by accident. Otherwise we'd have to
+        # purge all loggers from the root logger and that sucks
+        self.setup_logging()
+        # Initialize the base dispatcher
+        super(LavaDispatcher, self).__init__()
+        # And import the non-flat namespace commands
+        self.import_commands('lava_dev.commands')
+        self.args = None
+        self._history_needs_saving = True
+
+    def construct_parser(self):
+        """
+        Construct a parser for this dispatcher.
+
+        This is only used if the parser is not provided by the parent
+        dispatcher instance.
+        """
+        # Construct a basic parser
+        parser = super(LavaDispatcher, self).construct_parser()
+        # Add the --about command
+        parser.register("action", "about_lava", AboutLavaAction)
+        parser.add_argument(
+            "--about",
+            action="about_lava",
+            default=argparse.SUPPRESS,
+            help="Provide detailed information about LAVA and Linaro")
+        # Add the --verbose flag
+        parser.add_argument(
+            "-v", "--verbose",
+            default=False,
+            action="store_true",
+            help="be more verbose (displays all INFO messages)")
+        group = parser.add_argument_group(title="history management")
+        group.add_argument(
+            "-H", "--history",
+            type=argparse.FileType(mode="wb"),
+            metavar="TARBALL",
+            help="Save session history, as tarball, to TARBALL")
+        group = parser.add_argument_group(
+            title="logging and software development",
+            description="""\
+Log messages at the WARNING level or higher are sent to stderr.  DEBUG
+messages (when enabled by -D or -T) go to ~/.cache/lava/logs/debug.log.
+""")
+        # Add the --debug flag
+        group.add_argument(
+            "-D", "--debug",
+            action="store_true",
+            default=False,
+            help="enable DEBUG messages on the root logger")
+        # Add the --trace flag
+        group.add_argument(
+            "-T", "--trace",
+            metavar="LOGGER",
+            action="append",
+            default=[],
+            help=("enable DEBUG messages on the specified logger "
+                  "(can be used multiple times)"))
+        # Add the --pdb flag
+        group.add_argument(
+            "-P", "--pdb",
+            action="store_true",
+            default=False,
+            help="jump into pdb (python debugger) when a command crashes")
+        # Add the --debug-command-errors flag
+        group.add_argument(
+            "-C", "--debug-command-errors",
+            action="store_true",
+            default=False,
+            help="crash on CommandError, useful with --pdb")
+        # Add the --debug-interrupt flag
+        group.add_argument(
+            "-I", "--debug-interrupt",
+            action="store_true",
+            default=False,
+            help="crash on SIGINT/KeyboardInterrupt, useful with --pdb")
+        # Return the improved parser
+        return parser
+
+    def setup_logging(self):
+        """
+        Setup logging for the root dispatcher
+        """
+        self._setup_error_handler()
+        self._setup_warning_handler()
+
+    def dispatch(self, raw_args=None):
+        """
+        Enhanced version of Dispatcher.dispatch() that logs command duration
+        and handles all exceptions (swallowing them)
+        """
+        try:
+            return super(LavaDispatcher, self).dispatch(raw_args, True)
+        except SystemExit:
+            pass
+        except BaseException as exc:
+            if isinstance(exc, KeyboardInterrupt):
+                self.say(None, "Interrupted!")
+                if self.args is None or not self.args.debug_interrupt:
+                    return 1
+            elif isinstance(exc, CommandError):
+                if self.args is None or not self.args.debug_command_errors:
+                    return 1
+            else:
+                self.history.error(
+                    "Runaway exception while invoking command: {exc}",
+                    exc=str(exc))
+            if self.args and self.args.pdb:
+                self.logger.info("Starting pdb...")
+                pdb.post_mortem()
+            else:
+                self.say(None, "This command has crashed")
+                self.say(None, "The problem was: {0}", exc)
+                self.say(None, "See {0} for full traceback",
+                         self.error_log_pathname),
+            return 1
+        finally:
+            # Save history as tarball
+            if self._history_needs_saving and self.args and self.args.history:
+                self.history.debug("Saving history to {name}",
+                                   name=self.args.history.name)
+                with Scratch() as scratch:
+                    self.save_history_now(scratch)
+
+    def save_history_now(self, scratch):
+        self._history_needs_saving = False
+        if self.args is not None and self.args.history:
+            history.top.save_as_tarball(scratch, self.args.history)
+
+    def _setup_error_handler(self):
+        """
+        Add a handler that logs all ERRORs to a dedicated file
+        """
+        # TODO: support XDA dirs
+        pathname = os.path.expanduser("~/.cache/lava/logs/error.log")
+        self.error_log_pathname = pathname
+        if not os.path.exists(os.path.dirname(pathname)):
+            os.makedirs(os.path.dirname(pathname))
+        handler = logging.FileHandler(pathname, delay=True)
+        handler.setLevel(logging.ERROR)
+        handler.setFormatter(
+            logging.Formatter("%(levelname)s %(name)s: %(message)s"))
+        logging.getLogger().addHandler(handler)
+
+    def _setup_debug_handler(self):
+        """
+        Add a handler that logs everything to a dedicated file
+        """
+        # TODO: support XDA dirs
+        pathname = os.path.expanduser("~/.cache/lava/logs/debug.log")
+        self.say(None, "Note: debugging messages are saved to {0}", pathname)
+        if not os.path.exists(os.path.dirname(pathname)):
+            os.makedirs(os.path.dirname(pathname))
+        handler = logging.FileHandler(pathname)
+        handler.setLevel(logging.DEBUG)
+        handler.setFormatter(
+            logging.Formatter((
+                "%(asctime)s "
+                "[pid:%(process)s, thread:%(threadName)s, "
+                "reltime:%(relativeCreated)dms] "
+                "%(levelname)s %(name)s: %(message)s")))
+        logging.getLogger().addHandler(handler)
+
+    def _setup_warning_handler(self):
+        """
+        Add a handler that logs warnings, and other important stuff to stderr
+        """
+        handler = logging.StreamHandler(sys.stderr)
+        handler.setLevel(logging.WARN)
+        handler.setFormatter(
+            logging.Formatter("%(levelname)s: %(name)s: %(message)s"))
+        logging.getLogger().addHandler(handler)
+
+    def _setup_info_handler(self):
+        class OnlyInfoFilter(logging.Filterer):
+            def filter(self, record):
+                if record.levelno == logging.INFO:
+                    return 1
+                return 0
+
+        class SayHandler(logging.Handler):
+            def emit(inner_self, record):
+                self.say(None, "{0}", record.getMessage())
+        handler = SayHandler()
+        handler.setLevel(logging.INFO)
+        handler.setFormatter(logging.Formatter("%(message)s"))
+        handler.addFilter(OnlyInfoFilter())
+        logging.getLogger().addHandler(handler)
+
+    def _adjust_logging_level(self, args):
+        # XXX: store arguments as we want to look at some more esoteric options
+        # in dispatch() later on, this feels ugly, perhaps the dispatcher
+        # should just know the arguments
+        self.args = args
+        # Enable verbose message handler
+        if args.verbose:
+            logging.getLogger().setLevel(logging.INFO)
+            self._setup_info_handler()
+        # Enable debug.log if needed
+        if args.debug or args.trace:
+            self._setup_debug_handler()
+        # Enable application-wide debugging
+        if args.debug:
+            logging.getLogger().setLevel(logging.DEBUG)
+        # Enable trace loggers
+        for name in args.trace:
+            logging.getLogger(name).setLevel(logging.DEBUG)

=== modified file 'setup.py'
--- setup.py	2012-05-16 10:38:09 +0000
+++ setup.py	2012-05-23 22:00:25 +0000
@@ -34,6 +34,10 @@
     url='https://launchpad.net/lava-core',
     license="LGPLv3, GPLv3",
     test_suite="unittest2.collector",
+    entry_points="""
+    [console_scripts]
+    lava-dev = lava.core.main:LavaDispatcher.run
+    """,
     classifiers=[
         "Development Status :: 4 - Beta",
         "Intended Audience :: Developers",
@@ -45,7 +49,8 @@
     ],
     extras_require={},
     install_requires=[
-        'json-document',
+        'argparse >= 1.1',
+        'json-document >= 0.8',
         'json-schema-validator',
         'simplejson',
     ],