Merge lp:~jimbaker/pyjuju/generate-html into lp:pyjuju/ftests

[Jim Baker]

Implements waterfall.py, which writes to stdout an HTML summary of the results, using a waterfall style. This report links the output of failed and successful tests. This follows the review by Gustavo below. Note that the formatting is intentionally bare; it will be addressed in a future branch that adds CSS and expands f_tests.mustache in a minimal fashion.

A future stacked branch will implemented an opinionated driver that will manage working with bzr and driving churn; this driver will be called butler.py, to be in the spirit of that name.

[Gustavo Niemeyer (niemeyer)]

[1]

This tool is doing three different things:

1. Running bzr update
2. Running churn
3. Converting churn output to html

The tool we discussed should do 3 only.

Let it take as input a directory full of churn output directories, and
produce the output based on the directory name. So we run it like:

$ butler.py watefall/ > waterfall.html

Given that within the waterfall dir one might expect:

    /waterfall
        /341
            some-test-name.out
            other-test-name.out.FAILED
        /342
            some-test-name.out
            other-test-name.out

etc. butler will then have the 341, 342, etc in the rows, and the test
names in the columns, with OK/FAILED in the cells, as you've done.

Once this is in place, let's create a tiny but opinionated driver
that knows how to update a branch *revision by revision* and drive
churn and butler from outside. This driver must get as input _just_
the build directory, and will have three things inside it.

So

  $ whatever build/

Generates

  build/
    ensemble/
    waterfall/
    waterfall.html

But once you run

  $ whatever build/

again, it will notice that there's already content inside it,
and will just update revision by revision the branch between
the latest revision in the waterfall and the current known
tip of the trunk.

With that in place, we can put this logic in a machine, and have it
rsyncing this content out into a public web server.

Do not use bzrlib, btw. Use bzr itself. We're not doing anything
fancy, and the command line API won't change.

[2]

Please put a copy of pystache within the tree.

[3]

96 + <td>{{#failed}}<a href="{{path}}"><strong>FAILED</strong></a>{{/failed}}
97 + {{^failed}}Succeeded{{/failed}}

We have the succeeded output as well. Let's offer it so people can
investigate differences.

It might also be better to have "ok" (lowercased) rather than "Succeeded", to make the
"FAILED" one much more outstanding.

review: Needs Fixing

[Jim Baker (jimbaker)]

On Fri, Sep 2, 2011 at 6:42 PM, Gustavo Niemeyer <email address hidden>wrote:

> Review: Needs Fixing
> [1]
>
> This tool is doing three different things:
>
> 1. Running bzr update
> 2. Running churn
> 3. Converting churn output to html
>
> The tool we discussed should do 3 only.
>
> Let it take as input a directory full of churn output directories, and
> produce the output based on the directory name. So we run it like:
>
> $ butler.py watefall/ > waterfall.html
>
> Given that within the waterfall dir one might expect:
>
> /waterfall
> /341
> some-test-name.out
> other-test-name.out.FAILED
> /342
> some-test-name.out
> other-test-name.out
>
> etc. butler will then have the 341, 342, etc in the rows, and the test
> names in the columns, with OK/FAILED in the cells, as you've done.
>

Done. I believe the mechanism I have used in working with mustache templates
to generate this table is the best approach.

>
> Once this is in place, let's create a tiny but opinionated driver
> that knows how to update a branch *revision by revision* and drive
> churn and butler from outside. This driver must get as input _just_
> the build directory, and will have three things inside it.
>
> So
>
> $ whatever build/
>
> Generates
>
> build/
> ensemble/
> waterfall/
> waterfall.html
>
> But once you run
>
> $ whatever build/
>
> again, it will notice that there's already content inside it,
> and will just update revision by revision the branch between
> the latest revision in the waterfall and the current known
> tip of the trunk.
>
> With that in place, we can put this logic in a machine, and have it
> rsyncing this content out into a public web server.
>
> Do not use bzrlib, btw. Use bzr itself. We're not doing anything
> fancy, and the command line API won't change.
>

Sounds good. Given that butler is supposed to be managing the results of
everything, I renamed butler.py to waterfall.py, leaving the name butler
available for this opinionated driver. That will be done in a separate
branch.

>
> [2]
>
> Please put a copy of pystache within the tree.
>

Done. This actually involved some debugging effort because pystache, in its
interaction with MarkupSafe, was somehow behaving differently when placed
here, vs normal package install with tools like pip. Modified accordingly.
So pystache as added here just has package code + (renamed) license file.

> [3]
>
> 96 + <td>{{#failed}}<a
> href="{{path}}"><strong>FAILED</strong></a>{{/failed}}
> 97 + {{^failed}}Succeeded{{/failed}}
>
> We have the succeeded output as well. Let's offer it so people can
> investigate differences.
>

Done

>
> It might also be better to have "ok" (lowercased) rather than "Succeeded",
> to make the
> "FAILED" one much more outstanding.
>

Done. The mustache template will need some more work re styling, but it's
functional now. This can be done in a future branch.

[Gustavo Niemeyer (niemeyer)]

Looks great, thanks Jim. Only a couple of minor comments, and it can go in IMO.

Then, for the driver! I hope we can see a waterfall working this week still!

[4]

=== added file 'pystache-LICENSE'

Please rename this to LICENSE and put it within the pystache dir itself.

[5]

+OUTPUT_STATUS_RE = re.compile(r"(?P<name>.*?)(?P<status>\.out.FAILED|\.out)")

This regex makes things more complex than it helps. E.g.:

FAILED_SUFFIX = ".out.FAILED"
OK_SUFFIX = ".out"
(...)
    if name.endswith(FAILED_SUFFIX):
        test_name = name[:-len(FAILED_SUFFIX)]

etc.

review: Approve

[lp:~jimbaker/pyjuju/generate-html]

16. By Jim Baker on 2011-09-09

Addressed review comments

[Jim Baker (jimbaker)]

On Thu, Sep 8, 2011 at 6:10 PM, Gustavo Niemeyer <email address hidden>wrote:

> Review: Approve
>
> Looks great, thanks Jim. Only a couple of minor comments, and it can go in
> IMO.
>
> Then, for the driver! I hope we can see a waterfall working this week
> still!
>

Please see the corresponding butler branch in review

>
> [4]
>
> === added file 'pystache-LICENSE'
>
> Please rename this to LICENSE and put it within the pystache dir itself.
>

Renamed accordingly

>
> [5]
>
> +OUTPUT_STATUS_RE =
> re.compile(r"(?P<name>.*?)(?P<status>\.out.FAILED|\.out)")
>
> This regex makes things more complex than it helps. E.g.:
>
> FAILED_SUFFIX = ".out.FAILED"
> OK_SUFFIX = ".out"
> (...)
> if name.endswith(FAILED_SUFFIX):
> test_name = name[:-len(FAILED_SUFFIX)]
>
> etc.
>
>
Done

[Gustavo Niemeyer (niemeyer)]

Thanks Jim, please merge it on ftests.

review: Approve

=== added file 'f_tests.mustache'
--- f_tests.mustache	1970-01-01 00:00:00 +0000
+++ f_tests.mustache	2011-09-09 22:36:28 +0000
@@ -0,0 +1,24 @@
+<table>
+ <thead>
+  <tr>
+   <th>Run/Test</th>
+   {{#test_names}}
+   <th>{{name}}</th>
+   {{/test_names}}
+  </tr>
+ </thead>
+ <tbody>
+  {{#results}}
+   <tr>
+    <td>{{run_name}}</td>
+    {{#tests}}
+     <td>{{#test_name}}
+      <a href="{{path}}">
+       {{#failed}}<strong>FAILED</strong>
+       {{/failed}}{{^failed}}ok{{/failed}}</a>
+     {{/test_name}}</td>
+    {{/tests}}
+   </tr>
+  {{/results}}
+ </tbody>
+</table>

=== added directory 'pystache'
=== added file 'pystache/LICENSE'
--- pystache/LICENSE	1970-01-01 00:00:00 +0000
+++ pystache/LICENSE	2011-09-09 22:36:28 +0000
@@ -0,0 +1,20 @@
+Copyright (c) 2009 Chris Wanstrath
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

=== added file 'pystache/__init__.py'
--- pystache/__init__.py	1970-01-01 00:00:00 +0000
+++ pystache/__init__.py	2011-09-09 22:36:28 +0000
@@ -0,0 +1,8 @@
+from pystache.template import Template
+from pystache.view import View
+from pystache.loader import Loader
+
+def render(template, context=None, **kwargs):
+    context = context and context.copy() or {}
+    context.update(kwargs)
+    return Template(template, context).render()

=== added file 'pystache/loader.py'
--- pystache/loader.py	1970-01-01 00:00:00 +0000
+++ pystache/loader.py	2011-09-09 22:36:28 +0000
@@ -0,0 +1,47 @@
+import os
+
+class Loader(object):
+    
+    template_extension = 'mustache'
+    template_path = '.'
+    template_encoding = None
+    
+    def load_template(self, template_name, template_dirs=None, encoding=None, extension=None):
+        '''Returns the template string from a file or throws IOError if it non existent'''
+        if None == template_dirs:
+            template_dirs = self.template_path
+        
+        if encoding is not None:
+            self.template_encoding = encoding
+        
+        if extension is not None:
+            self.template_extension = extension
+        
+        file_name = template_name + '.' + self.template_extension
+
+        # Given a single directory we'll load from it
+        if isinstance(template_dirs, basestring):
+            file_path = os.path.join(template_dirs, file_name)
+
+            return self._load_template_file(file_path)
+            
+        # Given a list of directories we'll check each for our file
+        for path in template_dirs:
+            file_path = os.path.join(path, file_name)
+            if os.path.exists(file_path):
+                return self._load_template_file(file_path)
+                
+        raise IOError('"%s" not found in "%s"' % (template_name, ':'.join(template_dirs),))
+    
+    def _load_template_file(self, file_path):
+        '''Loads and returns the template file from disk'''
+        f = open(file_path, 'r')
+        
+        try:
+            template = f.read()
+            if self.template_encoding:
+                template = unicode(template, self.template_encoding)
+        finally:
+            f.close()
+        
+        return template
\ No newline at end of file

=== added file 'pystache/template.py'
--- pystache/template.py	1970-01-01 00:00:00 +0000
+++ pystache/template.py	2011-09-09 22:36:28 +0000
@@ -0,0 +1,178 @@
+import re
+import cgi
+import collections
+import os
+import copy
+
+
+# Modified from original version (0.3.1) to not use MarkupSafe. Using
+# MarkupSafe for HTML escaping will cause the expansion of certain
+# values (lists in particular) to fail in a rather spectacular and
+# recursive fashion. Rather curiously, this behavior only occurs when
+# the code is used directly, instead of being part of an installed
+# package.
+escape = lambda x: cgi.escape(unicode(x))
+literal = unicode
+
+
+class Modifiers(dict):
+    """Dictionary with a decorator for assigning functions to keys."""
+
+    def set(self, key):
+        """
+        Decorator function to set the given key to the decorated function.
+
+            >>> modifiers = {}
+            >>> @modifiers.set('P')
+            ... def render_tongue(self, tag_name=None, context=None):
+            ...     return ":P %s" % tag_name
+            >>> modifiers
+            {'P': <function render_tongue at 0x...>}
+        """
+
+        def setter(func):
+            self[key] = func
+            return func
+        return setter
+
+
+class Template(object):
+
+    tag_re = None
+
+    otag = '{{'
+
+    ctag = '}}'
+
+    modifiers = Modifiers()
+
+    def __init__(self, template=None, context=None, **kwargs):
+        from view import View
+
+        self.template = template
+
+        if kwargs:
+            context.update(kwargs)
+
+        self.view = context if isinstance(context, View) else View(context=context)
+        self._compile_regexps()
+
+    def _compile_regexps(self):
+        tags = {
+            'otag': re.escape(self.otag),
+            'ctag': re.escape(self.ctag)
+        }
+
+        section = r"%(otag)s[\#|^]([^\}]*)%(ctag)s\s*(.+?\s*)%(otag)s/\1%(ctag)s"
+        self.section_re = re.compile(section % tags, re.M|re.S)
+
+        tag = r"%(otag)s(#|=|&|!|>|\{)?(.+?)\1?%(ctag)s+"
+        self.tag_re = re.compile(tag % tags)
+
+    def _render_sections(self, template, view):
+        while True:
+            match = self.section_re.search(template)
+            if match is None:
+                break
+
+            section, section_name, inner = match.group(0, 1, 2)
+            section_name = section_name.strip()
+            it = self.view.get(section_name, None)
+            replacer = ''
+
+            # Callable
+            if it and isinstance(it, collections.Callable):
+                replacer = it(inner)
+            # Dictionary
+            elif it and hasattr(it, 'keys') and hasattr(it, '__getitem__'):
+                if section[2] != '^':
+                    replacer = self._render_dictionary(inner, it)
+            # Lists
+            elif it and hasattr(it, '__iter__'):
+                if section[2] != '^':
+                    replacer = self._render_list(inner, it)
+            # Other objects
+            elif it and isinstance(it, object):
+                if section[2] != '^':
+                    replacer = self._render_dictionary(inner, it)
+            # Falsey and Negated or Truthy and Not Negated
+            elif (not it and section[2] == '^') or (it and section[2] != '^'):
+                replacer = self._render_dictionary(inner, it)
+
+            template = literal(template.replace(section, replacer))
+
+        return template
+
+    def _render_tags(self, template):
+        while True:
+            match = self.tag_re.search(template)
+            if match is None:
+                break
+
+            tag, tag_type, tag_name = match.group(0, 1, 2)
+            tag_name = tag_name.strip()
+            func = self.modifiers[tag_type]
+            replacement = func(self, tag_name)
+            template = template.replace(tag, replacement)
+
+        return template
+
+    def _render_dictionary(self, template, context):
+        self.view.context_list.insert(0, context)
+        template = Template(template, self.view)
+        out = template.render()
+        self.view.context_list.pop(0)
+        return out
+
+    def _render_list(self, template, listing):
+        insides = []
+        for item in listing:
+            insides.append(self._render_dictionary(template, item))
+
+        return ''.join(insides)
+
+    @modifiers.set(None)
+    def _render_tag(self, tag_name):
+        raw = self.view.get(tag_name, '')
+
+        # For methods with no return value
+        if not raw and raw is not 0:
+            if tag_name == '.':
+                raw = self.view.context_list[0]
+            else:
+                return ''
+
+        return escape(raw)
+
+    @modifiers.set('!')
+    def _render_comment(self, tag_name):
+        return ''
+
+    @modifiers.set('>')
+    def _render_partial(self, template_name):
+        from pystache import Loader
+        markup = Loader().load_template(template_name, self.view.template_path, encoding=self.view.template_encoding)
+        template = Template(markup, self.view)
+        return template.render()
+
+    @modifiers.set('=')
+    def _change_delimiter(self, tag_name):
+        """Changes the Mustache delimiter."""
+        self.otag, self.ctag = tag_name.split(' ')
+        self._compile_regexps()
+        return ''
+
+    @modifiers.set('{')
+    @modifiers.set('&')
+    def render_unescaped(self, tag_name):
+        """Render a tag without escaping it."""
+        return literal(self.view.get(tag_name, ''))
+
+    def render(self, encoding=None):
+        template = self._render_sections(self.template, self.view)
+        result = self._render_tags(template)
+
+        if encoding is not None:
+            result = result.encode(encoding)
+
+        return result

=== added file 'pystache/view.py'
--- pystache/view.py	1970-01-01 00:00:00 +0000
+++ pystache/view.py	2011-09-09 22:36:28 +0000
@@ -0,0 +1,94 @@
+from pystache import Template
+import os.path
+import re
+from types import *
+
+def get_or_attr(context_list, name, default=None):
+    if not context_list:
+        return default
+
+    for obj in context_list:
+        try:
+            return obj[name]
+        except KeyError:
+            pass
+        except:
+            try:
+                return getattr(obj, name)
+            except AttributeError:
+                pass
+    return default
+
+class View(object):
+    
+    template_name = None
+    template_path = None
+    template = None
+    template_encoding = None
+    template_extension = 'mustache'
+    
+    def __init__(self, template=None, context=None, **kwargs):
+        self.template = template
+        context = context or {}
+        context.update(**kwargs)
+
+        self.context_list = [context]
+        
+    def get(self, attr, default=None):
+        attr = get_or_attr(self.context_list, attr, getattr(self, attr, default))
+        if hasattr(attr, '__call__') and type(attr) is UnboundMethodType:
+            return attr()
+        else:
+            return attr
+    
+    def get_template(self, template_name):
+        if not self.template:
+            from pystache import Loader
+            template_name = self._get_template_name(template_name)
+            self.template = Loader().load_template(template_name, self.template_path, encoding=self.template_encoding, extension=self.template_extension)
+        
+        return self.template
+
+    def _get_template_name(self, template_name=None):
+        """TemplatePartial => template_partial
+        Takes a string but defaults to using the current class' name or
+        the `template_name` attribute
+        """
+        if template_name:
+            return template_name
+
+        template_name = self.__class__.__name__
+
+        def repl(match):
+            return '_' + match.group(0).lower()
+
+        return re.sub('[A-Z]', repl, template_name)[1:]
+
+    def _get_context(self):
+        context = {}
+        for item in self.context_list:
+            if hasattr(item, 'keys') and hasattr(item, '__getitem__'):
+                context.update(item)
+        return context
+
+    def render(self, encoding=None):
+        return Template(self.get_template(self.template_name), self).render(encoding=encoding)
+
+    def __contains__(self, needle):
+        return needle in self.context or hasattr(self, needle)
+
+    def __getitem__(self, attr):
+        val = self.get(attr, None)
+
+        if not val and val is not 0:
+            raise KeyError("Key '%s' does not exist in View" % attr)
+        return val
+
+    def __getattr__(self, attr):
+        if attr == 'context':
+            return self._get_context()
+
+        raise AttributeError("Attribute '%s' does not exist in View" % attr)
+
+    def __str__(self):
+        return self.render()
\ No newline at end of file

=== added file 'pystache/waterfall.html'
=== added file 'waterfall.py'
--- waterfall.py	1970-01-01 00:00:00 +0000
+++ waterfall.py	2011-09-09 22:36:28 +0000
@@ -0,0 +1,90 @@
+# waterfall - Generate waterfall report from test runs
+
+import argparse
+import os
+import re
+
+from pystache import View
+
+
+FAILED_SUFFIX = ".out.FAILED"
+OK_SUFFIX = ".out"
+
+
+def main():
+    """Write waterfall summary of test results to stdout"""
+    parser = argparse.ArgumentParser(
+        description="Creates waterfall report from test runs")
+    parser.add_argument("waterfall",
+                        help="Directory of test runs resulting from churn")
+    args = parser.parse_args()
+    waterfall(args.waterfall)
+
+
+class FTests(View):
+    """Pull in template from f_tests.mustache.
+
+    Uses pystache's default class name to filename mapping."""
+
+
+def wrap_names(names):
+    """Wraps names in a dict for rendering by pystache."""
+    for name in names:
+        yield dict(name=name)
+
+
+def whipped(run_names, test_names, results):
+    """Returns the results as a dense version suitable for rendering."""
+    for run_name in run_names:
+        tests = [results.get((test_name, run_name), dict(empty=True))
+                 for test_name in test_names]
+        yield dict(run_name=run_name, tests=tests)
+
+
+def waterfall(waterfall_path):
+    """Given a directory of churn runs, writes to stdout a waterfall report.
+
+    `waterfall_path` contains a set of directories, one per run. Each
+    run contains one file per test. NOTE This function assumes there
+    are no non-run directories in `waterfall_path`. If that is
+    relaxed, some filtering obviously needs to be done here.
+
+    Other than providing an iterator of a listing of test results per
+    run, via the :func:`whipped`, all display formating is done
+    through `f_tests.mustache`. In particular, any desired styles
+    should be setup there.
+    """
+    # First, process test results, with an entry per run, test pair.
+    # The resulting `results` dict is sparse
+    results = {}
+    test_names = set()
+    run_names = sorted(os.listdir(waterfall_path))
+    for run_name in run_names:
+        run_path = os.path.join(waterfall_path, run_name)
+        for name in sorted(os.listdir(run_path)):
+            test_name = None
+            if name.endswith(FAILED_SUFFIX):
+                test_name = name[:-len(FAILED_SUFFIX)]
+                failed = True
+            elif name.endswith(OK_SUFFIX):
+                test_name = name[:-len(OK_SUFFIX)]
+                failed = False
+            if test_name:
+                test_names.add(test_name)
+                results[test_name, run_name] = {
+                    "run_name": run_name,
+                    "test_name": test_name,
+                    "failed": failed,
+                    "path": os.path.join(run_path, name)}
+
+    # Next, transform `results` into an equivalent dense version
+    # before passing onto pystache for rendering into HTML.
+    test_names = sorted(test_names)
+    wrapped_test_names = wrap_names(test_names)
+    dense_results = whipped(run_names, test_names, results)
+
+    print FTests(test_names=wrapped_test_names, results=dense_results).render()
+
+
+if __name__ == "__main__":
+    main()