Merge lp:~benji/lazr.restful/faster-wadl-generation into lp:lazr.restful

[Benji York]

Added an optimization to the WADL docstring handling that results in a 30%
decrease in WADL generation time for large files.

[Leonard Richardson (leonardr)]

The change itself looks fine. We should try to get something like this into epydoc, but this is a big enough improvement to justify some temporary messiness.

But, the code is much messier than it needs to be. It's bad to have lots of code outside classes and function definitions, since it's difficult to pinpoint exactly where that code "lives". Can you contain all that code in a class called _PydocParser, then instantiate it as a top-level object called _PYDOC_PARSER and invoke its parse_docstring() method as needed.

review: Approve

[Benji York (benji)]

On Thu, Nov 18, 2010 at 4:23 PM, Leonard Richardson
<email address hidden> wrote:
> Review: Approve
> The change itself looks fine. We should try to get something like this
> into epydoc, but this is a big enough improvement to justify some
> temporary messiness.

Agreed. I'll be contacting the epydoc project.

> But, the code is much messier than it needs to be. It's bad to have
> lots of code outside classes and function definitions, since it's
> difficult to pinpoint exactly where that code "lives". Can you contain
> all that code in a class called _PydocParser, then instantiate it as a
> top-level object called  _PYDOC_PARSER and invoke its
> parse_docstring() method as needed.

Done.
--
Benji York

[lp:~benji/lazr.restful/faster-wadl-generation]

157. By Benji York on 2010-11-18

encapsulate the various bits needed for the parse_docstring replacement

=== modified file 'src/lazr/restful/NEWS.txt'
--- src/lazr/restful/NEWS.txt	2010-10-25 16:31:09 +0000
+++ src/lazr/restful/NEWS.txt	2010-11-18 22:14:26 +0000
@@ -2,6 +2,12 @@
 NEWS for lazr.restful
 =====================
 
+0.15.0 (unreleased)
+===================
+
+Added an optimization to the WADL docstring handling that results in a 30%
+decrease in WADL generation time for large files.
+
 0.14.1 (2010-10-24)
 ===================
 

=== modified file 'src/lazr/restful/tales.py'
--- src/lazr/restful/tales.py	2010-08-24 18:57:50 +0000
+++ src/lazr/restful/tales.py	2010-11-18 22:14:26 +0000
@@ -12,7 +12,13 @@
 import urllib
 
 from epydoc.markup import DocstringLinker
-from epydoc.markup.restructuredtext import parse_docstring
+from epydoc.markup.restructuredtext import (
+    _DocumentPseudoWriter,
+    _EpydocReader,
+    ParsedRstDocstring,
+    )
+from docutils import io
+from docutils.core import Publisher
 
 from zope.component import (
     adapts, getGlobalSiteManager, getUtility, queryMultiAdapter)
@@ -61,6 +67,49 @@
         return indexterm
 
 
+class _PydocParser:
+    """Encapsulate the state/objects needed to parse docstrings."""
+
+    def __init__(self):
+        # Set up the instance we'll be using to render docstrings.
+        self.errors = []
+        self.writer = _DocumentPseudoWriter()
+        self.publisher = Publisher(_EpydocReader(self.errors),
+            writer=self.writer,
+            source_class=io.StringInput)
+        self.publisher.set_components('standalone', 'restructuredtext',
+            'pseudoxml')
+        settings_overrides={
+            'report_level':10000,
+            'halt_level':10000,
+            'warning_stream':None,
+            }
+        self.publisher.process_programmatic_settings(None,
+            settings_overrides, None)
+        self.publisher.set_destination()
+
+
+    def parse_docstring(self, docstring, errors):
+        """Parse a docstring for eventual transformation into HTML
+
+        This function is a replacement for parse_docstring from
+        epydoc.markup.restructuredtext.parse_docstring.  This function reuses
+        the Publisher instance while the original did not.  Using This
+        function yields significantly faster WADL generation for complex
+        systems.
+        """
+        # Clear any errors from previous calls.
+        del self.errors[:]
+        self.publisher.set_source(docstring, None)
+        self.publisher.publish()
+        # Move any errors into the caller-provided list.
+        errors[:] = self.errors[:]
+        return ParsedRstDocstring(self.writer.document)
+
+
+_PYDOC_PARSER = _PydocParser()
+
+
 WADL_DOC_TEMPLATE = (
     '<wadl:doc xmlns="http://www.w3.org/1999/xhtml">\n%s\n</wadl:doc>')
 
@@ -76,7 +125,7 @@
         return None
     doc = "%s\n%s" % (lines[0], textwrap.dedent("\n".join(lines[1:])))
     errors = []
-    parsed = parse_docstring(doc, errors)
+    parsed = _PYDOC_PARSER.parse_docstring(doc, errors)
     if len(errors) > 0:
         messages = [str(error) for error in errors]
         raise AssertionError(

=== modified file 'src/lazr/restful/version.txt'
--- src/lazr/restful/version.txt	2010-10-25 16:31:09 +0000
+++ src/lazr/restful/version.txt	2010-11-18 22:14:26 +0000
@@ -1,1 +1,1 @@
-0.14.1
+0.15.0