Merge lp:~mkanat/loggerhead/limit-pygments into lp:loggerhead

[Max Kanat-Alexander]

Make pygments only highlight files if they're smaller than 512K, to prevent overloading loggerhead.

A fix for bug 513044--limit pygments to only being used on files 512K or smaller.

[lp:~mkanat/loggerhead/limit-pygments]

409. By Max Kanat-Alexander on 2010-04-22

Don't put parens around the if condition

[Ian Clatworthy (ian-clatworthy)]

The value probably ought to be a configuration setting down the track but this looks a sensible improvement as is. If anything, I'd be tempted to make the default limit even lower, e.g. 256K.

review: Approve

=== modified file 'loggerhead/highlight.py'
--- loggerhead/highlight.py	2009-05-16 06:03:47 +0000
+++ loggerhead/highlight.py	2010-04-22 03:26:24 +0000
@@ -16,6 +16,8 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 #
 
+import cgi
+
 from pygments import highlight as _highlight_func
 from pygments.lexers import guess_lexer, guess_lexer_for_filename, TextLexer
 from pygments.formatters import HtmlFormatter
@@ -23,12 +25,19 @@
 
 DEFAULT_PYGMENT_STYLE = 'colorful'
 
+# Trying to highlight very large files using pygments was killing
+# loggerhead on launchpad.net, because pygments isn't very fast.
+# So we only highlight files if they're 512K or smaller.
+MAX_HIGHLIGHT_SIZE = 512000;
 
 def highlight(path, text, encoding, style=DEFAULT_PYGMENT_STYLE):
     """
     Returns a list of highlighted (i.e. HTML formatted) strings.
     """
 
+    if len(text) > MAX_HIGHLIGHT_SIZE:
+        return map(cgi.escape, text.split('\n'))
+
     formatter = HtmlFormatter(style=style, nowrap=True, classprefix='pyg-')
 
     try: