Merge lp:~jameinel/loggerhead/trunk-into-experimental into lp:~loggerhead-team/loggerhead/experimental

[John A Meinel]

Merge lp:loggerhead back into the experimental branch, to prepare it for future synchronization.

This merges lp:loggerhead back into the experimental branch, and resolves all the small conflicts therein.

It shouldn't be too controversial. The small conflicts were mostly because of similar code being done differently between the old pqm branch and the old trunk.

The one thing is that this changes the inventory page to show the View and Download links, rather than showing Raw and Download links.

[lp:~jameinel/loggerhead/trunk-into-experimental]

452. By John A Meinel on 2011-02-10

Bring in the load-test updates as well.

[Michael Hudson-Doyle (mwhudson)]

review: Approve

[Max Kanat-Alexander (mkanat)]

Why are you changing the color scheme and removing the search box?

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/10/2011 5:54 PM, Max Kanat-Alexander wrote:
> Why are you changing the color scheme and removing the search box?

I'm merging what was in trunk into experimental, and what is in trunk is
what was in pqm.

If you have suggestions for how to restore it, great! It certainly
wasn't intended to remove functionality, just get things up to speed.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1Ue2AACgkQJdeBCYSNAAN+6gCfa1M+pk2k1oB0VEBZCxJvujV5
x78AniD7Mfh/FeDFCBybtR29amwvtldU
=I4Ja
-----END PGP SIGNATURE-----

[Max Kanat-Alexander (mkanat)]

Ah, don't remove the search box, and don't change global.css. Merging what was in pqm into what was in experimental should not have been necessary, except for the fixes that you made to pqm.

[Robert Collins (lifeless)]

On Fri, Feb 11, 2011 at 1:11 PM, Max Kanat-Alexander
<email address hidden> wrote:
> Ah, don't remove the search box, and don't change global.css. Merging what was in pqm into what was in experimental should not have been necessary, except for the fixes that you made to pqm.

Actually merging trunk to experimental is a great idea because trunk
is where development is happening. It makes experimental reflect what
the merged branch will look like, when the issues with experimental
are resolved.

That said, the search box was and is a cute widget, but it scales very
poorly. I suspect we need a switch to control it (in trunk). It would
be nice to have it available, I think I missed that it was being
changed in the merge from pqm -> trunk.

[Max Kanat-Alexander (mkanat)]

Yes, of course experimental should be kept up.

However, you made "trunk" into something that's Launchpad-specific, so there is some stuff that doesn't apply to other people, on "trunk" now. We removed features on the pqm branch because Launchpad couldn't use them, but everybody else still uses them.

So during this one merge, it's important to not merge the Launchpad-specific removals (or the color scheme change) back into experimental (and in fact they should not normally be there on something called "trunk" either).

Most of the changes in this merge are fine, but this one merge needs to be looked over more carefully. I imagine future merges from trunk will be fine.

[Robert Collins (lifeless)]

On Fri, Feb 11, 2011 at 1:25 PM, Max Kanat-Alexander
<email address hidden> wrote:
> Yes, of course experimental should be kept up.
>
> However, you made "trunk" into something that's Launchpad-specific, so there is some stuff that doesn't apply to other people, on "trunk" now. We removed features on the pqm branch because Launchpad couldn't use them, but everybody else still uses them.

Actually, we didn't make trunk Launchpad /specific/. We made it
Launchpad /acceptable/ : thats a big difference. The implementation
may have missed some nuances however, which will get sorted out soon.

> So during this one merge, it's important to not merge the Launchpad-specific removals (or the color scheme change) back into experimental (and in fact they should not normally be there on something called "trunk" either).

Whats appropriate for trunk is appropriate for experimental; it would
be a mistake to filter these things at the experimental layer, rather
the mistakes need to be fi
> --
> https://code.launchpad.net/~jameinel/loggerhead/trunk-into-experimental/+merge/49310
> Your team Loggerhead-team is subscribed to branch lp:~loggerhead-team/loggerhead/experimental.
>

xed in trunk.

> Most of the changes in this merge are fine, but this one merge needs to be looked over more carefully. I imagine future merges from trunk will be fine.

A merge from trunk should always be a no brainer. As should this one.

-Rob

[Max Kanat-Alexander (mkanat)]

Right, so I'll try that again:

1) Don't change the color scheme to Launchpad's, because the upstream loggerhead color scheme is better.

2) Don't remove the search box from experimental.

3) Don't comment out the raw link in experimental.

[Robert Collins (lifeless)]

On Fri, Feb 11, 2011 at 2:17 PM, Max Kanat-Alexander
<email address hidden> wrote:
> Right, so I'll try that again:
>
> 1) Don't change the color scheme to Launchpad's, because the upstream loggerhead color scheme is better.

> 2) Don't remove the search box from experimental.
>
> 3) Don't comment out the raw link in experimental.

So these changes already hit trunk; I think we should:
 - propogate them to experimental
 - fix in trunk and propogate that fix likewise

otherwise we're just creating conflicts on future merges to experimental.

Could I get a screenshot of the two colour schemes to show to a few folk?

-Rob

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/10/2011 7:17 PM, Max Kanat-Alexander wrote:
> Right, so I'll try that again:
>
> 1) Don't change the color scheme to Launchpad's, because the upstream loggerhead color scheme is better.
>
> 2) Don't remove the search box from experimental.
>
> 3) Don't comment out the raw link in experimental.

I'll put together a proposal for the first two. But I'd like to ask for
more info on the 3rd.

I like having *access* to the raw content, but feel the link should be
on the View page. So the standard linking is:

Inventory => View => Annotate
                  => Raw

At least, IMO, leaving the HTML world of links and navigation should be
a last/final step, not a standard step while browsing.

Especially since you can also do neat things like share a link to an
exact line, rather than just the whole document, etc.

I'm certainly interested in hearing what you think, though. Is it just a
performance thing?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1UsK8ACgkQJdeBCYSNAAOxLgCfYc8zDN8qZvqB35lJzRIx8Nrg
7ZgAnjheqLoufFrrpADu69azI22Bovgr
=Vm81
-----END PGP SIGNATURE-----

[Max Kanat-Alexander (mkanat)]

Well, the raw controller was originally added with the idea that if your view was taking too long, you could get an instantaneous raw view--it was implemented for performance reasons. Also, the number of links on View is getting pretty large. However, I do agree that for navigational purposes, it does make more sense on the View page.

[Max Kanat-Alexander (mkanat)]

Also, you could put the link in both places--it's just a tiny icon on the Inventory page.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/10/2011 7:17 PM, Max Kanat-Alexander wrote:
> Right, so I'll try that again:
>
> 1) Don't change the color scheme to Launchpad's, because the upstream loggerhead color scheme is better.
>

I don't know how we will settle this one, but for now it is:
https://bugs.launchpad.net/loggerhead/+bug/718968

> 2) Don't remove the search box from experimental.
>

https://bugs.launchpad.net/loggerhead/+bug/718978

This should be as simple as a 'tal:condition' that uses some config
entry to set it as available. I imagine the default should be available
if bzr-search is available, but Launchpad would want it disabled.

> 3) Don't comment out the raw link in experimental.

https://bugs.launchpad.net/loggerhead/+bug/718982

If you have a chance, please look at the bugs to make sure I've captured
your concerns and the state of the discussion.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1ZlIMACgkQJdeBCYSNAAMSogCgtQKuhOSMCIy9PHIyHwQWxfID
hQ0AoJRMzhybOeIpMgmjUS8gOLrlkoxF
=hzIr
-----END PGP SIGNATURE-----

=== added file 'HACKING'
--- HACKING	1970-01-01 00:00:00 +0000
+++ HACKING	2011-02-10 23:12:12 +0000
@@ -0,0 +1,59 @@
+Loggerhead
+==========
+
+Overview
+--------
+
+This document attempts to give some hints for people that are wanting to work
+on Loggerhead.
+
+
+Testing
+-------
+
+You can run the loggerhead test suite as a bzr plugin. To run just the
+loggerhead tests::
+
+  bzr selftest -s bp.loggerhead
+
+
+Load Testing
+------------
+
+As a web service, Loggerhead will often be hit by multiple requests. We want
+to make sure that loggerhead can scale with many requests, without performing
+poorly or crashing under the load.
+
+There is a command ``bzr load-test-loggerhead`` that can be run to stress
+loggerhead. A script is given, describing what requests to make, against what
+URLs, and for what level of parallel activity.
+
+
+Load Testing Multiple Instances
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+One way that Launchpad provides both high availability and performance scaling
+is by running multiple instances of loggerhead, serving the same content. A
+proxy is then used to load balance the requests. This also allows us to shut
+down one instance for upgrading, without interupting service (requests are
+just routed to the other instance).
+
+However, multiple processes poses an even greater risk that caches will
+conflict. As such, it is useful to test that changes don't introduce coherency
+issues at load. ``bzr load-test-loggerhead`` can be configured with a script
+that will make requests against multiple loggerhead instances concurrently.
+
+To run multiple instances, it is often sufficient to just spawn multiple
+servers on different ports. For example::
+
+  $ bzr serve --http --port=8080 &
+  $ bzr serve --http --port=8081 &
+
+There is a simple example script already in the source tree::
+
+  $ bzr load-test-loggerhead load_test_scripts/multiple_instances.script
+
+
+
+.. vim: ft=rst tw=78
+

=== modified file 'NEWS'
--- NEWS	2011-01-11 00:09:54 +0000
+++ NEWS	2011-02-10 23:12:12 +0000
@@ -4,13 +4,26 @@
 dev [future]
 ------------
 
-    - Remove ``start-loggerhead`` and ``stop-loggerhead`` which were already
-      deprecated. (John Arbash Meinel)
+    - Add ``bzr load-test-loggerhead`` as a way to make sure loggerhead can
+      handle concurrent requests, etc. Scripts can be written that spawn
+      multiple threads, and issue concurrent requests.
+      (John Arbash Meinel)
+
+    - If we get a HEAD request, there is no reason to expand the template, we
+      shouldn't be returning body content anyway.
+      (John Arbash Meinel, #716201, #716217)
 
     - Make the /download/ URLs use revnos instead of revids. This makes
       download URLs simpler and allows you to have a single download URL
       for the latest version of a file. (Max Kanat-Alexander, #322295)
 
+    - Merge the pqm changes back into trunk, after trunk was reverted to an old
+      revision. (John Arbash Meinel, #716152)
+
+    - Remove ``start-loggerhead`` and ``stop-loggerhead`` which were already
+      deprecated. (John Arbash Meinel)
+
+
 1.18 [10Nov2010]
 ----------------
 

=== modified file '__init__.py'
--- __init__.py	2010-11-11 03:36:24 +0000
+++ __init__.py	2011-02-10 23:12:12 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009, 2010 Canonical Ltd
+# Copyright 2009, 2010, 2011 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -38,21 +38,11 @@
 if __name__ == 'bzrlib.plugins.loggerhead':
     import bzrlib
     from bzrlib.api import require_any_api
+    from bzrlib import commands
 
     require_any_api(bzrlib, bzr_compatible_versions)
 
-    # NB: Normally plugins should lazily load almost everything, but this
-    # seems reasonable to have in-line here: bzrlib.commands and options are
-    # normally loaded, and the rest of loggerhead won't be loaded until serve
-    # --http is run.
-
-    # transport_server_registry was added in bzr 1.16. When we drop support for
-    # older releases, we can remove the code to override cmd_serve.
-
-    try:
-        from bzrlib.transport import transport_server_registry
-    except ImportError:
-        transport_server_registry = None
+    from bzrlib.transport import transport_server_registry
 
     DEFAULT_HOST = '0.0.0.0'
     DEFAULT_PORT = 8080
@@ -83,7 +73,6 @@
 
 
 
-
     def _ensure_loggerhead_path():
         """Ensure that you can 'import loggerhead' and get the root."""
         # loggerhead internal code will try to 'import loggerhead', so
@@ -116,41 +105,27 @@
         app = HTTPExceptionHandler(app)
         serve(app, host=host, port=port)
 
-    if transport_server_registry is not None:
-        transport_server_registry.register('http', serve_http, help=HELP)
-    else:
-        import bzrlib.builtins
-        from bzrlib.commands import get_cmd_object, register_command
-        from bzrlib.option import Option
-
-        _original_command = get_cmd_object('serve')
-
-        class cmd_serve(bzrlib.builtins.cmd_serve):
-            __doc__ = _original_command.__doc__
-
-            takes_options = _original_command.takes_options + [
-                Option('http', help=HELP)]
-
-            def run(self, *args, **kw):
-                if 'http' in kw:
-                    from bzrlib.transport import get_transport
-                    allow_writes = kw.get('allow_writes', False)
-                    path = kw.get('directory', '.')
-                    port = kw.get('port', DEFAULT_PORT)
-                    # port might be an int already...
-                    if isinstance(port, basestring) and ':' in port:
-                        host, port = port.split(':')
-                    else:
-                        host = DEFAULT_HOST
-                    if allow_writes:
-                        transport = get_transport(path)
-                    else:
-                        transport = get_transport('readonly+' + path)
-                    serve_http(transport, host, port)
-                else:
-                    super(cmd_serve, self).run(*args, **kw)
-
-        register_command(cmd_serve)
+    transport_server_registry.register('http', serve_http, help=HELP)
+
+    class cmd_load_test_loggerhead(commands.Command):
+        """Run a load test against a live loggerhead instance.
+
+        Pass in the name of a script file to run. See loggerhead/load_test.py
+        for a description of the file format.
+        """
+
+        takes_args = ["filename"]
+
+        def run(self, filename):
+            from bzrlib.plugins.loggerhead.loggerhead import load_test
+            script = load_test.run_script(filename)
+            for thread_id in sorted(script._threads):
+                worker = script._threads[thread_id][0]
+                for url, success, time in worker.stats:
+                    self.outf.write(' %5.3fs %s %s\n'
+                                    % (time, str(success)[0], url))
+
+    commands.register_command(cmd_load_test_loggerhead)
 
     def load_tests(standard_tests, module, loader):
         _ensure_loggerhead_path()

=== added directory 'load_test_scripts'
=== added file 'load_test_scripts/multiple_instances.script'
--- load_test_scripts/multiple_instances.script	1970-01-01 00:00:00 +0000
+++ load_test_scripts/multiple_instances.script	2011-02-10 23:12:12 +0000
@@ -0,0 +1,19 @@
+{
+    "comment": "Connect to multiple loggerhead instances and make requests on each. One should be on :8080, one should be on :8081. Multiple threads will place requests on each.",
+    "parameters": {"base_url": "http://localhost"},
+    "requests": [
+        {"thread": "1", "relpath": ":8080/changes"},
+        {"thread": "2", "relpath": ":8080/files"},
+        {"thread": "3", "relpath": ":8081/files"},
+        {"thread": "4", "relpath": ":8081/changes"},
+        {"thread": "1", "relpath": ":8080/changes"},
+        {"thread": "2", "relpath": ":8080/files"},
+        {"thread": "3", "relpath": ":8081/files"},
+        {"thread": "4", "relpath": ":8081/changes"},
+        {"thread": "1", "relpath": ":8080/changes"},
+        {"thread": "2", "relpath": ":8080/files"},
+        {"thread": "3", "relpath": ":8081/files"},
+        {"thread": "4", "relpath": ":8081/changes"}
+    ]
+}
+

=== added file 'load_test_scripts/simple.script'
--- load_test_scripts/simple.script	1970-01-01 00:00:00 +0000
+++ load_test_scripts/simple.script	2011-02-10 23:12:12 +0000
@@ -0,0 +1,9 @@
+{
+    "comment": "A fairly trivial load test script. It just loads the main two pages from a loggerhead install running directly on a branch.",
+    "parameters": {"base_url": "http://localhost:8080"},
+    "requests": [
+        {"relpath": "/changes"},
+        {"relpath": "/files"}
+    ]
+}
+

=== modified file 'loggerhead/controllers/__init__.py'
--- loggerhead/controllers/__init__.py	2011-01-11 00:30:17 +0000
+++ loggerhead/controllers/__init__.py	2011-02-10 23:12:12 +0000
@@ -84,9 +84,9 @@
         if len(args) > 1:
             path = unicode('/'.join(args[1:]), 'utf-8')
         
-        return self.get_output(path, kwargs, start_response)
+        return self.get_output(path, kwargs, start_response, environ)
     
-    def get_output(self, path, kwargs, start_response):
+    def get_output(self, path, kwargs, start_response, environ):
         vals = {
             'static_url': self._branch.static_url,
             'branch': self._branch,
@@ -103,6 +103,9 @@
         if 'Content-Type' not in headers:
             headers['Content-Type'] = 'text/html'
         writer = start_response("200 OK", headers.items())
+        if environ.get('REQUEST_METHOD') == 'HEAD':
+            # No content for a HEAD request
+            return []
         template = load_template(self.template_path)
         z = time.time()
         w = BufferingWriter(writer, 8192)
@@ -151,4 +154,4 @@
 
     def tree_for(self, file_id, revid):
         file_revid = self._history.get_inventory(revid)[file_id].revision
-        return self._history._branch.repository.revision_tree(file_revid)
\ No newline at end of file
+        return self._history._branch.repository.revision_tree(file_revid)

=== modified file 'loggerhead/controllers/raw_ui.py'
--- loggerhead/controllers/raw_ui.py	2011-01-11 00:02:53 +0000
+++ loggerhead/controllers/raw_ui.py	2011-02-10 23:12:12 +0000
@@ -24,9 +24,12 @@
 
 from loggerhead.controllers import TemplatedBranchView
 
+# Note: RawUI should probably not be a TemplatedBranchView, because, quite
+#       frankly, it doesn't use a template...
+
 class RawUI(TemplatedBranchView):
 
-    def get_output(self, path, kwargs, start_response):
+    def get_output(self, path, kwargs, start_response, environ):
         # /raw/<revno>/[path]
         file_info = self.file_info(path, kwargs)
         revid = file_info['revid']
@@ -37,13 +40,13 @@
         tree = self.tree_for(file_id, revid)
         content = tree.get_file_lines(file_id)
         size = tree.inventory[file_id].text_size
-        
+
         mime_type = self.get_mime_type(content, file_info)
         headers = self.get_headers(mime_type, size, file_info)
-        
+
         self.log.info('/raw %s @ %s (%d bytes): %.3f secs',
               file_info['path'], revid, size, time.time() - self._call_start)
-        
+
         start_response('200 OK', headers)
         return content
 
@@ -64,4 +67,4 @@
             ('Content-Type', mime_type),
             ('Content-Length', str(size)),
             ('X-Content-Type-Options', 'nosniff'),
-        ]
\ No newline at end of file
+        ]

=== modified file 'loggerhead/controllers/view_ui.py'
--- loggerhead/controllers/view_ui.py	2010-12-03 17:11:21 +0000
+++ loggerhead/controllers/view_ui.py	2011-02-10 23:12:12 +0000
@@ -37,11 +37,11 @@
 class ViewUI(TemplatedBranchView):
 
     template_path = 'loggerhead.templates.view'
-    
+
     def text_lines(self, file_info):
         file_id = file_info['file_id']
         revid = file_info['revid']
-        
+
         tree = self.tree_for(file_id, revid)
         file_text = tree.get_file_text(file_id)
         encoding = 'utf-8'
@@ -54,7 +54,7 @@
         file_lines = bzrlib.osutils.split_lines(file_text)
         # This can throw bzrlib.errors.BinaryFile (which our caller catches).
         bzrlib.textfile.check_text_lines(file_lines)
-        
+
         if highlight is not None:
             hl_lines = highlight(file_info['filename'], file_text, encoding)
             # highlight strips off extra newlines at the end of the file.

=== added file 'loggerhead/load_test.py'
--- loggerhead/load_test.py	1970-01-01 00:00:00 +0000
+++ loggerhead/load_test.py	2011-02-10 23:12:12 +0000
@@ -0,0 +1,233 @@
+# Copyright 2011 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+"""Code to do some load testing against a loggerhead instance.
+
+This is basically meant to take a list of actions to take, and run it against a
+real host, and see how the results respond.::
+
+    {"parameters": {
+         "base_url": "http://localhost:8080",
+     },
+     "requests": [
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"}
+     ],
+    }
+
+All threads have a Queue length of 1. When a third request for a given thread
+is seen, no more requests are queued until that thread finishes its current
+job. So this results in all requests being issued sequentially::
+
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"}
+
+While this would cause all requests to be sent in parallel:
+
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "2", "relpath": "/changes"},
+        {"thread": "3", "relpath": "/changes"},
+        {"thread": "4", "relpath": "/changes"}
+
+This should keep 2 threads pipelined with activity, as long as they finish in
+approximately the same speed. We'll start the first thread running, and the
+second thread, and queue up both with a second request once the first finishes.
+When we get to the third request for thread "1", we block on queuing up more
+work until the first thread 1 request has finished.
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "2", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "2", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "2", "relpath": "/changes"}
+
+There is not currently a way to say "run all these requests keeping exactly 2
+threads active". Though if you know the load pattern, you could approximate
+this.
+"""
+
+import threading
+import time
+import Queue
+
+try:
+    import simplejson
+except ImportError:
+    import json as simplejson
+
+from bzrlib import (
+    errors,
+    transport,
+    urlutils,
+    )
+
+# This code will be doing multi-threaded requests against bzrlib.transport
+# code. We want to make sure to load everything ahead of time, so we don't get
+# lazy-import failures
+_ = transport.get_transport('http://example.com')
+
+
+class RequestDescription(object):
+    """Describes info about a request."""
+
+    def __init__(self, descrip_dict):
+        self.thread = descrip_dict.get('thread', '1')
+        self.relpath = descrip_dict['relpath']
+
+
+class RequestWorker(object):
+    """Process requests in a worker thread."""
+
+    _timer = time.time
+
+    def __init__(self, identifier, blocking_time=1.0, _queue_size=1):
+        self.identifier = identifier
+        self.queue = Queue.Queue(_queue_size)
+        self.start_time = self.end_time = None
+        self.stats = []
+        self.blocking_time = blocking_time
+
+    def step_next(self):
+        url = self.queue.get(True, self.blocking_time)
+        if url == '<noop>':
+            # This is usually an indicator that we want to stop, so just skip
+            # this one.
+            self.queue.task_done()
+            return
+        self.start_time = self._timer()
+        success = self.process(url)
+        self.end_time = self._timer()
+        self.update_stats(url, success)
+        self.queue.task_done()
+
+    def run(self, stop_event):
+        while not stop_event.isSet():
+            try:
+                self.step_next()
+            except Queue.Empty:
+                pass
+
+    def process(self, url):
+        base, path = urlutils.split(url)
+        t = transport.get_transport(base)
+        try:
+            # TODO: We should probably look into using some part of
+            #       blocking_timeout to decide when to stop trying to read
+            #       content
+            content = t.get_bytes(path)
+        except (errors.TransportError, errors.NoSuchFile):
+            return False
+        return True
+
+    def update_stats(self, url, success):
+        self.stats.append((url, success, self.end_time - self.start_time))
+
+
+class ActionScript(object):
+    """This tracks the actions that we want to perform."""
+
+    _worker_class = RequestWorker
+    _default_base_url = 'http://localhost:8080'
+    _default_blocking_timeout = 60.0
+
+    def __init__(self):
+        self.base_url = self._default_base_url
+        self.blocking_timeout = self._default_blocking_timeout
+        self._requests = []
+        self._threads = {}
+        self.stop_event = threading.Event()
+
+    @classmethod
+    def parse(cls, content):
+        script = cls()
+        json_dict = simplejson.loads(content)
+        if 'parameters' not in json_dict:
+            raise ValueError('Missing "parameters" section')
+        if 'requests' not in json_dict:
+            raise ValueError('Missing "requests" section')
+        param_dict = json_dict['parameters']
+        request_list = json_dict['requests']
+        base_url = param_dict.get('base_url', None)
+        if base_url is not None:
+            script.base_url = base_url
+        blocking_timeout = param_dict.get('blocking_timeout', None)
+        if blocking_timeout is not None:
+            script.blocking_timeout = blocking_timeout
+        for request_dict in request_list:
+            script.add_request(request_dict)
+        return script
+
+    def add_request(self, request_dict):
+        request = RequestDescription(request_dict)
+        self._requests.append(request)
+
+    def _get_worker(self, thread_id):
+        if thread_id in self._threads:
+            return self._threads[thread_id][0]
+        handler = self._worker_class(thread_id,
+                                     blocking_time=self.blocking_timeout/10.0)
+
+        t = threading.Thread(target=handler.run, args=(self.stop_event,),
+                             name='Thread-%s' % (thread_id,))
+        self._threads[thread_id] = (handler, t)
+        t.start()
+        return handler
+
+    def finish_queues(self):
+        """Wait for all queues of all children to finish."""
+        for h, t in self._threads.itervalues():
+            h.queue.join()
+
+    def stop_and_join(self):
+        """Stop all running workers, and return.
+
+        This will stop even if workers still have work items.
+        """
+        self.stop_event.set()
+        for h, t in self._threads.itervalues():
+            # Signal the queue that it should stop blocking, we don't have to
+            # wait for the queue to empty, because we may see stop_event before
+            # we see the <noop>
+            h.queue.put('<noop>')
+            # And join the controlling thread
+            for i in range(10):
+                t.join(self.blocking_timeout / 10.0)
+                if not t.isAlive():
+                    break
+
+    def _full_url(self, relpath):
+        return self.base_url + relpath
+
+    def run(self):
+        self.stop_event.clear()
+        for request in self._requests:
+            full_url = self._full_url(request.relpath)
+            worker = self._get_worker(request.thread)
+            worker.queue.put(full_url, True, self.blocking_timeout)
+        self.finish_queues()
+        self.stop_and_join()
+
+
+def run_script(filename):
+    with open(filename, 'rb') as f:
+        content = f.read()
+    script = ActionScript.parse(content)
+    script.run()
+    return script

=== modified file 'loggerhead/static/css/global.css'
--- loggerhead/static/css/global.css	2009-10-17 06:55:25 +0000
+++ loggerhead/static/css/global.css	2011-02-10 23:12:12 +0000
@@ -96,27 +96,27 @@
 	text-align:center;
 	color:#fff;
 	text-decoration:none;
-	background:#2f40a5;
+	background:#d18b39;
 }
 ul#submenuTabs li {
 	float:left;
 	display:inline;
 	margin-left:1px;
 	color:#fff;
-	background:#2f40a5;
+	background:#d18b39;
 }
 ul#submenuTabs li a:hover {
 	float:left;
 	display:inline;
 	color:#fff;
-	background:#2f40a5;
+	background:#d18b39;
 	text-decoration:underline;
 }
 ul#submenuTabs li#first a, ul#submenuTabs li#first a:link, ul#submenuTabs li#first a:visited, ul#submenuTabs li#first a:hover {
-	background:#2f40a5 url(../images/bg_submenuTabs.gif) top left no-repeat;
+	background:#d18b39 url(../images/bg_submenuTabs.gif) top left no-repeat;
 }
 ul#submenuTabs li#last a, ul#submenuTabs li#last a:link, ul#submenuTabs li#last a:visited, ul#submenuTabs li#last a:hover {
-	background:#2f40a5 url(../images/bg_submenuTabs.gif) bottom right no-repeat;
+	background:#d18b39 url(../images/bg_submenuTabs.gif) bottom right no-repeat;
 }
 
 /* Search & RSS */

=== modified file 'loggerhead/static/images/bg_Tabs.gif'
Binary files loggerhead/static/images/bg_Tabs.gif	2008-07-25 19:23:15 +0000 and loggerhead/static/images/bg_Tabs.gif	2011-02-10 23:12:12 +0000 differ
=== modified file 'loggerhead/static/images/bg_menuTabs.gif'
Binary files loggerhead/static/images/bg_menuTabs.gif	2008-07-25 19:23:15 +0000 and loggerhead/static/images/bg_menuTabs.gif	2011-02-10 23:12:12 +0000 differ
=== modified file 'loggerhead/static/images/bg_submenuTabs.gif'
Binary files loggerhead/static/images/bg_submenuTabs.gif	2008-07-25 19:23:15 +0000 and loggerhead/static/images/bg_submenuTabs.gif	2011-02-10 23:12:12 +0000 differ
=== modified file 'loggerhead/templates/inventory.pt'
--- loggerhead/templates/inventory.pt	2011-01-11 00:02:53 +0000
+++ loggerhead/templates/inventory.pt	2011-02-10 23:12:12 +0000
@@ -129,12 +129,19 @@
               <td class="date" tal:content="python:util.date_time(file.change.date)"></td>
               <td class="timedate2" tal:content="python:util.human_size(file.size)"></td>
               <td class="expcell">
+                <a class="view_link"
+                   tal:attributes="href python:url(['/view', revno_url, file.absolutepath]);
+                                   title string:View ${file/filename}">
+                   <img tal:attributes="src python:branch.static_url('/static/images/ico_planilla.gif')" alt="Diff" />
+                </a>
+<!--
                 <a class="raw_link"
                    tal:attributes="href python:url(['/raw', revno_url, file.absolutepath]);
                                    title string:Raw content of ${file/filename}">
                   <img tal:attributes="src python:branch.static_url('/static/images/ico_file.gif')"
                        alt="View raw file" />
                 </a>
+-->
                  <a class="download_link"
                     tal:attributes="href python:url(['/download', revno_url, file.absolutepath]);
                                     title string:Download ${file/filename} at revision ${file/change/revno}">

=== modified file 'loggerhead/templates/macros.pt'
--- loggerhead/templates/macros.pt	2009-10-21 19:17:16 +0000
+++ loggerhead/templates/macros.pt	2011-02-10 23:12:12 +0000
@@ -45,14 +45,6 @@
 
 <body>
     <!-- Loggerhead Content Area -->
-      <div id="finderBox">
-         <tal:search-box define="navigation navigation"
-                        replace="structure python:search_box(branch,
-                                 navigation)" />
-        <div>
-          <tal:feed-link replace="structure python:feed_link(branch, url)" />
-        </div>
-      </div>
 
       <tal:menu define="fileview_active fileview_active"
                 replace="structure python:menu(branch, url, fileview_active)" />

=== modified file 'loggerhead/templates/revision.pt'
--- loggerhead/templates/revision.pt	2010-11-30 10:18:40 +0000
+++ loggerhead/templates/revision.pt	2011-02-10 23:12:12 +0000
@@ -20,7 +20,6 @@
     </head>
 
     <body>
-
       <tal:block metal:fill-slot="heading">
         <h1>
           <tal:has-link condition="branch/branch_link">

=== modified file 'loggerhead/tests/__init__.py'
--- loggerhead/tests/__init__.py	2011-01-28 22:44:19 +0000
+++ loggerhead/tests/__init__.py	2011-02-10 23:12:12 +0000
@@ -21,6 +21,7 @@
             'history_db_tests',
             'test_controllers',
             'test_corners',
+            'test_load_test',
             'test_simple',
             'test_templating',
         ]]))

=== modified file 'loggerhead/tests/test_controllers.py'
--- loggerhead/tests/test_controllers.py	2011-01-31 21:27:06 +0000
+++ loggerhead/tests/test_controllers.py	2011-02-10 23:12:12 +0000
@@ -1,3 +1,10 @@
+from cStringIO import StringIO
+import logging
+
+from paste.httpexceptions import HTTPServerError
+
+from bzrlib import errors
+
 from loggerhead.apps.branch import BranchWSGIApp
 from loggerhead.controllers.annotate_ui import AnnotateUI
 from loggerhead.controllers.inventory_ui import InventoryUI
@@ -15,15 +22,50 @@
         tree.commit('')
         tree.branch.lock_read()
         self.addCleanup(tree.branch.unlock)
-        branch_app = BranchWSGIApp(tree.branch)
+        branch_app = BranchWSGIApp(tree.branch, '')
+        branch_app.log.setLevel(logging.CRITICAL)
+        # These are usually set in BranchWSGIApp.app(), which is set from env
+        # settings set by BranchesFromTransportRoot, so we fake it.
+        branch_app._static_url_base = '/'
+        branch_app._url_base = '/'
         return tree.branch, InventoryUI(branch_app, branch_app.get_history)
 
+    def consume_app(self, app, extra_environ=None):
+        env = {'SCRIPT_NAME': '/files', 'PATH_INFO': ''}
+        if extra_environ is not None:
+            env.update(extra_environ)
+        body = StringIO()
+        start = []
+        def start_response(status, headers, exc_info=None):
+            start.append((status, headers, exc_info))
+            return body.write
+        extra_content = list(app(env, start_response))
+        body.writelines(extra_content)
+        return start[0], body.getvalue()
+
     def test_get_filelist(self):
         bzrbranch, inv_ui = self.make_bzrbranch_and_inventory_ui_for_tree_shape(
             ['filename'])
         inv = bzrbranch.repository.get_inventory(bzrbranch.last_revision())
         self.assertEqual(1, len(inv_ui.get_filelist(inv, '', 'filename')))
 
+    def test_smoke(self):
+        bzrbranch, inv_ui = self.make_bzrbranch_and_inventory_ui_for_tree_shape(
+            ['filename'])
+        start, content = self.consume_app(inv_ui)
+        self.assertEqual(('200 OK', [('Content-Type', 'text/html')], None),
+                         start)
+        self.assertContainsRe(content, 'filename')
+
+    def test_no_content_for_HEAD(self):
+        bzrbranch, inv_ui = self.make_bzrbranch_and_inventory_ui_for_tree_shape(
+            ['filename'])
+        start, content = self.consume_app(inv_ui,
+                            extra_environ={'REQUEST_METHOD': 'HEAD'})
+        self.assertEqual(('200 OK', [('Content-Type', 'text/html')], None),
+                         start)
+        self.assertEqual('', content)
+
 
 class TestRevisionUI(BasicTests):
 

=== added file 'loggerhead/tests/test_load_test.py'
--- loggerhead/tests/test_load_test.py	1970-01-01 00:00:00 +0000
+++ loggerhead/tests/test_load_test.py	2011-02-10 23:12:12 +0000
@@ -0,0 +1,337 @@
+# Copyright 2011 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#
+
+"""Tests for the load testing code."""
+
+import socket
+import time
+import threading
+import Queue
+
+from bzrlib import tests
+from bzrlib.tests import http_server
+
+from bzrlib.plugins.loggerhead.loggerhead import load_test
+
+
+empty_script = """{
+    "parameters": {},
+    "requests": []
+}"""
+
+class TestRequestDescription(tests.TestCase):
+
+    def test_init_from_dict(self):
+        rd = load_test.RequestDescription({'thread': '10', 'relpath': '/foo'})
+        self.assertEqual('10', rd.thread)
+        self.assertEqual('/foo', rd.relpath)
+
+    def test_default_thread_is_1(self):
+        rd = load_test.RequestDescription({'relpath': '/bar'})
+        self.assertEqual('1', rd.thread)
+        self.assertEqual('/bar', rd.relpath)
+
+
+_cur_time = time.time()
+def one_sec_timer():
+    """Every time this timer is called, it increments by 1 second."""
+    global _cur_time
+    _cur_time += 1.0
+    return _cur_time
+
+
+class NoopRequestWorker(load_test.RequestWorker):
+
+    # Every call to _timer will increment by one
+    _timer = staticmethod(one_sec_timer)
+
+    # Ensure that process never does anything
+    def process(self, url):
+        return True
+
+
+class TestRequestWorkerInfrastructure(tests.TestCase):
+    """Tests various infrastructure bits, without doing actual requests."""
+
+    def test_step_next_tracks_time(self):
+        rt = NoopRequestWorker('id')
+        rt.queue.put('item')
+        rt.step_next()
+        self.assertTrue(rt.queue.empty())
+        self.assertEqual([('item', True, 1.0)], rt.stats)
+
+    def test_step_multiple_items(self):
+        rt = NoopRequestWorker('id')
+        rt.queue.put('item')
+        rt.step_next()
+        rt.queue.put('next-item')
+        rt.step_next()
+        self.assertTrue(rt.queue.empty())
+        self.assertEqual([('item', True, 1.0), ('next-item', True, 1.0)],
+                         rt.stats)
+
+    def test_step_next_does_nothing_for_noop(self):
+        rt = NoopRequestWorker('id')
+        rt.queue.put('item')
+        rt.step_next()
+        rt.queue.put('<noop>')
+        rt.step_next()
+        self.assertEqual([('item', True, 1.0)], rt.stats)
+
+    def test_step_next_will_timeout(self):
+        # We don't want step_next to block forever
+        rt = NoopRequestWorker('id', blocking_time=0.001)
+        self.assertRaises(Queue.Empty, rt.step_next)
+
+    def test_run_stops_for_stop_event(self):
+        rt = NoopRequestWorker('id', blocking_time=0.001, _queue_size=2)
+        rt.queue.put('item1')
+        rt.queue.put('item2')
+        event = threading.Event()
+        t = threading.Thread(target=rt.run, args=(event,))
+        t.start()
+        # Wait for the queue to be processed
+        rt.queue.join()
+        # Now we can queue up another one, and wait for it
+        rt.queue.put('item3')
+        rt.queue.join()
+        # Now set the stopping event
+        event.set()
+        # Add another item to the queue, which might get processed, but the
+        # next item won't
+        rt.queue.put('item4')
+        rt.queue.put('item5')
+        t.join()
+        self.assertEqual([('item1', True, 1.0), ('item2', True, 1.0),
+                          ('item3', True, 1.0)],
+                         rt.stats[:3])
+        # The last event might be item4 or might be item3, the important thing
+        # is that even though there are still queued events, we won't
+        # process anything past the first
+        self.assertNotEqual('item5', rt.stats[-1][0])
+
+
+class TestRequestWorker(tests.TestCaseWithTransport):
+
+    def setUp(self):
+        super(TestRequestWorker, self).setUp()
+        self.transport_readonly_server = http_server.HttpServer
+
+    def test_request_items(self):
+        rt = load_test.RequestWorker('id', blocking_time=0.01, _queue_size=2)
+        self.build_tree(['file1', 'file2'])
+        readonly_url1 = self.get_readonly_url('file1')
+        self.assertStartsWith(readonly_url1, 'http://')
+        readonly_url2 = self.get_readonly_url('file2')
+        rt.queue.put(readonly_url1)
+        rt.queue.put(readonly_url2)
+        rt.step_next()
+        rt.step_next()
+        self.assertEqual(readonly_url1, rt.stats[0][0])
+        self.assertEqual(readonly_url2, rt.stats[1][0])
+
+    def test_request_nonexistant_items(self):
+        rt = load_test.RequestWorker('id', blocking_time=0.01, _queue_size=2)
+        readonly_url1 = self.get_readonly_url('no-file1')
+        rt.queue.put(readonly_url1)
+        rt.step_next()
+        self.assertEqual(readonly_url1, rt.stats[0][0])
+        self.assertEqual(False, rt.stats[0][1])
+
+    def test_no_server(self):
+        s = socket.socket()
+        # Bind to a port, but don't listen on it
+        s.bind(('localhost', 0))
+        url = 'http://%s:%s/path' % s.getsockname()
+        rt = load_test.RequestWorker('id', blocking_time=0.01, _queue_size=2)
+        rt.queue.put(url)
+        rt.step_next()
+        self.assertEqual((url, False), rt.stats[0][:2])
+
+
+class NoActionScript(load_test.ActionScript):
+
+    _thread_class = NoopRequestWorker
+    _default_blocking_timeout = 0.01
+
+
+class TestActionScriptInfrastructure(tests.TestCase):
+
+    def test_parse_requires_parameters_and_requests(self):
+        self.assertRaises(ValueError,
+            load_test.ActionScript.parse, '')
+        self.assertRaises(ValueError,
+            load_test.ActionScript.parse, '{}')
+        self.assertRaises(ValueError,
+            load_test.ActionScript.parse, '{"parameters": {}}')
+        self.assertRaises(ValueError,
+            load_test.ActionScript.parse, '{"requests": []}')
+        load_test.ActionScript.parse(
+            '{"parameters": {}, "requests": [], "comment": "section"}')
+        script = load_test.ActionScript.parse(
+            empty_script)
+        self.assertIsNot(None, script)
+
+    def test_parse_default_base_url(self):
+        script = load_test.ActionScript.parse(empty_script)
+        self.assertEqual('http://localhost:8080', script.base_url)
+
+    def test_parse_find_base_url(self):
+        script = load_test.ActionScript.parse(
+            '{"parameters": {"base_url": "http://example.com"},'
+            ' "requests": []}')
+        self.assertEqual('http://example.com', script.base_url)
+
+    def test_parse_default_blocking_timeout(self):
+        script = load_test.ActionScript.parse(empty_script)
+        self.assertEqual(60.0, script.blocking_timeout)
+
+    def test_parse_find_blocking_timeout(self):
+        script = load_test.ActionScript.parse(
+            '{"parameters": {"blocking_timeout": 10.0},'
+            ' "requests": []}'
+        )
+        self.assertEqual(10.0, script.blocking_timeout)
+
+    def test_parse_finds_requests(self):
+        script = load_test.ActionScript.parse(
+            '{"parameters": {}, "requests": ['
+            ' {"relpath": "/foo"},'
+            ' {"relpath": "/bar"}'
+            ' ]}')
+        self.assertEqual(2, len(script._requests))
+        self.assertEqual("/foo", script._requests[0].relpath)
+        self.assertEqual("/bar", script._requests[1].relpath)
+
+    def test__get_worker(self):
+        script = NoActionScript()
+        # If an id is found, then we should create it
+        self.assertEqual({}, script._threads)
+        worker = script._get_worker('id')
+        self.assertTrue('id' in script._threads)
+        # We should have set the blocking timeout
+        self.assertEqual(script.blocking_timeout / 10.0,
+                         worker.blocking_time)
+
+        # Another request will return the identical object
+        self.assertIs(worker, script._get_worker('id'))
+
+        # And the stop event will stop the thread
+        script.stop_and_join()
+
+    def test__full_url(self):
+        script = NoActionScript()
+        self.assertEqual('http://localhost:8080/path',
+                         script._full_url('/path'))
+        self.assertEqual('http://localhost:8080/path/to/foo',
+                         script._full_url('/path/to/foo'))
+        script.base_url = 'http://example.com'
+        self.assertEqual('http://example.com/path/to/foo',
+                         script._full_url('/path/to/foo'))
+        script.base_url = 'http://example.com/base'
+        self.assertEqual('http://example.com/base/path/to/foo',
+                         script._full_url('/path/to/foo'))
+        script.base_url = 'http://example.com'
+        self.assertEqual('http://example.com:8080/path',
+                         script._full_url(':8080/path'))
+
+    def test_single_threaded(self):
+        script = NoActionScript.parse("""{
+            "parameters": {"base_url": ""},
+            "requests": [
+                {"thread": "1", "relpath": "first"},
+                {"thread": "1", "relpath": "second"},
+                {"thread": "1", "relpath": "third"},
+                {"thread": "1", "relpath": "fourth"}
+            ]}""")
+        script.run()
+        worker = script._get_worker("1")
+        self.assertEqual(["first", "second", "third", "fourth"],
+                         [s[0] for s in worker.stats])
+
+    def test_two_threads(self):
+        script = NoActionScript.parse("""{
+            "parameters": {"base_url": ""},
+            "requests": [
+                {"thread": "1", "relpath": "first"},
+                {"thread": "2", "relpath": "second"},
+                {"thread": "1", "relpath": "third"},
+                {"thread": "2", "relpath": "fourth"}
+            ]}""")
+        script.run()
+        worker = script._get_worker("1")
+        self.assertEqual(["first", "third"],
+                         [s[0] for s in worker.stats])
+        worker = script._get_worker("2")
+        self.assertEqual(["second", "fourth"],
+                         [s[0] for s in worker.stats])
+
+
+class TestActionScriptIntegration(tests.TestCaseWithTransport):
+
+    def setUp(self):
+        super(TestActionScriptIntegration, self).setUp()
+        self.transport_readonly_server = http_server.HttpServer
+
+    def test_full_integration(self):
+        self.build_tree(['first', 'second', 'third', 'fourth'])
+        url = self.get_readonly_url()
+        script = load_test.ActionScript.parse("""{
+            "parameters": {"base_url": "%s", "blocking_timeout": 0.1},
+            "requests": [
+                {"thread": "1", "relpath": "first"},
+                {"thread": "2", "relpath": "second"},
+                {"thread": "1", "relpath": "no-this"},
+                {"thread": "2", "relpath": "or-this"},
+                {"thread": "1", "relpath": "third"},
+                {"thread": "2", "relpath": "fourth"}
+            ]}""" % (url,))
+        script.run()
+        worker = script._get_worker("1")
+        self.assertEqual([("first", True), ('no-this', False),
+                          ("third", True)],
+                         [(s[0].rsplit('/', 1)[1], s[1])
+                          for s in worker.stats])
+        worker = script._get_worker("2")
+        self.assertEqual([("second", True), ('or-this', False),
+                          ("fourth", True)],
+                         [(s[0].rsplit('/', 1)[1], s[1])
+                          for s in worker.stats])
+
+
+class TestRunScript(tests.TestCaseWithTransport):
+
+    def setUp(self):
+        super(TestRunScript, self).setUp()
+        self.transport_readonly_server = http_server.HttpServer
+
+    def test_run_script(self):
+        self.build_tree(['file1', 'file2', 'file3', 'file4'])
+        url = self.get_readonly_url()
+        self.build_tree_contents([('localhost.script', """{
+    "parameters": {"base_url": "%s", "blocking_timeout": 0.1},
+    "requests": [
+        {"thread": "1", "relpath": "file1"},
+        {"thread": "2", "relpath": "file2"},
+        {"thread": "1", "relpath": "file3"},
+        {"thread": "2", "relpath": "file4"}
+    ]
+}""" % (url,))])
+        script = load_test.run_script('localhost.script')
+        worker = script._threads["1"][0]
+        self.assertEqual([("file1", True), ('file3', True)],
+                         [(s[0].rsplit('/', 1)[1], s[1])
+                          for s in worker.stats])
+        worker = script._threads["2"][0]
+        self.assertEqual([("file2", True), ("file4", True)],
+                         [(s[0].rsplit('/', 1)[1], s[1])
+                          for s in worker.stats])

=== modified file 'loggerhead/tests/test_simple.py'
--- loggerhead/tests/test_simple.py	2010-05-14 10:58:20 +0000
+++ loggerhead/tests/test_simple.py	2011-02-10 23:12:12 +0000
@@ -95,8 +95,18 @@
     def test_annotate(self):
         app = self.setUpLoggerhead()
         res = app.get('/annotate', params={'file_id': self.fileid})
-        for line in self.filecontents.splitlines():
-            res.mustcontain(cgi.escape(line))
+        # This now fails because pigments is highlighting the lines.
+        # Specifically, instead of getting:
+        # 'with&lt;htmlspecialchars' we are now getting
+        # '<span class='pyg-n'>with</span><span class='pyg-o'>&lt;</span>'
+        # '<span class='pyg-n'>htmlspecialchars</span>
+        # So pygments is breaking up the special characters with custom
+        # highlighting. The alternative is to figure out how to disable
+        # pygments for this test.
+        ## for line in self.filecontents.splitlines():
+        ##     res.mustcontain(cgi.escape(line))
+        self.assertContainsRe(res.body,
+            "with.*&lt;.*htmlspecialchars")
 
     def test_inventory(self):
         app = self.setUpLoggerhead()