Merge lp:~jameinel/launchpad/loggerhead-clear-cache into lp:launchpad

[John A Meinel]

When loggerhead finished with a branch, have it clear its caches, rather than waiting for the GC to do so.

This is a small tweak for the Launchpad Loggerhead app that drive bazaar.launchpad.net.

Specifically, every request to loggerhead gets a new bzrlib.branch.Branch object created. Which then loads data for all the indexes, etc. A lot of this gets cached on the assumption that the object will get re-used. However, these objects are *not* shared between Loggerhead requests. It is a known issue that they also don't go away by simple ref-count (they require the garbage collector to run to remove circular references).

So this is just helping things along. Once we've finished an HTTP request, directly request that caches at least get cleared out.

In doing a test of just spidering of the launchpad codebase using bazaar.launchpad.dev on my local machine. This dropped the peak memory of 'start-loggerhead.py' down from 500+MB down to around 130-200MB.

A different option would be to look into changing how branches are opened, and possibly caching them. It would theoretically help performance a lot, we wouldn't be re-reading all the same content over and over. However at this point, I'd rather drop a cache than add a new one.

[Robert Collins (lifeless)]

review: Approve

=== modified file 'lib/launchpad_loggerhead/app.py'
--- lib/launchpad_loggerhead/app.py	2012-06-29 08:40:05 +0000
+++ lib/launchpad_loggerhead/app.py	2012-08-15 21:05:07 +0000
@@ -286,6 +286,12 @@
                     served_url=None, private=private)
                 return view.app(environ, start_response)
             finally:
+                bzr_branch.repository.revisions.clear_cache()
+                bzr_branch.repository.signatures.clear_cache()
+                bzr_branch.repository.inventories.clear_cache()
+                if bzr_branch.repository.chk_bytes is not None:
+                    bzr_branch.repository.chk_bytes.clear_cache()
+                bzr_branch.repository.texts.clear_cache()
                 bzr_branch.unlock()
         finally:
             lp_server.stop_server()