Merge lp:~jtv/launchpad/message-sharing-rearrange-commits into lp:launchpad

[Jeroen T. Vermeulen]

[Jeroen T. Vermeulen (jtv)]

= Rearranging gc and commits in message-sharing migration =

Memory usage is still an issue in the message-sharing migration script. It commits and gc's periodically, but in phase 3 we still see increasing object counts.

This script is a minor tweak to give us better debug output: it makes the explicit garbage-collection runs happen just after a commit, when there's an absolute minimum number of objects still pinned in memory by the ongoing transaction.

There's no functional impact, but the change makes sense even outside the debugging context because it's... well, it's... uh, it's just better.

No lint, no tests (well, run {{{bin/test -vv -t message.sharing}}} if you feel like it) and no miracle cures for this one.

Jeroen

[Stuart Bishop (stub)]

Outside of debugging, we won't want the explicit garbage collection calls at all.

I agree this makes sense for debugging.

review: Approve

=== modified file 'lib/lp/translations/scripts/message_sharing_migration.py'
--- lib/lp/translations/scripts/message_sharing_migration.py	2009-10-30 11:18:59 +0000
+++ lib/lp/translations/scripts/message_sharing_migration.py	2009-11-13 15:15:24 +0000
@@ -7,7 +7,6 @@
     ]
 
 
-import gc
 import os
 
 from zope.component import getUtility
@@ -291,15 +290,6 @@
         if self.txn is None:
             return
 
-        if self.commit_count % 100 == 0 or not intermediate:
-            freed = gc.collect()
-            objcount = len(gc.get_objects())
-            garbage = len(gc.garbage)
-            memsize = open("/proc/%s/statm" % os.getpid()).read().split()[5]
-            log.debug(
-                "Freed: %d.  Object count: %d.  Garbage: %d.  Memory size: %s"
-                % (freed, objcount, garbage, memsize))
-
         self.commit_count += 1
 
         if intermediate and self.commit_count % 10 != 0: