Merge lp:~jtv/launchpad/scale-message-sharing-migration into lp:launchpad

[Jeroen T. Vermeulen]

Memory-footprint cuts for message sharing migration script.

[Jeroen T. Vermeulen (jtv)]

= Bug 400544 =

The message-sharing migration script was using far too much memory for large projects and packages. In previous branches (the latest still on approach for landing as I write this) we stopped the script from fetching msgid texts while merging. This branch should further reduce memory footprint.

It doesn't do this by putting less stuff in memory per se. It probably allocates more overall. But it breaks things up so that less needs to be in memory at the same time.

== Details ==

The diff begins with some whitespace cleanups in potmsgset.py. Pay no heed to them. In the same file, getAllTranslationMessages now returns a Storm result set. The caller can now append sorting clauses to it, which was more relevant in an earlier incarnation of this branch, but it's an improvement anyway so I left it in. There is no more default ordering, but I verified that no code relied on the ordering there was anyway.

In the script, moved elimination of certain troublesome duplications among TranslationMessages into a separate phase of execution. Since this brings the number of phases to 3, I also started numbering them clearly in the options list.

You'll note some new intermediate commits between phases. The commit function has evolved to skip intermediate aborts on dry runs; skip commits or aborts when no txn has been set; and to run the garbage collector after every commit in hopes of relieving memory load somewhat.

Next in the diff comes the script's new execution phase. It figures out which potmsgsets are "representative," and cleans up only those. They are the only ones that are a problem for the rest of the algorithm. There are probably few enough other cases that it's not worth going through a full cleanup.

Right below that you'll see how the _scrubPOTMsgSetTranslations invocation was removed from the previous first execution phase (which is now the second).

The extra intermediate commits I've added inside phases are a delicate matter. Underneath Storm and above the database connection, psycopg batches query result sets. If you commit transactions while iterating over a result set (especially if you're also removing rows that occur in it!) there's a risk of confusing the iteration. This is why there are no _endTransaction calls while iterating over result sets. Once the full result set has been loaded, this is no longer an issue.

_mapExistingMessages is gone. It was a helper for _scrubPOTMsgSetTranslations, but its approach involved loading all of a potmsgset's translations right up front. A serious memory footprint risk that was still in there because once upon a time, the resulting dict was reused elsewhere in the algorithm.

There still is a two-stage approach, but with a different division of labour: one loop retrieves the TranslationMessages a potmsgset has for a given language/variant pair, but keeps only its ids so as not to pin all those objects in the ORM cache. A second loop sorts out only the TranslationMessages for a single language, deleting duplicates as it finds them. Duplicates must have the same language, and a problematic potmsgset is one that has TranslationMessages in many languages, so this ...

[Brad Crittenden (bac)]

Hi Jeroen,

Thanks for this branch. Other than the following issues this branch looks great.

* As discussed on IRC, the parser option around 67 needs an action and a better help description to remove the ambiguous 'some'.

* Remove the assignment at line 158.

* The comment at line 291 makes no sense as written.

review: Approve (code)

=== modified file 'lib/lp/translations/model/potmsgset.py'
--- lib/lp/translations/model/potmsgset.py	2009-10-22 15:51:58 +0000
+++ lib/lp/translations/model/potmsgset.py	2009-10-28 14:58:15 +0000
@@ -15,7 +15,7 @@
 from zope.component import getUtility
 
 from sqlobject import ForeignKey, IntCol, StringCol, SQLObjectNotFound
-from storm.store import EmptyResultSet
+from storm.store import EmptyResultSet, Store
 
 from canonical.config import config
 from canonical.database.constants import DEFAULT, UTC_NOW
@@ -58,7 +58,7 @@
     u'translator_credits': (None, TranslationCreditsType.GNOME),
 
     # KDE credits messages.
-    u'Your emails': 
+    u'Your emails':
         (u'EMAIL OF TRANSLATORS', TranslationCreditsType.KDE_EMAILS),
     u'Your names':
         (u'NAME OF TRANSLATORS', TranslationCreditsType.KDE_NAMES),
@@ -66,7 +66,7 @@
     # Old KDE credits messages.
     u'_: EMAIL OF TRANSLATORS\nYour emails':
         (None, TranslationCreditsType.KDE_EMAILS),
-    u'_: NAME OF TRANSLATORS\nYour names': 
+    u'_: NAME OF TRANSLATORS\nYour names':
         (None, TranslationCreditsType.KDE_NAMES),
     }
 
@@ -1056,7 +1056,7 @@
         if self.is_translation_credit:
             translation = self.getSharedTranslationMessage(pofile.language)
             if translation is None:
-                message = self.updateTranslation(pofile, pofile.owner, 
+                message = self.updateTranslation(pofile, pofile.owner,
                     [credits_message_str], False,
                     datetime.datetime.now(pytz.UTC),
                     allow_credits=True)
@@ -1104,7 +1104,8 @@
 
     def getAllTranslationMessages(self):
         """See `IPOTMsgSet`."""
-        return TranslationMessage.selectBy(potmsgset=self, orderBy=['id'])
+        return Store.of(self).find(
+            TranslationMessage, TranslationMessage.potmsgset == self)
 
     def getAllTranslationTemplateItems(self):
         """See `IPOTMsgSet`."""

=== modified file 'lib/lp/translations/model/translationmessage.py'
--- lib/lp/translations/model/translationmessage.py	2009-07-19 04:41:14 +0000
+++ lib/lp/translations/model/translationmessage.py	2009-10-28 14:58:15 +0000
@@ -386,7 +386,7 @@
 
         for form in range(TranslationConstants.MAX_PLURAL_FORMS):
             msgstr_name = 'msgstr%d' % form
-            msgstr = getattr(self, msgstr_name)
+            msgstr = getattr(self, 'msgstr%dID' % form)
             if msgstr is None:
                 form_clause = "%s IS NULL" % msgstr_name
             else:
@@ -454,10 +454,10 @@
         """See `ITranslationMessage`."""
         store = Store.of(self)
 
-        forms_match = (TranslationMessage.msgstr0 == self.msgstr0)
+        forms_match = (TranslationMessage.msgstr0ID == self.msgstr0ID)
         for form in xrange(1, TranslationConstants.MAX_PLURAL_FORMS):
             form_name = 'msgstr%d' % form
-            form_value = getattr(self, form_name)
+            form_value = getattr(self, 'msgstr%dID' % form)
             forms_match = And(
                 forms_match,
                 getattr(TranslationMessage, form_name) == form_value)

=== modified file 'lib/lp/translations/scripts/message_sharing_migration.py'
--- lib/lp/translations/scripts/message_sharing_migration.py	2009-10-22 10:17:44 +0000
+++ lib/lp/translations/scripts/message_sharing_migration.py	2009-10-28 14:58:15 +0000
@@ -7,12 +7,16 @@
     ]
 
 
+import gc
+import os
+
 from zope.component import getUtility
 from zope.security.proxy import removeSecurityProxy
 
 from canonical.launchpad.utilities.orderingcheck import OrderingCheck
 from lp.translations.interfaces.potemplate import IPOTemplateSet
 from lp.translations.interfaces.translations import TranslationConstants
+from lp.translations.model.translationmessage import TranslationMessage
 from lp.registry.interfaces.product import IProductSet
 from lp.registry.interfaces.sourcepackagename import ISourcePackageNameSet
 from lp.services.scripts.base import (
@@ -153,14 +157,19 @@
 
     template_set = None
 
+    commit_count = 0
+
     def add_my_options(self):
         self.parser.add_option('-d', '--distribution', dest='distribution',
             help="Distribution to merge messages for.")
+        self.parser.add_option('-D', '--remove-duplicates',
+            dest='remove_duplicates', action='store_true',
+            help="Phase 1: Remove problematic duplicate TranslationMessages.")
         self.parser.add_option('-p', '--product', dest='product',
             help="Product to merge messages for.")
         self.parser.add_option('-P', '--merge-potmsgsets',
             action='store_true', dest='merge_potmsgsets',
-            help="Merge POTMsgSets.")
+            help="Phase 2: Merge POTMsgSets.")
         self.parser.add_option('-s', '--source-package', dest='sourcepackage',
             help="Source package name within a distribution.")
         self.parser.add_option('-t', '--template-names',
@@ -168,7 +177,7 @@
             help="Merge for templates with name matching this regex pattern.")
         self.parser.add_option('-T', '--merge-translationmessages',
             action='store_true', dest='merge_translationmessages',
-            help="Merge TranslationMessages.")
+            help="Phase 3: Merge TranslationMessages.")
         self.parser.add_option('-x', '--dry-run', dest='dry_run',
             action='store_true',
             help="Dry run, don't really make any changes.")
@@ -185,13 +194,14 @@
 
     def main(self):
         actions = (
+            self.options.remove_duplicates or
             self.options.merge_potmsgsets or
             self.options.merge_translationmessages)
 
         if not actions:
             raise LaunchpadScriptFailure(
-                "Select at least one action: merge POTMsgSets, "
-                "TranslationMessages, or both.")
+                "Select at least one action: remove duplicates, merge "
+                "POTMsgSets, and/or merge TranslationMessages.")
 
         if self.options.product and self.options.distribution:
             raise LaunchpadScriptFailure(
@@ -250,27 +260,85 @@
                     name, len(templates), number + 1, class_count))
             self.logger.debug("Templates: %s" % str(templates))
 
+            if self.options.remove_duplicates:
+                self.logger.info("Removing duplicate messages.")
+                self._removeDuplicateMessages(templates)
+                self._endTransaction(intermediate=True)
+
             if self.options.merge_potmsgsets:
+                self.logger.info("Merging POTMsgSets.")
                 self._mergePOTMsgSets(templates)
+                self._endTransaction(intermediate=True)
 
             if self.options.merge_translationmessages:
+                self.logger.info("Merging TranslationMessages.")
                 self._mergeTranslationMessages(templates)
 
             self._endTransaction()
 
         self.logger.info("Done.")
 
-    def _endTransaction(self):
+    def _endTransaction(self, intermediate=False):
+        """End this transaction and start a new one.
+
+        :param intermediate: Whether this is an intermediate commit.
+            Dry-run mode aborts transactions rather than committing
+            them; where doing that may break dependencies between steps
+            of the algorithm, pass `True` so that the abort can be
+            skipped.
+        """
+        if self.txn is None:
+            return
+
+        if self.commit_count % 100 == 0 or not intermediate:
+            freed = gc.collect()
+            objcount = len(gc.get_objects())
+            garbage = len(gc.garbage)
+            memsize = open("/proc/%s/statm" % os.getpid()).read().split()[5]
+            self.logger.debug(
+                "Freed: %d.  Object count: %d.  Garbage: %d.  Memory size: %s"
+                % (freed, objcount, garbage, memsize))
+
+        self.commit_count += 1
+
+        if intermediate and self.commit_count % 10 != 0:
+            return
+
         if self.options.dry_run:
-            self.txn.abort()
+            if not intermediate:
+                self.txn.abort()
         else:
             self.txn.commit()
-        self.txn.begin()
 
-    def _mergePOTMsgSets(self, potemplates):
-        """Merge POTMsgSets for given sequence of sharing templates."""
+    def _removeDuplicateMessages(self, potemplates):
+        """Get rid of duplicate `TranslationMessages` where needed."""
         self._setUpUtilities()
 
+        representatives = {}
+        order_check = OrderingCheck(cmp=self.compare_template_precedence)
+
+        for template in potemplates:
+            order_check.check(template)
+            for potmsgset in template.getPOTMsgSets(False, prefetch=False):
+                key = get_potmsgset_key(potmsgset)
+                if key not in representatives:
+                    representatives[key] = potmsgset
+
+        for representative in representatives.itervalues():
+            self._scrubPOTMsgSetTranslations(representative)
+            self._endTransaction(intermediate=True)
+
+    def _mapRepresentatives(self, potemplates):
+        """Map out POTMsgSets' subordinates and templates.
+
+        :param potemplates: An equivalence class of `POTemplate`s to
+            sort out.
+        :return: A tuple of dicts.  The first maps each `POTMsgSet`'s
+            key (as returned by `get_potmsgset_key`) to a list of its
+            subordinate `POTMsgSet`s.  The second maps each
+            representative `POTMsgSet` to its representative
+            `POTemplate`.
+        """
         # Map each POTMsgSet key (context, msgid, plural) to its
         # representative POTMsgSet.
         representatives = {}
@@ -302,21 +370,34 @@
                 else:
                     subordinates[representative] = []
 
+        return subordinates, representative_templates
+
+    def _mergePOTMsgSets(self, potemplates):
+        """Merge POTMsgSets for given sequence of sharing templates."""
+        self._setUpUtilities()
+
+        subordinates, representative_templates = self._mapRepresentatives(
+            potemplates)
+
+        num_representatives = len(subordinates)
+        representative_num = 0
+
         for representative, potmsgsets in subordinates.iteritems():
-            # Scrub the representative POTMsgSet of any duplicate
-            # translation messages.  We don't need do this for subordinates
-            # because the algorithm will refuse to add new duplicates to the
-            # representative POTMsgSet anyway.
-            self._scrubPOTMsgSetTranslations(representative)
-
-            seen_potmsgsets = set([representative])
+            representative_num += 1
+            self.logger.info("Message %d/%d: %d subordinate(s)." % (
+                representative_num, num_representatives, len(potmsgsets)))
+
+            seen_potmsgsets = set([representative.id])
+
+            potmsgset_deletions = 0
+            tm_deletions = 0
 
             # Merge each subordinate POTMsgSet into its representative.
             for subordinate in potmsgsets:
-                if subordinate in seen_potmsgsets:
+                if subordinate.id in seen_potmsgsets:
                     continue
 
-                seen_potmsgsets.add(subordinate)
+                seen_potmsgsets.add(subordinate.id)
 
                 for message in subordinate.getAllTranslationMessages():
                     message = removeSecurityProxy(message)
@@ -350,22 +431,40 @@
                             bequeathe_flags(
                                 message, twin,
                                 (clashing_current, clashing_imported))
+                            tm_deletions += 1
 
                 merge_translationtemplateitems(
                     subordinate, representative,
                     representative_templates[representative])
                 removeSecurityProxy(subordinate).destroySelf()
+                potmsgset_deletions += 1
+
+                self._endTransaction(intermediate=True)
+
+            self.logger.info(
+                "Deleted POTMsgSets: %d.  TranslationMessages: %d." % (
+                    potmsgset_deletions, tm_deletions))
 
     def _mergeTranslationMessages(self, potemplates):
         """Share `TranslationMessage`s between templates where possible."""
         self._setUpUtilities()
         order_check = OrderingCheck(cmp=self.compare_template_precedence)
         for template in potemplates:
+            deletions = 0
             order_check.check(template)
             for potmsgset in template.getPOTMsgSets(False, prefetch=False):
+                before = potmsgset.getAllTranslationMessages().count()
+
                 for message in potmsgset.getAllTranslationMessages():
                     removeSecurityProxy(message).shareIfPossible()
 
+                after = potmsgset.getAllTranslationMessages().count()
+                self._endTransaction(intermediate=True)
+
+                deletions += max(0, before - after)
+
+            self.logger.info("Removed TranslationMessages: %d" % deletions)
+
     def _getPOTMsgSetTranslationMessageKey(self, tm):
         """Return tuple that identifies a TranslationMessage in a POTMsgSet.
 
@@ -375,36 +474,12 @@
         that's sorted out in the nested dicts).
         """
         tm = removeSecurityProxy(tm)
-        msgstr_ids = [
+        msgstr_ids = tuple([
             getattr(tm, 'msgstr%dID' % form)
             for form in xrange(TranslationConstants.MAX_PLURAL_FORMS)
-            ]
+            ])
             
-        return (tm.language, tm.variant) + tuple(msgstr_ids)
-
-    def _mapExistingMessages(self, potmsgset):
-        """Map out the existing TranslationMessages for `potmsgset`.
-
-        :return: a dict mapping message keys (as returned by
-        `_getPOTMsgSetTranslationMessageKey`) to further dicts, which
-        map templates to TranslationMessages for that key and template.
-        Each list should normally have only one message in it, but in
-        the transition period this isn't always the case.
-        """
-        existing_tms = {}
-
-        for tm in potmsgset.getAllTranslationMessages():
-            key = self._getPOTMsgSetTranslationMessageKey(tm)
-            if key not in existing_tms:
-                existing_tms[key] = {}
-            per_template = existing_tms[key]
-
-            if tm.potemplate not in per_template:
-                per_template[tm.potemplate] = []
-
-            per_template[tm.potemplate].append(tm)
-
-        return existing_tms
+        return (tm.potemplateID, tm.languageID, tm.variant) + msgstr_ids
 
     def _scrubPOTMsgSetTranslations(self, potmsgset):
         """Map out translations for `potmsgset`, and eliminate duplicates.
@@ -416,34 +491,53 @@
         # XXX JeroenVermeulen 2009-06-15
         # spec=message-sharing-prevent-duplicates: We're going to have a
         # unique index again at some point that will prevent this.  When
-        # it becomes impossible to test this function, it can be
-        # removed.
-        translations = self._mapExistingMessages(potmsgset)
-
-        for key, per_template in translations.iteritems():
-            for template, tms in per_template.iteritems():
-                assert len(tms) > 0, "Empty-list entry in translations dict."
-                if len(tms) == 1:
-                    continue
-                self.logger.info(
-                    "Cleaning up %s identical '%s' messages for: \"%s\"" % (
-                    len(tms), tms[0].language.getFullCode(),
-                    potmsgset.singular_text))
-                for tm in tms[1:]:
-                    assert tm != tms[0], "Duplicate is listed twice."
-                    assert tm.potmsgset == tms[0].potmsgset, (
+        # it becomes impossible to test this function, this whole
+        # migration phase can be scrapped.
+        tms = potmsgset.getAllTranslationMessages().order_by(
+            TranslationMessage.languageID, TranslationMessage.variant,
+            TranslationMessage.id)
+
+        ids_per_language = {}
+        for tm in tms:
+            language = (tm.language, tm.variant)
+            if language not in ids_per_language:
+                ids_per_language[language] = []
+            ids_per_language[language].append(tm.id)
+
+        self._endTransaction(intermediate=True)
+
+        deletions = 0
+
+        for ids in ids_per_language.itervalues():
+            translations = {}
+
+            for tm_id in ids:
+                tm = TranslationMessage.get(tm_id)
+                key = self._getPOTMsgSetTranslationMessageKey(tm)
+
+                if key in translations:
+                    self.logger.info(
+                    "Cleaning up identical '%s' message for: \"%s\"" % (
+                        tm.language.getFullCode(), potmsgset.singular_text))
+
+                    existing_tm = translations[key]
+                    assert tm != existing_tm, (
+                        "Message is duplicate of itself.")
+                    assert tm.potmsgset == existing_tm.potmsgset, (
                         "Different potmsgsets considered identical.")
-                    assert tm.potemplate == tms[0].potemplate, (
+                    assert tm.potemplate == existing_tm.potemplate, (
                         "Different potemplates considered identical.")
 
-                    # Transfer any current/imported flags to the first of
-                    # the identical messages, and delete the duplicates.
-                    bequeathe_flags(tm, tms[0])
-                    removeSecurityProxy(tm).sync()
-
-                per_template[template] = [tms[0]]
-
-        return translations
+                    # Transfer any current/imported flags to the existing
+                    # message, and delete the duplicate.
+                    bequeathe_flags(tm, existing_tm)
+                    deletions += 1
+                else:
+                    translations[key] = tm
+
+            self._endTransaction(intermediate=True)
+
+        self.logger.info("Deleted TranslationMessages: %d" % deletions)
 
     def _findClashes(self, message, target_potmsgset, target_potemplate):
         """What would clash if we moved `message` to the target environment?

=== modified file 'lib/lp/translations/scripts/tests/test_message_sharing_migration.py'
--- lib/lp/translations/scripts/tests/test_message_sharing_migration.py	2009-10-22 11:25:01 +0000
+++ lib/lp/translations/scripts/tests/test_message_sharing_migration.py	2009-10-28 14:58:15 +0000
@@ -18,6 +18,7 @@
     IPOFileTranslatorSet)
 from lp.translations.model.pomsgid import POMsgID
 from lp.translations.model.potemplate import POTemplate
+from lp.translations.model.potranslation import POTranslation
 from lp.translations.scripts.message_sharing_migration import (
     MessageSharingMerge)
 from canonical.testing import LaunchpadZopelessLayer
@@ -361,40 +362,6 @@
         self.assertEqual(tms, expected_tms)
         self.assertEqual(len(tms), 3)
 
-    def test_duplicatesAreCleanedUp(self):
-        # The POTMsgSet merging function cleans up any duplicate
-        # TranslationMessages that might get in the way.
-        trunk_message, stable_message = self._makeTranslationMessages(
-            'snaggle', 'snaggle')
-        trunk_message.is_current = False
-        trunk_message.sync()
-
-        potmsgset = trunk_message.potmsgset
-
-        stable_message.is_imported = True
-        stable_message.potemplate = trunk_message.potemplate
-        stable_message.potmsgset = potmsgset
-        stable_message.sync()
-
-        # We've set up a situation where trunk has two identical
-        # messages (one of which is current, the other imported) and
-        # stable has none.
-        self.assertEqual(self._getTranslations(), ('snaggle', None))
-        tms = set(potmsgset.getAllTranslationMessages())
-        self.assertEqual(tms, set([trunk_message, stable_message]))
-
-        self.script._mergePOTMsgSets(self.templates)
-
-        # The duplicates have been cleaned up.
-        self.assertEqual(potmsgset.getAllTranslationMessages().count(), 1)
-        
-        # The is_current and is_imported flags from the duplicate
-        # messages have been merged into a single, current, imported
-        # message.
-        message = self._getMessage(potmsgset, self.trunk_template)
-        self.assertTrue(message.is_current)
-        self.assertTrue(message.is_imported)
-
 
 class TestTranslationMessageNonMerging(TestCaseWithFactory,
                                        TranslatedProductMixin):
@@ -586,51 +553,61 @@
         self.assertEqual(poft.latest_message, stable_message)
 
 
-class TestMapMessages(TestCaseWithFactory, TranslatedProductMixin):
-    """Test _mapExistingMessages and friends."""
+class TestRemoveDuplicates(TestCaseWithFactory, TranslatedProductMixin):
+    """Test _scrubPOTMsgSetTranslations and friends."""
     layer = LaunchpadZopelessLayer
 
     def setUp(self):
         self.layer.switchDbUser('postgres')
-        super(TestMapMessages, self).setUp(user='[e-mail]')
-        super(TestMapMessages, self).setUpProduct()
-
-    def test_NoMessagesToMap(self):
-        # Mapping an untranslated POTMsgSet produces an empty dict.
-        empty = self.script._mapExistingMessages(self.trunk_potmsgset)
-        self.assertEqual(empty, {})
-
-    def test_MapSharedMessage(self):
-        # Map existing, shared translation for a POTMsgSet.
-        message = self._makeTranslationMessage(
-            pofile=self.trunk_pofile, potmsgset=self.trunk_potmsgset,
-            text='winslow', diverged=False)
-
-        map = self.script._mapExistingMessages(message.potmsgset)
-        key = self.script._getPOTMsgSetTranslationMessageKey(message)
-        expected = {key: { self.trunk_template: [message]}}
-
-    def test_MapDivergedMessage(self):
-        # Map existing, diverged translation for a POTMsgSet.
-        message = self._makeTranslationMessage(
-            pofile=self.trunk_pofile, potmsgset=self.trunk_potmsgset,
-            text='winslow', diverged=True)
-
-        map = self.script._mapExistingMessages(message.potmsgset)
-        key = self.script._getPOTMsgSetTranslationMessageKey(message)
-        expected = {key: {self.trunk_template: [message]}}
-
-        self.assertEqual(map, expected)
+        super(TestRemoveDuplicates, self).setUp(user='[e-mail]')
+        super(TestRemoveDuplicates, self).setUpProduct()
+
+    def test_duplicatesAreCleanedUp(self):
+        # The duplicates removal function cleans up any duplicate
+        # TranslationMessages that might get in the way of merging.
+        trunk_message, stable_message = self._makeTranslationMessages(
+            'snaggle', 'snaggle')
+        trunk_message.is_current = False
+        trunk_message.sync()
+
+        potmsgset = trunk_message.potmsgset
+
+        stable_message.is_imported = True
+        stable_message.potemplate = trunk_message.potemplate
+        stable_message.potmsgset = potmsgset
+        stable_message.sync()
+
+        # We've set up a situation where trunk has two identical
+        # messages (one of which is current, the other imported) and
+        # stable has none.
+        self.assertEqual(self._getTranslations(), ('snaggle', None))
+        tms = set(potmsgset.getAllTranslationMessages())
+        self.assertEqual(tms, set([trunk_message, stable_message]))
+
+        self.script._removeDuplicateMessages(self.templates)
+
+        # The duplicates have been cleaned up.
+        self.assertEqual(potmsgset.getAllTranslationMessages().count(), 1)
+        
+        # The is_current and is_imported flags from the duplicate
+        # messages have been merged into a single, current, imported
+        # message.
+        message = self._getMessage(potmsgset, self.trunk_template)
+        self.assertTrue(message.is_current)
+        self.assertTrue(message.is_imported)
 
     def test_ScrubPOTMsgSetTranslationsWithoutDuplication(self):
         # _scrubPOTMsgSetTranslations eliminates duplicated
         # TranslationMessages.  If it doesn't find any, nothing happens.
-        message = self._makeTranslationMessage(
+        self._makeTranslationMessage(
             pofile=self.trunk_pofile, potmsgset=self.trunk_potmsgset,
             text='gbzidh', diverged=False)
-        map = self.script._scrubPOTMsgSetTranslations(self.trunk_potmsgset)
-        key = self.script._getPOTMsgSetTranslationMessageKey(message)
-        self.assertEqual(map, {key: {None: [message]}})
+
+        self.script._scrubPOTMsgSetTranslations(self.trunk_potmsgset)
+
+        message1, message2 = self._getMessages()
+        self.assertIsNot(None, message1)
+        self.assertIs(None, message2)
 
     def test_ScrubPOTMsgSetTranslationsWithDuplication(self):
         # If there are duplicate TranslationMessages, one inherits all
@@ -646,14 +623,14 @@
         message2.is_imported = True
         message2.potmsgset = self.trunk_potmsgset
         message2.potemplate = self.trunk_template
+        ids = (message1.id, message2.id)
 
-        map = self.script._scrubPOTMsgSetTranslations(self.trunk_potmsgset)
+        self.script._scrubPOTMsgSetTranslations(self.trunk_potmsgset)
 
         message, no_message = self._getMessages()
 
-        # The resulting map has only one of the identical messages.
-        key = self.script._getPOTMsgSetTranslationMessageKey(message)
-        self.assertEqual(map, {key: {self.trunk_template: [message]}})
+        # One of the two messages is now gone.
+        self.assertIs(None, no_message)
 
         # The remaining message combines the flags from both its
         # predecessors.
@@ -763,16 +740,17 @@
         transaction.commit()
         gc.collect()
 
-    def _listLoadedPOMsgIDs(self, ignore_list=None):
-        """Return the set of all `POMsgID` objects that are in memory.
+    def _listLoadedObjects(self, of_class, ignore_list=None):
+        """Return the set of objects of a given type that are in memory.
 
+        :param of_class: A class to filter for.
         :param ignore_list: A previous return value.  Any POMsgIDs that
             were already in that list are ignored here.
         """
         pomsgids = set([
             whatever
             for whatever in gc.get_objects()
-            if isinstance(whatever, POMsgID)
+            if isinstance(whatever, of_class)
             ])
         if ignore_list is not None:
             pomsgids -= ignore_list
@@ -805,19 +783,47 @@
     def test_merging_loads_no_msgids(self):
         # Migration does not load actual msgids into memory.
         self._flushDbObjects()
-        msgids_before = self._listLoadedPOMsgIDs()
-
-        self._makeTranslationMessages('x', 'y', trunk_diverged=True)
-        self._makeTranslationMessages('1', '2', stable_diverged=True)
-
-        self._resetReferences()
-        self.assertNotEqual([], self.templates)
-        self.assertEqual(set(), self._listLoadedPOMsgIDs(msgids_before))
-        
-        self.script._mergePOTMsgSets(self.templates)
-        self.script._mergeTranslationMessages(self.templates)
-
-        self.assertEqual(set(), self._listLoadedPOMsgIDs(msgids_before))
+        msgids_before = self._listLoadedObjects(POMsgID)
+
+        self._makeTranslationMessages('x', 'y', trunk_diverged=True)
+        self._makeTranslationMessages('1', '2', stable_diverged=True)
+
+        self._resetReferences()
+        self.assertNotEqual([], self.templates)
+        self.assertEqual(
+            set(), self._listLoadedObjects(POMsgID, msgids_before))
+        
+        self.script._mergePOTMsgSets(self.templates)
+        self.script._mergeTranslationMessages(self.templates)
+
+        self.assertEqual(
+            set(), self._listLoadedObjects(POMsgID, msgids_before))
+
+    def test_merging_loads_no_potranslations(self):
+        # Migration does not load actual POTranslations into memory.
+        self._flushDbObjects()
+        potranslations_before = self._listLoadedObjects(POTranslation)
+
+        self._makeTranslationMessages('x', 'y', trunk_diverged=True)
+        self._makeTranslationMessages('1', '2', stable_diverged=True)
+
+        self._resetReferences()
+        self.assertNotEqual([], self.templates)
+        self.assertEqual(
+            set(),
+            self._listLoadedObjects(POTranslation, potranslations_before))
+        
+        self.script._mergePOTMsgSets(self.templates)
+
+        self.assertEqual(
+            set(),
+            self._listLoadedObjects(POTranslation, potranslations_before))
+        
+        self.script._mergeTranslationMessages(self.templates)
+
+        self.assertEqual(
+            set(),
+            self._listLoadedObjects(POTranslation, potranslations_before))
 
 
 def test_suite():