Merge lp:~danilo/launchpad/bug-814580 into lp:launchpad

[Данило Шеган]

Merge or split messages when translation templates are renamed or moved to a different product or source package.

= Bug 814580: migrate translations on POTemplate changes =

== Proposed fix ==

This extends the TranslationSharingJob to do translation splitting/merging even when a single template is modified (i.e. renamed or moved to a different parent).
We use the existing code for translation merging and only add new code for finding which templates are affected, and do the similar thing for translation splitting.

I haven't fixed the lint yet so as not to make the diff larger. I'll happily do that before landing.

== Tests ==

bin/test -cvvt translationsplitter -t translationpackaging

== Demo and Q/A ==

Rename a PO template, note how TranslationSharingJob is created (table PackagingJob, in the process of being renamed).
Test how the job works by executing cronscripts/run_jobs.py -vv packaging_translations

(also move the po template to a different project/sourcepackage and confirm the same happens)

Use the POTemplate:+admin page to rename it or move it to a different project/source package.

= Launchpad lint =

Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/translations/utilities/translationsplitter.py
  lib/lp/translations/interfaces/translationsharingjob.py
  lib/lp/translations/translationmerger.py
  lib/lp/translations/model/translationsharingjob.py
  lib/lp/translations/model/translationpackagingjob.py
  lib/lp/translations/tests/test_translationpackagingjob.py
  lib/lp/translations/configure.zcml
  lib/lp/translations/tests/test_translationsplitter.py

./lib/lp/translations/translationmerger.py
     571: local variable 'total_ids' is assigned to but never used
./lib/lp/translations/tests/test_translationpackagingjob.py
      63: local variable 'package' is assigned to but never used
     196: local variable 'recorder' is assigned to but never used
     207: local variable 'recorder' is assigned to but never used
     215: local variable 'other_packaging' is assigned to but never used
     229: local variable 'job' is assigned to but never used
     227: local variable 'recorder' is assigned to but never used
     236: local variable 'recorder' is assigned to but never used
     253: local variable 'recorder' is assigned to but never used
     257: local variable 'job2' is assigned to but never used
./lib/lp/translations/tests/test_translationsplitter.py
      81: local variable 'ubuntu_template' is assigned to but never used
      82: local variable 'ubuntu_sequence' is assigned to but never used
      95: local variable 'upstream_translation' is assigned to but never used
     112: local variable 'upstream_translation' is assigned to but never used
     143: local variable 'upstream_message' is assigned to but never used

[Henning Eggers (henninge)]

Looks good to me.
I just wonder if "translationmerger.py" and "translationsplitter.py" should not be in the same directory?

review: Approve

[Данило Шеган (danilo)]

They should be, imho as well. However, these branches have taken enough of my time already.

Also, with the new db patch deployment process, I'll still have to land this on devel myself once db patch gets deployed, so keeping it as 'work in progress' and will land this MP directly once possible.

=== modified file 'lib/lp/translations/configure.zcml'
--- lib/lp/translations/configure.zcml	2011-08-01 12:02:47 +0000
+++ lib/lp/translations/configure.zcml	2011-08-01 12:02:49 +0000
@@ -154,6 +154,10 @@
         for="lp.registry.interfaces.packaging.IPackaging
              lazr.lifecycle.interfaces.IObjectEvent"
         handler=".model.translationsharingjob.schedule_packaging_job" />
+    <subscriber
+        for="lp.translations.interfaces.potemplate.IPOTemplate
+             lazr.lifecycle.interfaces.IObjectModifiedEvent"
+        handler=".model.translationsharingjob.schedule_potemplate_job" />
     <facet
         facet="translations">
 
@@ -643,6 +647,10 @@
         class="lp.translations.model.translationpackagingjob.TranslationSplitJob">
         <allow interface='lp.services.job.interfaces.job.IRunnableJob'/>
     </class>
+    <class
+        class="lp.translations.model.translationpackagingjob.TranslationTemplateChangeJob">
+        <allow interface='lp.services.job.interfaces.job.IRunnableJob'/>
+    </class>
     <utility
         component="lp.translations.model.translationtemplatesbuildjob.TranslationTemplatesBuildJob"
         provides="lp.buildmaster.interfaces.buildfarmjob.IBuildFarmJob"

=== modified file 'lib/lp/translations/interfaces/translationsharingjob.py'
--- lib/lp/translations/interfaces/translationsharingjob.py	2011-08-01 12:02:47 +0000
+++ lib/lp/translations/interfaces/translationsharingjob.py	2011-08-01 12:02:49 +0000
@@ -19,3 +19,6 @@
 
     sourcepackagename = Attribute(
         _("The sourcepackagename of the Packaging."))
+
+    potemplate = Attribute(
+        _("The POTemplate to pass around as the relevant template."))

=== modified file 'lib/lp/translations/model/translationpackagingjob.py'
--- lib/lp/translations/model/translationpackagingjob.py	2011-08-01 12:02:47 +0000
+++ lib/lp/translations/model/translationpackagingjob.py	2011-08-01 12:02:49 +0000
@@ -10,6 +10,7 @@
 __all__ = [
     'TranslationMergeJob',
     'TranslationSplitJob',
+    'TranslationTemplateChangeJob',
     ]
 
 import logging
@@ -17,6 +18,7 @@
 from lazr.lifecycle.interfaces import (
     IObjectCreatedEvent,
     IObjectDeletedEvent,
+    IObjectModifiedEvent,
     )
 import transaction
 from zope.interface import (
@@ -40,7 +42,10 @@
     TransactionManager,
     TranslationMerger,
     )
-from lp.translations.utilities.translationsplitter import TranslationSplitter
+from lp.translations.utilities.translationsplitter import (
+    TranslationSplitter,
+    TranslationTemplateSplitter,
+    )
 
 
 class TranslationPackagingJob(TranslationSharingJobDerived, BaseRunnableJob):
@@ -117,3 +122,31 @@
             'Splitting %s and %s', self.productseries.displayname,
             self.sourcepackage.displayname)
         TranslationSplitter(self.productseries, self.sourcepackage).split()
+
+
+class TranslationTemplateChangeJob(TranslationPackagingJob):
+    """Job for merging/splitting translations when template is changed."""
+
+    implements(IRunnableJob)
+
+    class_job_type = TranslationSharingJobType.TEMPLATE_CHANGE
+
+    create_on_event = IObjectModifiedEvent
+
+    @classmethod
+    def forPOTemplate(cls, potemplate):
+        """Create a TranslationTemplateChangeJob for a POTemplate.
+
+        :param potemplate: The `POTemplate` to create the job for.
+        :return: A `TranslationTemplateChangeJob`.
+        """
+        return cls.create(potemplate=potemplate)
+
+    def run(self):
+        """See `IRunnableJob`."""
+        logger = logging.getLogger()
+        logger.info("Sanitizing translations for '%s'" % (
+                self.potemplate.displayname))
+        TranslationTemplateSplitter(self.potemplate).split()
+        tm = TransactionManager(transaction.manager, False)
+        TranslationMerger.mergeModifiedTemplates(self.potemplate, tm)

=== modified file 'lib/lp/translations/model/translationsharingjob.py'
--- lib/lp/translations/model/translationsharingjob.py	2011-08-01 12:02:47 +0000
+++ lib/lp/translations/model/translationsharingjob.py	2011-08-01 12:02:49 +0000
@@ -37,6 +37,7 @@
 from lp.translations.interfaces.translationsharingjob import (
     ITranslationSharingJob,
     )
+from lp.translations.model.potemplate import POTemplate
 
 
 class TranslationSharingJobType(DBEnumeratedType):
@@ -54,6 +55,12 @@
         Split translations between productseries and sourcepackage.
         """)
 
+    TEMPLATE_CHANGE = DBItem(2, """
+        Split/merge translations for a single translation template.
+
+        Split/merge translations for a single translation template.
+        """)
+
 
 class TranslationSharingJob(StormBase):
     """Base class for jobs related to a packaging."""
@@ -82,8 +89,12 @@
 
     sourcepackagename = Reference(sourcepackagename_id, SourcePackageName.id)
 
+    potemplate_id = Int('potemplate')
+
+    potemplate = Reference(potemplate_id, POTemplate.id)
+
     def __init__(self, job, job_type, productseries, distroseries,
-                 sourcepackagename):
+                 sourcepackagename, potemplate=None):
         """"Constructor.
 
         :param job: The `Job` to use for storing basic job state.
@@ -96,6 +107,7 @@
         self.distroseries = distroseries
         self.sourcepackagename = sourcepackagename
         self.productseries = productseries
+        self.potemplate = potemplate
 
 
 class RegisteredSubclass(type):
@@ -143,16 +155,18 @@
         self.job = job
 
     @classmethod
-    def create(cls, productseries, distroseries, sourcepackagename):
+    def create(cls, productseries=None, distroseries=None,
+               sourcepackagename=None, potemplate=None):
         """"Create a TranslationPackagingJob backed by TranslationSharingJob.
 
         :param productseries: The ProductSeries side of the Packaging.
         :param distroseries: The distroseries of the Packaging sourcepackage.
         :param sourcepackagename: The name of the Packaging sourcepackage.
+        :param potemplate: POTemplate to restrict to (if any).
         """
         context = TranslationSharingJob(
             Job(), cls.class_job_type, productseries,
-            distroseries, sourcepackagename)
+            distroseries, sourcepackagename, potemplate)
         return cls(context)
 
     @classmethod
@@ -170,6 +184,27 @@
                 job_class.forPackaging(packaging)
 
     @classmethod
+    def schedulePOTemplateJob(cls, potemplate, event):
+        """Event subscriber to create a TranslationSharingJob on events.
+
+        :param potemplate: The `POTemplate` to create
+            a `TranslationSharingJob` for.
+        :param event: The event itself.
+        """
+        if ('name' not in event.edited_fields and
+            'productseries' not in event.edited_fields and
+            'distroseries' not in event.edited_fields and
+            'sourcepackagename' not in event.edited_fields):
+            # Ignore changes to POTemplates that are neither renames,
+            # nor moves to a different package/project.
+            return
+        for event_type, job_classes in cls._event_types.iteritems():
+            if not event_type.providedBy(event):
+                continue
+            for job_class in job_classes:
+                job_class.forPOTemplate(potemplate)
+
+    @classmethod
     def iterReady(cls, extra_clauses):
         """See `IJobSource`.
 
@@ -207,3 +242,4 @@
 
 #make accessible to zcml
 schedule_packaging_job = TranslationSharingJobDerived.schedulePackagingJob
+schedule_potemplate_job = TranslationSharingJobDerived.schedulePOTemplateJob

=== modified file 'lib/lp/translations/tests/test_translationpackagingjob.py'
--- lib/lp/translations/tests/test_translationpackagingjob.py	2011-08-01 12:02:47 +0000
+++ lib/lp/translations/tests/test_translationpackagingjob.py	2011-08-01 12:02:49 +0000
@@ -8,12 +8,17 @@
 
 import transaction
 from zope.component import getUtility
+from zope.event import notify
+
+from lazr.lifecycle.event import ObjectModifiedEvent
+from lazr.lifecycle.snapshot import Snapshot
 
 from canonical.launchpad.webapp.testing import verifyObject
 from canonical.testing.layers import (
     LaunchpadZopelessLayer,
     )
 from lp.registry.interfaces.packaging import IPackagingUtil
+from lp.translations.interfaces.potemplate import IPOTemplate
 from lp.translations.model.translationsharingjob import (
     TranslationSharingJob,
     TranslationSharingJobDerived,
@@ -36,6 +41,7 @@
     TranslationMergeJob,
     TranslationPackagingJob,
     TranslationSplitJob,
+    TranslationTemplateChangeJob,
     )
 from lp.translations.tests.test_translationsplitter import (
     make_shared_potmsgset,
@@ -101,20 +107,32 @@
 
 class JobFinder:
 
-    def __init__(self, productseries, sourcepackage, job_class):
-        self.productseries = productseries
-        self.sourcepackagename = sourcepackage.sourcepackagename
-        self.distroseries = sourcepackage.distroseries
+    def __init__(self, productseries, sourcepackage, job_class,
+                 potemplate=None):
+        if potemplate is None:
+            self.productseries = productseries
+            self.sourcepackagename = sourcepackage.sourcepackagename
+            self.distroseries = sourcepackage.distroseries
+            self.potemplate = None
+        else:
+            self.potemplate = potemplate
         self.job_type = job_class.class_job_type
 
     def find(self):
-        return list(TranslationSharingJobDerived.iterReady([
-            TranslationSharingJob.productseries_id == self.productseries.id,
-            (TranslationSharingJob.sourcepackagename_id ==
-             self.sourcepackagename.id),
-            TranslationSharingJob.distroseries_id == self.distroseries.id,
-            TranslationSharingJob.job_type == self.job_type,
-            ]))
+        if self.potemplate is None:
+            return list(TranslationSharingJobDerived.iterReady([
+              TranslationSharingJob.productseries_id == self.productseries.id,
+              (TranslationSharingJob.sourcepackagename_id ==
+               self.sourcepackagename.id),
+              TranslationSharingJob.distroseries_id == self.distroseries.id,
+              TranslationSharingJob.job_type == self.job_type,
+              ]))
+        else:
+            return list(
+                TranslationSharingJobDerived.iterReady([
+                    TranslationSharingJob.potemplate_id == self.potemplate.id,
+                    TranslationSharingJob.job_type == self.job_type,
+                    ]))
 
 
 class TestTranslationPackagingJob(TestCaseWithFactory):
@@ -275,3 +293,62 @@
                 packaging.distroseries)
         (job,) = finder.find()
         self.assertIsInstance(job, TranslationSplitJob)
+
+
+class TestTranslationTemplateChangeJob(TestCaseWithFactory):
+
+    layer = LaunchpadZopelessLayer
+
+    def test_modifyPOTemplate_makes_job(self):
+        """Creating a Packaging should make a TranslationMergeJob."""
+        potemplate = self.factory.makePOTemplate()
+        finder = JobFinder(
+            None, None, TranslationTemplateChangeJob, potemplate)
+        self.assertEqual([], finder.find())
+        with person_logged_in(potemplate.owner):
+            snapshot = Snapshot(potemplate, providing=IPOTemplate)
+            potemplate.name = self.factory.getUniqueString()
+            notify(ObjectModifiedEvent(potemplate, snapshot, ["name"]))
+
+        (job,) = finder.find()
+        self.assertIsInstance(job, TranslationTemplateChangeJob)
+
+    def test_splits_and_merges(self):
+        """Changing a template makes the translations split and then
+        re-merged in the new target sharing set."""
+        potemplate = self.factory.makePOTemplate(name='template')
+        other_ps = self.factory.makeProductSeries(
+            product=potemplate.productseries.product)
+        old_shared = self.factory.makePOTemplate(name='template',
+                                                 productseries=other_ps)
+        new_shared = self.factory.makePOTemplate(name='renamed',
+                                                 productseries=other_ps)
+
+        # Set up shared POTMsgSets and translations.
+        potmsgset = self.factory.makePOTMsgSet(potemplate, sequence=1)
+        potmsgset.setSequence(old_shared, 1)
+        self.factory.makeCurrentTranslationMessage(potmsgset=potmsgset)
+
+        # This is the identical English message in the new_shared template.
+        target_potmsgset = self.factory.makePOTMsgSet(
+            new_shared, sequence=1, singular=potmsgset.singular_text)
+
+        # Rename the template and confirm that messages are now shared
+        # with new_shared instead of old_shared.
+        potemplate.name = 'renamed'
+        job = TranslationTemplateChangeJob.create(potemplate=potemplate)
+
+        self.becomeDbUser('rosettaadmin')
+        job.run()
+
+        # New POTMsgSet is now different from the old one (it's been split),
+        # but matches the target potmsgset (it's been merged into it).
+        new_potmsgset = potemplate.getPOTMsgSets()[0]
+        self.assertNotEqual(potmsgset, new_potmsgset)
+        self.assertEqual(target_potmsgset, new_potmsgset)
+
+        # Translations have been merged as well.
+        self.assertContentEqual(
+            [tm.translations for tm in potmsgset.getAllTranslationMessages()],
+            [tm.translations
+             for tm in new_potmsgset.getAllTranslationMessages()])

=== modified file 'lib/lp/translations/tests/test_translationsplitter.py'
--- lib/lp/translations/tests/test_translationsplitter.py	2011-02-25 20:23:40 +0000
+++ lib/lp/translations/tests/test_translationsplitter.py	2011-08-01 12:02:49 +0000
@@ -13,6 +13,7 @@
     )
 from lp.translations.utilities.translationsplitter import (
     TranslationSplitter,
+    TranslationTemplateSplitter,
     )
 
 
@@ -153,3 +154,183 @@
             upstream_item.potmsgset.getAllTranslationMessages().count(),
             ubuntu_item.potmsgset.getAllTranslationMessages().count(),
         )
+
+
+class TestTranslationTemplateSplitterBase:
+
+    layer = ZopelessDatabaseLayer
+
+    def getPOTMsgSetAndTemplateToSplit(self, splitter):
+        return [(tti1.potmsgset, tti1.potemplate)
+                for tti1, tti2 in splitter.findShared()]
+
+    def setUpSharingTemplates(self, other_side=False):
+        """Sets up two sharing templates with one sharing message and
+        one non-sharing message in each template."""
+        template1 = self.makePOTemplate()
+        template2 = self.makeSharingTemplate(template1, other_side)
+
+        shared_potmsgset = self.factory.makePOTMsgSet(template1, sequence=1)
+        shared_potmsgset.setSequence(template2, 1)
+
+        # POTMsgSets appearing in only one of the templates are not returned.
+        self.factory.makePOTMsgSet(template1, sequence=2)
+        self.factory.makePOTMsgSet(template2, sequence=2)
+        return template1, template2, shared_potmsgset
+
+    def makePOTemplate(self):
+        raise NotImplementedError('Subclasses should implement this.')
+
+    def makeSharingTemplate(self, template, other_side=False):
+        raise NotImplementedError('Subclasses should implement this.')
+
+    def test_findShared_renamed(self):
+        """Shared POTMsgSets are included for a renamed template."""
+        template1, template2, shared_potmsgset = self.setUpSharingTemplates()
+
+        splitter = TranslationTemplateSplitter(template2)
+        self.assertContentEqual([], splitter.findShared())
+
+        template2.name = 'renamed'
+        self.assertContentEqual(
+            [(shared_potmsgset, template1)],
+            self.getPOTMsgSetAndTemplateToSplit(splitter))
+
+    def test_findShared_moved_product(self):
+        """Moving a template to a different product splits its messages."""
+        template1, template2, shared_potmsgset = self.setUpSharingTemplates()
+
+        splitter = TranslationTemplateSplitter(template2)
+        self.assertContentEqual([], splitter.findShared())
+
+        # Move the template to a different product entirely.
+        template2.productseries = self.factory.makeProduct().development_focus
+        template2.distroseries = None
+        template2.sourcepackagename = None
+        self.assertContentEqual(
+            [(shared_potmsgset, template1)],
+            self.getPOTMsgSetAndTemplateToSplit(splitter))
+
+    def test_findShared_moved_distribution(self):
+        """Moving a template to a different distribution gets it split."""
+        template1, template2, shared_potmsgset = self.setUpSharingTemplates()
+
+        splitter = TranslationTemplateSplitter(template2)
+        self.assertContentEqual([], splitter.findShared())
+
+        # Move the template to a different distribution entirely.
+        sourcepackage = self.factory.makeSourcePackage()
+        template2.distroseries = sourcepackage.distroseries
+        template2.sourcepackagename = sourcepackage.sourcepackagename
+        template2.productseries = None
+        self.assertContentEqual(
+            [(shared_potmsgset, template1)],
+            self.getPOTMsgSetAndTemplateToSplit(splitter))
+
+    def test_findShared_moved_to_nonsharing_target(self):
+        """Moving a template to a target not sharing with the existing
+        upstreams and source package gets it split."""
+        template1, template2, shared_potmsgset = self.setUpSharingTemplates(
+            other_side=True)
+
+        splitter = TranslationTemplateSplitter(template2)
+        self.assertContentEqual([], splitter.findShared())
+
+        # Move the template to a different distribution entirely.
+        sourcepackage = self.factory.makeSourcePackage()
+        template2.distroseries = sourcepackage.distroseries
+        template2.sourcepackagename = sourcepackage.sourcepackagename
+        template2.productseries = None
+        self.assertContentEqual(
+            [(shared_potmsgset, template1)],
+            self.getPOTMsgSetAndTemplateToSplit(splitter))
+
+    def test_split_messages(self):
+        """Splitting messages works properly."""
+        template1, template2, shared_potmsgset = self.setUpSharingTemplates()
+
+        splitter = TranslationTemplateSplitter(template2)
+        self.assertContentEqual([], splitter.findShared())
+
+        # Move the template to a different product entirely.
+        template2.productseries = self.factory.makeProduct().development_focus
+        template2.distroseries = None
+        template2.sourcepackagename = None
+
+        other_item, this_item = splitter.findShared()[0]
+
+        splitter.split()
+
+        self.assertNotEqual(other_item.potmsgset, this_item.potmsgset)
+        self.assertEqual(shared_potmsgset, other_item.potmsgset)
+        self.assertNotEqual(shared_potmsgset, this_item.potmsgset)
+
+
+class TestProductTranslationTemplateSplitter(
+    TestCaseWithFactory, TestTranslationTemplateSplitterBase):
+    """Templates in a product get split appropriately."""
+
+    def makePOTemplate(self):
+        return self.factory.makePOTemplate(
+            name='template',
+            side=TranslationSide.UPSTREAM)
+
+    def makeSharingTemplate(self, template, other_side=False):
+        if other_side:
+            template2 = self.factory.makePOTemplate(
+                name='template',
+                side=TranslationSide.UBUNTU)
+            self.factory.makePackagingLink(
+                productseries=template.productseries,
+                distroseries=template2.distroseries,
+                sourcepackagename=template2.sourcepackagename)
+            return template2
+        else:
+            product = template.productseries.product
+            other_series = self.factory.makeProductSeries(product=product)
+            return self.factory.makePOTemplate(name='template',
+                                               productseries=other_series)
+
+
+class TestDistributionTranslationTemplateSplitter(
+    TestCaseWithFactory, TestTranslationTemplateSplitterBase):
+    """Templates in a distribution get split appropriately."""
+
+    def makePOTemplate(self):
+        return self.factory.makePOTemplate(
+            name='template',
+            side=TranslationSide.UBUNTU)
+
+    def makeSharingTemplate(self, template, other_side=False):
+        if other_side:
+            template2 = self.factory.makePOTemplate(
+                name='template',
+                side=TranslationSide.UPSTREAM)
+            self.factory.makePackagingLink(
+                productseries=template2.productseries,
+                distroseries=template.distroseries,
+                sourcepackagename=template.sourcepackagename)
+            return template2
+        else:
+            distro = template.distroseries.distribution
+            other_series = self.factory.makeDistroRelease(distribution=distro)
+            return self.factory.makePOTemplate(
+                name='template',
+                distroseries=other_series,
+                sourcepackagename=template.sourcepackagename)
+
+    def test_findShared_moved_sourcepackage(self):
+        """Moving a template to a different source package gets it split."""
+        template1, template2, shared_potmsgset = self.setUpSharingTemplates()
+
+        splitter = TranslationTemplateSplitter(template2)
+        self.assertContentEqual([], splitter.findShared())
+
+        # Move the template to a different source package inside the
+        # same distroseries.
+        sourcepackage = self.factory.makeSourcePackage(
+            distroseries=template2.distroseries)
+        template2.sourcepackagename = sourcepackage.sourcepackagename
+        self.assertContentEqual(
+            [(shared_potmsgset, template1)],
+            self.getPOTMsgSetAndTemplateToSplit(splitter))

=== modified file 'lib/lp/translations/translationmerger.py'
--- lib/lp/translations/translationmerger.py	2011-05-27 21:12:25 +0000
+++ lib/lp/translations/translationmerger.py	2011-08-01 12:02:49 +0000
@@ -387,6 +387,26 @@
             merger = cls(templates, tm)
             merger.mergePOTMsgSets()
 
+    @classmethod
+    def mergeModifiedTemplates(cls, potemplate, tm):
+        subset = getUtility(IPOTemplateSet).getSharingSubset(
+            distribution=potemplate.distribution,
+            sourcepackagename=potemplate.sourcepackagename,
+            product=potemplate.product)
+        templates = list(subset.getSharingPOTemplates(potemplate.name))
+        templates.sort(key=methodcaller('sharingKey'), reverse=True)
+        merger = cls(templates, tm)
+        merger.mergeAll()
+
+    def mergeAll(self):
+        """Properly merge POTMsgSets and TranslationMessages."""
+        self._removeDuplicateMessages()
+        self.tm.endTransaction(intermediate=True)
+        self.mergePOTMsgSets()
+        self.tm.endTransaction(intermediate=True)
+        self.mergeTranslationMessages()
+        self.tm.endTransaction()
+
     def __init__(self, potemplates, tm):
         """Constructor.
 

=== modified file 'lib/lp/translations/utilities/translationsplitter.py'
--- lib/lp/translations/utilities/translationsplitter.py	2011-05-12 20:21:58 +0000
+++ lib/lp/translations/utilities/translationsplitter.py	2011-08-01 12:02:49 +0000
@@ -6,50 +6,30 @@
 
 import logging
 
-from storm.locals import ClassAlias, Store
+from storm.expr import (
+    And,
+    Join,
+    LeftJoin,
+    Not,
+    Or,
+    )
+from storm.locals import (
+    ClassAlias,
+    Store,
+    )
 import transaction
 
+from lp.registry.model.distroseries import DistroSeries
+from lp.registry.model.packaging import Packaging
+from lp.registry.model.productseries import ProductSeries
 from lp.translations.model.potemplate import POTemplate
 from lp.translations.model.translationtemplateitem import (
     TranslationTemplateItem,
     )
 
 
-class TranslationSplitter:
-    """Split translations for a productseries, sourcepackage pair.
-
-    If a productseries and sourcepackage were linked in error, and then
-    unlinked, they may still share some translations.  This class breaks those
-    associations.
-    """
-
-    def __init__(self, productseries, sourcepackage):
-        """Constructor.
-
-        :param productseries: The `ProductSeries` to split from.
-        :param sourcepackage: The `SourcePackage` to split from.
-        """
-        self.productseries = productseries
-        self.sourcepackage = sourcepackage
-
-    def findShared(self):
-        """Provide tuples of upstream, ubuntu for each shared POTMsgSet."""
-        store = Store.of(self.productseries)
-        UpstreamItem = ClassAlias(TranslationTemplateItem, 'UpstreamItem')
-        UpstreamTemplate = ClassAlias(POTemplate, 'UpstreamTemplate')
-        UbuntuItem = ClassAlias(TranslationTemplateItem, 'UbuntuItem')
-        UbuntuTemplate = ClassAlias(POTemplate, 'UbuntuTemplate')
-        return store.find(
-            (UpstreamItem, UbuntuItem),
-            UpstreamItem.potmsgsetID == UbuntuItem.potmsgsetID,
-            UbuntuItem.potemplateID == UbuntuTemplate.id,
-            UbuntuTemplate.sourcepackagenameID ==
-                self.sourcepackage.sourcepackagename.id,
-            UbuntuTemplate.distroseriesID ==
-                self.sourcepackage.distroseries.id,
-            UpstreamItem.potemplateID == UpstreamTemplate.id,
-            UpstreamTemplate.productseriesID == self.productseries.id,
-        )
+class TranslationSplitterBase:
+    """Base class for translation splitting jobs."""
 
     @staticmethod
     def splitPOTMsgSet(ubuntu_item):
@@ -86,9 +66,151 @@
         """Split the translations for the ProductSeries and SourcePackage."""
         logger = logging.getLogger()
         shared = enumerate(self.findShared(), 1)
+        total = 0
         for num, (upstream_item, ubuntu_item) in shared:
             self.splitPOTMsgSet(ubuntu_item)
             self.migrateTranslations(upstream_item.potmsgset, ubuntu_item)
             if num % 100 == 0:
                 logger.info('%d entries split.  Committing...', num)
                 transaction.commit()
+            total = num
+
+        if total % 100 != 0 or total == 0:
+            transaction.commit()
+            logger.info('%d entries split.', total)
+
+
+class TranslationSplitter(TranslationSplitterBase):
+    """Split translations for a productseries, sourcepackage pair.
+
+    If a productseries and sourcepackage were linked in error, and then
+    unlinked, they may still share some translations.  This class breaks those
+    associations.
+    """
+
+    def __init__(self, productseries, sourcepackage):
+        """Constructor.
+
+        :param productseries: The `ProductSeries` to split from.
+        :param sourcepackage: The `SourcePackage` to split from.
+        """
+        self.productseries = productseries
+        self.sourcepackage = sourcepackage
+
+    def findShared(self):
+        """Provide tuples of upstream, ubuntu for each shared POTMsgSet."""
+        store = Store.of(self.productseries)
+        UpstreamItem = ClassAlias(TranslationTemplateItem, 'UpstreamItem')
+        UpstreamTemplate = ClassAlias(POTemplate, 'UpstreamTemplate')
+        UbuntuItem = ClassAlias(TranslationTemplateItem, 'UbuntuItem')
+        UbuntuTemplate = ClassAlias(POTemplate, 'UbuntuTemplate')
+        return store.find(
+            (UpstreamItem, UbuntuItem),
+            UpstreamItem.potmsgsetID == UbuntuItem.potmsgsetID,
+            UbuntuItem.potemplateID == UbuntuTemplate.id,
+            UbuntuTemplate.sourcepackagenameID ==
+                self.sourcepackage.sourcepackagename.id,
+            UbuntuTemplate.distroseriesID ==
+                self.sourcepackage.distroseries.id,
+            UpstreamItem.potemplateID == UpstreamTemplate.id,
+            UpstreamTemplate.productseriesID == self.productseries.id,
+        )
+
+
+class TranslationTemplateSplitter(TranslationSplitterBase):
+    """Split translations for an extracted potemplate.
+
+    When a POTemplate is removed from a set of sharing templates,
+    it keeps sharing POTMsgSets with other templates.  This class
+    removes those associations.
+    """
+
+    def __init__(self, potemplate):
+        """Constructor.
+
+        :param potemplate: The `POTemplate` to sanitize.
+        """
+        self.potemplate = potemplate
+
+    def findShared(self):
+        """Provide tuples of (other, this) items for each shared POTMsgSet.
+
+        Only return those that are shared but shouldn't be because they
+        are now in non-sharing templates.
+        """
+        store = Store.of(self.potemplate)
+        ThisItem = ClassAlias(TranslationTemplateItem, 'ThisItem')
+        OtherItem = ClassAlias(TranslationTemplateItem, 'OtherItem')
+        OtherTemplate = ClassAlias(POTemplate, 'OtherTemplate')
+
+        tables = [
+            OtherTemplate,
+            Join(OtherItem, OtherItem.potemplateID == OtherTemplate.id),
+            Join(ThisItem,
+                 And(ThisItem.potmsgsetID == OtherItem.potmsgsetID,
+                     ThisItem.potemplateID == self.potemplate.id)),
+            ]
+
+        if self.potemplate.productseries is not None:
+            # If the template is now in a product, we look for all
+            # effectively sharing templates that are in *different*
+            # products, or that are in a sourcepackage which is not
+            # linked (through Packaging table) with this product.
+            ps = self.potemplate.productseries
+            productseries_join = LeftJoin(
+                ProductSeries,
+                ProductSeries.id == OtherTemplate.productseriesID)
+            packaging_join = LeftJoin(
+                Packaging,
+                And(Packaging.productseriesID == ps.id,
+                    (Packaging.sourcepackagenameID ==
+                     OtherTemplate.sourcepackagenameID),
+                    Packaging.distroseriesID == OtherTemplate.distroseriesID
+                    ))
+            tables.extend([productseries_join, packaging_join])
+            # Template should not be sharing if...
+            other_clauses = Or(
+                # The name is different, or...
+                OtherTemplate.name != self.potemplate.name,
+                # It's in a different product, or...
+                And(Not(ProductSeries.id == None),
+                    ProductSeries.productID != ps.productID),
+                # There is no link between this product series and
+                # a source package the template is in.
+                And(Not(OtherTemplate.distroseriesID == None),
+                    Packaging.id == None))
+        else:
+            # If the template is now in a source package, we look for all
+            # effectively sharing templates that are in *different*
+            # distributions or source packages, or that are in a product
+            # which is not linked with this source package.
+            ds = self.potemplate.distroseries
+            spn = self.potemplate.sourcepackagename
+            distroseries_join = LeftJoin(
+                DistroSeries,
+                DistroSeries.id == OtherTemplate.distroseriesID)
+            packaging_join = LeftJoin(
+                Packaging,
+                And(Packaging.distroseriesID == ds.id,
+                    Packaging.sourcepackagenameID == spn.id,
+                    Packaging.productseriesID == OtherTemplate.productseriesID
+                    ))
+            tables.extend([distroseries_join, packaging_join])
+            # Template should not be sharing if...
+            other_clauses = Or(
+                # The name is different, or...
+                OtherTemplate.name != self.potemplate.name,
+                # It's in a different distribution or source package, or...
+                And(Not(DistroSeries.id == None),
+                    Or(DistroSeries.distributionID != ds.distributionID,
+                       OtherTemplate.sourcepackagenameID != spn.id)),
+                # There is no link between this source package and
+                # a product the template is in.
+                And(Not(OtherTemplate.productseriesID == None),
+                    Packaging.id == None))
+
+        return store.using(*tables).find(
+            (OtherItem, ThisItem),
+            OtherTemplate.id != self.potemplate.id,
+            other_clauses,
+            )