Merge lp:~jtv/launchpad/bug-662552-suggestive-potemplates into lp:launchpad

[Jeroen T. Vermeulen]

Use suggestive-potemplates cache.

= Bug 662552 =

Translations has been getting a lot of timeouts lately. The page that probably makes users suffer most is POFile:+translate, which is slow mainly because it collects and displays "global suggestions." These are translations that have been entered or used for a given English string elsewhere in Launchpad. They're one of those features that make Launchpad Translations so useful.

Part of the work when gathering global suggestions is to find matching English strings and their translations. Another part is finding out whether their templates belong to projects or distributions that officially use Launchpad for translations, and whether they occur in enabled templates.

This second part is done identically several times for each message on a translation page, and does not change between page requests.

So I did the obvious, some time ago: create a little cache table that lists the templates that qualify as sources for global suggestions. This is the "suggestive templates cache." The cache is present and kept up to date on production, but not used yet. This branch changes that, simplifying the global-suggestions query to a narrower join with less sensitivity to locks. (And an XXX disappears in the process).

Most of the diff deals with tests that are affected by the extra caching. Generally all that's needed is to initialize the cache at the beginning of the test; I didn't want to add more sample data here that needs to be kept consistent and restored after every test.

The best way to test this is to run all Translations tests:
{{{
./bin/test -vvc lp.translations
}}}

To Q/A, check that the translation pages still show global suggestions, i.e. suggestions that are not directly related to the context that you're translating. For instance, until we land the Recife feature branch, suggestions made (but not used) in Ubuntu packages show up in their upstream projects only because of global suggestions.

When doing Q/A, it's important to ensure that the server's configuration has been updated to re-enable global suggestions. Oops levels should end up below what they were on 2010-10-26, and preferably less even than that. The easiest thing to observe is probably the difference in page timings between staging and production.

There's a bit of lint left, though none that I would seem to have caused. I'd be happy to clean it up during or after review. For now I did not want to pollute the diff.

Jeroen

[Abel Deuring (adeuring)]

review: Approve (code)

=== modified file 'lib/lp/translations/doc/potmsgset.txt'
--- lib/lp/translations/doc/potmsgset.txt	2010-10-18 22:24:59 +0000
+++ lib/lp/translations/doc/potmsgset.txt	2010-10-26 10:49:44 +0000
@@ -701,10 +701,11 @@
 On the other, we have a translation template for the evolution package in
 Ubuntu Hoary distribution.
 
+    >>> templateset = getUtility(IPOTemplateSet)
     >>> ubuntu = getUtility(IDistributionSet)['ubuntu']
     >>> ubuntu_hoary = ubuntu.getSeries('hoary')
     >>> evo_hoary_package = ubuntu_hoary.getSourcePackage('evolution')
-    >>> evo_distro_template = getUtility(IPOTemplateSet).getSubset(
+    >>> evo_distro_template = templateset.getSubset(
     ...     sourcepackagename=evo_hoary_package.sourcepackagename,
     ...     distroseries=ubuntu_hoary).getPOTemplateByName('evolution-2.2')
     >>> print evo_distro_template.title
@@ -724,6 +725,16 @@
     >>> evo_distro_template.iscurrent
     True
 
+The "suggestive templates" cache is up to date.
+
+    >>> def refresh_suggestive_templates_cache():
+    ...     """Update the `SuggestivePOTemplate` cache."""
+    ...     templateset.wipeSuggestivePOTemplatesCache()
+    ...     templateset.populateSuggestivePOTemplatesCache()
+
+    >>> refresh_suggestive_templates_cache()
+    >>> transaction.commit()
+
 We have the same message in both templates but with different
 translations in Spanish:
 
@@ -765,6 +776,7 @@
 are not available as suggestions anymore:
 
     >>> evo_distro_template.iscurrent = False
+    >>> refresh_suggestive_templates_cache()
     >>> transaction.commit()
     >>> suggestions = (
     ...     evo_product_message.getExternallyUsedTranslationMessages(spanish))
@@ -790,6 +802,7 @@
 And products not using translations officially have the same behaviour.
 
     >>> evolution.translations_usage = ServiceUsage.NOT_APPLICABLE
+    >>> refresh_suggestive_templates_cache()
     >>> transaction.commit()
     >>> suggestions = evo_distro_message.getExternallyUsedTranslationMessages(
     ...    spanish)
@@ -800,6 +813,7 @@
 
     >>> ubuntu.translations_usage = ServiceUsage.LAUNCHPAD
     >>> evolution.translations_usage = ServiceUsage.LAUNCHPAD
+    >>> refresh_suggestive_templates_cache()
     >>> transaction.commit()
 
 
@@ -885,6 +899,7 @@
 we get no suggestions.
 
     >>> potmsgset_translated.potemplate.iscurrent = False
+    >>> refresh_suggestive_templates_cache()
     >>> transaction.commit()
 
     >>> wiki_submissions = (
@@ -900,6 +915,7 @@
     # suggestions just due to the change to the official_rosetta flag.
     >>> potmsgset_translated.potemplate.iscurrent = True
     >>> ubuntu.translations_usage = ServiceUsage.NOT_APPLICABLE
+    >>> refresh_suggestive_templates_cache()
     >>> transaction.commit()
 
     >>> wiki_submissions = (
 
=== modified file 'lib/lp/translations/model/potmsgset.py'
--- lib/lp/translations/model/potmsgset.py	2010-09-07 15:26:49 +0000
+++ lib/lp/translations/model/potmsgset.py	2010-10-26 10:49:44 +0000
@@ -361,31 +361,18 @@
             query = ["(NOT %s)" % in_use_clause]
         query.append('TranslationMessage.language = %s' % sqlvalues(language))
 
-        # XXX j.c.sackett 2010-08-30 bug=627631 Once data migration has
-        # happened for the usage enums, this sql needs to be updated
-        # to check for the translations_usage, not official_rosetta.
         query.append('''
             potmsgset IN (
                 SELECT POTMsgSet.id
                 FROM POTMsgSet
                 JOIN TranslationTemplateItem ON
                     TranslationTemplateItem.potmsgset = POTMsgSet.id
-                JOIN POTemplate ON
-                    TranslationTemplateItem.potemplate = POTemplate.id
-                LEFT JOIN ProductSeries ON
-                    POTemplate.productseries = ProductSeries.id
-                LEFT JOIN Product ON
-                    ProductSeries.product = Product.id
-                LEFT JOIN DistroSeries ON
-                    POTemplate.distroseries = DistroSeries.id
-                LEFT JOIN Distribution ON
-                    DistroSeries.distribution = Distribution.id
+                JOIN SuggestivePOTemplate ON
+                    TranslationTemplateItem.potemplate =
+                        SuggestivePOTemplate.potemplate
                 WHERE
                     POTMsgSet.id <> %s AND
-                    msgid_singular = %s AND
-                    POTemplate.iscurrent AND
-                    (Product.official_rosetta OR
-                        Distribution.official_rosetta)
+                    msgid_singular = %s
             )''' % sqlvalues(self, self.msgid_singular))
 
         # Subquery to find the ids of TranslationMessages that are
 
=== modified file 'lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt'
--- lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt	2009-09-15 18:53:26 +0000
+++ lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt	2010-10-26 10:49:44 +0000
@@ -4,7 +4,15 @@
 form will suggest translations from this alternative language as well as those
 in the form's main language.
 
+    >>> from zope.component import getUtility
     >>> from canonical.launchpad.testing.pages import extract_url_parameter
+    >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
+
+    >>> login(ANONYMOUS)
+    >>> utility = getUtility(IPOTemplateSet)
+    >>> dummy = utility.populateSuggestivePOTemplatesCache()
+    >>> logout()
+
     >>> def get_alternative_languages_widget(browser):
     ...     """Check and return alternative languages widget offered on page.
     ...
 
=== modified file 'lib/lp/translations/stories/standalone/xx-pofile-translate-dismiss-suggestions.txt'
--- lib/lp/translations/stories/standalone/xx-pofile-translate-dismiss-suggestions.txt	2009-10-29 17:46:00 +0000
+++ lib/lp/translations/stories/standalone/xx-pofile-translate-dismiss-suggestions.txt	2010-10-26 10:49:44 +0000
@@ -4,6 +4,16 @@
 translation of a message. If the current translation is good enough, new
 suggestions can be dismissed to keep them off the page.
 
+    >>> from zope.component import getUtility
+    >>> from canonical.launchpad.testing.pages import extract_url_parameter
+    >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
+
+    >>> login(ANONYMOUS)
+    >>> utility = getUtility(IPOTemplateSet)
+    >>> dummy = utility.populateSuggestivePOTemplatesCache()
+    >>> logout()
+
+
 First the admin (or anybody with edit rights) adds a translation.
 
     >>> admin_browser.open('http://translations.launchpad.dev/alsa-utils'
 
=== modified file 'lib/lp/translations/stories/standalone/xx-pofile-translate-legal-warning.txt'
--- lib/lp/translations/stories/standalone/xx-pofile-translate-legal-warning.txt	2010-03-18 18:55:02 +0000
+++ lib/lp/translations/stories/standalone/xx-pofile-translate-legal-warning.txt	2010-10-26 10:49:44 +0000
@@ -10,6 +10,14 @@
 On a Hoary Evolution sourcepackage, we can see a suggestion coming
 from the upstream project.
 
+    >>> from zope.component import getUtility
+    >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
+
+    >>> login(ANONYMOUS)
+    >>> utility = getUtility(IPOTemplateSet)
+    >>> dummy = utility.populateSuggestivePOTemplatesCache()
+    >>> logout()
+
     >>> browser = setupBrowser(auth='Basic [e-mail]:test')
     >>> browser.open('http://translations.launchpad.dev/ubuntu/hoary/+source/evolution/+pots/evolution-2.2/es/3/+translate')
     >>> print extract_text(find_tag_by_id(
 
=== modified file 'lib/lp/translations/stories/standalone/xx-pofile-translate.txt'
--- lib/lp/translations/stories/standalone/xx-pofile-translate.txt	2010-10-05 00:08:16 +0000
+++ lib/lp/translations/stories/standalone/xx-pofile-translate.txt	2010-10-26 10:49:44 +0000
@@ -1,5 +1,3 @@
-
-
 Translation Submissions
 =======================
 
@@ -17,6 +15,14 @@
 xx-pofile-translate-newlines-check.txt
 xx-pofile-translate-performance.txt
 
+    >>> from zope.component import getUtility
+    >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
+
+    >>> login(ANONYMOUS)
+    >>> utility = getUtility(IPOTemplateSet)
+    >>> dummy = utility.populateSuggestivePOTemplatesCache()
+    >>> logout()
+
 
 Anonymous access
 ----------------
 
=== modified file 'lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt'
--- lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt	2010-10-18 22:24:59 +0000
+++ lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt	2010-10-26 10:49:44 +0000
@@ -4,6 +4,16 @@
 Here we are going to check the basic behaviour of the translation form
 when we render just one message with all available information for it.
 
+The suggestive-templates cache needs to be up to date.
+
+    >>> from zope.component import getUtility
+    >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
+
+    >>> login(ANONYMOUS)
+    >>> utility = getUtility(IPOTemplateSet)
+    >>> dummy = utility.populateSuggestivePOTemplatesCache()
+    >>> logout()
+
 
 Getting there
 -------------
 
=== modified file 'lib/lp/translations/stories/translationgroups/60-translation-suggestions.txt'
--- lib/lp/translations/stories/translationgroups/60-translation-suggestions.txt	2010-10-18 22:24:59 +0000
+++ lib/lp/translations/stories/translationgroups/60-translation-suggestions.txt	2010-10-26 10:49:44 +0000
@@ -16,6 +16,7 @@
     >>> from lp.registry.interfaces.distribution import IDistributionSet
     >>> from lp.registry.interfaces.person import IPersonSet
     >>> from lp.services.worlddata.interfaces.language import ILanguageSet
+    >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
     >>> from lp.translations.interfaces.translator import ITranslatorSet
 
     >>> login('[e-mail]')
@@ -25,6 +26,10 @@
     >>> ubuntu_spanish_reviewer = getUtility(ITranslatorSet).new(
     ...     translationgroup=ubuntu.translationgroup, language=spanish,
     ...     translator=carlos)
+
+    >>> utility = getUtility(IPOTemplateSet)
+    >>> dummy = utility.populateSuggestivePOTemplatesCache()
+
     >>> logout()
 
 Let's add a new suggestion as a person without privileges.
 
=== modified file 'lib/lp/translations/tests/test_potmsgset.py'
--- lib/lp/translations/tests/test_potmsgset.py	2010-10-04 19:50:45 +0000
+++ lib/lp/translations/tests/test_potmsgset.py	2010-10-26 10:49:44 +0000
@@ -25,6 +25,7 @@
 from lp.registry.interfaces.product import IProductSet
 from lp.services.worlddata.interfaces.language import ILanguageSet
 from lp.testing import TestCaseWithFactory
+from lp.translations.interfaces.potemplate import IPOTemplateSet
 from lp.translations.interfaces.potmsgset import (
     POTMsgSetInIncompatibleTemplatesError,
     TranslationCreditsType,
@@ -64,6 +65,10 @@
         self.potmsgset = self.factory.makePOTMsgSet(self.devel_potemplate)
         self.potmsgset.setSequence(self.devel_potemplate, 1)
 
+    def _refreshSuggestiveTemplatesCache(self):
+        """Refresh the `SuggestivePOTemplate` cache."""
+        getUtility(IPOTemplateSet).populateSuggestivePOTemplatesCache()
+
     def test_TranslationTemplateItem(self):
         self.potmsgset.setSequence(self.stable_potemplate, 1)
 
@@ -333,6 +338,7 @@
         external_potmsgset.setSequence(external_template, 1)
         external_pofile = self.factory.makePOFile('sr', external_template)
         serbian = external_pofile.language
+        self._refreshSuggestiveTemplatesCache()
 
         transaction.commit()
 
@@ -393,6 +399,7 @@
         external_potmsgset.setSequence(external_template, 1)
         external_pofile = self.factory.makePOFile('sr', external_template)
         serbian = external_pofile.language
+        self._refreshSuggestiveTemplatesCache()
 
         transaction.commit()
 
 
=== modified file 'lib/lp/translations/tests/test_suggestions.py'
--- lib/lp/translations/tests/test_suggestions.py	2010-10-04 19:50:45 +0000
+++ lib/lp/translations/tests/test_suggestions.py	2010-10-26 10:49:44 +0000
@@ -20,6 +20,7 @@
 from lp.app.enums import ServiceUsage
 from lp.services.worlddata.interfaces.language import ILanguageSet
 from lp.testing.factory import LaunchpadObjectFactory
+from lp.translations.interfaces.potemplate import IPOTemplateSet
 from lp.translations.interfaces.translationmessage import (
     TranslationValidationStatus,
     )
@@ -50,6 +51,11 @@
         self.nl = getUtility(ILanguageSet).getLanguageByCode('nl')
         self.foo_nl = factory.makePOFile('nl', potemplate=self.foo_template)
         self.bar_nl = factory.makePOFile('nl', potemplate=self.bar_template)
+        self._refreshSuggestiveTemplatesCache()
+
+    def _refreshSuggestiveTemplatesCache(self):
+        """Update the `SuggestivePOTemplate` cache."""
+        getUtility(IPOTemplateSet).populateSuggestivePOTemplatesCache()
 
     def test_NoSuggestions(self):
         # When a msgid string is unique and nobody has submitted any
 
=== modified file 'lib/lp/translations/tests/test_translatablemessage.py'
--- lib/lp/translations/tests/test_translatablemessage.py	2010-10-04 19:50:45 +0000
+++ lib/lp/translations/tests/test_translatablemessage.py	2010-10-26 10:49:44 +0000
@@ -13,10 +13,12 @@
 
 import pytz
 import transaction
+from zope.component import getUtility
 
 from canonical.testing.layers import ZopelessDatabaseLayer
 from lp.app.enums import ServiceUsage
 from lp.testing import TestCaseWithFactory
+from lp.translations.interfaces.potemplate import IPOTemplateSet
 from lp.translations.model.translatablemessage import TranslatableMessage
 
 
@@ -153,6 +155,12 @@
 
         self.message = TranslatableMessage(self.potmsgset, self.pofile)
 
+        self._refreshSuggestiveTemplatesCache()
+
+    def _refreshSuggestiveTemplatesCache(self):
+        """Refresh the `SuggestivePOTemplate` cache."""
+        getUtility(IPOTemplateSet).populateSuggestivePOTemplatesCache()
+
     def test_getExternalTranslations(self):
         transaction.commit()
         externals = self.message.getExternalTranslations()