Merge lp:~jtv/launchpad/bug-662552-suggestive-potemplates into lp:launchpad

Proposed by Jeroen T. Vermeulen on 2010-10-26
Status: Merged
Approved by: Jeroen T. Vermeulen on 2010-10-26
Approved revision: no longer in the source branch.
Merged at revision: 11800
Proposed branch: lp:~jtv/launchpad/bug-662552-suggestive-potemplates
Merge into: lp:launchpad
Diff against target: 335 lines (+91/-20)
11 files modified
lib/lp/translations/doc/potmsgset.txt (+17/-1)
lib/lp/translations/model/potmsgset.py (+4/-17)
lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt (+8/-0)
lib/lp/translations/stories/standalone/xx-pofile-translate-dismiss-suggestions.txt (+10/-0)
lib/lp/translations/stories/standalone/xx-pofile-translate-legal-warning.txt (+8/-0)
lib/lp/translations/stories/standalone/xx-pofile-translate.txt (+8/-2)
lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt (+10/-0)
lib/lp/translations/stories/translationgroups/60-translation-suggestions.txt (+5/-0)
lib/lp/translations/tests/test_potmsgset.py (+7/-0)
lib/lp/translations/tests/test_suggestions.py (+6/-0)
lib/lp/translations/tests/test_translatablemessage.py (+8/-0)
To merge this branch: bzr merge lp:~jtv/launchpad/bug-662552-suggestive-potemplates
Reviewer Review Type Date Requested Status
Abel Deuring (community) code 2010-10-26 Approve on 2010-10-26
Review via email: mp+39354@code.launchpad.net

Commit message

Use suggestive-potemplates cache.

Description of the change

= Bug 662552 =

Translations has been getting a lot of timeouts lately. The page that probably makes users suffer most is POFile:+translate, which is slow mainly because it collects and displays "global suggestions." These are translations that have been entered or used for a given English string elsewhere in Launchpad. They're one of those features that make Launchpad Translations so useful.

Part of the work when gathering global suggestions is to find matching English strings and their translations. Another part is finding out whether their templates belong to projects or distributions that officially use Launchpad for translations, and whether they occur in enabled templates.

This second part is done identically several times for each message on a translation page, and does not change between page requests.

So I did the obvious, some time ago: create a little cache table that lists the templates that qualify as sources for global suggestions. This is the "suggestive templates cache." The cache is present and kept up to date on production, but not used yet. This branch changes that, simplifying the global-suggestions query to a narrower join with less sensitivity to locks. (And an XXX disappears in the process).

Most of the diff deals with tests that are affected by the extra caching. Generally all that's needed is to initialize the cache at the beginning of the test; I didn't want to add more sample data here that needs to be kept consistent and restored after every test.

The best way to test this is to run all Translations tests:
{{{
./bin/test -vvc lp.translations
}}}

To Q/A, check that the translation pages still show global suggestions, i.e. suggestions that are not directly related to the context that you're translating. For instance, until we land the Recife feature branch, suggestions made (but not used) in Ubuntu packages show up in their upstream projects only because of global suggestions.

When doing Q/A, it's important to ensure that the server's configuration has been updated to re-enable global suggestions. Oops levels should end up below what they were on 2010-10-26, and preferably less even than that. The easiest thing to observe is probably the difference in page timings between staging and production.

There's a bit of lint left, though none that I would seem to have caused. I'd be happy to clean it up during or after review. For now I did not want to pollute the diff.

Jeroen

To post a comment you must log in.
Abel Deuring (adeuring) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/translations/doc/potmsgset.txt'
2--- lib/lp/translations/doc/potmsgset.txt 2010-10-18 22:24:59 +0000
3+++ lib/lp/translations/doc/potmsgset.txt 2010-10-26 10:49:44 +0000
4@@ -701,10 +701,11 @@
5 On the other, we have a translation template for the evolution package in
6 Ubuntu Hoary distribution.
7
8+ >>> templateset = getUtility(IPOTemplateSet)
9 >>> ubuntu = getUtility(IDistributionSet)['ubuntu']
10 >>> ubuntu_hoary = ubuntu.getSeries('hoary')
11 >>> evo_hoary_package = ubuntu_hoary.getSourcePackage('evolution')
12- >>> evo_distro_template = getUtility(IPOTemplateSet).getSubset(
13+ >>> evo_distro_template = templateset.getSubset(
14 ... sourcepackagename=evo_hoary_package.sourcepackagename,
15 ... distroseries=ubuntu_hoary).getPOTemplateByName('evolution-2.2')
16 >>> print evo_distro_template.title
17@@ -724,6 +725,16 @@
18 >>> evo_distro_template.iscurrent
19 True
20
21+The "suggestive templates" cache is up to date.
22+
23+ >>> def refresh_suggestive_templates_cache():
24+ ... """Update the `SuggestivePOTemplate` cache."""
25+ ... templateset.wipeSuggestivePOTemplatesCache()
26+ ... templateset.populateSuggestivePOTemplatesCache()
27+
28+ >>> refresh_suggestive_templates_cache()
29+ >>> transaction.commit()
30+
31 We have the same message in both templates but with different
32 translations in Spanish:
33
34@@ -765,6 +776,7 @@
35 are not available as suggestions anymore:
36
37 >>> evo_distro_template.iscurrent = False
38+ >>> refresh_suggestive_templates_cache()
39 >>> transaction.commit()
40 >>> suggestions = (
41 ... evo_product_message.getExternallyUsedTranslationMessages(spanish))
42@@ -790,6 +802,7 @@
43 And products not using translations officially have the same behaviour.
44
45 >>> evolution.translations_usage = ServiceUsage.NOT_APPLICABLE
46+ >>> refresh_suggestive_templates_cache()
47 >>> transaction.commit()
48 >>> suggestions = evo_distro_message.getExternallyUsedTranslationMessages(
49 ... spanish)
50@@ -800,6 +813,7 @@
51
52 >>> ubuntu.translations_usage = ServiceUsage.LAUNCHPAD
53 >>> evolution.translations_usage = ServiceUsage.LAUNCHPAD
54+ >>> refresh_suggestive_templates_cache()
55 >>> transaction.commit()
56
57
58@@ -885,6 +899,7 @@
59 we get no suggestions.
60
61 >>> potmsgset_translated.potemplate.iscurrent = False
62+ >>> refresh_suggestive_templates_cache()
63 >>> transaction.commit()
64
65 >>> wiki_submissions = (
66@@ -900,6 +915,7 @@
67 # suggestions just due to the change to the official_rosetta flag.
68 >>> potmsgset_translated.potemplate.iscurrent = True
69 >>> ubuntu.translations_usage = ServiceUsage.NOT_APPLICABLE
70+ >>> refresh_suggestive_templates_cache()
71 >>> transaction.commit()
72
73 >>> wiki_submissions = (
74
75=== modified file 'lib/lp/translations/model/potmsgset.py'
76--- lib/lp/translations/model/potmsgset.py 2010-09-07 15:26:49 +0000
77+++ lib/lp/translations/model/potmsgset.py 2010-10-26 10:49:44 +0000
78@@ -361,31 +361,18 @@
79 query = ["(NOT %s)" % in_use_clause]
80 query.append('TranslationMessage.language = %s' % sqlvalues(language))
81
82- # XXX j.c.sackett 2010-08-30 bug=627631 Once data migration has
83- # happened for the usage enums, this sql needs to be updated
84- # to check for the translations_usage, not official_rosetta.
85 query.append('''
86 potmsgset IN (
87 SELECT POTMsgSet.id
88 FROM POTMsgSet
89 JOIN TranslationTemplateItem ON
90 TranslationTemplateItem.potmsgset = POTMsgSet.id
91- JOIN POTemplate ON
92- TranslationTemplateItem.potemplate = POTemplate.id
93- LEFT JOIN ProductSeries ON
94- POTemplate.productseries = ProductSeries.id
95- LEFT JOIN Product ON
96- ProductSeries.product = Product.id
97- LEFT JOIN DistroSeries ON
98- POTemplate.distroseries = DistroSeries.id
99- LEFT JOIN Distribution ON
100- DistroSeries.distribution = Distribution.id
101+ JOIN SuggestivePOTemplate ON
102+ TranslationTemplateItem.potemplate =
103+ SuggestivePOTemplate.potemplate
104 WHERE
105 POTMsgSet.id <> %s AND
106- msgid_singular = %s AND
107- POTemplate.iscurrent AND
108- (Product.official_rosetta OR
109- Distribution.official_rosetta)
110+ msgid_singular = %s
111 )''' % sqlvalues(self, self.msgid_singular))
112
113 # Subquery to find the ids of TranslationMessages that are
114
115=== modified file 'lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt'
116--- lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt 2009-09-15 18:53:26 +0000
117+++ lib/lp/translations/stories/standalone/xx-pofile-translate-alternative-language.txt 2010-10-26 10:49:44 +0000
118@@ -4,7 +4,15 @@
119 form will suggest translations from this alternative language as well as those
120 in the form's main language.
121
122+ >>> from zope.component import getUtility
123 >>> from canonical.launchpad.testing.pages import extract_url_parameter
124+ >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
125+
126+ >>> login(ANONYMOUS)
127+ >>> utility = getUtility(IPOTemplateSet)
128+ >>> dummy = utility.populateSuggestivePOTemplatesCache()
129+ >>> logout()
130+
131 >>> def get_alternative_languages_widget(browser):
132 ... """Check and return alternative languages widget offered on page.
133 ...
134
135=== modified file 'lib/lp/translations/stories/standalone/xx-pofile-translate-dismiss-suggestions.txt'
136--- lib/lp/translations/stories/standalone/xx-pofile-translate-dismiss-suggestions.txt 2009-10-29 17:46:00 +0000
137+++ lib/lp/translations/stories/standalone/xx-pofile-translate-dismiss-suggestions.txt 2010-10-26 10:49:44 +0000
138@@ -4,6 +4,16 @@
139 translation of a message. If the current translation is good enough, new
140 suggestions can be dismissed to keep them off the page.
141
142+ >>> from zope.component import getUtility
143+ >>> from canonical.launchpad.testing.pages import extract_url_parameter
144+ >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
145+
146+ >>> login(ANONYMOUS)
147+ >>> utility = getUtility(IPOTemplateSet)
148+ >>> dummy = utility.populateSuggestivePOTemplatesCache()
149+ >>> logout()
150+
151+
152 First the admin (or anybody with edit rights) adds a translation.
153
154 >>> admin_browser.open('http://translations.launchpad.dev/alsa-utils'
155
156=== modified file 'lib/lp/translations/stories/standalone/xx-pofile-translate-legal-warning.txt'
157--- lib/lp/translations/stories/standalone/xx-pofile-translate-legal-warning.txt 2010-03-18 18:55:02 +0000
158+++ lib/lp/translations/stories/standalone/xx-pofile-translate-legal-warning.txt 2010-10-26 10:49:44 +0000
159@@ -10,6 +10,14 @@
160 On a Hoary Evolution sourcepackage, we can see a suggestion coming
161 from the upstream project.
162
163+ >>> from zope.component import getUtility
164+ >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
165+
166+ >>> login(ANONYMOUS)
167+ >>> utility = getUtility(IPOTemplateSet)
168+ >>> dummy = utility.populateSuggestivePOTemplatesCache()
169+ >>> logout()
170+
171 >>> browser = setupBrowser(auth='Basic carlos@canonical.com:test')
172 >>> browser.open('http://translations.launchpad.dev/ubuntu/hoary/+source/evolution/+pots/evolution-2.2/es/3/+translate')
173 >>> print extract_text(find_tag_by_id(
174
175=== modified file 'lib/lp/translations/stories/standalone/xx-pofile-translate.txt'
176--- lib/lp/translations/stories/standalone/xx-pofile-translate.txt 2010-10-05 00:08:16 +0000
177+++ lib/lp/translations/stories/standalone/xx-pofile-translate.txt 2010-10-26 10:49:44 +0000
178@@ -1,5 +1,3 @@
179-
180-
181 Translation Submissions
182 =======================
183
184@@ -17,6 +15,14 @@
185 xx-pofile-translate-newlines-check.txt
186 xx-pofile-translate-performance.txt
187
188+ >>> from zope.component import getUtility
189+ >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
190+
191+ >>> login(ANONYMOUS)
192+ >>> utility = getUtility(IPOTemplateSet)
193+ >>> dummy = utility.populateSuggestivePOTemplatesCache()
194+ >>> logout()
195+
196
197 Anonymous access
198 ----------------
199
200=== modified file 'lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt'
201--- lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt 2010-10-18 22:24:59 +0000
202+++ lib/lp/translations/stories/standalone/xx-translationmessage-translate.txt 2010-10-26 10:49:44 +0000
203@@ -4,6 +4,16 @@
204 Here we are going to check the basic behaviour of the translation form
205 when we render just one message with all available information for it.
206
207+The suggestive-templates cache needs to be up to date.
208+
209+ >>> from zope.component import getUtility
210+ >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
211+
212+ >>> login(ANONYMOUS)
213+ >>> utility = getUtility(IPOTemplateSet)
214+ >>> dummy = utility.populateSuggestivePOTemplatesCache()
215+ >>> logout()
216+
217
218 Getting there
219 -------------
220
221=== modified file 'lib/lp/translations/stories/translationgroups/60-translation-suggestions.txt'
222--- lib/lp/translations/stories/translationgroups/60-translation-suggestions.txt 2010-10-18 22:24:59 +0000
223+++ lib/lp/translations/stories/translationgroups/60-translation-suggestions.txt 2010-10-26 10:49:44 +0000
224@@ -16,6 +16,7 @@
225 >>> from lp.registry.interfaces.distribution import IDistributionSet
226 >>> from lp.registry.interfaces.person import IPersonSet
227 >>> from lp.services.worlddata.interfaces.language import ILanguageSet
228+ >>> from lp.translations.interfaces.potemplate import IPOTemplateSet
229 >>> from lp.translations.interfaces.translator import ITranslatorSet
230
231 >>> login('foo.bar@canonical.com')
232@@ -25,6 +26,10 @@
233 >>> ubuntu_spanish_reviewer = getUtility(ITranslatorSet).new(
234 ... translationgroup=ubuntu.translationgroup, language=spanish,
235 ... translator=carlos)
236+
237+ >>> utility = getUtility(IPOTemplateSet)
238+ >>> dummy = utility.populateSuggestivePOTemplatesCache()
239+
240 >>> logout()
241
242 Let's add a new suggestion as a person without privileges.
243
244=== modified file 'lib/lp/translations/tests/test_potmsgset.py'
245--- lib/lp/translations/tests/test_potmsgset.py 2010-10-04 19:50:45 +0000
246+++ lib/lp/translations/tests/test_potmsgset.py 2010-10-26 10:49:44 +0000
247@@ -25,6 +25,7 @@
248 from lp.registry.interfaces.product import IProductSet
249 from lp.services.worlddata.interfaces.language import ILanguageSet
250 from lp.testing import TestCaseWithFactory
251+from lp.translations.interfaces.potemplate import IPOTemplateSet
252 from lp.translations.interfaces.potmsgset import (
253 POTMsgSetInIncompatibleTemplatesError,
254 TranslationCreditsType,
255@@ -64,6 +65,10 @@
256 self.potmsgset = self.factory.makePOTMsgSet(self.devel_potemplate)
257 self.potmsgset.setSequence(self.devel_potemplate, 1)
258
259+ def _refreshSuggestiveTemplatesCache(self):
260+ """Refresh the `SuggestivePOTemplate` cache."""
261+ getUtility(IPOTemplateSet).populateSuggestivePOTemplatesCache()
262+
263 def test_TranslationTemplateItem(self):
264 self.potmsgset.setSequence(self.stable_potemplate, 1)
265
266@@ -333,6 +338,7 @@
267 external_potmsgset.setSequence(external_template, 1)
268 external_pofile = self.factory.makePOFile('sr', external_template)
269 serbian = external_pofile.language
270+ self._refreshSuggestiveTemplatesCache()
271
272 transaction.commit()
273
274@@ -393,6 +399,7 @@
275 external_potmsgset.setSequence(external_template, 1)
276 external_pofile = self.factory.makePOFile('sr', external_template)
277 serbian = external_pofile.language
278+ self._refreshSuggestiveTemplatesCache()
279
280 transaction.commit()
281
282
283=== modified file 'lib/lp/translations/tests/test_suggestions.py'
284--- lib/lp/translations/tests/test_suggestions.py 2010-10-04 19:50:45 +0000
285+++ lib/lp/translations/tests/test_suggestions.py 2010-10-26 10:49:44 +0000
286@@ -20,6 +20,7 @@
287 from lp.app.enums import ServiceUsage
288 from lp.services.worlddata.interfaces.language import ILanguageSet
289 from lp.testing.factory import LaunchpadObjectFactory
290+from lp.translations.interfaces.potemplate import IPOTemplateSet
291 from lp.translations.interfaces.translationmessage import (
292 TranslationValidationStatus,
293 )
294@@ -50,6 +51,11 @@
295 self.nl = getUtility(ILanguageSet).getLanguageByCode('nl')
296 self.foo_nl = factory.makePOFile('nl', potemplate=self.foo_template)
297 self.bar_nl = factory.makePOFile('nl', potemplate=self.bar_template)
298+ self._refreshSuggestiveTemplatesCache()
299+
300+ def _refreshSuggestiveTemplatesCache(self):
301+ """Update the `SuggestivePOTemplate` cache."""
302+ getUtility(IPOTemplateSet).populateSuggestivePOTemplatesCache()
303
304 def test_NoSuggestions(self):
305 # When a msgid string is unique and nobody has submitted any
306
307=== modified file 'lib/lp/translations/tests/test_translatablemessage.py'
308--- lib/lp/translations/tests/test_translatablemessage.py 2010-10-04 19:50:45 +0000
309+++ lib/lp/translations/tests/test_translatablemessage.py 2010-10-26 10:49:44 +0000
310@@ -13,10 +13,12 @@
311
312 import pytz
313 import transaction
314+from zope.component import getUtility
315
316 from canonical.testing.layers import ZopelessDatabaseLayer
317 from lp.app.enums import ServiceUsage
318 from lp.testing import TestCaseWithFactory
319+from lp.translations.interfaces.potemplate import IPOTemplateSet
320 from lp.translations.model.translatablemessage import TranslatableMessage
321
322
323@@ -153,6 +155,12 @@
324
325 self.message = TranslatableMessage(self.potmsgset, self.pofile)
326
327+ self._refreshSuggestiveTemplatesCache()
328+
329+ def _refreshSuggestiveTemplatesCache(self):
330+ """Refresh the `SuggestivePOTemplate` cache."""
331+ getUtility(IPOTemplateSet).populateSuggestivePOTemplatesCache()
332+
333 def test_getExternalTranslations(self):
334 transaction.commit()
335 externals = self.message.getExternalTranslations()