Move statistics updating outside the web browser code

Bug #877195 reported by Данило Шеган
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
Benji York
Ubuntu Translations
Fix Released
Undecided
Unassigned

Bug Description

At the moment, we have a lot of problems with translation statistics not being updated real-time even though we make every attempt to do so. I am suspecting this fails mostly due to timeouts in POSTs when we issue POFile.updateStatistics() call.

We have so far solved this with band-aids like cronscripts/rosetta-pofile-stats.py and cronscripts/rosetta-pofile-stats-daily.py, which have their own problems (bug 622668 and bug 781274). These scripts became even more important since message-sharing was introduced, because POFile.updateStatistics() updated the statistics only on a single POFile that was being touched, even though changes on its messages affect "sharing" POFiles as well. pofile-stats-daily script specifically has logic to find these "sharing" files, though it might be incomplete.

Instead of trying to fix our band-aids, we should fix the root cause instead. We know that sometimes it's going to be impossible to both save all the changes on a single +translate page *and* calculate the statistics (eg. ddtp-ubuntu templates are over 40k messages total) in a single POST, and we also know that statistics being very up-to-date real-time is not essential (we have gotten translators used to them being a week or so behind; having them eg. 5 minute behind at *most* would be a huge improvement).

So, we should instead decouple translation saving from statistics update. I propose we move any updateStatistics calls from the browser code (though, there should be only one) to a job which is scheduled to run whenever there are changes to a POFile. We should also extend updateStatistics call to support finding all the sharing POFiles as well (all the potentially touched POFiles).
The same should be done for POTemplate.newPOFile callsite which is called from within the browser code as well.

Other updateStatistics callsites (eg. in model/pofile.py and another one in model/potemplate.py importFromQueue) are related to imports which are already de-coupled from the browser code, so can (and should) stay "inline". Having them fix the statistics for all the sharing POFiles would be nothing but an improvement.

Once all of the above is done, we should be able to kill rosetta-pofile-stats-daily.py script, and leave rosetta-pofile-stats.py script around just in case we want to run it manually (we should remove it from the crontab or at least make is less common). Since it would go over all the POFiles, we should ensure that one doesn't try to find all the sharing POFiles when it's calling updateStatistics on one POFile.

Fixing this bug would solve bug 781274 (a critical regression) and bug 622668, thus marking this one as critical and adding notes there.

Related branches

description: updated
Changed in launchpad:
importance: High → Critical
Benji York (benji)
Changed in launchpad:
assignee: nobody → Benji York (benji)
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
Changed in launchpad:
status: Triaged → Fix Committed
Benji York (benji)
tags: added: qa-untestable
removed: qa-needstesting
Revision history for this message
Данило Шеган (danilo) wrote :

FWIW, this might fix bug #373269 as well (comment #1 there is actually this bug report itself :).

William Grant (wgrant)
Changed in launchpad:
status: Fix Committed → In Progress
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
removed: qa-untestable
Changed in launchpad:
status: In Progress → Fix Committed
Revision history for this message
Gary Poster (gary) wrote :

QA report: running run_jobs.py pofile_stats produced OOPS-ccca226ef6eeb1499bdb5ee0c2957372

tags: added: qa-bad
removed: qa-needstesting
tags: added: qa-ok
removed: qa-bad
Revision history for this message
Gary Poster (gary) wrote :

Marked qa-ok only to keep qa pipeline from clogging.

devpad:/srv/oops-amqp/qastaging/2011-11-23/OOPS-519709ea5af08d4e8d3d9b50b522083b

ProgrammingError: permission denied for relation productseries

Revision history for this message
Gary Poster (gary) wrote :

See bug 894177 for permission problem. This bug can be marked fix released.

Raphaël Badin (rvb)
Changed in launchpad:
status: Fix Committed → Fix Released
Revision history for this message
Dirk Stöcker (stoecker) wrote :

Actually the situation is worse now. When I translate texts in the interface ATM, then the Untranslated column still shows the same amount of untranslated strings. Before only the "Needs review" was affected.

The "untranslated" column is the major information point whether work is to be done or not. When the numbers there are wrong, then people will overlook work to do. The simply will assume the numbers are wrong also when new strings have been added.

Benji York (benji)
Changed in launchpad:
status: Fix Released → In Progress
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
removed: qa-ok
Changed in launchpad:
status: In Progress → Fix Committed
Revision history for this message
Gary Poster (gary) wrote :

Dirk is right. The fix is in progress. The new bug, given another number because it is a separate regression and because people are reporting it, is bug 897565

Benji York (benji)
tags: added: qa-ok
removed: qa-needstesting
Benji York (benji)
Changed in launchpad:
status: Fix Committed → Fix Released
Milo Casagrande (milo)
Changed in ubuntu-translations:
status: New → Fix Released
Revision history for this message
zoff99 (zoff) wrote :

its not fixed, now its even worse
suggestions are not shown now in any column!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.