Merge lp:~henninge/launchpad/devel-bug-597539-translationmessage-pofile into lp:launchpad

[Henning Eggers]

Removed remaining uses of TranslationMessage.pofile.

= Bug 597539 =

With the advent of message sharing, the "pofile" attribute of a TranslationMessage became obsolete. TranslationMessage now has a "language" attribute which can be used to find the right POFile if you also know the POTemplate. Since a TranslationMessage can now be shared by multiple POFiles, a direct link makes no sense now.

A lot of code was still using this link, though, since the column in the database still exists and holds values. Since TranslationMessage was still setting pofile upon creation, it most likely pointed to the first POFile that this message was translated in and that mostly worked.

This branch cleans this mess up by hunting down all these uses of the pofile attribute and tries to find an equivalent replacement. Please excuse the diff size, it contains file deletions an repetitive changes in templates and such.

== Proposed fix ==

For each occurrence, one of these fixes was used:

- Use the new "language" attribute if the "pofile" was only used to get to the language. Luckily there were a lot of those cases.
- In view code it was mostly possible to use "browser_pofile" which caches the POFile that the message was currently being viewed in. Some extra code had to be added to make sure it is always set.
- Sometimes the POFile was actually known and there was no reason to use TranslationMessage.pofile.
- Sometimes getOnePOFile will do which just gets any POFile associated with the translation message.
- A new query to find all TranslationMessages for a POFile was introduced.
- Some scripts did not receive nice treatment, one was crippled, the other one plain deleted. ;)

== Implementation details ==

Notes for some specific files.

=== modified file 'lib/lp/translations/browser/tests/translationmessage-views.txt'
Using POFile statistics to show that a form submission succeeded is overkilll, so I removed them.

=== modified file 'lib/lp/translations/browser/translationmessage.py'
I introduced a new parameter 'local_to_pofile' which is passed in from the top to mark local translations explicitly instead of deriving that from the "pofile" attribute. The last chunk is the core of this change.

=== modified file 'lib/lp/translations/doc/gettext-check-messages.txt'
=== modified file 'lib/lp/translations/scripts/gettext_check_messages.py'
This script operates purely on the TranslationMessage table with no relation to a POFile. Finding an imported message is not possible without that information so it had be ridded of that functionality. Obviously this also affected the test. The change has just been commented out because that script will be revisited later anyway in our current feature work and maybe we come up with a better solution.

=== modified file 'lib/lp/translations/tests/test_doc.py'
=== removed file 'lib/lp/translations/doc/remove-upstream-translations-script.txt'
=== removed file 'scripts/rosetta/remove-upstream-translations.py'
This script is not needed anymore so why bother fixing it? Yeah for code deletion!

=== modified file 'lib/lp/translations/interfaces/pofile.py'
=== modified file 'lib/lp/translations/model/pofile.py'
=== modified file 'lib/lp/translations/tests/test_pofile.py'
As POFile.translation_messages cannot be a simple join anymore, I introduced getTranslationMessages to retrieve the messages. This could also be used (with an extra condition) to replace another direct SQL query on the pofile column. Complete with test.

=== modified file 'lib/lp/translations/interfaces/translationmessage.py'
=== modified file 'lib/lp/translations/model/translationmessage.py'
The method ensureBrowserPOFile is simple but had to be in the model class because otherwise assigment to browser_pofile would not be possible.

=== modified file 'lib/lp/translations/model/potmsgset.py'
This is at the heart of this branch because it stops TranslationMessage.pofile from being assigned a value during its creation.

== Test ==

Just run all translation tests, it takes about half an hour.

bin/test -vvcm lp.translations

== QA ==

When this branch is on staging, the pofile column needs to b NULL'ed.

ALTER TABLE translationmessage DISABLE TRIGGER ALL;
UPDATE translationmessage SET pofile=NULL;
ALTER TABLE translationmessage ENABLE TRIGGER ALL;

Then various +translate pages need to be opened and also some imports would be great and similar. Check that
- nothing oopses
- the pofile column remains NULL'ed.

[Jeroen T. Vermeulen (jtv)]

Hi Henning,

Thanks for finally ridding us of this blemish. Excellent cover letter! There are a few things that need fixing.

Points discussed on IRC:

Good job on cutting down redundant testing in translationmessage-views.txt. Don't introduce actions in doctests with "Let's" though.

I really don't like the idea of keeping dead and broken code in gettext_check_messages.py. When we overhaul the script, it will help if all the code that's in there is valid. If we want to look up how it used to work, we have bzr.

The docstring for ensureBrowserPOFile is a bit ambivalent: it "ensures" and "makes sure" that the browser_pofile is set… "if possible." Which is it? The only scenario I can imagine it coming up empty is where all templates that contained a potmsgset have been deleted. (If a template merely stops using a potmsgset, there will still be a TTI linking the two). That could happen with manual database maintenance. Some 36% of our POTMsgSets (1.3 million) and 17% of our TMs (11.5 million) are orphans. You're adding a check to ensure that these don't show up as external suggestions; we should also clean them out of the database.

For convenience, ensureBrowserPOFile can also return self.browser_pofile.

In getTranslationMessages, the Coalesce is a bit hard to indent nicely. How about e.g. creating a variable "applicable_template = Coalesce(TranslationMessage.potemplateID, self.potemplate.id)" and then having the query say "applicable_template == self.potemplate.id"?

In updateTranslation you pass pofile=None to the TranslationMessage constructor. It's a bit moot with the Recife changes coming up, but better remove that argument altogether.

Points not yet discussed on IRC:

In pofiletranslator.txt, I'd say that the oldest TM "belongs to" pofile #1, rather than "references" it. The reference is obsolete!

Nice job fixing up all those empty result sets we generated.

In test_pofile.py, you use factory.makeTranslationMessage a lot. We now also have more specific factory methods that are 100% on the new model and don't use updateTranslation: makeSuggestion, makeCurrentTranslationMessage, makeDivergedTranslationMessage. Please use those where possible.

Actually test_getTranslationMessages_baseline rolls a bunch of tests into one. That tends to weigh down diagnosis and maintenance—it's similar to the statistics check you removed from that doctest above. Could you break it down into separate "I'm creating a {suggestion,shared,diverged,obsolete} message, and it shows up in getTranslationMessages" tests?

In test_getTranslationMessages_condition, you're testing two factors at the same time: the addition of a query condition, and the effect of this particular condition on different types of messages. All you really want to test here is that the query condition works properly, which is 3 tests with one message each: message matching the condition is found, message not matching the condition is not found, and message matching the condition but for another pofile is still not found (i.e. the query condition narrows the search criteria rather than replacing them). If that's a lot of setUp work, consider giving getTranslationMessages a...

review: Approve (code)

[Henning Eggers (henninge)]

> Hi Henning,
>
> Thanks for finally ridding us of this blemish. Excellent cover letter! There
> are a few things that need fixing.

Yes, and it was fun, too. In parts at least. ;-)

>
> Points discussed on IRC:
>
> Good job on cutting down redundant testing in translationmessage-views.txt.
> Don't introduce actions in doctests with "Let's" though.

Oh, it felt like I was in Rome, you know...
Fixed. ;-)

>
> I really don't like the idea of keeping dead and broken code in
> gettext_check_messages.py. When we overhaul the script, it will help if all
> the code that's in there is valid. If we want to look up how it used to work,
> we have bzr.

Old code is gone.

>
> The docstring for ensureBrowserPOFile is a bit ambivalent: it "ensures" and
> "makes sure" that the browser_pofile is set… "if possible."

Yes, I kinda new that was bad. Improved.

> You're adding a check to ensure that
> these don't show up as external suggestions;

Yes, added. It had to go into four places, where ever ensureBrowserPofile is
used. Since one was in a loop, I added an extra function for it.

>
> For convenience, ensureBrowserPOFile can also return self.browser_pofile.

It does now.

>
> In getTranslationMessages, the Coalesce is a bit hard to indent nicely. How
> about e.g. creating a variable "applicable_template =
> Coalesce(TranslationMessage.potemplateID, self.potemplate.id)" and then having
> the query say "applicable_template == self.potemplate.id"?

Yes, that looks better. Thanks.

>
> In updateTranslation you pass pofile=None to the TranslationMessage
> constructor. It's a bit moot with the Recife changes coming up, but better
> remove that argument altogether.

Removed.

>
>
> Points not yet discussed on IRC:
>
> In pofiletranslator.txt, I'd say that the oldest TM "belongs to" pofile #1,
> rather than "references" it. The reference is obsolete!

Good point. ;)

>
> Nice job fixing up all those empty result sets we generated.

Yeah, they made my eyes hurt.

>
> In test_pofile.py, you use factory.makeTranslationMessage a lot. We now also
> have more specific factory methods that are 100% on the new model and don't
> use updateTranslation: makeSuggestion, makeCurrentTranslationMessage,
> makeDivergedTranslationMessage. Please use those where possible.

Jeroen ...
We don't have those ...
This is *not* the recife branch. ;-P

>
> Actually test_getTranslationMessages_baseline rolls a bunch of tests into one.

Thank you very much for those suggestions for splitting up the tests. I applied them all and like it much better. Great!

Thanks for your review!

Henning

[Henning Eggers (henninge)]

Incremental diff.

=== modified file 'lib/lp/translations/browser/tests/translationmessage-views.txt'
--- lib/lp/translations/browser/tests/translationmessage-views.txt	2010-07-23 09:32:52 +0000
+++ lib/lp/translations/browser/tests/translationmessage-views.txt	2010-09-03 16:06:39 +0000
@@ -37,7 +37,7 @@
 
 Here we can see that it's lacking that information.
 
-    >>> print translationmessage_page_view.context.pofile.language.pluralforms
+    >>> print translationmessage_page_view.context.language.pluralforms
     None
 
 And the view class detects it correctly.
@@ -61,7 +61,7 @@
 
 We have the plural form information for this language.
 
-    >>> print translationmessage_page_view.context.pofile.language.pluralforms
+    >>> print translationmessage_page_view.context.language.pluralforms
     2
 
 And thus, the view class should know that it doesn't lacks the plural forms
@@ -228,25 +228,7 @@
 
 == Submitting translations ==
 
-It's time to check the submission of translations and the IPOFile statistics
-update.
-
-But first, let's see current values.
-
-    >>> translationmessage = TranslationMessage.get(1)
-    >>> pofile = translationmessage.pofile
-    >>> pofile.updateStatistics()
-    (7, 0, 1, 2)
-    >>> pofile.currentCount()
-    7
-    >>> pofile.updatesCount()
-    0
-    >>> pofile.rosettaCount()
-    1
-    >>> pofile.unreviewedCount()
-    2
-
-Now we do the submit.
+A new translation is submitted through the view.
 
     >>> server_url = '/'.join(
     ...     [canonical_url(translationmessage), '+translate'])
@@ -287,17 +269,7 @@
     >>> translationmessage_page_view.initialize()
     >>> transaction.commit()
 
-This time we didn't get any problem with the submission, and we can see that
-statistics were updated accordingly.
-
-    >>> pofile.currentCount()
-    6
-    >>> pofile.updatesCount()
-    1
-    >>> pofile.rosettaCount()
-    2
-    >>> pofile.unreviewedCount()
-    2
+This time we didn't get any problem with the submission.
 
 Now, let's see how the system prevents a submission that has a timestamp older
 than when last current translation was submitted.
@@ -715,7 +687,6 @@
     >>> def submit_translation(
     ...     translationmessage, translation, force_suggestion=False):
     ...     global year_tick
-    ...     pofile = translationmessage.pofile
     ...     potmsgset = translationmessage.potmsgset
     ...     server_url = '/'.join(
     ...         [canonical_url(translationmessage), '+translate'])
@@ -723,7 +694,8 @@
     ...             datetime.now(UTC).strftime('-%m-%dT%H:%M:%S+00:00'))
     ...     year_tick += 1
     ...     msgset_id = 'msgset_' + str(potmsgset.id)
-    ...     msgset_id_lang = msgset_id + '_' + pofile.language.code
+    ...     language_code = translationmessage.language.code
+    ...     msgset_id_lang = msgset_id + '_' + language_code
     ...     form = {
     ...         'lock_timestamp': now,
     ...         'alt': None,

=== modified file 'lib/lp/translations/browser/translationmessage.py'
--- lib/lp/translations/browser/translationmessage.py	2010-08-24 10:45:57 +0000
+++ lib/lp/translations/browser/translationmessage.py	2010-09-03 16:06:39 +0000
@@ -926,7 +926,7 @@
         self.force_suggestion = force_suggestion
         self.force_diverge = force_diverge
         self.user_is_official_translator = (
-            current_translation_message.pofile.canEditTranslations(self.user))
+            self.pofile.canEditTranslations(self.user))
         self.form_is_writeable = form_is_writeable
         if self.context.is_imported:
             # The imported translation matches the current one.
@@ -1033,28 +1033,41 @@
                 # Imported one matches the current one.
                 imported_submission = None
             elif self.imported_translationmessage is not None:
-                imported_submission = (
-                    convert_translationmessage_to_submission(
-                        message=self.imported_translationmessage,
-                        current_message=self.context,
-                        plural_form=index,
-                        pofile=self.imported_translationmessage.pofile,
-                        legal_warning_needed=False,
-                        is_empty=False,
-                        packaged=True))
+                pofile = (
+                    self.imported_translationmessage.ensureBrowserPOFile())
+                if pofile is None:
+                    imported_submission = None
+                else:
+                    imported_submission = (
+                        convert_translationmessage_to_submission(
+                            message=self.imported_translationmessage,
+                            current_message=self.context,
+                            plural_form=index,
+                            pofile=pofile,
+                            legal_warning_needed=False,
+                            is_empty=False,
+                            packaged=True,
+                            local_to_pofile=True))
             else:
                 imported_submission = None
 
-            if (self.context.potemplate is not None and
-                self.shared_translationmessage is not None):
-                shared_submission = (
-                    convert_translationmessage_to_submission(
-                        message=self.shared_translationmessage,
-                        current_message=self.context,
-                        plural_form=index,
-                        pofile=self.shared_translationmessage.pofile,
-                        legal_warning_needed=False,
-                        is_empty=False))
+            diverged_and_have_shared = (
+                self.context.potemplate is not None and
+                self.shared_translationmessage is not None) 
+            if diverged_and_have_shared:
+                pofile = self.shared_translationmessage.ensureBrowserPOFile()
+                if pofile is None:
+                    shared_submission = None
+                else:
+                    shared_submission = (
+                        convert_translationmessage_to_submission(
+                            message=self.shared_translationmessage,
+                            current_message=self.context,
+                            plural_form=index,
+                            pofile=pofile,
+                            legal_warning_needed=False,
+                            is_empty=False,
+                            local_to_pofile=True))
             else:
                 shared_submission = None
 
@@ -1137,6 +1150,24 @@
             else:
                 self.can_confirm_and_dismiss = True
 
+    def _setOnePOFile(self, messages):
+        """Return a list of messages that all have a browser_pofile set.
+        
+        If a pofile cannot be found for a message, it is not included in
+        the resulting list.
+        """ 
+        result = []
+        for message in messages:
+            if message.browser_pofile is None:
+                pofile = message.getOnePOFile()
+                if pofile is None:
+                    # Do not include in result.
+                    continue
+                else:
+                    message.setPOFile(pofile)
+            result.append(message)
+        return result
+    
     def _buildAllSuggestions(self):
         """Builds all suggestions and puts them into suggestions_block.
 
@@ -1193,26 +1224,18 @@
 
             # Get a list of translations which are _used_ as translations
             # for this same message in a different translation template.
-            externally_used = sorted(
+            externally_used = self._setOnePOFile(sorted(
                 potmsgset.getExternallyUsedTranslationMessages(language),
                 key=operator.attrgetter("date_created"),
-                reverse=True)
-            for suggestion in externally_used:
-                pofile = suggestion.getOnePOFile()
-                if suggestion.browser_pofile is None:
-                    suggestion.setPOFile(pofile)
+                reverse=True))
 
             # Get a list of translations which are suggested as
             # translations for this same message in a different translation
             # template, but are not used.
-            externally_suggested = sorted(
+            externally_suggested = self._setOnePOFile(sorted(
                 potmsgset.getExternallySuggestedTranslationMessages(language),
                 key=operator.attrgetter("date_created"),
-                reverse=True)
-            for suggestion in externally_suggested:
-                pofile = suggestion.getOnePOFile()
-                if suggestion.browser_pofile is None:
-                    suggestion.setPOFile(pofile)
+                reverse=True))
         else:
             # Don't show suggestions for anonymous users.
             local = externally_used = externally_suggested = []
@@ -1249,7 +1272,7 @@
                 self.seen_translations.add(imported.translations[index])
             local_suggestions = (
                 self._buildTranslationMessageSuggestions(
-                    'Suggestions', local, index))
+                    'Suggestions', local, index, local_to_pofile=True))
             externally_used_suggestions = (
                 self._buildTranslationMessageSuggestions(
                     'Used in', externally_used, index, legal_warning=True))
@@ -1272,7 +1295,8 @@
                 len(alternate_language_suggestions.submissions))
 
     def _buildTranslationMessageSuggestions(self, title, suggestions, index,
-                                            legal_warning=False):
+                                            legal_warning=False,
+                                            local_to_pofile=False):
         """Build filtered list of submissions to be shown in the view.
 
         `title` is the title for the suggestion type, `suggestions` is
@@ -1282,7 +1306,8 @@
             title, self.context,
             suggestions[:self.max_entries],
             self.user_is_official_translator, self.form_is_writeable,
-            index, self.seen_translations, legal_warning=legal_warning)
+            index, self.seen_translations, legal_warning=legal_warning,
+            local_to_pofile=local_to_pofile)
         self.seen_translations = iterable_submissions.seen_translations
         return iterable_submissions
 
@@ -1524,10 +1549,11 @@
 
     def __init__(self, title, translation, submissions,
                  user_is_official_translator, form_is_writeable,
-                 plural_form, seen_translations=None, legal_warning=False):
+                 plural_form, seen_translations=None, legal_warning=False,
+                 local_to_pofile=False):
         self.title = title
         self.potmsgset = translation.potmsgset
-        self.pofile = translation.pofile
+        self.pofile = translation.browser_pofile
         self.user_is_official_translator = user_is_official_translator
         self.form_is_writeable = form_is_writeable
         self.submissions = []
@@ -1557,7 +1583,8 @@
                     plural_form,
                     self.pofile,
                     legal_warning,
-                    is_empty=False))
+                    is_empty=False,
+                    local_to_pofile=local_to_pofile))
         self.seen_translations = seen_translations
 
 
@@ -1566,7 +1593,7 @@
 
 def convert_translationmessage_to_submission(
     message, current_message, plural_form, pofile, legal_warning_needed,
-    is_empty=False, packaged=False):
+    is_empty=False, packaged=False, local_to_pofile=False):
     """Turn a TranslationMessage to an object used for rendering a submission.
 
     :param message: A TranslationMessage.
@@ -1578,10 +1605,10 @@
 
     submission = Submission()
     submission.translationmessage = message
-    for attribute in ['id', 'language', 'potmsgset', 'pofile',
-                      'date_created']:
+    for attribute in ['id', 'language', 'potmsgset', 'date_created']:
         setattr(submission, attribute, getattr(message, attribute))
 
+    submission.pofile = message.browser_pofile
     submission.person = message.submitter
 
     submission.is_empty = is_empty
@@ -1589,7 +1616,7 @@
     submission.suggestion_text = text_to_html(
         message.translations[plural_form],
         message.potmsgset.flags)
-    submission.is_local_to_pofile = (message.pofile == pofile)
+    submission.is_local_to_pofile = local_to_pofile
     submission.legal_warning = legal_warning_needed and (
         message.origin == RosettaTranslationOrigin.SCM)
     submission.suggestion_html_id = (

=== modified file 'lib/lp/translations/doc/gettext-check-messages.txt'
--- lib/lp/translations/doc/gettext-check-messages.txt	2010-02-19 16:54:42 +0000
+++ lib/lp/translations/doc/gettext-check-messages.txt	2010-09-03 16:06:39 +0000
@@ -100,7 +100,6 @@
     INFO Messages checked: 1
     INFO Validation errors: 0
     INFO Messages disabled: 0
-    INFO Messages unmasked: 0
     INFO Commit points: ...
 
 
@@ -118,14 +117,14 @@
     DEBUG Checking messages matching:  id=...
     DEBUG Checking message ...
     INFO ... (current): format specifications ... are not the same
-    INFO ...: unmasked ...
+    DEBUG Commit point.
+    COMMIT
     DEBUG Commit point.
     COMMIT
     INFO Done.
     INFO Messages checked: 1
     INFO Validation errors: 1
     INFO Messages disabled: 1
-    INFO Messages unmasked: 1
     INFO Commit points: ...
 
 The failed message is demoted to a mere suggestion.
@@ -133,45 +132,14 @@
     >>> current_message.is_current
     False
 
-In this case, there was a perfectly good imported message that was being
-masked by the invalid current translation.  So in addition to disabling
-the bad current message, the script activates the good imported one.
-
-    >>> imported_message.is_current
-    True
-
-If the imported message is also bad, this is reported and the imported
-message is not activated.
-
-    >>> imported_message.is_current = False
-    >>> current_message.is_current = True
-    >>> imported_message.translations = [u'%s %s i']
-
-    >>> run_checker(["-w id=%s" % quote(current_message.id)])
-    DEBUG Checking messages matching:  id=...
-    DEBUG Checking message ...
-    INFO ... (current): format specifications ... are not the same
-    DEBUG Commit point.
-    COMMIT
-    DEBUG Commit point.
-    COMMIT
-    INFO Done.
-    INFO Messages checked: 1
-    INFO Validation errors: 2
-    INFO Messages disabled: 1
-    INFO Messages unmasked: 0
-    INFO Commit points: ...
-
-    >>> current_message.is_current
-    False
-    >>> imported_message.is_current
-    False
-
 
 == Output ==
 
 Besides current messages, the script's output also distinguishes
-imported ones, and ones that are completely unused.
+imported ones, and ones that are completely unused. The imported message
+happens to produce validation errors.
+
+    >>> imported_message.translations = [u'%s %s i']
 
 In this example we'd like to see a nicely predictable ordering, so we
 add a sort order using the -o option.
@@ -188,9 +156,8 @@
     COMMIT
     INFO Done.
     INFO Messages checked: 2
-    INFO Validation errors: 3
+    INFO Validation errors: 2
     INFO Messages disabled: 0
-    INFO Messages unmasked: 0
     INFO Commit points: 2
 
 In this case the imported message is checked twice: once all by itself
@@ -223,9 +190,8 @@
     ABORT
     INFO Done.
     INFO Messages checked: 1
-    INFO Validation errors: 2
+    INFO Validation errors: 1
     INFO Messages disabled: 1
-    INFO Messages unmasked: 0
     INFO Commit points: 2
 
 
@@ -251,7 +217,6 @@
     COMMIT
     INFO Done.
     INFO Messages checked: 2
-    INFO Validation errors: 3
+    INFO Validation errors: 2
     INFO Messages disabled: 0
-    INFO Messages unmasked: 0
     INFO Commit points: 3

=== modified file 'lib/lp/translations/doc/pofile.txt'
--- lib/lp/translations/doc/pofile.txt	2010-08-06 07:40:52 +0000
+++ lib/lp/translations/doc/pofile.txt	2010-09-03 16:06:39 +0000
@@ -773,7 +773,7 @@
     Found %i invalid file.
     >>> print pofile_sr.language.code
     sr
-    >>> print translation_message.pofile.language.code
+    >>> print translation_message.language.code
     sr
 
 This entry is in fact one that is not used anymore, that means, its sequence

=== modified file 'lib/lp/translations/doc/potmsgset.txt'
--- lib/lp/translations/doc/potmsgset.txt	2010-08-06 06:51:37 +0000
+++ lib/lp/translations/doc/potmsgset.txt	2010-09-03 16:06:39 +0000
@@ -472,7 +472,7 @@
 
 A TranslationMessage knows what language it is in.
 
-    >>> print pt_BR_dummy_current.pofile.language.code
+    >>> print pt_BR_dummy_current.language.code
     pt_BR
 
 Using another dummy pofile we'll get a POTMsgset that's not a singular
@@ -621,7 +621,8 @@
 
     >>> translationmessage = TranslationMessage.get(2)
     >>> potmsgset = translationmessage.potmsgset
-    >>> pofile = translationmessage.pofile
+    >>> from lp.translations.model.pofile import POFile
+    >>> pofile = POFile.get(1)
     >>> translationmessage.date_reviewed.isoformat()
     '2005-04-07T13:19:17.601068+00:00'
     >>> potmsgset.isTranslationNewerThan(pofile,
@@ -660,8 +661,9 @@
     ...             usage.append('Upstream')
     ...         if not usage:
     ...             usage.append('None')
+    ...         pofile = suggestion.getOnePOFile()
     ...         lines.append('%s: %s (%s)' % (
-    ...             suggestion.pofile.title,
+    ...             pofile.title,
     ...             suggestion.translations[0],
     ...             ' & '.join(usage)))
     ...     for line in sorted(lines):

=== removed file 'lib/lp/translations/doc/remove-upstream-translations-script.txt'
--- lib/lp/translations/doc/remove-upstream-translations-script.txt	2009-07-01 20:45:39 +0000
+++ lib/lp/translations/doc/remove-upstream-translations-script.txt	1970-01-01 00:00:00 +0000
@@ -1,142 +0,0 @@
-This test checks the admin script 'remove-upstream-translations-script.py'.
-That script allows us to remove all translations comming from upstream
-to fix broken data that would pollute our suggestions database.
-
-It was developed to fix bug #32610.
-
-First we do some needed initializations.
-
-    >>> from canonical.launchpad.ftests.harness import LaunchpadTestSetup
-    >>> LaunchpadTestSetup(dbuser='rosettaadmin').setUp()
-    >>> import subprocess, sys
-
-Now, we must be sure that we don't break if the user doesn't give us the
-required arguments.
-
-    >>> process = subprocess.Popen([
-    ...     sys.executable,
-    ...     'scripts/rosetta/remove-upstream-translations.py',
-    ...     '-p', 'evolution', '-vvv'],
-    ...     stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-    ...     stderr=subprocess.STDOUT
-    ...     )
-    >>> (output, empty) = process.communicate()
-    >>> print output
-    WARNING Nothing to do. Exiting...
-    <BLANKLINE>
-
-We can remove all translations from a concrete potemplate.
-
-    >>> process = subprocess.Popen([
-    ...     sys.executable,
-    ...     'scripts/rosetta/remove-upstream-translations.py',
-    ...     '-d', 'ubuntu', '-r', 'hoary', '-n', 'evolution', '-vvv'],
-    ...     stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-    ...     stderr=subprocess.STDOUT
-    ...     )
-    >>> (output, empty) = process.communicate()
-    >>> print output
-    DEBUG   Processing Spanish (es) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 8 submissions
-    DEBUG   Processing Japanese (ja) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 0 submissions
-    DEBUG   Processing Xhosa (xh) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 0 submissions
-    DEBUG   Processing Spanish (es) translation of man in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 0 submissions
-    DEBUG   Processing Spanish (es) translation of disabled-template in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 0 submissions
-    DEBUG   Removed 8 submissions in total.
-    <BLANKLINE>
-
-We need to force a DB reset because the changes are done from an external
-script and the test system is not able to detect the database changes.
-
-    >>> LaunchpadTestSetup().force_dirty_database()
-    >>> LaunchpadTestSetup().tearDown()
-
-Or with a concrete language.
-
-    >>> LaunchpadTestSetup(dbuser='rosettaadmin').setUp()
-
-    >>> process = subprocess.Popen([
-    ...     sys.executable,
-    ...     'scripts/rosetta/remove-upstream-translations.py',
-    ...     '-d', 'ubuntu', '-r', 'hoary', '-n', 'evolution', '-l', 'es',
-    ...     '-vvv'],
-    ...     stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-    ...     stderr=subprocess.STDOUT
-    ...     )
-    >>> (output, empty) = process.communicate()
-    >>> print output
-    DEBUG   Processing Spanish (es) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 8 submissions
-    DEBUG   Processing Spanish (es) translation of man in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 0 submissions
-    DEBUG   Processing Spanish (es) translation of disabled-template in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 0 submissions
-    DEBUG   Removed 8 submissions in total.
-    <BLANKLINE>
-
-We need to force a DB reset because the changes are done from an external
-script and the test system is not able to detect the database changes.
-
-    >>> LaunchpadTestSetup().force_dirty_database()
-    >>> LaunchpadTestSetup().tearDown()
-
-Or even with a concrete template name.
-
-    >>> LaunchpadTestSetup(dbuser='rosettaadmin').setUp()
-
-    >>> process = subprocess.Popen([
-    ...     sys.executable,
-    ...     'scripts/rosetta/remove-upstream-translations.py',
-    ...     '-d', 'ubuntu', '-r', 'hoary', '-n', 'evolution', '-t', 'evolution-2.2',
-    ...     '-vvv'],
-    ...     stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-    ...     stderr=subprocess.STDOUT
-    ...     )
-    >>> (output, empty) = process.communicate()
-    >>> print output
-    DEBUG   Processing Spanish (es) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 8 submissions
-    DEBUG   Processing Japanese (ja) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 0 submissions
-    DEBUG   Processing Xhosa (xh) translation of evolution-2.2 in Ubuntu Hoary package "evolution"...
-    DEBUG   Removed 0 submissions
-    DEBUG   Removed 8 submissions in total.
-    <BLANKLINE>
-
-We need to force a DB reset because the changes are done from an external
-script and the test system is not able to detect the database changes.
-
-    >>> LaunchpadTestSetup().force_dirty_database()
-    >>> LaunchpadTestSetup().tearDown()
-
-And it also works for products!
-
-    >>> LaunchpadTestSetup(dbuser='rosettaadmin').setUp()
-
-    >>> import subprocess, sys
-    >>> process = subprocess.Popen([
-    ...     sys.executable,
-    ...     'scripts/rosetta/remove-upstream-translations.py',
-    ...     '-p', 'evolution', '-s', 'trunk', '-vvv'],
-    ...     stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-    ...     stderr=subprocess.STDOUT
-    ...     )
-    >>> (output, empty) = process.communicate()
-    >>> print output
-    DEBUG   Processing Spanish (es) translation of evolution-2.2 in Evolution trunk...
-    DEBUG   Removed 9 submissions
-    DEBUG   Processing Portuguese (Brazil) (pt_BR) translation of evolution-2.2-test in Evolution trunk...
-    DEBUG   Removed 5 submissions
-    DEBUG   Removed 14 submissions in total.
-    <BLANKLINE>
-
-We need to force a DB reset because the changes are done from an external
-script and the test system is not able to detect the database changes.
-
-    >>> LaunchpadTestSetup().force_dirty_database()
-    >>> LaunchpadTestSetup().tearDown()
-

=== modified file 'lib/lp/translations/interfaces/pofile.py'
--- lib/lp/translations/interfaces/pofile.py	2010-08-20 20:31:18 +0000
+++ lib/lp/translations/interfaces/pofile.py	2010-09-03 16:06:39 +0000
@@ -201,6 +201,13 @@
         `date_created` with newest first.
         """
 
+    def getTranslationMessages(condition=None):
+        """Get TranslationMessages in this `IPOFile`.
+
+        If condition is None, return all messages, else narrow the result
+        set down using the condition.
+        """
+
     def makeTranslatableMessage(potmsgset):
         """Factory method for an `ITranslatableMessage` object.
 

=== modified file 'lib/lp/translations/interfaces/translationmessage.py'
--- lib/lp/translations/interfaces/translationmessage.py	2010-08-20 20:31:18 +0000
+++ lib/lp/translations/interfaces/translationmessage.py	2010-09-03 16:06:39 +0000
@@ -229,6 +229,12 @@
     def getOnePOFile():
         """Get any POFile containing this translation."""
 
+    def ensureBrowserPOFile():
+        """Assign the result of getOnePOFile to browser_pofile.
+
+        If browser_pofile is already set, it is left unchanged.
+        """
+
     def isHidden(pofile):
         """Whether this is an unused, hidden suggestion in `pofile`.
 

=== modified file 'lib/lp/translations/model/pofile.py'
--- lib/lp/translations/model/pofile.py	2010-08-24 10:45:57 +0000
+++ lib/lp/translations/model/pofile.py	2010-09-03 16:06:39 +0000
@@ -21,11 +21,11 @@
     BoolCol,
     ForeignKey,
     IntCol,
-    SQLMultipleJoin,
     StringCol,
     )
 from storm.expr import (
     And,
+    Coalesce,
     Exists,
     In,
     Join,
@@ -36,7 +36,10 @@
     SQL,
     )
 from storm.info import ClassAlias
-from storm.store import Store
+from storm.store import (
+    EmptyResultSet,
+    Store,
+    )
 from zope.component import (
     getAdapter,
     getUtility,
@@ -55,6 +58,7 @@
     )
 from canonical.launchpad import helpers
 from canonical.launchpad.interfaces.launchpad import ILaunchpadCelebrities
+from canonical.launchpad.interfaces.lpstorm import IStore
 from canonical.launchpad.readonly import is_read_only
 from canonical.launchpad.webapp.interfaces import (
     DEFAULT_FLAVOR,
@@ -525,9 +529,26 @@
     from_sourcepackagename = ForeignKey(foreignKey='SourcePackageName',
         dbName='from_sourcepackagename', notNull=False, default=None)
 
-    # joins
-    translation_messages = SQLMultipleJoin(
-        'TranslationMessage', joinColumn='pofile', orderBy='id')
+    @property
+    def translation_messages(self):
+        """See `IPOFile`."""
+        return self.getTranslationMessages()
+
+    def getTranslationMessages(self, condition=None):
+        """See `IPOFile`."""
+        applicable_template = Coalesce(
+            TranslationMessage.potemplateID, self.potemplate.id)
+        clauses = [
+            TranslationTemplateItem.potmsgsetID == TranslationMessage.potmsgsetID,
+            TranslationTemplateItem.potemplate == self.potemplate,
+            TranslationMessage.language == self.language,
+            applicable_template == self.potemplate.id,
+            ]
+        if condition is not None:
+            clauses.append(condition)
+
+        return IStore(self).find(
+            TranslationMessage, *clauses).order_by(TranslationMessage.id)
 
     @property
     def title(self):
@@ -1086,9 +1107,9 @@
             entry_to_import.setErrorOutput(None)
 
         # Prepare the mail notification.
-        msgsets_imported = TranslationMessage.select(
-            'was_obsolete_in_last_import IS FALSE AND pofile=%s' %
-            (sqlvalues(self.id))).count()
+        msgsets_imported = self.getTranslationMessages(
+            TranslationMessage.was_obsolete_in_last_import == False
+            ).count()
 
         replacements = collect_import_info(entry_to_import, self, warnings)
         replacements.update({
@@ -1354,16 +1375,13 @@
         """See `IPOFile`."""
         return self.potemplate.translationpermission
 
-    def emptySelectResults(self):
-        return POFile.select("1=2")
-
     def getTranslationsFilteredBy(self, person):
         """See `IPOFile`."""
         return None
 
     def getPOTMsgSetTranslated(self):
         """See `IPOFile`."""
-        return self.emptySelectResults()
+        return EmptyResultSet()
 
     def getPOTMsgSetUntranslated(self):
         """See `IPOFile`."""
@@ -1371,15 +1389,19 @@
 
     def getPOTMsgSetWithNewSuggestions(self):
         """See `IPOFile`."""
-        return self.emptySelectResults()
+        return EmptyResultSet()
 
     def getPOTMsgSetChangedInLaunchpad(self):
         """See `IPOFile`."""
-        return self.emptySelectResults()
+        return EmptyResultSet()
 
     def getPOTMsgSetWithErrors(self):
         """See `IPOFile`."""
-        return self.emptySelectResults()
+        return EmptyResultSet()
+
+    def getTranslationMessages(self, condition=None):
+        """See `IPOFile`."""
+        return EmptyResultSet()
 
     def hasMessageID(self, msgid):
         """See `IPOFile`."""

=== modified file 'lib/lp/translations/model/potmsgset.py'
--- lib/lp/translations/model/potmsgset.py	2010-08-20 20:31:18 +0000
+++ lib/lp/translations/model/potmsgset.py	2010-09-03 16:06:39 +0000
@@ -846,7 +846,6 @@
                 matching_message = TranslationMessage(
                     potmsgset=self,
                     potemplate=pofile.potemplate,
-                    pofile=pofile,
                     language=pofile.language,
                     origin=origin,
                     submitter=submitter,

=== modified file 'lib/lp/translations/model/translationmessage.py'
--- lib/lp/translations/model/translationmessage.py	2010-08-24 10:45:57 +0000
+++ lib/lp/translations/model/translationmessage.py	2010-09-03 16:06:39 +0000
@@ -123,7 +123,7 @@
                 'This translation message already exists in the database.')
 
         self.id = None
-        self.pofile = pofile
+        self.pofile = None
         self.browser_pofile = pofile
         self.potemplate = pofile.potemplate
         self.language = pofile.language
@@ -160,6 +160,10 @@
         """See `ITranslationMessage`."""
         return None
 
+    def ensureBrowserPOFile(self):
+        """See `ITranslationMessage`."""
+        return self.browser_pofile
+
     @property
     def all_msgstrs(self):
         """See `ITranslationMessage`."""
@@ -380,6 +384,12 @@
         else:
             return None
 
+    def ensureBrowserPOFile(self):
+        """See `ITranslationMessage`."""
+        if self.browser_pofile is None:
+            self.browser_pofile = self.getOnePOFile()
+        return self.browser_pofile
+
     def _getSharedEquivalent(self):
         """Get shared message that otherwise exactly matches this one.
         """

=== modified file 'lib/lp/translations/scripts/gettext_check_messages.py'
--- lib/lp/translations/scripts/gettext_check_messages.py	2010-08-20 20:31:18 +0000
+++ lib/lp/translations/scripts/gettext_check_messages.py	2010-09-03 16:06:39 +0000
@@ -42,7 +42,6 @@
     _check_count = 0
     _error_count = 0
     _disable_count = 0
-    _unmask_count = 0
     _commit_count = 0
 
     _commit_interval = timedelta(0, 3)
@@ -73,10 +72,9 @@
         self.logger.info("Messages checked: %d" % self._check_count)
         self.logger.info("Validation errors: %d" % self._error_count)
         self.logger.info("Messages disabled: %d" % self._disable_count)
-        self.logger.info("Messages unmasked: %d" % self._unmask_count)
         self.logger.info("Commit points: %d" % self._commit_count)
 
-    def _log_bad_message(self, bad_message, unmasked_message, error):
+    def _log_bad_message(self, bad_message, error):
         """Report gettext validation error for active message."""
         currency_markers = []
         if bad_message.is_current:
@@ -87,9 +85,6 @@
             currency_markers.append('unused')
         currency = ', '.join(currency_markers)
         self.logger.info("%d (%s): %s" % (bad_message.id, currency, error))
-        if unmasked_message is not None:
-            self.logger.info(
-                "%s: unmasked %s." % (bad_message.id, unmasked_message.id))
 
     def _check_message_for_error(self, translationmessage):
         """Return error message for `translationmessage`, if any.
@@ -108,39 +103,19 @@
 
         return None
 
-    def _get_imported_alternative(self, translationmessage):
-        """Look for a valid, imported alternative for this message."""
-        if translationmessage.is_imported:
-            return None
-
-        potmsgset = translationmessage.potmsgset
-        pofile = translationmessage.pofile
-        return potmsgset.getImportedTranslationMessage(
-            pofile.potemplate, pofile.language)
-
     def _check_and_fix(self, translationmessage):
         """Check message against gettext, and fix it if necessary."""
         error = self._check_message_for_error(translationmessage)
         if error is None:
             return
 
-        imported = self._get_imported_alternative(translationmessage)
-        if imported is not None:
-            # There is also an imported message that the current message
-            # was previously masking.  If that one passes checks, we can
-            # activate it instead.  Disabling the current message
-            # "unmasks" the imported one.
-            imported_error = self._check_message_for_error(imported)
-            if imported_error is not None:
-                imported = None
+        # Here would be the place to check if another message can be used
+        # instead of the bad one.
 
-        self._log_bad_message(translationmessage, imported, error)
+        self._log_bad_message(translationmessage, error)
         if translationmessage.is_current:
             translationmessage.is_current = False
             self._disable_count += 1
-            if imported is not None:
-                imported.is_current = True
-                self._unmask_count += 1
 
     def _do_commit(self):
         """Commit ongoing transaction, start a new one."""

=== modified file 'lib/lp/translations/templates/currenttranslationmessage-translate-one.pt'
--- lib/lp/translations/templates/currenttranslationmessage-translate-one.pt	2010-04-23 00:38:35 +0000
+++ lib/lp/translations/templates/currenttranslationmessage-translate-one.pt	2010-09-03 16:06:39 +0000
@@ -5,7 +5,7 @@
   omit-tag="">
 
 <tal:language-code
-  define="language_code context/pofile/language/code">
+  define="language_code context/language/code">
 <tr class="translation">
   <td class="icon left right">
     <a tal:attributes="
@@ -176,7 +176,7 @@
     <tr tal:attributes="class string:secondary translation ${view/html_id}">
       <th colspan="3">
         <label class="language-code">Current
-          <span tal:replace="context/pofile/language/englishname">
+          <span tal:replace="context/language/englishname">
             Welsh
           </span><span tal:replace="string:[${plural_index}]"
                        tal:condition="view/is_plural">[0]</span>:
@@ -219,8 +219,8 @@
             <div tal:content="structure translation_dictionary/current_translation"
                  tal:condition="not: view/user_is_official_translator"
                  tal:attributes="
-                   lang context/pofile/language/dashedcode;
-                   dir context/pofile/language/abbreviated_text_dir;
+                   lang context/language/dashedcode;
+                   dir context/language/abbreviated_text_dir;
                    id string:${translation_dictionary/html_id_translation}">
               current translation
             </div>
@@ -229,8 +229,8 @@
               tal:content="structure translation_dictionary/current_translation"
               tal:condition="view/user_is_official_translator"
               tal:attributes="
-                lang context/pofile/language/dashedcode;
-                dir context/pofile/language/abbreviated_text_dir;
+                lang context/language/dashedcode;
+                dir context/language/abbreviated_text_dir;
                 id string:${translation_dictionary/html_id_translation};
                 for string:${translation_dictionary/html_id_translation}_radiobutton">
               current translation
@@ -503,8 +503,8 @@
             tal:attributes="
               name string:${translation_dictionary/html_id_translation}_new;
               id string:${translation_dictionary/html_id_translation}_new;
-              lang context/pofile/language/dashedcode;
-              dir context/pofile/language/abbreviated_text_dir;
+              lang context/language/dashedcode;
+              dir context/language/abbreviated_text_dir;
               value translation_dictionary/submitted_translation;
             "
             class="translate expandable"
@@ -526,8 +526,8 @@
                 tal:attributes="
                   id string:${translation_dictionary/html_id_translation}_new;
                   name string:${translation_dictionary/html_id_translation}_new;
-                  lang context/pofile/language/dashedcode;
-                  dir context/pofile/language/abbreviated_text_dir;
+                  lang context/language/dashedcode;
+                  dir context/language/abbreviated_text_dir;
                 ">
 <tal:content replace="translation_dictionary/submitted_translation" /></textarea>
             </tal:with-content>
@@ -544,8 +544,8 @@
                 tal:attributes="
                   id string:${translation_dictionary/html_id_translation}_new;
                   name string:${translation_dictionary/html_id_translation}_new;
-                  lang context/pofile/language/dashedcode;
-                  dir context/pofile/language/abbreviated_text_dir;
+                  lang context/language/dashedcode;
+                  dir context/language/abbreviated_text_dir;
                 "></textarea>
             </tal:without-content>
           </tal:multi-line>

=== modified file 'lib/lp/translations/templates/translationmessage-translate.pt'
--- lib/lp/translations/templates/translationmessage-translate.pt	2010-05-18 18:04:00 +0000
+++ lib/lp/translations/templates/translationmessage-translate.pt	2010-09-03 16:06:39 +0000
@@ -84,7 +84,7 @@
         <!-- Paging doodads. -->
         <tal:navigation
           replace="structure view/batchnav/@@+navigation-links-lower" />
-      <tal:status replace="structure context/pofile/@@+access" />
+      <tal:status replace="structure context/browser_pofile/@@+access" />
       </tal:havepluralforms>
       <metal:pofile-js-footer
         use-macro="context/@@+translations-macros/pofile-js-footer" />

=== modified file 'lib/lp/translations/tests/pofiletranslator.txt'
--- lib/lp/translations/tests/pofiletranslator.txt	2009-08-13 15:12:16 +0000
+++ lib/lp/translations/tests/pofiletranslator.txt	2010-09-03 16:06:39 +0000
@@ -24,24 +24,13 @@
     >>> language_id = 387
     >>> potmsgset_id = 1
 
-
-Note that our oldest TranslationMessage references pofile #1
-
-    >>> cur.execute("""
-    ...     SELECT pofile
-    ...     FROM TranslationMessage
-    ...     ORDER BY date_created
-    ...     LIMIT 1""")
-    >>> cur.fetchone()[0]
-    1
-
+Note that our oldest TranslationMessage belongs to pofile #1.
 
 Stub has so far not translated anything in this pofile
 
     >>> pofiletranslator(stub_id, pofile_id) is None
     True
 
-
 If we add a message, the cache is updated
 
     >>> cur.execute("""

=== modified file 'lib/lp/translations/tests/test_doc.py'
--- lib/lp/translations/tests/test_doc.py	2010-08-20 20:31:18 +0000
+++ lib/lp/translations/tests/test_doc.py	2010-09-03 16:06:39 +0000
@@ -27,12 +27,8 @@
 
 special = {
     'pofile-views.txt': LayeredDocFileSuite(
-            '../browser/tests/pofile-views.txt',
-            setUp=setUp, tearDown=tearDown, layer=LaunchpadFunctionalLayer
-            ),
-    'remove-upstream-translations-script.txt': LayeredDocFileSuite(
-        '../doc/remove-upstream-translations-script.txt',
-        setUp=setGlobs, stdout_logging=False, layer=None
+        '../browser/tests/pofile-views.txt',
+        setUp=setUp, tearDown=tearDown, layer=LaunchpadFunctionalLayer
         ),
     'poexport-queue.txt': LayeredDocFileSuite(
         '../doc/poexport-queue.txt',

=== modified file 'lib/lp/translations/tests/test_pofile.py'
--- lib/lp/translations/tests/test_pofile.py	2010-08-20 20:31:18 +0000
+++ lib/lp/translations/tests/test_pofile.py	2010-09-03 16:06:39 +0000
@@ -1729,8 +1729,7 @@
 
     def test_makeTranslatableMessage(self):
         # TranslatableMessages can be created from the PO file
-        potmsgset = self.factory.makePOTMsgSet(self.potemplate,
-                                                    sequence=1)
+        potmsgset = self.factory.makePOTMsgSet(self.potemplate, sequence=1)
         message = self.pofile.makeTranslatableMessage(potmsgset)
         verifyObject(ITranslatableMessage, message)
 
@@ -1754,6 +1753,113 @@
                 "(sequence=0) to the end of the file.")
 
 
+class TestPOFileTranslationMessages(TestCaseWithFactory):
+    """Test PO file getTranslationMessages method."""
+
+    layer = ZopelessDatabaseLayer
+
+    def setUp(self):
+        super(TestPOFileTranslationMessages, self).setUp()
+        self.pofile = self.factory.makePOFile('eo')
+        self.potemplate = self.pofile.potemplate
+        self.potmsgset = self.factory.makePOTMsgSet(
+            self.potemplate, sequence=1)
+
+    def test_getTranslationMessages_current_shared(self):
+        # A shared message is included in this POFile's messages.
+        message = self.factory.makeTranslationMessage(
+            potmsgset=self.potmsgset, pofile=self.pofile, force_shared=True)
+        
+        self.assertEqual(
+            [message], list(self.pofile.getTranslationMessages()))
+        
+    def test_getTranslationMessages_current_diverged(self):
+        # A diverged message is included in this POFile's messages.
+        message = self.factory.makeTranslationMessage(
+            potmsgset=self.potmsgset, pofile=self.pofile, force_diverged=True)
+        
+        self.assertEqual(
+            [message], list(self.pofile.getTranslationMessages()))
+        
+    def test_getTranslationMessages_suggestion(self):
+        # A suggestion is included in this POFile's messages.
+        message = self.factory.makeTranslationMessage(
+            potmsgset=self.potmsgset, pofile=self.pofile)
+        
+        self.assertEqual(
+            [message], list(self.pofile.getTranslationMessages()))
+        
+    def test_getTranslationMessages_obsolete(self):
+        # A message on an obsolete POTMsgSEt is included in this
+        # POFile's messages.
+        potmsgset = self.factory.makePOTMsgSet(self.potemplate, sequence=0)
+        message = self.factory.makeTranslationMessage(
+            potmsgset=potmsgset, pofile=self.pofile, force_shared=True)
+        
+        self.assertEqual(
+            [message], list(self.pofile.getTranslationMessages()))
+        
+    def test_getTranslationMessages_other_pofile(self):
+        # A message from another POFiles is not included.
+        other_pofile = self.factory.makePOFile('de')
+        self.factory.makeTranslationMessage(
+            potmsgset=self.potmsgset, pofile=other_pofile)
+        
+        self.assertEqual([], list(self.pofile.getTranslationMessages()))
+        
+    def test_getTranslationMessages_condition_matches(self):
+        # A message matching the given condition is included.
+        # Diverged messages are linked to a specific POTemplate.
+        message = self.factory.makeTranslationMessage(
+            potmsgset=self.potmsgset, pofile=self.pofile, force_diverged=True)
+        
+        self.assertContentEqual(
+            [message],
+            self.pofile.getTranslationMessages(
+                "TranslationMessage.potemplate IS NOT NULL"))
+       
+    def test_getTranslationMessages_condition_matches_not(self):
+        # A message not matching the given condition is excluded.
+        # Shared messages are not linked to a POTemplate.
+        self.factory.makeTranslationMessage(
+            potmsgset=self.potmsgset, pofile=self.pofile, force_shared=True)
+        
+        self.assertContentEqual(
+            [],
+            self.pofile.getTranslationMessages(
+                "TranslationMessage.potemplate IS NOT NULL"))
+       
+    def test_getTranslationMessages_condition_matches_in_other_pofile(self):
+        # A message matching given condition but located in another POFile
+        # is not included.
+        other_pofile = self.factory.makePOFile('de')
+        self.factory.makeTranslationMessage(
+            potmsgset=self.potmsgset, pofile=other_pofile,
+            force_diverged=True)
+        
+        self.assertContentEqual(
+            [],
+            self.pofile.getTranslationMessages(
+                "TranslationMessage.potemplate IS NOT NULL"))
+       
+    def test_getTranslationMessages_diverged_elsewhere(self):
+        # Diverged messages from sharing POTemplates are not included.
+        # Create a sharing potemplate in another product series and share
+        # potmsgset in both templates.
+        other_series = self.factory.makeProductSeries(
+            product=self.potemplate.productseries.product)
+        other_template = self.factory.makePOTemplate(
+            productseries=other_series, name=self.potemplate.name)
+        other_pofile = other_template.getPOFileByLang(
+            self.pofile.language.code)
+        self.potmsgset.setSequence(other_template, 1)
+        self.factory.makeTranslationMessage(
+            potmsgset=self.potmsgset, pofile=other_pofile,
+            force_diverged=True)
+        
+        self.assertEqual([], list(self.pofile.getTranslationMessages()))
+       
+
 class TestPOFileToTranslationFileDataAdapter(TestCaseWithFactory):
     """Test POFile being adapted to IPOFileToTranslationFileData."""
 

=== removed file 'scripts/rosetta/remove-upstream-translations.py'
--- scripts/rosetta/remove-upstream-translations.py	2010-08-06 07:02:32 +0000
+++ scripts/rosetta/remove-upstream-translations.py	1970-01-01 00:00:00 +0000
@@ -1,227 +0,0 @@
-#!/usr/bin/python -S
-#
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-# pylint: disable-msg=W0403
-"""Remove all translations from upstream.
-
-This script is useful to recover from breakages after importing bad
-.po files like the one reported at #32610.
-"""
-
-import _pythonpath
-
-import sys
-import logging
-from optparse import OptionParser
-
-from zope.component import getUtility
-
-from canonical.config import config
-from canonical.database.constants import UTC_NOW
-from canonical.lp import initZopeless
-from canonical.launchpad.scripts import (
-    execute_zcml_for_scripts, logger, logger_options)
-from canonical.launchpad.interfaces import ILaunchpadCelebrities
-from lp.registry.interfaces.product import IProductSet
-from lp.registry.interfaces.distribution import IDistributionSet
-from lp.registry.interfaces.distroseries import IDistroSeriesSet
-from lp.registry.interfaces.sourcepackagename import (
-    ISourcePackageNameSet)
-from lp.translations.interfaces.potemplate import IPOTemplateSet
-from lp.translations.interfaces.translationmessage import (
-    RosettaTranslationOrigin)
-
-
-
-logger_name = 'remove-upstream-translations'
-
-def parse_options(args):
-    """Parse a set of command line options.
-
-    Return an optparse.Values object.
-    """
-    parser = OptionParser()
-
-    parser.add_option("-p", "--product", dest="product",
-        help="The product where we should look for translations.")
-    parser.add_option("-s", "--series", dest="series",
-        help="The product series where we should look for translations.")
-    parser.add_option("-d", "--distro", dest="distro",
-        help="The distribution where we should look for translations.")
-    parser.add_option("-r", "--distroseries", dest="distroseries",
-        help="The distribution series where we should look for translations."
-        )
-    parser.add_option("-n", "--sourcepackagename", dest="sourcepackagename",
-        help="The distribution where we should look for translations.")
-    parser.add_option("-t", "--potemplatename", dest="potemplatename",
-        help="The PO Template name where we should look for translations.")
-    parser.add_option("-l", "--language-code", dest="languagecode",
-        help="The language code where we should look for translations.")
-
-    # Add the verbose/quiet options.
-    logger_options(parser)
-
-    (options, args) = parser.parse_args(args)
-
-    return options
-
-def remove_upstream_entries(ztm, potemplates, lang_code=None):
-    """Remove all translations that came from upstream.
-
-    :arg ztm: Zope transaction manager.
-    :arg potemplates: A set of potemplates that we should process.
-    :arg lang_code: A string with a language code where we should do the
-        removal.
-
-    If lang_code is None, we process all available languages.
-    """
-
-    logger_object = logging.getLogger(logger_name)
-
-    items_deleted = 0
-    # All changes should be logged as done by Rosetta Expert team.
-    rosetta_experts = getUtility(ILaunchpadCelebrities).rosetta_experts
-
-    for potemplate in potemplates:
-        if lang_code is None:
-            pofiles = sorted(
-                list(potemplate.pofiles),
-                key=lambda p: p.language.code)
-        else:
-            pofile = potemplate.getPOFileByLang(lang_code)
-            if pofile is None:
-                pofiles = []
-            else:
-                pofiles = [pofile]
-
-        for pofile in pofiles:
-            logger_object.debug('Processing %s...' % pofile.title)
-            pofile_items_deleted = 0
-            for message in pofile.translation_messages:
-                active_changed = False
-                if message.origin == RosettaTranslationOrigin.SCM:
-                    if message.is_current:
-                        active_changed = True
-                    message.destroySelf()
-                    pofile_items_deleted += 1
-                if active_changed:
-                    message.pofile.date_changed = UTC_NOW
-                    message.pofile.lasttranslator = rosetta_experts
-                    message.reviewer = rosetta_experts
-                    message.date_reviewed = UTC_NOW
-
-            items_deleted += pofile_items_deleted
-            logger_object.debug(
-                 'Removed %d submissions' % pofile_items_deleted)
-            pofile.updateStatistics()
-            ztm.commit()
-
-    # We finished the removal process, is time to notify the amount of entries
-    # that we removed.
-    logger_object.debug(
-        'Removed %d submissions in total.' % items_deleted)
-
-
-def main(argv):
-    options = parse_options(argv[1:])
-    logger_object = logger(options, logger_name)
-
-    execute_zcml_for_scripts()
-    ztm = initZopeless(dbuser=config.rosetta.admin_dbuser)
-
-    product = None
-    series = None
-    distro = None
-    distroseries = None
-    sourcepackagename = None
-    potemplatename = None
-    language_code = None
-    if options.product is not None:
-        productset = getUtility(IProductSet)
-        product = productset.getByName(options.product)
-        if product is None:
-            logger_object.error(
-                'The %s product does not exist.' % options.product)
-            return 1
-
-    if options.series is not None:
-        if product is None:
-            logger_object.error(
-                'You need to specify a product if you want to select a'
-                ' productseries.')
-            return 1
-
-        series = product.getSeries(options.series)
-        if series is None:
-            logger_object.error(
-                'The %s series does not exist inside %s product.' % (
-                    options.series, options.product))
-            return 1
-
-    if options.distro is not None:
-        if product is not None:
-            logger_object.error(
-                'You cannot mix distributions and products.')
-            return 1
-        distroset = getUtility(IDistributionSet)
-        distro = distroset.getByName(options.distro)
-        if distro is None:
-            logger_object.error(
-                'The %s distribution does not exist.' % options.distro)
-            return 1
-
-    if options.distroseries is not None:
-        if distro is None:
-            logger_object.error(
-                'You need to specify a distribution if you want to select a'
-                ' sourcepackagename.')
-        distroseriesset = getUtility(IDistroSeriesSet)
-        distroseries = distroseriesset.queryByName(
-            distro, options.distroseries)
-        if distroseries is None:
-            logger_object.error(
-                'The %s distribution does not exist.' % options.distroseries)
-            return 1
-
-    if options.sourcepackagename is not None:
-        if distroseries is None:
-            logger_object.error(
-                'You need to specify a distribution series if you want to'
-                ' select a sourcepackagename.')
-            return 1
-        sourcepackagenameset = getUtility(ISourcePackageNameSet)
-        sourcepackagename = sourcepackagenameset.queryByName(
-            options.sourcepackagename)
-        if sourcepackagename is None:
-            logger_object.error(
-                'The %s sourcepackagename does not exist.' % (
-                    options.sourcepackagename))
-            return 1
-
-    potemplateset = getUtility(IPOTemplateSet)
-    if series is None and distroseries is None:
-        if options.potemplatename is None:
-            logger_object.warning('Nothing to do. Exiting...')
-            return 0
-        else:
-            potemplates = potemplateset.getAllByName(
-                options.potemplatename)
-    else:
-        potemplate_subset = potemplateset.getSubset(
-            distroseries=distroseries, sourcepackagename=sourcepackagename,
-            productseries=series)
-        if options.potemplatename is not None:
-            potemplate = potemplate_subset.getPOTemplateByName(
-                options.potemplatename)
-            potemplates = [potemplate]
-        else:
-            # Get a list from the subset of potemplates to be able to do
-            # transaction commits.
-            potemplates = list(potemplate_subset)
-
-    remove_upstream_entries(ztm, potemplates, options.languagecode)
-
-if __name__ == '__main__':
-    sys.exit(main(sys.argv))