Merge ~cjwatson/launchpad:remove-xpi-importer into launchpad:master

Proposed by Colin Watson
Status: Merged
Approved by: Colin Watson
Approved revision: 2dfbcd277821d0cda66f498d079b36ae523eb0aa
Merge reported by: Otto Co-Pilot
Merged at revision: not available
Proposed branch: ~cjwatson/launchpad:remove-xpi-importer
Merge into: launchpad:master
Diff against target: 3848 lines (+193/-518)
16 files modified
dev/null (+0/-53)
lib/lp/translations/configure.zcml (+0/-9)
lib/lp/translations/doc/poexport-language-pack.txt (+16/-153)
lib/lp/translations/doc/sourcepackagerelease-translations.txt (+1/-2)
lib/lp/translations/scripts/language_pack.py (+1/-25)
lib/lp/translations/scripts/tests/test_validate_translations_file.py (+0/-38)
lib/lp/translations/scripts/validate_translations_file.py (+0/-24)
lib/lp/translations/tests/test_autoapproval.py (+1/-24)
lib/lp/translations/tests/test_potmsgset.py (+3/-146)
lib/lp/translations/tests/test_translationbranchapprover.py (+0/-11)
lib/lp/translations/tests/test_translationimportqueue.py (+0/-1)
lib/lp/translations/utilities/tests/test_translation_importer.py (+2/-18)
lib/lp/translations/utilities/tests/test_xpi_po_exporter.py (+169/-7)
lib/lp/translations/utilities/translation_import.py (+0/-2)
utilities/sourcedeps.cache (+0/-4)
utilities/sourcedeps.conf (+0/-1)
Reviewer Review Type Date Requested Status
Kristian Glass (community) Approve
Review via email: mp+380059@code.launchpad.net

Commit message

Remove XPI import support

Description of the change

As far as I can tell, this hasn't been used on production since about 2011, possibly due to changes in the Firefox release model. It's sufficiently complicated that it isn't worth keeping if it isn't being used, although we can reintroduce it later if necessary.

The initial motivation for this was that it seems surprisingly difficult to parse DTDs in modern Python without the non-Python-3-friendly copy of parts of the old python-xml package that we've been keeping around in sourcecode/old_xmlplus, but it turned into a larger opportunity to prune unused code.

I had to start by rearranging test_xpi_po_exporter to avoid using the XPI importer, as it makes some sense to preserve vestigial support for exporting XPI templates in PO format in order that there's some way to extract existing information from the production database.

To post a comment you must log in.
Revision history for this message
Kristian Glass (doismellburning) wrote :

Nicely done

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/lib/lp/translations/configure.zcml b/lib/lp/translations/configure.zcml
index 9f91581..3693a4b 100644
--- a/lib/lp/translations/configure.zcml
+++ b/lib/lp/translations/configure.zcml
@@ -291,15 +291,6 @@
291 <allow291 <allow
292 interface="lp.translations.interfaces.translationimporter.ITranslationFormatImporter"/>292 interface="lp.translations.interfaces.translationimporter.ITranslationFormatImporter"/>
293 </class>293 </class>
294 <class
295 class="lp.translations.utilities.mozilla_xpi_importer.MozillaXpiImporter">
296 <allow
297 interface="lp.translations.interfaces.translationimporter.ITranslationFormatImporter"/>
298 </class>
299 <subscriber
300 for="lp.translations.interfaces.translationimportqueue.ITranslationImportQueueEntry"
301 provides="lp.translations.interfaces.translationimporter.ITranslationFormatImporter"
302 factory="lp.translations.utilities.mozilla_xpi_importer.MozillaXpiImporter"/>
303294
304 <!-- PO File -->295 <!-- PO File -->
305296
diff --git a/lib/lp/translations/doc/poexport-language-pack.txt b/lib/lp/translations/doc/poexport-language-pack.txt
index 94b3624..062abc8 100644
--- a/lib/lp/translations/doc/poexport-language-pack.txt
+++ b/lib/lp/translations/doc/poexport-language-pack.txt
@@ -33,9 +33,6 @@ This is handy for examining the tar files that are generated.
33 ... if not member.isreg():33 ... if not member.isreg():
34 ... # Not a regular file. No size to print.34 ... # Not a regular file. No size to print.
35 ... size = '-'35 ... size = '-'
36 ... elif member.name.endswith('.xpi'):
37 ... # XPI file. Binary, so don't try counting lines.
38 ... size = 'bin'
39 ... else:36 ... else:
40 ... size = len(tarfile.extractfile(member).readlines())37 ... size = len(tarfile.extractfile(member).readlines())
41 ... print("| %5s | %s" % (size, member.name))38 ... print("| %5s | %s" % (size, member.name))
@@ -102,15 +99,17 @@ And one of the included .po files look like what we expected.
102 '# traducci\xc3\xb3n de es.po al Spanish\n'99 '# traducci\xc3\xb3n de es.po al Spanish\n'
103100
104101
105Language pack with XPI translations102Base language pack export using Librarian with date limits
106-----------------------------------103----------------------------------------------------------
107104
108Launchpad supports XPI file imports. However, we don't have an export105Launchpad is also able to generate a tarball of all files for a
109process ready, and thus, we do it with an external script that does that106distribution series that only includes translation files which have been
110last part based on .po files and the original en-US.xpi file. To achieve107changed since a certain date.
111that, we export all translations for XPI files in a special directory:
112xpi/translation_domain/
113108
109First we need to set up some data to test with, and for this we need
110some DB classes.
111
112 >>> from StringIO import StringIO
114 >>> from lp.registry.interfaces.distribution import IDistributionSet113 >>> from lp.registry.interfaces.distribution import IDistributionSet
115 >>> from lp.registry.interfaces.person import IPersonSet114 >>> from lp.registry.interfaces.person import IPersonSet
116 >>> from lp.registry.model.sourcepackagename import SourcePackageName115 >>> from lp.registry.model.sourcepackagename import SourcePackageName
@@ -126,122 +125,6 @@ Get the Grumpy distro series.
126125
127 >>> series = getUtility(IDistributionSet)['ubuntu'].getSeries('grumpy')126 >>> series = getUtility(IDistributionSet)['ubuntu'].getSeries('grumpy')
128127
129
130Sample data initialization
131..........................
132
133We need to import an XPI template and a translation to see those files
134exported as part of language packs.
135
136 >>> from lp.translations.enums import RosettaImportStatus
137 >>> from lp.translations.interfaces.translationimportqueue import (
138 ... ITranslationImportQueue)
139 >>> from lp.translations.utilities.tests.test_xpi_import \
140 ... import get_en_US_xpi_file_to_import
141
142We are going to import translations for mozilla-firefox package in
143grumpy distro series.
144
145 >>> series = getUtility(IDistributionSet)['ubuntu'].getSeries('grumpy')
146 >>> spn = SourcePackageName.byName('mozilla-firefox')
147 >>> pot_header = 'Content-Type: text/plain; charset=UTF-8\n'
148 >>> firefox_template = POTemplate(
149 ... name='firefox', translation_domain='firefox',
150 ... distroseries=series, sourcepackagename=spn,
151 ... owner=mark, languagepack=True, path='en-US.xpi',
152 ... header=pot_header)
153
154Attach the en-US.xpi (the template) file.
155
156 >>> en_US_xpi = get_en_US_xpi_file_to_import('en-US')
157 >>> translation_import_queue = getUtility(ITranslationImportQueue)
158 >>> by_maintainer = True
159 >>> template_entry = translation_import_queue.addOrUpdateEntry(
160 ... firefox_template.path, en_US_xpi, by_maintainer,
161 ... mark, distroseries=series, sourcepackagename=spn,
162 ... potemplate=firefox_template)
163
164Attach the es.xpi file (the translation) file.
165
166 >>> es_xpi = get_en_US_xpi_file_to_import('en-US')
167 >>> firefox_es_translation = firefox_template.newPOFile('es')
168 >>> translation_entry = translation_import_queue.addOrUpdateEntry(
169 ... 'es.xpi', es_xpi, by_maintainer,
170 ... mark, distroseries=series, sourcepackagename=spn,
171 ... potemplate=firefox_template,
172 ... pofile=firefox_es_translation)
173
174Before we are ready to import the attached files, we need to approve
175them first.
176
177 >>> template_entry.setStatus(
178 ... RosettaImportStatus.APPROVED, rosetta_experts)
179 >>> translation_entry.setStatus(
180 ... RosettaImportStatus.APPROVED, rosetta_experts)
181
182Given that the files are attached to Librarian, we need to commit the
183transaction to make sure it's stored properly and available.
184
185 >>> transaction.commit()
186
187We do now the import from the queue:
188
189 >>> (subject, body) = firefox_template.importFromQueue(template_entry)
190 >>> (subject, body) = firefox_es_translation.importFromQueue(
191 ... translation_entry)
192 >>> flush_database_caches()
193 >>> transaction.commit()
194
195
196Language pack export with XPI files
197...................................
198
199We are now ready to get an exported language pack with XPI files.
200
201 >>> language_pack = export_language_pack(
202 ... distribution_name='ubuntu',
203 ... series_name='grumpy',
204 ... component=None,
205 ... force_utf8=True,
206 ... output_file=None,
207 ... logger=logger)
208 >>> transaction.commit()
209
210We get other entries in language pack + en-US.xpi file and the
211translations in .po file format.
212
213 >>> tarfile = bytes_to_tarfile(language_pack.file.read())
214 >>> examine_tarfile(tarfile)
215 | - | rosetta-grumpy
216 | 1 | rosetta-grumpy/mapping.txt
217 | 1 | rosetta-grumpy/timestamp.txt
218 | - | rosetta-grumpy/xpi
219 | - | rosetta-grumpy/xpi/firefox
220 | bin | rosetta-grumpy/xpi/firefox/en-US.xpi
221 | 94 | rosetta-grumpy/xpi/firefox/en.po
222 | 102 | rosetta-grumpy/xpi/firefox/es.po
223
224We got a valid en-US.xpi file.
225
226 >>> fh = tarfile.extractfile('rosetta-grumpy/xpi/firefox/en-US.xpi')
227 >>> from zipfile import ZipFile
228 >>> zip = ZipFile(fh, 'r')
229 >>> sorted(zip.namelist())
230 ['chrome.manifest', 'chrome/en-US.jar', 'copyover3.png', 'install.rdf']
231
232
233Base language pack export using Librarian with date limits
234----------------------------------------------------------
235
236Launchpad is also able to generate a tarball of all files for a
237distribution series that only includes translation files which have been
238changed since a certain date.
239
240First we need to set up some data to test with, and for this we need
241some DB classes.
242
243 >>> from StringIO import StringIO
244
245Get a source package name to go with our distro series.128Get a source package name to go with our distro series.
246129
247 >>> spn = SourcePackageName.byName('evolution')130 >>> spn = SourcePackageName.byName('evolution')
@@ -319,12 +202,9 @@ Check that the log looks ok.
319202
320 >>> print(logger.getLogBuffer())203 >>> print(logger.getLogBuffer())
321 DEBUG Selecting PO files for export204 DEBUG Selecting PO files for export
322 INFO Number of PO files to export: 4205 INFO Number of PO files to export: 2
323 DEBUG Exporting PO file ... (1/4)206 DEBUG Exporting PO file ... (1/2)
324 DEBUG Exporting PO file ... (2/4)207 DEBUG Exporting PO file ... (2/2)
325 DEBUG Exporting PO file ... (3/4)
326 DEBUG Exporting PO file ... (4/4)
327 INFO Exporting XPI template files.
328 INFO Adding timestamp file208 INFO Adding timestamp file
329 INFO Adding mapping file209 INFO Adding mapping file
330 INFO Done.210 INFO Done.
@@ -342,13 +222,8 @@ Check that the log looks ok.
342 | - | rosetta-grumpy/es222 | - | rosetta-grumpy/es
343 | - | rosetta-grumpy/es/LC_MESSAGES223 | - | rosetta-grumpy/es/LC_MESSAGES
344 | 21 | rosetta-grumpy/es/LC_MESSAGES/test.po224 | 21 | rosetta-grumpy/es/LC_MESSAGES/test.po
345 | 2 | rosetta-grumpy/mapping.txt225 | 1 | rosetta-grumpy/mapping.txt
346 | 1 | rosetta-grumpy/timestamp.txt226 | 1 | rosetta-grumpy/timestamp.txt
347 | - | rosetta-grumpy/xpi
348 | - | rosetta-grumpy/xpi/firefox
349 | bin | rosetta-grumpy/xpi/firefox/en-US.xpi
350 | 94 | rosetta-grumpy/xpi/firefox/en.po
351 | 102 | rosetta-grumpy/xpi/firefox/es.po
352227
353Check the files look OK.228Check the files look OK.
354229
@@ -431,21 +306,15 @@ should get only files that were updated after 2000-01-02.
431 >>> tarfile = bytes_to_tarfile(language_pack.file.read())306 >>> tarfile = bytes_to_tarfile(language_pack.file.read())
432307
433Now, there is only one file exported for the 'test' domain, the one that308Now, there is only one file exported for the 'test' domain, the one that
434had the modification date after the last generated language pack. We309had the modification date after the last generated language pack.
435ignore the xpi entries because those are outside the scope of this test.
436310
437 >>> examine_tarfile(tarfile)311 >>> examine_tarfile(tarfile)
438 | - | rosetta-grumpy312 | - | rosetta-grumpy
439 | - | rosetta-grumpy/cy313 | - | rosetta-grumpy/cy
440 | - | rosetta-grumpy/cy/LC_MESSAGES314 | - | rosetta-grumpy/cy/LC_MESSAGES
441 | 21 | rosetta-grumpy/cy/LC_MESSAGES/test.po315 | 21 | rosetta-grumpy/cy/LC_MESSAGES/test.po
442 | 2 | rosetta-grumpy/mapping.txt316 | 1 | rosetta-grumpy/mapping.txt
443 | 1 | rosetta-grumpy/timestamp.txt317 | 1 | rosetta-grumpy/timestamp.txt
444 | - | rosetta-grumpy/xpi
445 | - | rosetta-grumpy/xpi/firefox
446 | bin | rosetta-grumpy/xpi/firefox/en-US.xpi
447 | 94 | rosetta-grumpy/xpi/firefox/en.po
448 | 102 | rosetta-grumpy/xpi/firefox/es.po
449318
450There is another situation where a translation file is exported again as319There is another situation where a translation file is exported again as
451part of a language pack update, even without being changed. It is re-320part of a language pack update, even without being changed. It is re-
@@ -488,13 +357,8 @@ template has. That's why we get both translations:
488 | - | rosetta-grumpy/es357 | - | rosetta-grumpy/es
489 | - | rosetta-grumpy/es/LC_MESSAGES358 | - | rosetta-grumpy/es/LC_MESSAGES
490 | 21 | rosetta-grumpy/es/LC_MESSAGES/test.po359 | 21 | rosetta-grumpy/es/LC_MESSAGES/test.po
491 | 2 | rosetta-grumpy/mapping.txt360 | 1 | rosetta-grumpy/mapping.txt
492 | 1 | rosetta-grumpy/timestamp.txt361 | 1 | rosetta-grumpy/timestamp.txt
493 | - | rosetta-grumpy/xpi
494 | - | rosetta-grumpy/xpi/firefox
495 | bin | rosetta-grumpy/xpi/firefox/en-US.xpi
496 | 94 | rosetta-grumpy/xpi/firefox/en.po
497 | 102 | rosetta-grumpy/xpi/firefox/es.po
498362
499363
500Script arguments and concurrency364Script arguments and concurrency
@@ -543,7 +407,6 @@ different distribution and series combinations.
543 /var/lock/launchpad-language-pack-exporter__ubuntu__hoary.lock407 /var/lock/launchpad-language-pack-exporter__ubuntu__hoary.lock
544 INFO Exporting translations for series hoary of distribution ubuntu.408 INFO Exporting translations for series hoary of distribution ubuntu.
545 INFO Number of PO files to export: 12409 INFO Number of PO files to export: 12
546 INFO Exporting XPI template files.
547 INFO Adding timestamp file410 INFO Adding timestamp file
548 INFO Adding mapping file411 INFO Adding mapping file
549 INFO Done.412 INFO Done.
diff --git a/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz b/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz
index 4473202..5a26175 100644
550Binary files a/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz and b/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz differ413Binary files a/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz and b/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz differ
diff --git a/lib/lp/translations/doc/sourcepackagerelease-translations.txt b/lib/lp/translations/doc/sourcepackagerelease-translations.txt
index 039d644..0a03aff 100644
--- a/lib/lp/translations/doc/sourcepackagerelease-translations.txt
+++ b/lib/lp/translations/doc/sourcepackagerelease-translations.txt
@@ -83,11 +83,10 @@ And the queue should have 2 entries, with exactly the same contents.
83 >>> queue_entries = translation_import_queue.getAllEntries(target=sp_test)83 >>> queue_entries = translation_import_queue.getAllEntries(target=sp_test)
8484
85 >>> queue_entries.count()85 >>> queue_entries.count()
86 286 1
8787
88 >>> for entry in queue_entries:88 >>> for entry in queue_entries:
89 ... print(entry.path, entry.importer.name)89 ... print(entry.path, entry.importer.name)
90 something/en-US.xpi maria
91 po/es.po maria90 po/es.po maria
9291
93Commit, so the uploaded translations become available to the scripts.92Commit, so the uploaded translations become available to the scripts.
diff --git a/lib/lp/translations/scripts/language_pack.py b/lib/lp/translations/scripts/language_pack.py
index 34d28c7..77fe80a 100644
--- a/lib/lp/translations/scripts/language_pack.py
+++ b/lib/lp/translations/scripts/language_pack.py
@@ -32,9 +32,6 @@ from lp.services.librarian.interfaces.client import (
32from lp.services.tarfile_helpers import LaunchpadWriteTarFile32from lp.services.tarfile_helpers import LaunchpadWriteTarFile
33from lp.translations.enums import LanguagePackType33from lp.translations.enums import LanguagePackType
34from lp.translations.interfaces.languagepack import ILanguagePackSet34from lp.translations.interfaces.languagepack import ILanguagePackSet
35from lp.translations.interfaces.translationfileformat import (
36 TranslationFileFormat,
37 )
38from lp.translations.interfaces.vpoexport import IVPOExportSet35from lp.translations.interfaces.vpoexport import IVPOExportSet
3936
4037
@@ -93,7 +90,6 @@ def export(distroseries, component, update, force_utf8, logger):
9390
94 # XXX JeroenVermeulen 2008-02-06: Is there anything here that we can unify91 # XXX JeroenVermeulen 2008-02-06: Is there anything here that we can unify
95 # with the export-queue code?92 # with the export-queue code?
96 xpi_templates_to_export = set()
97 path_prefix = 'rosetta-%s' % distroseries.name93 path_prefix = 'rosetta-%s' % distroseries.name
9894
99 pofiles = export_set.get_distroseries_pofiles(95 pofiles = export_set.get_distroseries_pofiles(
@@ -137,14 +133,7 @@ def export(distroseries, component, update, force_utf8, logger):
137133
138 domain = potemplate.translation_domain.encode('ascii')134 domain = potemplate.translation_domain.encode('ascii')
139 code = pofile.getFullLanguageCode().encode('UTF-8')135 code = pofile.getFullLanguageCode().encode('UTF-8')
140136 path = os.path.join(path_prefix, code, 'LC_MESSAGES', '%s.po' % domain)
141 if potemplate.source_file_format == TranslationFileFormat.XPI:
142 xpi_templates_to_export.add(potemplate)
143 path = os.path.join(
144 path_prefix, 'xpi', domain, '%s.po' % code)
145 else:
146 path = os.path.join(
147 path_prefix, code, 'LC_MESSAGES', '%s.po' % domain)
148137
149 try:138 try:
150 # We don't want obsolete entries here, it makes no sense for a139 # We don't want obsolete entries here, it makes no sense for a
@@ -160,19 +149,6 @@ def export(distroseries, component, update, force_utf8, logger):
160149
161 store.invalidate(pofile)150 store.invalidate(pofile)
162151
163 logger.info("Exporting XPI template files.")
164 librarian_client = getUtility(ILibrarianClient)
165 for template in xpi_templates_to_export:
166 if template.source_file is None:
167 logger.warning(
168 "%s doesn't have source file registered." % potemplate.title)
169 continue
170 domain = template.translation_domain.encode('ascii')
171 archive.add_file(
172 os.path.join(path_prefix, 'xpi', domain, 'en-US.xpi'),
173 librarian_client.getFileByAlias(
174 template.source_file.id).read())
175
176 logger.info("Adding timestamp file")152 logger.info("Adding timestamp file")
177 # Is important that the timestamp contain the date when the export153 # Is important that the timestamp contain the date when the export
178 # started, not when it finished because that notes how old is the154 # started, not when it finished because that notes how old is the
diff --git a/lib/lp/translations/scripts/tests/test_validate_translations_file.py b/lib/lp/translations/scripts/tests/test_validate_translations_file.py
index 8311718..4f1fe28 100644
--- a/lib/lp/translations/scripts/tests/test_validate_translations_file.py
+++ b/lib/lp/translations/scripts/tests/test_validate_translations_file.py
@@ -16,9 +16,6 @@ from lp.translations.scripts.validate_translations_file import (
16 UnknownFileType,16 UnknownFileType,
17 ValidateTranslationsFile,17 ValidateTranslationsFile,
18 )18 )
19from lp.translations.utilities.tests.xpi_helpers import (
20 get_en_US_xpi_file_to_import,
21 )
2219
2320
24class TestValidateTranslationsFile(TestCase):21class TestValidateTranslationsFile(TestCase):
@@ -45,30 +42,6 @@ class TestValidateTranslationsFile(TestCase):
45 self.assertRaises(42 self.assertRaises(
46 UnknownFileType, validator._validateContent, 'foo.bar', 'content')43 UnknownFileType, validator._validateContent, 'foo.bar', 'content')
4744
48 def test_validate_dtd_good(self):
49 validator = self._makeValidator()
50 result = validator._validateContent(
51 'test.dtd', '<!ENTITY a.translatable.string "A string">\n')
52 self.assertTrue(result)
53
54 def test_validate_dtd_bad(self):
55 validator = self._makeValidator()
56 result = validator._validateContent(
57 'test.dtd', '<!ENTIT etc.')
58 self.assertFalse(result)
59
60 def test_validate_xpi_manifest_good(self):
61 validator = self._makeValidator()
62 result = validator._validateContent(
63 'chrome.manifest', 'locale foo nl jar:chrome/nl.jar!/foo/')
64 self.assertTrue(result)
65
66 def test_validate_xpi_manifest_bad(self):
67 # XPI manifests must not begin with newline.
68 validator = self._makeValidator()
69 result = validator._validateContent('chrome.manifest', '\nlocale')
70 self.assertFalse(result)
71
72 def test_validate_po_good(self):45 def test_validate_po_good(self):
73 validator = self._makeValidator()46 validator = self._makeValidator()
74 result = validator._validateContent('nl.po', self._strip(r"""47 result = validator._validateContent('nl.po', self._strip(r"""
@@ -110,17 +83,6 @@ class TestValidateTranslationsFile(TestCase):
110 result = validator._validateContent('test.pot', 'garble')83 result = validator._validateContent('test.pot', 'garble')
111 self.assertFalse(result)84 self.assertFalse(result)
11285
113 def test_validate_xpi_good(self):
114 validator = self._makeValidator()
115 xpi_content = get_en_US_xpi_file_to_import('en-US').read()
116 result = validator._validateContent('pl.xpi', xpi_content)
117 self.assertTrue(result)
118
119 def test_validate_xpi_bad(self):
120 validator = self._makeValidator()
121 result = validator._validateContent('de.xpi', 'garble')
122 self.assertFalse(result)
123
124 def test_script(self):86 def test_script(self):
125 test_input = os.path.join(self._findTestData(), 'minimal.pot')87 test_input = os.path.join(self._findTestData(), 'minimal.pot')
126 script = 'scripts/rosetta/validate-translations-file.py'88 script = 'scripts/rosetta/validate-translations-file.py'
diff --git a/lib/lp/translations/scripts/validate_translations_file.py b/lib/lp/translations/scripts/validate_translations_file.py
index bcf2ff6..f9d9303 100644
--- a/lib/lp/translations/scripts/validate_translations_file.py
+++ b/lib/lp/translations/scripts/validate_translations_file.py
@@ -8,18 +8,12 @@ __all__ = [
8 'ValidateTranslationsFile',8 'ValidateTranslationsFile',
9 ]9 ]
1010
11from cStringIO import StringIO
12import logging11import logging
13from optparse import OptionParser12from optparse import OptionParser
14import os.path13import os.path
1514
16from lp.services import scripts15from lp.services import scripts
17from lp.translations.utilities.gettext_po_parser import POParser16from lp.translations.utilities.gettext_po_parser import POParser
18from lp.translations.utilities.mozilla_dtd_parser import DtdFile
19from lp.translations.utilities.mozilla_xpi_importer import (
20 MozillaZipImportParser,
21 )
22from lp.translations.utilities.xpi_manifest import XpiManifest
2317
2418
25class UnknownFileType(Exception):19class UnknownFileType(Exception):
@@ -31,37 +25,19 @@ def validate_unknown_file_type(filename, content):
31 raise UnknownFileType("Unrecognized file type for '%s'." % filename)25 raise UnknownFileType("Unrecognized file type for '%s'." % filename)
3226
3327
34def validate_dtd(filename, content):
35 """Validate XPI DTD file."""
36 DtdFile(filename, filename, content)
37
38
39def validate_po(filename, content):28def validate_po(filename, content):
40 """Validate a gettext PO or POT file."""29 """Validate a gettext PO or POT file."""
41 POParser().parse(content)30 POParser().parse(content)
4231
4332
44def validate_xpi(filename, content):
45 """Validate an XPI archive."""
46 MozillaZipImportParser(filename, StringIO(content))
47
48
49def validate_xpi_manifest(filename, content):
50 """Validate XPI manifest."""
51 XpiManifest(content)
52
53
54class ValidateTranslationsFile:33class ValidateTranslationsFile:
55 """Parse translations files to see if they are well-formed."""34 """Parse translations files to see if they are well-formed."""
5635
57 name = 'validate-translations-file'36 name = 'validate-translations-file'
5837
59 validators = {38 validators = {
60 'dtd': validate_dtd,
61 'manifest': validate_xpi_manifest,
62 'po': validate_po,39 'po': validate_po,
63 'pot': validate_po,40 'pot': validate_po,
64 'xpi': validate_xpi,
65 }41 }
6642
67 def __init__(self, test_args=None):43 def __init__(self, test_args=None):
diff --git a/lib/lp/translations/stories/standalone/xx-translations-xpi-import.txt b/lib/lp/translations/stories/standalone/xx-translations-xpi-import.txt
68deleted file mode 10064444deleted file mode 100644
index 2f0472b..0000000
--- a/lib/lp/translations/stories/standalone/xx-translations-xpi-import.txt
+++ /dev/null
@@ -1,55 +0,0 @@
1= Demonstrate import of Firefox XPI file =
2
3To import translations into Firefox product, we must first import en-US.xpi
4file, which is an equivalent of a PO Template.
5
6Lets start with Firefox product inside trunk revision.
7
8 >>> browser = setupBrowser(auth='Basic carlos@canonical.com:test')
9 >>> browser.open('http://translations.launchpad.test/firefox/trunk')
10
11Since we still don't have any POTemplates assigned, we must use the general
12translations upload link.
13
14 >>> browser.getLink('upload').click()
15 >>> print(browser.url)
16 http://translations.launchpad.test/firefox/trunk/+translations-upload
17
18Get the XPI file we are going to upload.
19
20 >>> from lp.translations.utilities.tests import test_xpi_import
21 >>> xpifile = test_xpi_import.get_en_US_xpi_file_to_import('en-US')
22
23Now, lets upload this file.
24
25 >>> browser.getControl('File:').add_file(
26 ... xpifile, 'application/zip', 'en-US.xpi')
27 >>> browser.getControl('Upload').click()
28
29 >>> print(browser.url)
30 http://translations.launchpad.test/firefox/trunk/+translations-upload
31 >>> for tag in find_tags_by_class(browser.contents, 'message'):
32 ... print(extract_text(tag.renderContents()))
33 Thank you for your upload. It will be automatically reviewed...
34
35Lets check the import queue to edit this entry and set the name.
36
37 >>> browser.getLink('Translation Import Queue').click()
38 >>> print(browser.getLink(url='en-US.xpi').url)
39 http://.../en-US.xpi
40 >>> browser.getLink(url='/+imports/').click()
41 >>> print(browser.url)
42 http://translations.launchpad.test/+imports/...
43 >>> qid = int(browser.url.rsplit('/', 1)[-1])
44
45All new entries need to get a template name to identify them in the context
46where will be imported. In this case, it's 'firefox'.
47
48 >>> browser.getControl('File Type').value = ['POT']
49 >>> browser.getControl('Name').value = 'firefox'
50 >>> browser.getControl('Translation domain').value = 'firefox'
51 >>> browser.getControl('Approve').click()
52 >>> print(browser.url)
53 http://translations.launchpad.test/firefox/trunk/+imports
54 >>> browser.getControl(name='field.status_%d' % qid).value
55 ['APPROVED']
diff --git a/lib/lp/translations/tests/test_autoapproval.py b/lib/lp/translations/tests/test_autoapproval.py
index a0ec5e4..03c60b3 100644
--- a/lib/lp/translations/tests/test_autoapproval.py
+++ b/lib/lp/translations/tests/test_autoapproval.py
@@ -1179,10 +1179,7 @@ class TestAutoBlocking(TestCaseWithFactory):
1179 translation target as `same_target_as`. This lets you create an1179 translation target as `same_target_as`. This lets you create an
1180 entry for the same translation target as another one.1180 entry for the same translation target as another one.
1181 """1181 """
1182 if suffix == '.xpi':1182 basename = self.factory.getUniqueString()
1183 basename = 'en-US'
1184 else:
1185 basename = self.factory.getUniqueString()
11861183
1187 filename = basename + suffix1184 filename = basename + suffix
1188 if directory is None:1185 if directory is None:
@@ -1227,26 +1224,6 @@ class TestAutoBlocking(TestCaseWithFactory):
12271224
1228 self.assertEqual(len(old_blocklist), len(new_blocklist))1225 self.assertEqual(len(old_blocklist), len(new_blocklist))
12291226
1230 def test_getBlockableDirectories_checks_xpi_templates(self):
1231 old_blocklist = self.queue._getBlockableDirectories()
1232
1233 self._makeTemplateEntry(
1234 suffix='.xpi', status=RosettaImportStatus.BLOCKED)
1235
1236 new_blocklist = self.queue._getBlockableDirectories()
1237
1238 self.assertEqual(len(old_blocklist) + 1, len(new_blocklist))
1239
1240 def test_getBlockableDirectories_ignores_xpi_translations(self):
1241 old_blocklist = self.queue._getBlockableDirectories()
1242
1243 self._makeTranslationEntry(
1244 'lt.xpi', status=RosettaImportStatus.BLOCKED)
1245
1246 new_blocklist = self.queue._getBlockableDirectories()
1247
1248 self.assertEqual(len(old_blocklist), len(new_blocklist))
1249
1250 def test_isBlockable_none(self):1227 def test_isBlockable_none(self):
1251 blocklist = self.queue._getBlockableDirectories()1228 blocklist = self.queue._getBlockableDirectories()
1252 entry = self._makeTranslationEntry('nl.po')1229 entry = self._makeTranslationEntry('nl.po')
diff --git a/lib/lp/translations/tests/test_potmsgset.py b/lib/lp/translations/tests/test_potmsgset.py
index c5aaae5..6f0a71b 100644
--- a/lib/lp/translations/tests/test_potmsgset.py
+++ b/lib/lp/translations/tests/test_potmsgset.py
@@ -15,7 +15,6 @@ from zope.security.proxy import removeSecurityProxy
1515
16from lp.app.enums import ServiceUsage16from lp.app.enums import ServiceUsage
17from lp.app.interfaces.launchpad import ILaunchpadCelebrities17from lp.app.interfaces.launchpad import ILaunchpadCelebrities
18from lp.services.propertycache import get_property_cache
19from lp.testing import TestCaseWithFactory18from lp.testing import TestCaseWithFactory
20from lp.testing.layers import (19from lp.testing.layers import (
21 DatabaseFunctionalLayer,20 DatabaseFunctionalLayer,
@@ -82,55 +81,17 @@ class TestTranslationSharedPOTMsgSets(TestCaseWithFactory):
82 self.assertEqual(devel_potmsgsets, [self.potmsgset])81 self.assertEqual(devel_potmsgsets, [self.potmsgset])
83 self.assertEqual(devel_potmsgsets, stable_potmsgsets)82 self.assertEqual(devel_potmsgsets, stable_potmsgsets)
8483
85 def test_POTMsgSetInIncompatiblePOTemplates(self):
86 # Make sure a POTMsgSet cannot be used in two POTemplates with
87 # different incompatible source_file_format (like XPI and PO).
88 self.devel_potemplate.source_file_format = TranslationFileFormat.PO
89 self.stable_potemplate.source_file_format = TranslationFileFormat.XPI
90
91 potmsgset = self.potmsgset
92
93 self.assertRaises(POTMsgSetInIncompatibleTemplatesError,
94 potmsgset.setSequence, self.stable_potemplate, 1)
95
96 # If the two file formats are compatible, it works.
97 self.stable_potemplate.source_file_format = (
98 TranslationFileFormat.KDEPO)
99 potmsgset.setSequence(self.stable_potemplate, 1)
100
101 devel_potmsgsets = list(self.devel_potemplate.getPOTMsgSets())
102 stable_potmsgsets = list(self.stable_potemplate.getPOTMsgSets())
103 self.assertEqual(devel_potmsgsets, stable_potmsgsets)
104
105 # We hack the POTemplate manually to make data inconsistent
106 # in database.
107 self.stable_potemplate.source_file_format = TranslationFileFormat.XPI
108 transaction.commit()
109
110 # We remove the security proxy to be able to get a callable for
111 # properties like `uses_english_msgids` and `singular_text`.
112 naked_potmsgset = removeSecurityProxy(potmsgset)
113
114 self.assertRaises(POTMsgSetInIncompatibleTemplatesError,
115 naked_potmsgset.__getattribute__,
116 "uses_english_msgids")
117
118 self.assertRaises(POTMsgSetInIncompatibleTemplatesError,
119 naked_potmsgset.__getattribute__, "singular_text")
120
121 def test_POTMsgSetUsesEnglishMsgids(self):84 def test_POTMsgSetUsesEnglishMsgids(self):
122 """Test that `uses_english_msgids` property works correctly."""85 """Test that `uses_english_msgids` property works correctly."""
86 # XXX cjwatson 2020-03-01: We currently have no file formats with
87 # importers for which uses_english_msgids would be False (XPI used
88 # to be such a case).
12389
124 # Gettext PO format uses English strings as msgids.90 # Gettext PO format uses English strings as msgids.
125 self.devel_potemplate.source_file_format = TranslationFileFormat.PO91 self.devel_potemplate.source_file_format = TranslationFileFormat.PO
126 transaction.commit()92 transaction.commit()
127 self.assertTrue(self.potmsgset.uses_english_msgids)93 self.assertTrue(self.potmsgset.uses_english_msgids)
12894
129 # Mozilla XPI format doesn't use English strings as msgids.
130 self.devel_potemplate.source_file_format = TranslationFileFormat.XPI
131 transaction.commit()
132 self.assertFalse(self.potmsgset.uses_english_msgids)
133
134 def test_getCurrentTranslationMessageOrDummy_returns_upstream_tm(self):95 def test_getCurrentTranslationMessageOrDummy_returns_upstream_tm(self):
135 pofile = self.factory.makePOFile('nl')96 pofile = self.factory.makePOFile('nl')
136 message = self.factory.makeCurrentTranslationMessage(pofile=pofile)97 message = self.factory.makeCurrentTranslationMessage(pofile=pofile)
@@ -1004,110 +965,6 @@ class TestPOTMsgSetText(TestCaseWithFactory):
1004 english_msgid, TranslationFileFormat.PO)965 english_msgid, TranslationFileFormat.PO)
1005 self.assertEqual(english_msgid, potmsgset.singular_text)966 self.assertEqual(english_msgid, potmsgset.singular_text)
1006967
1007 def test_singular_text_xpi(self):
1008 # Mozilla XPI format uses English strings as msgids if no English
1009 # pofile exists.
1010 symbolic_msgid = self.factory.getUniqueString()
1011 potmsgset = self._makePOTMsgSet(
1012 symbolic_msgid, TranslationFileFormat.XPI)
1013 self.assertEqual(symbolic_msgid, potmsgset.singular_text)
1014
1015 def test_singular_text_xpi_english(self):
1016 # Mozilla XPI format uses English strings as msgids if no English
1017 # pofile exists.
1018 # POTMsgSet singular_text is read from a shared English translation.
1019 symbolic_msgid = self.factory.getUniqueString()
1020 english_msgid = self.factory.getUniqueString()
1021 potmsgset, potemplate = self._makePOTMsgSetAndTemplate(
1022 symbolic_msgid, TranslationFileFormat.XPI)
1023 en_pofile = self.factory.makePOFile('en', potemplate)
1024 self.factory.makeCurrentTranslationMessage(
1025 pofile=en_pofile, potmsgset=potmsgset,
1026 translations=[english_msgid])
1027
1028 self.assertEqual(english_msgid, potmsgset.singular_text)
1029
1030 def test_singular_text_xpi_english_diverged(self):
1031 # A diverged (translation.potemplate != None) English translation
1032 # is not used as a singular_text.
1033 symbolic_msgid = self.factory.getUniqueString()
1034 english_msgid = self.factory.getUniqueString()
1035 diverged_msgid = self.factory.getUniqueString()
1036 potmsgset, potemplate = self._makePOTMsgSetAndTemplate(
1037 symbolic_msgid, TranslationFileFormat.XPI)
1038 en_pofile = self.factory.makePOFile('en', potemplate)
1039 self.factory.makeCurrentTranslationMessage(
1040 pofile=en_pofile, potmsgset=potmsgset,
1041 translations=[english_msgid])
1042 self.factory.makeCurrentTranslationMessage(
1043 pofile=en_pofile, potmsgset=potmsgset,
1044 translations=[diverged_msgid], diverged=True)
1045
1046 self.assertEqual(english_msgid, potmsgset.singular_text)
1047
1048 def _setUpSharingWithUbuntu(self):
1049 """Create a potmsgset shared in upstream and Ubuntu."""
1050 productseries = self.factory.makeProductSeries()
1051
1052 # Create the source package that this product is linked to.
1053 distroseries = self.factory.makeUbuntuDistroSeries()
1054 distroseries.distribution.translation_focus = distroseries
1055 sourcepackagename = self.factory.makeSourcePackageName()
1056 sourcepackage = self.factory.makeSourcePackage(
1057 distroseries=distroseries, sourcepackagename=sourcepackagename)
1058 sourcepackage.setPackaging(productseries, self.factory.makePerson())
1059
1060 # Create two sharing templates.
1061 self.potmsgset, upstream_potemplate = self._makePOTMsgSetAndTemplate(
1062 None, TranslationFileFormat.XPI, productseries)
1063 ubuntu_potemplate = self.factory.makePOTemplate(
1064 distroseries=distroseries, sourcepackagename=sourcepackagename,
1065 name=upstream_potemplate.name)
1066 ubuntu_potemplate.source_file_format = TranslationFileFormat.XPI
1067 self.potmsgset.setSequence(ubuntu_potemplate, 1)
1068
1069 # The pofile is automatically created for all sharing templates.
1070 self.upstream_pofile = self.factory.makePOFile(
1071 'en', upstream_potemplate, create_sharing=True)
1072 self.ubuntu_pofile = ubuntu_potemplate.getPOFileByLang('en')
1073 self.assertIsNot(None, self.ubuntu_pofile)
1074
1075 def test_singular_text_xpi_english_uses_upstream(self):
1076 # POTMsgSet singular_text is read from the upstream English
1077 # translation.
1078 self._setUpSharingWithUbuntu()
1079 # Create different "English translations" for this potmsgset.
1080 ubuntu_msgid = self.factory.getUniqueString()
1081 upstream_msgid = self.factory.getUniqueString()
1082
1083 self.factory.makeCurrentTranslationMessage(
1084 pofile=self.upstream_pofile, potmsgset=self.potmsgset,
1085 translations=[upstream_msgid])
1086 self.factory.makeCurrentTranslationMessage(
1087 pofile=self.ubuntu_pofile, potmsgset=self.potmsgset,
1088 translations=[ubuntu_msgid])
1089
1090 # makeCurrentTranslationMessage calls singular_text and caches the
1091 # upstream msgid, causing the test to pass even without the
1092 # Ubuntu message being present.
1093 del get_property_cache(self.potmsgset).singular_text
1094 self.assertEqual(upstream_msgid, self.potmsgset.singular_text)
1095
1096 def test_singular_text_xpi_english_falls_back_to_ubuntu(self):
1097 # POTMsgSet singular_text is read from the Ubuntu English
1098 # translation if no upstream one exists. This is a safeguard against
1099 # old or broken data.
1100 self._setUpSharingWithUbuntu()
1101
1102 # Create different "English translations" for this potmsgset.
1103 ubuntu_msgid = self.factory.getUniqueString()
1104
1105 self.factory.makeCurrentTranslationMessage(
1106 pofile=self.ubuntu_pofile, potmsgset=self.potmsgset,
1107 translations=[ubuntu_msgid])
1108
1109 self.assertEqual(ubuntu_msgid, self.potmsgset.singular_text)
1110
1111968
1112class TestPOTMsgSetTranslationCredits(TestCaseWithFactory):969class TestPOTMsgSetTranslationCredits(TestCaseWithFactory):
1113 """Test methods related to TranslationCredits."""970 """Test methods related to TranslationCredits."""
diff --git a/lib/lp/translations/tests/test_translationbranchapprover.py b/lib/lp/translations/tests/test_translationbranchapprover.py
index 4f309c0..43e4b36 100644
--- a/lib/lp/translations/tests/test_translationbranchapprover.py
+++ b/lib/lp/translations/tests/test_translationbranchapprover.py
@@ -111,17 +111,6 @@ class TestTranslationBranchApprover(TestCaseWithFactory):
111 self.assertEqual(111 self.assertEqual(
112 translation_domain, entry.potemplate.translation_domain)112 translation_domain, entry.potemplate.translation_domain)
113113
114 def test_new_template_domain_with_xpi(self):
115 # For xpi files, template files are always called "en-US.xpi" so
116 # the approver won't use that string for a domain. It'll fall
117 # back to the next possibility, which is the directory.
118 translation_domain = self.factory.getUniqueString()
119 template_path = translation_domain + '/en-US.xpi'
120 entry = self._upload_file(template_path)
121 self._createApprover(template_path).approve(entry)
122 self.assertEqual(
123 translation_domain, entry.potemplate.translation_domain)
124
125 def test_template_name(self):114 def test_template_name(self):
126 # The name is derived from the file name and must be a valid name.115 # The name is derived from the file name and must be a valid name.
127 translation_domain = (u'Invalid-Name_with illegal#Characters')116 translation_domain = (u'Invalid-Name_with illegal#Characters')
diff --git a/lib/lp/translations/tests/test_translationimportqueue.py b/lib/lp/translations/tests/test_translationimportqueue.py
index e3b3e4c..c4ed52e 100644
--- a/lib/lp/translations/tests/test_translationimportqueue.py
+++ b/lib/lp/translations/tests/test_translationimportqueue.py
@@ -407,7 +407,6 @@ class TestTranslationImportQueue(TestCaseWithFactory):
407 files = dict((407 files = dict((
408 self._makeFile('pot'),408 self._makeFile('pot'),
409 self._makeFile('po'),409 self._makeFile('po'),
410 self._makeFile('xpi'),
411 ))410 ))
412 tarfile_content = LaunchpadWriteTarFile.files_to_stream(files)411 tarfile_content = LaunchpadWriteTarFile.files_to_stream(files)
413 self.import_queue.addOrUpdateEntriesFromTarball(412 self.import_queue.addOrUpdateEntriesFromTarball(
diff --git a/lib/lp/translations/utilities/mozilla_dtd_parser.py b/lib/lp/translations/utilities/mozilla_dtd_parser.py
414deleted file mode 100644413deleted file mode 100644
index bdc4782..0000000
--- a/lib/lp/translations/utilities/mozilla_dtd_parser.py
+++ /dev/null
@@ -1,150 +0,0 @@
1# Copyright 2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Importer for DTD files as found in XPI archives."""
5
6__metaclass__ = type
7__all__ = [
8 'DtdFile'
9 ]
10
11from old_xmlplus.parsers.xmlproc import (
12 dtdparser,
13 utils,
14 xmldtd,
15 )
16
17from lp.translations.interfaces.translationimporter import (
18 TranslationFormatInvalidInputError,
19 TranslationFormatSyntaxError,
20 )
21from lp.translations.interfaces.translations import TranslationConstants
22from lp.translations.utilities.translation_common_format import (
23 TranslationMessageData,
24 )
25
26
27class MozillaDtdConsumer(xmldtd.WFCDTD):
28 """Mozilla DTD translatable message parser.
29
30 msgids are stored as entities. This class extracts it along
31 with translations, comments and source references.
32 """
33 def __init__(self, parser, filename, chrome_path, messages):
34 self.started = False
35 self.last_comment = None
36 self.chrome_path = chrome_path
37 self.messages = messages
38 self.filename = filename
39 xmldtd.WFCDTD.__init__(self, parser)
40
41 def dtd_start(self):
42 """See `xmldtd.WFCDTD`."""
43 self.started = True
44
45 def dtd_end(self):
46 """See `xmldtd.WFCDTD`."""
47 self.started = False
48
49 def handle_comment(self, contents):
50 """See `xmldtd.WFCDTD`."""
51 if not self.started:
52 return
53
54 if self.last_comment is not None:
55 self.last_comment += contents
56 elif len(contents) > 0:
57 self.last_comment = contents
58
59 if self.last_comment and not self.last_comment.endswith('\n'):
60 # Comments must end always with a new line.
61 self.last_comment += '\n'
62
63 def new_general_entity(self, name, value):
64 """See `xmldtd.WFCDTD`."""
65 if not self.started:
66 return
67
68 message = TranslationMessageData()
69 message.msgid_singular = name
70 # CarlosPerelloMarin 20070326: xmldtd parser does an inline
71 # parsing which means that the content is all in a single line so we
72 # don't have a way to show the line number with the source reference.
73 message.file_references_list = ["%s(%s)" % (self.filename, name)]
74 message.addTranslation(TranslationConstants.SINGULAR_FORM, value)
75 message.singular_text = value
76 message.context = self.chrome_path
77 message.source_comment = self.last_comment
78 self.messages.append(message)
79 self.started += 1
80 self.last_comment = None
81
82
83class DtdErrorHandler(utils.ErrorCounter):
84 """Error handler for the DTD parser."""
85 filename = None
86
87 def error(self, msg):
88 raise TranslationFormatSyntaxError(
89 filename=self.filename, message=msg)
90
91 def fatal(self, msg):
92 raise TranslationFormatInvalidInputError(
93 filename=self.filename, message=msg)
94
95
96class DummyDtdFile:
97 """"File" returned when DTD SYSTEM entity tries to include a file."""
98 done = False
99
100 def read(self, *args, **kwargs):
101 """Minimally satisfy attempt to read an included DTD file."""
102 if self.done:
103 return ''
104 else:
105 self.done = True
106 return '<!-- SYSTEM entities not supported. -->'
107
108 def close(self):
109 """Satisfy attempt to close file."""
110 pass
111
112
113class DtdInputSourceFactoryStub:
114 """Replace the class the DTD parser uses to include other DTD files."""
115
116 def create_input_source(self, sysid):
117 """Minimally satisfy attempt to open an included DTD file.
118
119 This is called when the DTD parser hits a SYSTEM entity.
120 """
121 return DummyDtdFile()
122
123
124class DtdFile:
125 """Class for reading translatable messages from a .dtd file.
126
127 It uses DTDParser which fills self.messages with parsed messages.
128 """
129 def __init__(self, filename, chrome_path, content):
130 self.messages = []
131 self.filename = filename
132 self.chrome_path = chrome_path
133
134 # .dtd files are supposed to be using UTF-8 encoding, if the file is
135 # using another encoding, it's against the standard so we reject it
136 try:
137 content = content.decode('utf-8')
138 except UnicodeDecodeError:
139 raise TranslationFormatInvalidInputError(
140 'Content is not valid UTF-8 text')
141
142 error_handler = DtdErrorHandler()
143 error_handler.filename = filename
144
145 parser = dtdparser.DTDParser()
146 parser.set_error_handler(error_handler)
147 parser.set_inputsource_factory(DtdInputSourceFactoryStub())
148 dtd = MozillaDtdConsumer(parser, filename, chrome_path, self.messages)
149 parser.set_dtd_consumer(dtd)
150 parser.parse_string(content)
diff --git a/lib/lp/translations/utilities/mozilla_xpi_importer.py b/lib/lp/translations/utilities/mozilla_xpi_importer.py
151deleted file mode 1006440deleted file mode 100644
index 3c76c79..0000000
--- a/lib/lp/translations/utilities/mozilla_xpi_importer.py
+++ /dev/null
@@ -1,423 +0,0 @@
1# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4__metaclass__ = type
5
6__all__ = [
7 'MozillaXpiImporter',
8 'MozillaZipImportParser',
9 ]
10
11from cStringIO import StringIO
12import textwrap
13
14from zope.component import getUtility
15from zope.interface import implementer
16
17from lp.services.librarian.interfaces.client import ILibrarianClient
18from lp.translations.interfaces.translationfileformat import (
19 TranslationFileFormat,
20 )
21from lp.translations.interfaces.translationimporter import (
22 ITranslationFormatImporter,
23 TranslationFormatInvalidInputError,
24 TranslationFormatSyntaxError,
25 )
26from lp.translations.interfaces.translations import TranslationConstants
27from lp.translations.utilities.mozilla_dtd_parser import DtdFile
28from lp.translations.utilities.mozilla_zip import MozillaZipTraversal
29from lp.translations.utilities.translation_common_format import (
30 TranslationFileData,
31 TranslationMessageData,
32 )
33from lp.translations.utilities.xpi_header import XpiHeader
34
35
36def add_source_comment(message, comment):
37 """Add the given comment inside message.source_comment."""
38 if message.source_comment:
39 message.source_comment += comment
40 else:
41 message.source_comment = comment
42
43 if not message.source_comment.endswith('\n'):
44 message.source_comment += '\n'
45
46
47class MozillaZipImportParser(MozillaZipTraversal):
48 """XPI and jar parser for import purposes.
49
50 Looks for DTD and properties files, and parses them for messages.
51 All messages found are left in `self.messages`.
52 """
53
54 # List of ITranslationMessageData representing messages found.
55 messages = None
56
57 def _begin(self):
58 """Overridable hook for `MozillaZipTraversal`."""
59 self.messages = []
60
61 def _finish(self):
62 """Overridable hook for `MozillaZipTraversal`."""
63 # Eliminate duplicate messages.
64 seen_messages = set()
65 deletions = []
66 for index, message in enumerate(self.messages):
67 identifier = (message.msgid_singular, message.context)
68 if identifier in seen_messages:
69 # This message is a duplicate. Mark it for removal.
70 deletions.append(index)
71 else:
72 seen_messages.add(identifier)
73 for index in reversed(deletions):
74 del self.messages[index]
75
76 for message in self.messages:
77 message.file_references = ', '.join(message.file_references_list)
78
79 def _processTranslatableFile(self, entry, locale_code, xpi_path,
80 chrome_path, filename_suffix):
81 """Overridable hook for `MozillaZipTraversal`.
82
83 This implementation is only interested in DTD and properties
84 files.
85 """
86 if filename_suffix == '.dtd':
87 parser = DtdFile
88 elif filename_suffix == '.properties':
89 parser = PropertyFile
90 else:
91 # We're not interested in other file types here.
92 return
93
94 # Parse file, subsume its messages.
95 content = self.archive.read(entry)
96 parsed_file = parser(
97 filename=xpi_path, chrome_path=chrome_path, content=content)
98 if parsed_file is not None:
99 self.extend(parsed_file.messages)
100
101 def _isTemplate(self):
102 """Is this a template?"""
103 name = self.filename
104 return name is not None and name.startswith('en-US.xpi')
105
106 def _processNestedJar(self, zip_instance):
107 """Overridable hook for `MozillaZipTraversal`.
108
109 This implementation complements `self.messages` with those found in
110 the jar file we just parsed.
111 """
112 self.extend(zip_instance.messages)
113
114 def _isCommandKeyMessage(self, message):
115 """Whether the message represents a command key shortcut."""
116 return (
117 self._isTemplate() and
118 message.translations and (
119 message.msgid_singular.endswith('.commandkey') or
120 message.msgid_singular.endswith('.key')))
121
122 def _isAccessKeyMessage(self, message):
123 """Whether the message represents an access key shortcut."""
124 return (
125 self._isTemplate() and
126 message.translations and (
127 message.msgid_singular.endswith('.accesskey')))
128
129 def extend(self, newdata):
130 """Complement `self.messages` with messages found in contained file.
131
132 :param newdata: a sequence representing the messages found in a
133 contained file.
134 """
135 for message in newdata:
136 # Special case accesskeys and commandkeys:
137 # these are single letter messages, lets display
138 # the value as a source comment.
139 if self._isCommandKeyMessage(message):
140 comment = u'\n'.join(textwrap.wrap(
141 u"""Select the shortcut key that you want to use. It
142 should be translated, but often shortcut keys (for
143 example Ctrl + KEY) are not changed from the original. If
144 a translation already exists, please don't change it if
145 you are not sure about it. Please find the context of
146 the key from the end of the 'Located in' text below."""))
147 add_source_comment(message, comment)
148 elif self._isAccessKeyMessage(message):
149 comment = u'\n'.join(textwrap.wrap(
150 u"""Select the access key that you want to use. These have
151 to be translated in a way that the selected character is
152 present in the translated string of the label being
153 referred to, for example 'i' in 'Edit' menu item in
154 English. If a translation already exists, please don't
155 change it if you are not sure about it. Please find the
156 context of the key from the end of the 'Located in' text
157 below."""))
158 add_source_comment(message, comment)
159 self.messages.append(message)
160
161
162def valid_property_msgid(msgid):
163 """Whether the given msgid follows the restrictions to be valid.
164
165 Checks done are:
166 - It cannot have white spaces.
167 """
168 return u' ' not in msgid
169
170
171class PropertyFile:
172 """Class for reading translatable messages from a .properties file.
173
174 The file format is described at:
175 http://www.mozilla.org/projects/l10n/mlp_chrome.html#text
176 """
177
178 license_block_text = u'END LICENSE BLOCK'
179
180 def __init__(self, filename, chrome_path, content):
181 """Constructs a dictionary from a .properties file.
182
183 :arg filename: The file name where the content came from.
184 :arg content: The file content that we want to parse.
185 """
186 self.filename = filename
187 self.chrome_path = chrome_path
188 self.messages = []
189
190 # Parse the content.
191 self.parse(content)
192
193 def parse(self, content):
194 """Parse given content as a property file.
195
196 Once the parse is done, self.messages has a list of the available
197 `ITranslationMessageData`s.
198 """
199
200 # .properties files are supposed to be unicode-escaped, but we know
201 # that there are some .xpi language packs that instead, use UTF-8.
202 # That's against the specification, but Mozilla applications accept
203 # it anyway, so we try to support it too.
204 # To do this support, we read the text as being in UTF-8
205 # because unicode-escaped looks like ASCII files.
206 try:
207 content = content.decode('utf-8')
208 except UnicodeDecodeError:
209 raise TranslationFormatInvalidInputError(
210 'Content is not valid unicode-escaped text')
211
212 line_num = 0
213 is_multi_line_comment = False
214 last_comment = None
215 last_comment_line_num = 0
216 ignore_comment = False
217 is_message = False
218 translation = u''
219 for line in content.splitlines():
220 # Now, to "normalize" all to the same encoding, we encode to
221 # unicode-escape first, and then decode it to unicode
222 # XXX: Danilo 2006-08-01: we _might_ get performance
223 # improvements if we reimplement this to work directly,
224 # though, it will be hard to beat C-based de/encoder.
225 # This call unescapes everything so we don't need to care about
226 # quotes escaping.
227 try:
228 string = line.encode('raw-unicode_escape')
229 line = string.decode('unicode_escape')
230 except UnicodeDecodeError as exception:
231 raise TranslationFormatInvalidInputError(
232 filename=self.filename, line_number=line_num,
233 message=str(exception))
234
235 line_num += 1
236 if not is_multi_line_comment:
237 # Remove any white space before the useful data, like
238 # ' # foo'.
239 line = line.lstrip()
240 if len(line) == 0:
241 # It's an empty line. Reset any previous comment we have.
242 last_comment = None
243 last_comment_line_num = 0
244 ignore_comment = False
245 elif line.startswith(u'#') or line.startswith(u'//'):
246 # It's a whole line comment.
247 ignore_comment = False
248 line = line[1:].strip()
249 if last_comment:
250 last_comment += line
251 elif len(line) > 0:
252 last_comment = line
253
254 if last_comment and not last_comment.endswith('\n'):
255 # Comments must end always with a new line.
256 last_comment += '\n'
257
258 last_comment_line_num = line_num
259 continue
260
261 # Unescaped URLs are a common mistake: the "//" starts an
262 # end-of-line comment. To work around that, treat "://" as
263 # a special case.
264 just_saw_colon = False
265
266 while line:
267 if is_multi_line_comment:
268 if line.startswith(u'*/'):
269 # The comment ended, we jump the closing tag and
270 # continue with the parsing.
271 line = line[2:]
272 is_multi_line_comment = False
273 last_comment_line_num = line_num
274 if ignore_comment:
275 last_comment = None
276 ignore_comment = False
277
278 # Comments must end always with a new line.
279 last_comment += '\n'
280 elif line.startswith(self.license_block_text):
281 # It's a comment with a licence notice, this
282 # comment can be ignored.
283 ignore_comment = True
284 # Jump the whole tag
285 line = line[len(self.license_block_text):]
286 else:
287 # Store the character.
288 if last_comment is None:
289 last_comment = line[0]
290 elif last_comment_line_num == line_num:
291 last_comment += line[0]
292 else:
293 last_comment = u'%s\n%s' % (last_comment, line[0])
294 last_comment_line_num = line_num
295 # Jump the processed char.
296 line = line[1:]
297 continue
298 elif line.startswith(u'/*'):
299 # It's a multi line comment
300 is_multi_line_comment = True
301 ignore_comment = False
302 last_comment_line_num = line_num
303 # Jump the comment starting tag
304 line = line[2:]
305 continue
306 elif line.startswith(u'//') and not just_saw_colon:
307 # End-of-line comment.
308 last_comment = '%s\n' % line[2:].strip()
309 last_comment_line_num = line_num
310 # On to next line.
311 break
312 elif is_message:
313 # Store the char and continue.
314 head_char = line[0]
315 translation += head_char
316 line = line[1:]
317 just_saw_colon = (head_char == ':')
318 continue
319 elif u'=' in line:
320 # Looks like a message string.
321 (key, value) = line.split('=', 1)
322 # Remove leading and trailing white spaces.
323 key = key.strip()
324
325 if valid_property_msgid(key):
326 is_message = True
327 # Jump the msgid, control chars and leading white
328 # space.
329 line = value.lstrip()
330 continue
331 else:
332 raise TranslationFormatSyntaxError(
333 line_number=line_num,
334 message=u"invalid msgid: '%s'" % key)
335 else:
336 # Got a line that is not a valid message nor a valid
337 # comment. Ignore it because main en-US.xpi catalog from
338 # Firefox has such line/error. We follow the 'be strict
339 # with what you export, be permisive with what you import'
340 # policy.
341 break
342 if is_message:
343 # We just parsed a message, so we need to add it to the list
344 # of messages.
345 if ignore_comment or last_comment_line_num < line_num - 1:
346 # We must ignore the comment or either the comment is not
347 # the last thing before this message or is not in the same
348 # line as this message.
349 last_comment = None
350 ignore_comment = False
351
352 message = TranslationMessageData()
353 message.msgid_singular = key
354 message.context = self.chrome_path
355 message.file_references_list = [
356 "%s:%d(%s)" % (self.filename, line_num, key)]
357 value = translation.strip()
358 message.addTranslation(
359 TranslationConstants.SINGULAR_FORM, value)
360 message.singular_text = value
361 message.source_comment = last_comment
362 self.messages.append(message)
363
364 # Reset status vars.
365 last_comment = None
366 last_comment_line_num = 0
367 is_message = False
368 translation = u''
369
370
371@implementer(ITranslationFormatImporter)
372class MozillaXpiImporter:
373 """Support class to import Mozilla .xpi files."""
374
375 def __init__(self):
376 self.basepath = None
377 self.productseries = None
378 self.distroseries = None
379 self.sourcepackagename = None
380 self.by_maintainer = False
381 self._translation_file = None
382
383 def getFormat(self, file_contents):
384 """See `ITranslationFormatImporter`."""
385 return TranslationFileFormat.XPI
386
387 priority = 0
388
389 # using "application/x-xpinstall" would trigger installation in
390 # firefox.
391 content_type = 'application/zip'
392
393 file_extensions = ['.xpi']
394 template_suffix = 'en-US.xpi'
395
396 uses_source_string_msgids = True
397
398 def parse(self, translation_import_queue_entry):
399 """See `ITranslationFormatImporter`."""
400 self._translation_file = TranslationFileData()
401 self.basepath = translation_import_queue_entry.path
402 self.productseries = translation_import_queue_entry.productseries
403 self.distroseries = translation_import_queue_entry.distroseries
404 self.sourcepackagename = (
405 translation_import_queue_entry.sourcepackagename)
406 self.by_maintainer = translation_import_queue_entry.by_maintainer
407
408 librarian_client = getUtility(ILibrarianClient)
409 content = librarian_client.getFileByAlias(
410 translation_import_queue_entry.content.id).read()
411
412 parser = MozillaZipImportParser(self.basepath, StringIO(content))
413 if parser.header is None:
414 raise TranslationFormatInvalidInputError("No install.rdf found")
415
416 self._translation_file.header = parser.header
417 self._translation_file.messages = parser.messages
418
419 return self._translation_file
420
421 def getHeaderFromString(self, header_string):
422 """See `ITranslationFormatImporter`."""
423 return XpiHeader(header_string)
diff --git a/lib/lp/translations/utilities/mozilla_zip.py b/lib/lp/translations/utilities/mozilla_zip.py
424deleted file mode 1006440deleted file mode 100644
index 489a6a2..0000000
--- a/lib/lp/translations/utilities/mozilla_zip.py
+++ /dev/null
@@ -1,170 +0,0 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4__metaclass__ = type
5
6__all__ = [
7 'MozillaZipTraversal',
8 ]
9
10from cStringIO import StringIO
11from os.path import (
12 basename,
13 splitext,
14 )
15from zipfile import (
16 BadZipfile,
17 ZipFile,
18 )
19
20from lp.translations.interfaces.translationimporter import (
21 TranslationFormatInvalidInputError,
22 )
23from lp.translations.utilities.xpi_header import XpiHeader
24from lp.translations.utilities.xpi_manifest import (
25 make_jarpath,
26 XpiManifest,
27 )
28
29
30class MozillaZipTraversal:
31 """Traversal of an XPI file, or a jar file inside an XPI file.
32
33 To traverse and process an XPI file, derive a class from this one
34 and replace any hooks that you may need.
35
36 If an XPI manifest is provided, traversal will be restricted to
37 directories it lists as containing localizable resources.
38 """
39
40 def __init__(self, filename, archive, xpi_path=None, manifest=None):
41 """Open zip (or XPI, or jar) file and scan its contents.
42
43 :param filename: Name of this zip (XPI/jar) archive.
44 :param archive: File-like object containing this zip archive.
45 :param xpi_path: Full path of this file inside the XPI archive.
46 Leave out for the XPI archive itself.
47 :param manifest: `XpiManifest` representing the XPI archive's
48 manifest file, if any.
49 """
50 self.filename = filename
51 self.header = None
52 self.last_translator = None
53 self.manifest = manifest
54 try:
55 self.archive = ZipFile(archive, 'r')
56 except BadZipfile as exception:
57 raise TranslationFormatInvalidInputError(
58 filename=filename, message=str(exception))
59
60 if xpi_path is None:
61 # This is the main XPI file.
62 xpi_path = ''
63 contained_files = set(self.archive.namelist())
64 if manifest is None:
65 # Look for a manifest.
66 for filename in ['chrome.manifest', 'en-US.manifest']:
67 if filename in contained_files:
68 manifest_content = self.archive.read(filename)
69 self.manifest = XpiManifest(manifest_content)
70 break
71 if 'install.rdf' in contained_files:
72 rdf_content = self.archive.read('install.rdf')
73 self.header = XpiHeader(rdf_content)
74
75 # Strip trailing newline to avoid doubling it.
76 xpi_path = xpi_path.rstrip('/')
77
78 self._begin()
79
80 # Process zipped files. Sort by path to keep ordering deterministic.
81 # Ordering matters in sequence numbering (which in turn shows up in
82 # the UI), but also for consistency in duplicates resolution and for
83 # automated testing.
84 for entry in sorted(self.archive.namelist()):
85 self._processEntry(entry, xpi_path)
86
87 self._finish()
88
89 def _processEntry(self, entry, xpi_path):
90 """Read one zip archive entry, figure out what to do with it."""
91 rootname, suffix = splitext(entry)
92 if basename(rootname) == '':
93 # If filename starts with a dot, that's not really a suffix.
94 suffix = ''
95
96 if suffix == '.jar':
97 jarpath = make_jarpath(xpi_path, entry)
98 if not self.manifest or self.manifest.containsLocales(jarpath):
99 # If this is a jar file that may contain localizable
100 # resources, don't process it in the normal way; recurse
101 # by creating another parser instance.
102 content = self.archive.read(entry)
103 nested_instance = self.__class__(
104 filename=entry, archive=StringIO(content),
105 xpi_path=jarpath, manifest=self.manifest)
106
107 self._processNestedJar(nested_instance)
108 return
109
110 # Construct XPI path; identical to "entry" if previous xpi_path
111 # was empty. XPI paths use slashes as separators, regardless of
112 # what the native filesystem uses.
113 xpi_path = '/'.join([xpi_path, entry]).lstrip('/')
114
115 if self.manifest is None:
116 # No manifest, so we don't have chrome paths. Process
117 # everything just to be sure.
118 chrome_path = None
119 locale_code = None
120 else:
121 chrome_path, locale_code = self.manifest.getChromePathAndLocale(
122 xpi_path)
123 if chrome_path is None:
124 # Not in a directory containing localizable resources.
125 return
126
127 self._processTranslatableFile(
128 entry, locale_code, xpi_path, chrome_path, suffix)
129
130 def _begin(self):
131 """Overridable hook: optional pre-traversal actions."""
132
133 def _processTranslatableFile(self, entry, locale_code, xpi_path,
134 chrome_path, filename_suffix):
135 """Overridable hook: process a file entry.
136
137 Called only for files that may be localizable. If there is a
138 manifest, that means the file must be in a location (or subtree)
139 named by a "locale" entry.
140
141 :param entry: Full name of file inside this zip archive,
142 including path relative to the archive's root.
143 :param locale_code: Code for locale this file belongs to, e.g.
144 "en-US".
145 :param xpi_path: Full path of this file inside the XPI archive,
146 e.g. "jar:locale/en-US.jar!/data/messages.dtd".
147 :param chrome_path: File's chrome path. This is a kind of
148 "normalized" path used in XPI to describe a virtual
149 directory hierarchy. The zip archive's actual layout (which
150 the XPI paths describe) may be different.
151 :param filename_suffix: File name suffix or "extension" of the
152 translatable file. This may be e.g. ".dtd" or ".xhtml," or
153 the empty string if the filename does not contain a dot.
154 """
155 raise NotImplementedError(
156 "XPI traversal class provides no _processTranslatableFile().")
157
158 def _processNestedJar(self, zip_instance):
159 """Overridable hook: handle a nested jar file.
160
161 :param zip_instance: An instance of the same class as self, which
162 has just parsed the nested jar file.
163 """
164 raise NotImplementedError(
165 "XPI traversal class provides no _processNestedJar().")
166
167 def _finish(self):
168 """Overridable hook: post-traversal actions."""
169 raise NotImplementedError(
170 "XPI traversal class provides no _finish().")
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/chrome.manifest b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/chrome.manifest
171deleted file mode 1006440deleted file mode 100644
index 99c5dc3..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/chrome.manifest
+++ /dev/null
@@ -1,5 +0,0 @@
1locale mac en-US jar:chrome/en-US.jar!/mac/
2locale unix en-US jar:chrome/en-US.jar!/unix/
3locale win en-US jar:chrome/en-US.jar!/win/
4override chrome://foo/bar/splat.dtd
5locale main en-US jar:chrome/en-US.jar!/
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.dtd b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.dtd
6deleted file mode 1006440deleted file mode 100644
index ae5af22..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.dtd
+++ /dev/null
@@ -1,2 +0,0 @@
1<!-- This message id also occurs elsewhere in this file -->
2<!ENTITY foozilla.clashing.key "This message is Mac-specific, and comes from DTD.">
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.properties b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.properties
3deleted file mode 1006440deleted file mode 100644
index cb23d48..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.properties
+++ /dev/null
@@ -1 +0,0 @@
1foozilla.clashing.key=This message is Mac-specific, and comes from properties.
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.dtd b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.dtd
2deleted file mode 1006440deleted file mode 100644
index 9e608fa..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.dtd
+++ /dev/null
@@ -1,6 +0,0 @@
1<!-- This message id also occurs elsewhere in this file -->
2<!ENTITY foozilla.regular.message "A non-clashing message.">
3<!ENTITY foozilla.clashing.key "This message is in the main DTD.">
4
5<!-- Clashing msgid within same file. Should be ignored. -->
6<!ENTITY foozilla.regular.message "This message should be ignored.">
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.properties b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.properties
7deleted file mode 1006440deleted file mode 100644
index 0a317b1..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.properties
+++ /dev/null
@@ -1 +0,0 @@
1foozilla.clashing.key=This message is in the main properties file.
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.dtd b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.dtd
2deleted file mode 1006440deleted file mode 100644
index 7388985..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.dtd
+++ /dev/null
@@ -1,2 +0,0 @@
1<!-- This message id also occurs elsewhere in this file -->
2<!ENTITY foozilla.clashing.key "This message is Unix-specific, and comes from DTD.">
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.properties b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.properties
3deleted file mode 1006440deleted file mode 100644
index 6888f4d..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.properties
+++ /dev/null
@@ -1 +0,0 @@
1foozilla.clashing.key=This message is Unix-specific, and comes from properties.
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.dtd b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.dtd
2deleted file mode 1006440deleted file mode 100644
index ee2dbd6..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.dtd
+++ /dev/null
@@ -1,2 +0,0 @@
1<!-- This message id also occurs elsewhere in this file -->
2<!ENTITY foozilla.clashing.key "This message is Windows-specific, and comes from DTD.">
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.properties b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.properties
3deleted file mode 1006440deleted file mode 100644
index aacf3d8..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.properties
+++ /dev/null
@@ -1 +0,0 @@
1foozilla.clashing.key=This message is Windows-specific, and comes from properties.
diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/install.rdf b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/install.rdf
2deleted file mode 1006440deleted file mode 100644
index 872390f..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/install.rdf
+++ /dev/null
@@ -1,19 +0,0 @@
1<?xml version="1.0"?>
2<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
3 xmlns:em="http://www.mozilla.org/2004/em-rdf#">
4 <Description about="urn:mozilla:install-manifest"
5 em:id="langpack-en-US@firefox.mozilla.org"
6 em:name="English U.S. (en-US) Language Pack"
7 em:version="2.0"
8 em:type="8"
9 em:creator="Jeroen Vermeulen">
10
11 <em:targetApplication>
12 <Description>
13 <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id><!-- firefox -->
14 <em:minVersion>2.0</em:minVersion>
15 <em:maxVersion>2.0.0.*</em:maxVersion>
16 </Description>
17 </em:targetApplication>
18 </Description>
19</RDF>
diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/chrome.manifest b/lib/lp/translations/utilities/tests/firefox-data/en-US/chrome.manifest
20deleted file mode 1006440deleted file mode 100644
index 5c362be..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/chrome.manifest
+++ /dev/null
@@ -1 +0,0 @@
1locale main en-US jar:chrome/en-US.jar!/
diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/copyover3.png b/lib/lp/translations/utilities/tests/firefox-data/en-US/copyover3.png
2deleted file mode 1006440deleted file mode 100644
index cdac869..0000000
3Binary files a/lib/lp/translations/utilities/tests/firefox-data/en-US/copyover3.png and /dev/null differ1Binary files a/lib/lp/translations/utilities/tests/firefox-data/en-US/copyover3.png and /dev/null differ
diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/copyover1.foo b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/copyover1.foo
4deleted file mode 1006442deleted file mode 100644
index d39d61f..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/copyover1.foo
+++ /dev/null
@@ -1,3 +0,0 @@
1This file is copied directly over to resulting translated XPI files.
2
3We only need to make sure the contents doesn't change a bit.
diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/copyover2.foo b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/copyover2.foo
4deleted file mode 1006440deleted file mode 100644
index c82082f..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/copyover2.foo
+++ /dev/null
@@ -1,7 +0,0 @@
1This is another file which should be blindly copied without any
2changes in the content, no matter how much one tries.
3
4And for some binary checks, lets add some UTF-8 encoded data. For
5example, name of "Carlos PerellĆ³ MarĆ­n" would be written as "ŠšŠ°Ń€Š»Š¾Ń
6ŠŸŠµŃ€ŠµŃ™Š¾ ŠœŠ°Ń€ŠøŠ½" in Serbian (which is phonetic, and is read exactly the
7same minus accents).
diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.dtd b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.dtd
8deleted file mode 1006440deleted file mode 100644
index 4efb0fe..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.dtd
+++ /dev/null
@@ -1,5 +0,0 @@
1<!-- This is a DTD file inside a subdirectory -->
2
3<!ENTITY foozilla.menu.title "MENU">
4<!ENTITY foozilla.menu.accesskey "M">
5<!ENTITY foozilla.menu.commandkey "m">
diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.properties b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.properties
6deleted file mode 1006440deleted file mode 100644
index e6c200b..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.properties
+++ /dev/null
@@ -1,6 +0,0 @@
1# This is a Properties file inside a subdirectory
2
3# Translators, what you are seeing now is a lovely,
4# awesome, multiline comment aimed at you directly
5# from the streets of a .properties file
6foozilla_something=SomeZilla
diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.dtd b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.dtd
7deleted file mode 1006440deleted file mode 100644
index 0828f76..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.dtd
+++ /dev/null
@@ -1,6 +0,0 @@
1<!ENTITY foozilla.name "FooZilla!">
2<!-- Translators, don't play with fire! -->
3<!ENTITY foozilla.play.fire "Do you want to play with fire?">
4<!-- This is just a comment, not a comment for translators -->
5
6<!ENTITY foozilla.play.ice "Play with ice?">
diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.properties b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.properties
7deleted file mode 1006440deleted file mode 100644
index a4b7b40..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.properties
+++ /dev/null
@@ -1,5 +0,0 @@
1foozilla.title=FooZilla Zilla Thingy
2# Translators, if you're older than six, don't translate this
3foozilla.happytitle=http://foozillingy.happy.net/
4foozilla.nocomment=No Comment // (Except this one)
5foozilla.utf8=Š”Š°Š½=Day
diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/install.rdf b/lib/lp/translations/utilities/tests/firefox-data/en-US/install.rdf
6deleted file mode 1006440deleted file mode 100644
index a34cc54..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/install.rdf
+++ /dev/null
@@ -1,21 +0,0 @@
1<?xml version="1.0"?>
2<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
3 xmlns:em="http://www.mozilla.org/2004/em-rdf#">
4 <Description about="urn:mozilla:install-manifest"
5 em:id="langpack-en-US@firefox.mozilla.org"
6 em:name="English U.S. (en-US) Language Pack"
7 em:version="2.0"
8 em:type="8"
9 em:creator="Danilo Å egan">
10 <em:contributor>Š”Š°Š½ŠøŠ»Š¾ ŠØŠµŠ³Š°Š½</em:contributor>
11 <em:contributor>Carlos PerellĆ³ MarĆ­n &lt;carlos@canonical.com&gt;</em:contributor>
12
13 <em:targetApplication>
14 <Description>
15 <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id><!-- firefox -->
16 <em:minVersion>2.0</em:minVersion>
17 <em:maxVersion>2.0.0.*</em:maxVersion>
18 </Description>
19 </em:targetApplication>
20 </Description>
21</RDF>
diff --git a/lib/lp/translations/utilities/tests/firefox-data/no-manifest/en-US-jar/file.txt b/lib/lp/translations/utilities/tests/firefox-data/no-manifest/en-US-jar/file.txt
22deleted file mode 1006440deleted file mode 100644
index 2d4b8bd..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/no-manifest/en-US-jar/file.txt
+++ /dev/null
@@ -1 +0,0 @@
1This is a translatable text file.
diff --git a/lib/lp/translations/utilities/tests/firefox-data/no-manifest/no-jar.txt b/lib/lp/translations/utilities/tests/firefox-data/no-manifest/no-jar.txt
2deleted file mode 1006440deleted file mode 100644
index 80a1d56..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/no-manifest/no-jar.txt
+++ /dev/null
@@ -1 +0,0 @@
1This is a file in an XPI but not in a jar.
diff --git a/lib/lp/translations/utilities/tests/firefox-data/system-entity/chrome.manifest b/lib/lp/translations/utilities/tests/firefox-data/system-entity/chrome.manifest
2deleted file mode 1006440deleted file mode 100644
index 5c362be..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/system-entity/chrome.manifest
+++ /dev/null
@@ -1 +0,0 @@
1locale main en-US jar:chrome/en-US.jar!/
diff --git a/lib/lp/translations/utilities/tests/firefox-data/system-entity/en-US-jar/test.dtd b/lib/lp/translations/utilities/tests/firefox-data/system-entity/en-US-jar/test.dtd
2deleted file mode 1006440deleted file mode 100644
index 7af5a5a..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/system-entity/en-US-jar/test.dtd
+++ /dev/null
@@ -1,9 +0,0 @@
1<!-- Test SYSTEM handling. -->
2
3<!ENTITY firststring "First translatable string">
4
5<!ENTITY % includedFile SYSTEM "chrome://includedFile.dtd">
6%includedFile;
7
8<!-- Parser will only get here if that last tag didn't break things. -->
9<!ENTITY secondstring "Second translatable string">
diff --git a/lib/lp/translations/utilities/tests/firefox-data/system-entity/install.rdf b/lib/lp/translations/utilities/tests/firefox-data/system-entity/install.rdf
10deleted file mode 1006440deleted file mode 100644
index 1484c26..0000000
--- a/lib/lp/translations/utilities/tests/firefox-data/system-entity/install.rdf
+++ /dev/null
@@ -1,17 +0,0 @@
1<?xml version="1.0"?>
2<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
3 xmlns:em="http://www.mozilla.org/2004/em-rdf#">
4 <Description about="urn:mozilla:install-manifest"
5 em:id="langpack-en-US@firefox.mozilla.org"
6 em:name="English U.S. (en-US) Language Pack"
7 em:version="2.0"
8 em:type="8">
9 <em:targetApplication>
10 <Description>
11 <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id><!-- firefox -->
12 <em:minVersion>2.0</em:minVersion>
13 <em:maxVersion>2.0.0.*</em:maxVersion>
14 </Description>
15 </em:targetApplication>
16 </Description>
17</RDF>
diff --git a/lib/lp/translations/utilities/tests/test_mozilla_xpi_importer.py b/lib/lp/translations/utilities/tests/test_mozilla_xpi_importer.py
18deleted file mode 1006440deleted file mode 100644
index ce147a7..0000000
--- a/lib/lp/translations/utilities/tests/test_mozilla_xpi_importer.py
+++ /dev/null
@@ -1,48 +0,0 @@
1# Copyright 2009-2017 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Mozilla XPI importer tests."""
5
6__metaclass__ = type
7
8from io import BytesIO
9import unittest
10
11from zope.interface.verify import verifyObject
12
13from lp.testing.layers import LaunchpadZopelessLayer
14from lp.translations.interfaces.translationfileformat import (
15 TranslationFileFormat,
16 )
17from lp.translations.interfaces.translationimporter import (
18 ITranslationFormatImporter,
19 )
20from lp.translations.utilities.mozilla_xpi_importer import MozillaXpiImporter
21
22
23class MozillaXpiImporterTestCase(unittest.TestCase):
24 """Class test for mozilla's .xpi file imports"""
25
26 layer = LaunchpadZopelessLayer
27
28 def setUp(self):
29 self.importer = MozillaXpiImporter()
30
31 def testInterface(self):
32 """Check whether the object follows the interface."""
33 self.assertTrue(
34 verifyObject(ITranslationFormatImporter, self.importer))
35
36 def testFormat(self):
37 """Check that MozillaXpiImporter handles the XPI file format."""
38 format = self.importer.getFormat(BytesIO(b''))
39 self.assertTrue(
40 format == TranslationFileFormat.XPI,
41 'MozillaXpiImporter format expected XPI but got %s' % format.name)
42
43 def testHasAlternativeMsgID(self):
44 """Check that MozillaXpiImporter has an alternative msgid."""
45 self.assertTrue(
46 self.importer.uses_source_string_msgids,
47 "MozillaXpiImporter format says it's not using alternative msgid"
48 " when it really does!")
diff --git a/lib/lp/translations/utilities/tests/test_mozilla_zip.py b/lib/lp/translations/utilities/tests/test_mozilla_zip.py
49deleted file mode 1006440deleted file mode 100644
index d7eee80..0000000
--- a/lib/lp/translations/utilities/tests/test_mozilla_zip.py
+++ /dev/null
@@ -1,113 +0,0 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""`MozillaZipTraversal` tests."""
5
6__metaclass__ = type
7
8import unittest
9
10from lp.testing.layers import LaunchpadZopelessLayer
11from lp.translations.interfaces.translationimporter import (
12 TranslationFormatInvalidInputError,
13 )
14from lp.translations.utilities.mozilla_zip import MozillaZipTraversal
15from lp.translations.utilities.tests.xpi_helpers import (
16 get_en_US_xpi_file_to_import,
17 )
18
19
20class TraversalRecorder(MozillaZipTraversal):
21 """XPI "parser": records traversal of an XPI or jar file.
22
23 Does nothing but keep track of the structure of nested zip files it
24 traverses, and the various parameters for each translatable file.
25
26 Produces a nice list of tuples (representing parameters for a
27 translatable file) and lists (representing nested jar files). Each
28 zip file's traversal, including nested ones, is concluded with a
29 string containing a full stop (".").
30 """
31 traversal = None
32
33 def _begin(self):
34 self.traversal = []
35
36 def _processTranslatableFile(self, entry, locale_code, xpi_path,
37 chrome_path, filename_suffix):
38 record = (entry, locale_code, xpi_path, chrome_path, filename_suffix)
39 self.traversal.append(record)
40
41 def _processNestedJar(self, nested_recorder):
42 self.traversal.append(nested_recorder.traversal)
43
44 def _finish(self):
45 self.traversal.append('.')
46
47
48class MozillaZipTraversalTestCase(unittest.TestCase):
49 """Test Mozilla XPI/jar traversal."""
50
51 layer = LaunchpadZopelessLayer
52
53 def test_InvalidXpiFile(self):
54 # If the "XPI" file isn't really a zip file, that's a
55 # TranslationFormatInvalidInputError.
56 self.assertRaises(
57 TranslationFormatInvalidInputError,
58 TraversalRecorder,
59 'foo.xpi', __file__)
60
61 def test_XpiTraversal(self):
62 """Test a typical traversal of XPI file, with nested jar file."""
63 xpi_archive = get_en_US_xpi_file_to_import('en-US')
64 record = TraversalRecorder('', xpi_archive)
65 self.assertEqual(record.traversal, [
66 [
67 ('copyover1.foo', 'en-US',
68 'jar:chrome/en-US.jar!/copyover1.foo',
69 'main/copyover1.foo', '.foo'
70 ),
71 ('subdir/copyover2.foo', 'en-US',
72 'jar:chrome/en-US.jar!/subdir/copyover2.foo',
73 'main/subdir/copyover2.foo', '.foo'
74 ),
75 ('subdir/test2.dtd', 'en-US',
76 'jar:chrome/en-US.jar!/subdir/test2.dtd',
77 'main/subdir/test2.dtd', '.dtd'
78 ),
79 ('subdir/test2.properties', 'en-US',
80 'jar:chrome/en-US.jar!/subdir/test2.properties',
81 'main/subdir/test2.properties', '.properties'
82 ),
83 ('test1.dtd', 'en-US',
84 'jar:chrome/en-US.jar!/test1.dtd',
85 'main/test1.dtd', '.dtd'
86 ),
87 ('test1.properties', 'en-US',
88 'jar:chrome/en-US.jar!/test1.properties',
89 'main/test1.properties', '.properties'
90 ),
91 '.'
92 ],
93 '.'
94 ])
95
96 def test_XpiTraversalWithoutManifest(self):
97 """Test traversal of an XPI file without manifest."""
98 xpi_archive = get_en_US_xpi_file_to_import('no-manifest')
99 record = TraversalRecorder('', xpi_archive)
100 # Without manifest, there is no knowledge of locale or chrome
101 # paths, so those are None.
102 self.assertEqual(record.traversal, [
103 [
104 ('file.txt', None,
105 'jar:chrome/en-US.jar!/file.txt', None, '.txt'
106 ),
107 '.'
108 ],
109 ('no-jar.txt', None,
110 'no-jar.txt', None, '.txt'
111 ),
112 '.'
113 ])
diff --git a/lib/lp/translations/utilities/tests/test_translation_importer.py b/lib/lp/translations/utilities/tests/test_translation_importer.py
index ce43cd8..dcf203b 100644
--- a/lib/lp/translations/utilities/tests/test_translation_importer.py
+++ b/lib/lp/translations/utilities/tests/test_translation_importer.py
@@ -72,9 +72,6 @@ class TranslationImporterTestCase(TestCaseWithFactory):
72 None,72 None,
73 importer.getTranslationFormatImporter(73 importer.getTranslationFormatImporter(
74 TranslationFileFormat.KDEPO))74 TranslationFileFormat.KDEPO))
75 self.assertIsNot(
76 None,
77 importer.getTranslationFormatImporter(TranslationFileFormat.XPI))
7875
79 def testGetTranslationFileFormatByFileExtension(self):76 def testGetTranslationFileFormatByFileExtension(self):
80 """Checked whether file format precedence works correctly."""77 """Checked whether file format precedence works correctly."""
@@ -94,10 +91,6 @@ class TranslationImporterTestCase(TestCaseWithFactory):
94 importer.getTranslationFileFormat(91 importer.getTranslationFileFormat(
95 ".po", BytesIO(b'msgid "_: kde context\nmessage"\nmsgstr ""')))92 ".po", BytesIO(b'msgid "_: kde context\nmessage"\nmsgstr ""')))
9693
97 self.assertEqual(
98 TranslationFileFormat.XPI,
99 importer.getTranslationFileFormat(".xpi", BytesIO(b"")))
100
101 def testNoConflictingPriorities(self):94 def testNoConflictingPriorities(self):
102 """Check that no two importers for the same file extension have95 """Check that no two importers for the same file extension have
103 exactly the same priority."""96 exactly the same priority."""
@@ -111,13 +104,12 @@ class TranslationImporterTestCase(TestCaseWithFactory):
111 def testFileExtensionsWithImporters(self):104 def testFileExtensionsWithImporters(self):
112 """Check whether we get the right list of file extensions handled."""105 """Check whether we get the right list of file extensions handled."""
113 self.assertEqual(106 self.assertEqual(
114 ['.po', '.pot', '.xpi'],107 ['.po', '.pot'],
115 TranslationImporter().supported_file_extensions)108 TranslationImporter().supported_file_extensions)
116109
117 def testTemplateSuffixes(self):110 def testTemplateSuffixes(self):
118 """Check for changes in filename suffixes that identify templates."""111 """Check for changes in filename suffixes that identify templates."""
119 self.assertEqual(112 self.assertEqual(['.pot'], TranslationImporter().template_suffixes)
120 ['.pot', 'en-US.xpi'], TranslationImporter().template_suffixes)
121113
122 def _assertIsNotTemplate(self, path):114 def _assertIsNotTemplate(self, path):
123 self.assertFalse(115 self.assertFalse(
@@ -137,12 +129,8 @@ class TranslationImporterTestCase(TestCaseWithFactory):
137 self._assertIsTemplate("bar.pot")129 self._assertIsTemplate("bar.pot")
138 self._assertIsTemplate("foo/bar.pot")130 self._assertIsTemplate("foo/bar.pot")
139 self._assertIsTemplate("foo.bar.pot")131 self._assertIsTemplate("foo.bar.pot")
140 self._assertIsTemplate("en-US.xpi")
141 self._assertIsTemplate("translations/en-US.xpi")
142132
143 self._assertIsNotTemplate("pt_BR.po")133 self._assertIsNotTemplate("pt_BR.po")
144 self._assertIsNotTemplate("pt_BR.xpi")
145 self._assertIsNotTemplate("pt-BR.xpi")
146134
147 def testHiddenFilesRecognition(self):135 def testHiddenFilesRecognition(self):
148 # Hidden files and directories (leading dot) are recognized.136 # Hidden files and directories (leading dot) are recognized.
@@ -189,13 +177,9 @@ class TranslationImporterTestCase(TestCaseWithFactory):
189 self._assertIsTranslation("po/el.po")177 self._assertIsTranslation("po/el.po")
190 self._assertIsTranslation("po/package-el.po")178 self._assertIsTranslation("po/package-el.po")
191 self._assertIsTranslation("po/package-zh_TW.po")179 self._assertIsTranslation("po/package-zh_TW.po")
192 self._assertIsTranslation("en-GB.xpi")
193 self._assertIsTranslation("translations/en-GB.xpi")
194180
195 self._assertIsNotTranslation("hi.pot")181 self._assertIsNotTranslation("hi.pot")
196 self._assertIsNotTranslation("po/hi.pot")182 self._assertIsNotTranslation("po/hi.pot")
197 self._assertIsNotTranslation("en-US.xpi")
198 self._assertIsNotTranslation("translations/en-US.xpi")
199183
200 def testIsIdenticalTranslation(self):184 def testIsIdenticalTranslation(self):
201 """Test `is_identical_translation`."""185 """Test `is_identical_translation`."""
diff --git a/lib/lp/translations/utilities/tests/test_xpi_dtd_format.py b/lib/lp/translations/utilities/tests/test_xpi_dtd_format.py
202deleted file mode 100644186deleted file mode 100644
index ddf0c70..0000000
--- a/lib/lp/translations/utilities/tests/test_xpi_dtd_format.py
+++ /dev/null
@@ -1,45 +0,0 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4__metaclass__ = type
5
6import unittest
7
8from lp.translations.interfaces.translationimporter import (
9 TranslationFormatInvalidInputError,
10 )
11from lp.translations.utilities.mozilla_dtd_parser import DtdFile
12
13
14class DtdFormatTestCase(unittest.TestCase):
15 """Test class for dtd file format."""
16
17 def test_DtdSyntaxError(self):
18 # Syntax errors in a DTD file are reported as translation format
19 # errors.
20 content = '<!ENTITY foo "gah"></ENTITY>'
21 self.assertRaises(
22 TranslationFormatInvalidInputError, DtdFile, 'test.dtd', None,
23 content)
24
25 def test_UTF8DtdFileTest(self):
26 """This test makes sure that we handle UTF-8 encoding files."""
27
28 content = (
29 '<!ENTITY utf8.message "\xc2\xbfQuieres? \xc2\xa1S\xc3\xad!">')
30
31 dtd_file = DtdFile('test.dtd', None, content)
32
33 # There is a single message.
34 self.assertEqual(len(dtd_file.messages), 1)
35 message = dtd_file.messages[0]
36
37 self.assertEqual([u'\xbfQuieres? \xa1S\xed!'], message.translations)
38
39 def test_Latin1DtdFileTest(self):
40 """This test makes sure that we detect bad encodings."""
41
42 content = '<!ENTITY latin1.message "\xbfQuieres? \xa1S\xed!">\n'
43
44 self.assertRaises(TranslationFormatInvalidInputError, DtdFile, None,
45 'test.dtd', content)
diff --git a/lib/lp/translations/utilities/tests/test_xpi_import.py b/lib/lp/translations/utilities/tests/test_xpi_import.py
46deleted file mode 1006440deleted file mode 100644
index 1bbe439..0000000
--- a/lib/lp/translations/utilities/tests/test_xpi_import.py
+++ /dev/null
@@ -1,365 +0,0 @@
1# Copyright 2009-2017 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Functional tests for XPI file format"""
5__metaclass__ = type
6
7import re
8import unittest
9
10from zope.component import getUtility
11
12from lp.app.interfaces.launchpad import ILaunchpadCelebrities
13from lp.registry.interfaces.person import IPersonSet
14from lp.registry.interfaces.product import IProductSet
15from lp.testing.layers import LaunchpadZopelessLayer
16from lp.translations.enums import RosettaImportStatus
17from lp.translations.interfaces.potemplate import IPOTemplateSet
18from lp.translations.utilities.mozilla_xpi_importer import MozillaXpiImporter
19from lp.translations.utilities.tests.helpers import (
20 import_pofile_or_potemplate,
21 )
22from lp.translations.utilities.tests.xpi_helpers import (
23 access_key_source_comment,
24 command_key_source_comment,
25 get_en_US_xpi_file_to_import,
26 )
27
28
29def unwrap(text):
30 """Remove line breaks and any other wrapping artifacts from text."""
31 return re.sub('\s+', ' ', text.strip())
32
33
34class XpiTestCase(unittest.TestCase):
35 """XPI file import into Launchpad."""
36
37 layer = LaunchpadZopelessLayer
38
39 def setUp(self):
40 # Get the importer.
41 self.importer = getUtility(IPersonSet).getByName('mark')
42
43 # Get the Firefox template.
44 firefox_product = getUtility(IProductSet).getByName('firefox')
45 firefox_productseries = firefox_product.getSeries('trunk')
46 firefox_potemplate_subset = getUtility(IPOTemplateSet).getSubset(
47 productseries=firefox_productseries)
48 self.firefox_template = firefox_potemplate_subset.new(
49 name='firefox',
50 translation_domain='firefox',
51 path='en-US.xpi',
52 owner=self.importer)
53 self.spanish_firefox = self.firefox_template.newPOFile('es')
54 self.spanish_firefox.path = 'translations/es.xpi'
55
56 def setUpTranslationImportQueueForTemplate(self, subdir):
57 """Return an ITranslationImportQueueEntry for testing purposes.
58
59 :param subdir: subdirectory in firefox-data to get XPI data from.
60 """
61 # Get the file to import.
62 en_US_xpi = get_en_US_xpi_file_to_import(subdir)
63 return import_pofile_or_potemplate(
64 file_contents=en_US_xpi,
65 person=self.importer,
66 potemplate=self.firefox_template)
67
68 def setUpTranslationImportQueueForTranslation(self, subdir):
69 """Return an ITranslationImportQueueEntry for testing purposes.
70
71 :param subdir: subdirectory in firefox-data to get XPI data from.
72 """
73 # Get the file to import. Given the way XPI file format works, we can
74 # just use the same template file like a translation one.
75 es_xpi = get_en_US_xpi_file_to_import(subdir)
76 return import_pofile_or_potemplate(
77 file_contents=es_xpi,
78 person=self.importer,
79 pofile=self.spanish_firefox,
80 by_maintainer=True)
81
82 def _assertXpiMessageInvariant(self, message):
83 """Check whether invariant part of all messages are correct."""
84 # msgid and singular_text are always different except for the keyboard
85 # shortcuts which are the 'accesskey' and 'commandkey' ones.
86 self.assertFalse(
87 (message.msgid_singular.msgid == message.singular_text and
88 message.msgid_singular.msgid not in (
89 u'foozilla.menu.accesskey', u'foozilla.menu.commandkey')),
90 'msgid and singular_text should be different but both are %s' % (
91 message.msgid_singular.msgid))
92
93 # Plural forms should be None as this format is not able to handle
94 # them.
95 self.assertIsNone(message.msgid_plural)
96 self.assertIsNone(message.plural_text)
97
98 # There is no way to know whether a comment is from a
99 # translator or a developer comment, so we have comenttext
100 # always as None and store all comments as source comments.
101 self.assertEqual(message.commenttext, u'')
102
103 # This format doesn't support any functionality like .po flags.
104 self.assertEqual(message.flagscomment, u'')
105
106 def test_TemplateImport(self):
107 """Test XPI template file import."""
108 # Prepare the import queue to handle a new .xpi import.
109 entry = self.setUpTranslationImportQueueForTemplate('en-US')
110
111 # The status is now IMPORTED:
112 self.assertEqual(entry.status, RosettaImportStatus.IMPORTED)
113
114 # Let's validate the content of the messages.
115 potmsgsets = list(self.firefox_template.getPOTMsgSets())
116
117 messages_msgid_list = []
118 for message in potmsgsets:
119 messages_msgid_list.append(message.msgid_singular.msgid)
120
121 # Check the common values for all messages.
122 self._assertXpiMessageInvariant(message)
123
124 if message.msgid_singular.msgid == u'foozilla.name':
125 # It's a normal message that lacks any comment.
126
127 self.assertEqual(message.singular_text, u'FooZilla!')
128 self.assertEqual(
129 message.filereferences,
130 u'jar:chrome/en-US.jar!/test1.dtd(foozilla.name)')
131 self.assertIsNone(message.sourcecomment)
132
133 elif message.msgid_singular.msgid == u'foozilla.play.fire':
134 # This one is also a normal message that has a comment.
135
136 self.assertEqual(
137 message.singular_text, u'Do you want to play with fire?')
138 self.assertEqual(
139 message.filereferences,
140 u'jar:chrome/en-US.jar!/test1.dtd(foozilla.play.fire)')
141 self.assertEqual(
142 message.sourcecomment,
143 u" Translators, don't play with fire! \n")
144
145 elif message.msgid_singular.msgid == u'foozilla.utf8':
146 # Now, we can see that special UTF-8 chars are extracted
147 # correctly.
148 self.assertEqual(
149 message.singular_text, u'\u0414\u0430\u043d=Day')
150 self.assertEqual(
151 message.filereferences,
152 u'jar:chrome/en-US.jar!/test1.properties:5' +
153 u'(foozilla.utf8)')
154 self.assertIsNone(message.sourcecomment)
155 elif message.msgid_singular.msgid == u'foozilla.menu.accesskey':
156 # access key is a special notation that is supposed to be
157 # translated with a key shortcut.
158 self.assertEqual(
159 message.singular_text, u'M')
160 self.assertEqual(
161 message.filereferences,
162 u'jar:chrome/en-US.jar!/subdir/test2.dtd' +
163 u'(foozilla.menu.accesskey)')
164 # The comment shows the key used when there is no translation,
165 # which is noted as the en_US translation.
166 self.assertEqual(
167 unwrap(message.sourcecomment),
168 unwrap(access_key_source_comment))
169 elif message.msgid_singular.msgid == u'foozilla.menu.commandkey':
170 # command key is a special notation that is supposed to be
171 # translated with a key shortcut.
172 self.assertEqual(
173 message.singular_text, u'm')
174 self.assertEqual(
175 message.filereferences,
176 u'jar:chrome/en-US.jar!/subdir/test2.dtd' +
177 u'(foozilla.menu.commandkey)')
178 # The comment shows the key used when there is no translation,
179 # which is noted as the en_US translation.
180 self.assertEqual(
181 unwrap(message.sourcecomment),
182 unwrap(command_key_source_comment))
183
184 # Check that we got all messages.
185 self.assertEqual(
186 [u'foozilla.happytitle', u'foozilla.menu.accesskey',
187 u'foozilla.menu.commandkey', u'foozilla.menu.title',
188 u'foozilla.name', u'foozilla.nocomment', u'foozilla.play.fire',
189 u'foozilla.play.ice', u'foozilla.title', u'foozilla.utf8',
190 u'foozilla_something'],
191 sorted(messages_msgid_list))
192
193 def test_TwiceTemplateImport(self):
194 """Test a template import done twice."""
195 # Prepare the import queue to handle a new .xpi import.
196 entry = self.setUpTranslationImportQueueForTemplate('en-US')
197
198 # The status is now IMPORTED:
199 self.assertEqual(entry.status, RosettaImportStatus.IMPORTED)
200
201 # Retrieve the number of messages we got in this initial import.
202 first_import_potmsgsets = self.firefox_template.getPOTMsgSets(
203 ).count()
204
205 # Force the entry to be imported again:
206 entry.setStatus(RosettaImportStatus.APPROVED,
207 getUtility(ILaunchpadCelebrities).rosetta_experts)
208 # Now, we tell the PO template to import from the file data it has.
209 (subject, body) = self.firefox_template.importFromQueue(entry)
210
211 # Retrieve the number of messages we got in this second import.
212 second_import_potmsgsets = self.firefox_template.getPOTMsgSets(
213 ).count()
214
215 # Both must match.
216 self.assertEqual(first_import_potmsgsets, second_import_potmsgsets)
217
218 def test_TranslationImport(self):
219 """Test XPI translation file import."""
220 # Prepare the import queue to handle a new .xpi import.
221 template_entry = self.setUpTranslationImportQueueForTemplate('en-US')
222 translation_entry = self.setUpTranslationImportQueueForTranslation(
223 'en-US')
224
225 # The status is now IMPORTED:
226 self.assertEqual(
227 translation_entry.status, RosettaImportStatus.IMPORTED)
228 self.assertEqual(template_entry.status, RosettaImportStatus.IMPORTED)
229
230 # Let's validate the content of the messages.
231 potmsgsets = list(self.firefox_template.getPOTMsgSets())
232
233 messages = [message.msgid_singular.msgid for message in potmsgsets]
234 messages.sort()
235 self.assertEqual(
236 [u'foozilla.happytitle',
237 u'foozilla.menu.accesskey',
238 u'foozilla.menu.commandkey',
239 u'foozilla.menu.title',
240 u'foozilla.name',
241 u'foozilla.nocomment',
242 u'foozilla.play.fire',
243 u'foozilla.play.ice',
244 u'foozilla.title',
245 u'foozilla.utf8',
246 u'foozilla_something'],
247 messages)
248
249 potmsgset = self.firefox_template.getPOTMsgSetByMsgIDText(
250 u'foozilla.name', context='main/test1.dtd')
251 translation = potmsgset.getCurrentTranslation(
252 self.firefox_template, self.spanish_firefox.language,
253 self.firefox_template.translation_side)
254
255 # It's a normal message that lacks any comment.
256 self.assertEqual(potmsgset.singular_text, u'FooZilla!')
257
258 # With this first import, upstream and Ubuntu translations must match.
259 self.assertEqual(
260 translation.translations,
261 potmsgset.getOtherTranslation(
262 self.spanish_firefox.language,
263 self.firefox_template.translation_side).translations)
264
265 potmsgset = self.firefox_template.getPOTMsgSetByMsgIDText(
266 u'foozilla.menu.accesskey', context='main/subdir/test2.dtd')
267
268 # access key is a special notation that is supposed to be
269 # translated with a key shortcut.
270 self.assertEqual(potmsgset.singular_text, u'M')
271 # The comment shows the key used when there is no translation,
272 # which is noted as the en_US translation.
273 self.assertEqual(
274 unwrap(potmsgset.sourcecomment),
275 unwrap(access_key_source_comment))
276 # But for the translation import, we get the key directly.
277 self.assertEqual(
278 potmsgset.getOtherTranslation(
279 self.spanish_firefox.language,
280 self.firefox_template.translation_side).translations,
281 [u'M'])
282
283 potmsgset = self.firefox_template.getPOTMsgSetByMsgIDText(
284 u'foozilla.menu.commandkey', context='main/subdir/test2.dtd')
285 # command key is a special notation that is supposed to be
286 # translated with a key shortcut.
287 self.assertEqual(
288 potmsgset.singular_text, u'm')
289 # The comment shows the key used when there is no translation,
290 # which is noted as the en_US translation.
291 self.assertEqual(
292 unwrap(potmsgset.sourcecomment),
293 unwrap(command_key_source_comment))
294 # But for the translation import, we get the key directly.
295 self.assertEqual(
296 potmsgset.getOtherTranslation(
297 self.spanish_firefox.language,
298 self.firefox_template.translation_side).translations,
299 [u'm'])
300
301 def test_GetLastTranslator(self):
302 """Tests whether we extract last translator information correctly."""
303 translation_entry = self.setUpTranslationImportQueueForTranslation(
304 'en-US')
305 importer = MozillaXpiImporter()
306 translation_file = importer.parse(translation_entry)
307
308 # Let's try with the translation file, it has valid Last Translator
309 # information.
310 name, email = translation_file.header.getLastTranslator()
311 self.assertEqual(name, u'Carlos Perell\xf3 Mar\xedn')
312 self.assertEqual(email, u'carlos@canonical.com')
313
314 def test_Contexts(self):
315 """Test that message context in XPI file is set to chrome path."""
316 queue_entry = self.setUpTranslationImportQueueForTranslation(
317 'clashing_ids')
318 importer = MozillaXpiImporter()
319 template = importer.parse(queue_entry)
320
321 messages = sorted([
322 (message.msgid_singular, message.context, message.singular_text)
323 for message in template.messages])
324 self.assertEqual(
325 [
326 (u'foozilla.clashing.key',
327 u'mac/extra.dtd',
328 u'This message is Mac-specific, and comes from DTD.'),
329 (u'foozilla.clashing.key',
330 u'mac/extra.properties',
331 u'This message is Mac-specific, and comes from properties.'),
332 (u'foozilla.clashing.key',
333 u'main/main.dtd',
334 u'This message is in the main DTD.'),
335 (u'foozilla.clashing.key',
336 u'main/main.properties',
337 u'This message is in the main properties file.'),
338 (u'foozilla.clashing.key',
339 u'unix/extra.dtd',
340 u'This message is Unix-specific, and comes from DTD.'),
341 (u'foozilla.clashing.key',
342 u'unix/extra.properties',
343 u'This message is Unix-specific, and comes from properties.'),
344 (u'foozilla.clashing.key',
345 u'win/extra.dtd',
346 u'This message is Windows-specific, and comes from DTD.'),
347 (u'foozilla.clashing.key',
348 u'win/extra.properties',
349 u'This message is Windows-specific, '
350 'and comes from properties.'),
351 (u'foozilla.regular.message',
352 u'main/main.dtd',
353 u'A non-clashing message.'),
354 ],
355 messages)
356
357 def test_SystemEntityIsIgnored(self):
358 """Test handling of SYSTEM entities in DTD files."""
359 self.setUpTranslationImportQueueForTemplate('system-entity')
360 msgids = [
361 (potmsgset.msgid_singular.msgid, potmsgset.singular_text)
362 for potmsgset in self.firefox_template.getPOTMsgSets()]
363 self.assertEqual(msgids, [
364 ('firststring', 'First translatable string'),
365 ('secondstring', 'Second translatable string')])
diff --git a/lib/lp/translations/utilities/tests/test_xpi_manifest.py b/lib/lp/translations/utilities/tests/test_xpi_manifest.py
366deleted file mode 1006440deleted file mode 100644
index b5348df..0000000
--- a/lib/lp/translations/utilities/tests/test_xpi_manifest.py
+++ /dev/null
@@ -1,307 +0,0 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Unit tests for XPI manifests."""
5
6__metaclass__ = type
7
8import unittest
9
10from lp.translations.interfaces.translationimporter import (
11 TranslationFormatSyntaxError,
12 )
13from lp.translations.utilities.xpi_manifest import XpiManifest
14
15
16class XpiManifestTestCase(unittest.TestCase):
17 """Test `XpiManifest`."""
18
19 def test_TrivialParse(self):
20 # Parse and use minimal manifest.
21 manifest = XpiManifest("locale chromepath en-US directory/")
22 self.assertEqual(len(manifest._locales), 1)
23 chrome_path, locale = manifest.getChromePathAndLocale(
24 'directory/file.dtd')
25 self.assertIsNotNone(chrome_path, "Failed to match simple path")
26 self.assertEqual(
27 chrome_path, "chromepath/file.dtd", "Bad chrome path")
28
29 def test_NonMatch(self):
30 # Failure to match path.
31 manifest = XpiManifest("locale chromepath en-US directory/")
32 chrome_path, locale = manifest.getChromePathAndLocale(
33 'nonexistent/file')
34 self.assertIsNone(chrome_path, "Unexpected path match.")
35 self.assertIsNone(locale, "Got locale without a match.")
36
37 def test_NoUsefulLines(self):
38 # Parse manifest without useful data. Lines that don't match what
39 # we're looking for are ignored.
40 manifest = XpiManifest("""
41 There are no usable
42 locale lines
43 in this file.
44 """.lstrip())
45 self.assertEqual(len(manifest._locales), 0)
46 chrome_path, locale = manifest.getChromePathAndLocale('lines')
47 self.assertIsNone(chrome_path, "Empty manifest matched a path.")
48 chrome_path, locale = manifest.getChromePathAndLocale('')
49 self.assertIsNone(chrome_path, "Matched empty path.")
50
51 def _checkSortOrder(self, manifest):
52 """Verify that manifest is sorted by increasing path length."""
53 last_entry = None
54 for entry in manifest._locales:
55 if last_entry is not None:
56 self.assertFalse(len(entry.path) < len(last_entry.path),
57 "Manifest entries not sorted by increasing path length.")
58 last_entry = entry
59
60 def test_MultipleLines(self):
61 # Parse manifest file with multiple entries.
62 manifest = XpiManifest("""
63 locale foo en-US foodir/
64 locale bar en-US bardir/
65 locale ixx en-US ixxdir/
66 locale gna en-US gnadir/
67 """.lstrip())
68 self.assertEqual(len(manifest._locales), 4)
69 self._checkSortOrder(manifest)
70 for dir in ['gna', 'bar', 'ixx', 'foo']:
71 path = "%sdir/file.html" % dir
72 chrome_path, locale = manifest.getChromePathAndLocale(path)
73 self.assertEqual(chrome_path, "%s/file.html" % dir,
74 "Bad chrome path in multi-line parse.")
75 self.assertEqual(
76 locale, 'en-US', "Bad locale in multi-line parse.")
77
78 def test_MultipleLocales(self):
79 # Different locales.
80 dirs = {
81 'foo': 'en-US',
82 'bar': 'es',
83 'ixx': 'zh_CN',
84 'zup': 'zh_TW',
85 'gna': 'pt',
86 'gnu': 'pt_BR'
87 }
88 manifest_text = '\n'.join([
89 "locale %s %s %sdir/\n" % (dir, locale, dir)
90 for dir, locale in dirs.iteritems()
91 ])
92 manifest = XpiManifest(manifest_text)
93 self._checkSortOrder(manifest)
94 for dir, dirlocale in dirs.iteritems():
95 path = "%sdir/file.html" % dir
96 chrome_path, locale = manifest.getChromePathAndLocale(path)
97 self.assertEqual(chrome_path, "%s/file.html" % dir,
98 "Bad chrome path in multi-line parse.")
99 self.assertEqual(locale, dirlocale, "Locales got mixed up.")
100
101 def test_IgnoredLines(self):
102 # Ignored lines: anything that doesn't start with "locale" or doesn't
103 # have the right number of arguments. The one correct line is picked
104 # out though.
105 manifest = XpiManifest("""
106 nonlocale obsolete fr foodir/
107 anotherline
108
109 #locale obsolete fr foodir/
110 locale okay fr foodir/
111 locale overlong fr foordir/ etc. etc. etc.
112 locale incomplete fr
113 """.lstrip())
114 self.assertEqual(len(manifest._locales), 1)
115 chrome_path, locale = manifest.getChromePathAndLocale('foodir/x')
116 self.assertIsNotNone(chrome_path, "Garbage lines messed up match.")
117 self.assertEqual(chrome_path, "okay/x", "Matched wrong line.")
118 self.assertEqual(locale, "fr", "Inexplicably mismatched locale.")
119
120 def test_DuplicateLines(self):
121 # The manifest ignores redundant lines with the same path.
122 manifest = XpiManifest("""
123 locale dup fy boppe
124 locale dup fy boppe
125 """.lstrip())
126 self.assertEqual(len(manifest._locales), 1)
127
128 def _checkLookup(self, manifest, path, chrome_path, locale):
129 """Helper: look up `path` in `manifest`, expect given output."""
130 found_chrome_path, found_locale = manifest.getChromePathAndLocale(
131 path)
132 self.assertIsNotNone(found_chrome_path, "No match found for " + path)
133 self.assertEqual(found_chrome_path, chrome_path)
134 self.assertEqual(found_locale, locale)
135
136 def test_NormalizedLookup(self):
137 # Both sides of a path lookup are normalized, so that a matching
138 # prefix is recognized in a path even if the two have some meaningless
139 # differences in their spelling.
140 manifest = XpiManifest("locale x nn //a/dir")
141 self._checkLookup(manifest, "a//dir///etc", 'x/etc', 'nn')
142
143 def _checkNormalize(self, bad_path, good_path):
144 """Test that `bad_path` normalizes to `good_path`."""
145 self.assertEqual(XpiManifest._normalizePath(bad_path), good_path)
146
147 def test_Normalize(self):
148 # These paths are all wrong or difficult for one reason or another.
149 # Check that the normalization of paths renders those little
150 # imperfections irrelevant to path lookup.
151 self._checkNormalize('x/', 'x/')
152 self._checkNormalize('x', 'x')
153 self._checkNormalize('/x', 'x')
154 self._checkNormalize('//x', 'x')
155 self._checkNormalize('/x/', 'x/')
156 self._checkNormalize('x//', 'x/')
157 self._checkNormalize('x///', 'x/')
158 self._checkNormalize('x/y/', 'x/y/')
159 self._checkNormalize('x/y', 'x/y')
160 self._checkNormalize('x//y/', 'x/y/')
161
162 def test_PathBoundaries(self):
163 # Paths can only match on path boundaries, where the slashes are
164 # supposed to be.
165 manifest = XpiManifest("""
166 locale short el /ploink/squit
167 locale long he /ploink/squittle
168 """.lstrip())
169 self._checkSortOrder(manifest)
170 self._checkLookup(manifest, 'ploink/squit/x', 'short/x', 'el')
171 self._checkLookup(manifest, '/ploink/squittle/x', 'long/x', 'he')
172
173 def test_Overlap(self):
174 # Path matching looks for longest prefix. Make sure this works right,
175 # even when nested directories are in "overlapping" manifest entries.
176 manifest = XpiManifest("""
177 locale foo1 ca a/
178 locale foo2 ca a/b/
179 locale foo3 ca a/b/c/x1
180 locale foo4 ca a/b/c/x2
181 """.lstrip())
182 self._checkSortOrder(manifest)
183 self._checkLookup(manifest, 'a/bb', 'foo1/bb', 'ca')
184 self._checkLookup(manifest, 'a/bb/c', 'foo1/bb/c', 'ca')
185 self._checkLookup(manifest, 'a/b/y', 'foo2/y', 'ca')
186 self._checkLookup(manifest, 'a/b/c/', 'foo2/c/', 'ca')
187 self._checkLookup(manifest, 'a/b/c/x12', 'foo2/c/x12', 'ca')
188 self._checkLookup(manifest, 'a/b/c/x1/y', 'foo3/y', 'ca')
189 self._checkLookup(manifest, 'a/b/c/x2/y', 'foo4/y', 'ca')
190
191 def test_JarLookup(self):
192 # Simple, successful lookup of a correct path inside a jar file.
193 manifest = XpiManifest("""
194 locale foo en_GB jar:foo.jar!/dir/
195 locale bar id jar:bar.jar!/
196 """.lstrip())
197 self._checkSortOrder(manifest)
198 self._checkLookup(
199 manifest, 'jar:foo.jar!/dir/file', 'foo/file', 'en_GB')
200 self._checkLookup(
201 manifest, 'jar:bar.jar!/dir/file', 'bar/dir/file', 'id')
202
203 def test_JarNormalization(self):
204 # Various badly-formed or corner-case paths. All get normalized.
205 self._checkNormalize('jar:jarless/path', 'jarless/path')
206 self._checkNormalize(
207 'jar:foo.jar!/contained/file', 'jar:foo.jar!/contained/file')
208 self._checkNormalize(
209 'foo.jar!contained/file', 'jar:foo.jar!/contained/file')
210 self._checkNormalize(
211 'jar:foo.jar!//contained/file', 'jar:foo.jar!/contained/file')
212 self._checkNormalize('splat.jar!', 'jar:splat.jar!/')
213 self._checkNormalize('dir/x.jar!dir', 'jar:dir/x.jar!/dir')
214
215 def test_NestedJarNormalization(self):
216 # Test that paths with jars inside jars are normalized correctly.
217 self._checkNormalize(
218 'jar:dir/x.jar!/y.jar!/dir', 'jar:dir/x.jar!/y.jar!/dir')
219 self._checkNormalize(
220 'dir/x.jar!y.jar!dir', 'jar:dir/x.jar!/y.jar!/dir')
221 self._checkNormalize(
222 'dir/x.jar!/dir/y.jar!', 'jar:dir/x.jar!/dir/y.jar!/')
223
224 def test_JarMixup(self):
225 # Two jar files can have files for the same locale. Two locales can
226 # have files in the same jar file. Two translations in different
227 # places can have the same chrome path.
228 manifest = XpiManifest("""
229 locale serbian sr jar:translations.jar!/sr/
230 locale croatian hr jar:translations.jar!/hr/
231 locale docs sr jar:docs.jar!/sr/
232 locale docs hr jar:docs.jar!/hr/
233 """.lstrip())
234 self._checkSortOrder(manifest)
235 self._checkLookup(
236 manifest, 'jar:translations.jar!/sr/x', 'serbian/x', 'sr')
237 self._checkLookup(
238 manifest, 'jar:translations.jar!/hr/x', 'croatian/x', 'hr')
239 self._checkLookup(manifest, 'jar:docs.jar!/sr/x', 'docs/x', 'sr')
240 self._checkLookup(manifest, 'jar:docs.jar!/hr/x', 'docs/x', 'hr')
241
242 def test_NestedJars(self):
243 # Jar files can be contained in jar files.
244 manifest = XpiManifest("""
245 locale x it jar:dir/x.jar!/subdir/y.jar!/
246 locale y it jar:dir/x.jar!/subdir/y.jar!/deep/
247 locale z it jar:dir/x.jar!/subdir/z.jar!/
248 """.lstrip())
249 self._checkSortOrder(manifest)
250 self._checkLookup(
251 manifest, 'jar:dir/x.jar!/subdir/y.jar!/foo', 'x/foo', 'it')
252 self._checkLookup(
253 manifest, 'jar:dir/x.jar!/subdir/y.jar!/deep/foo', 'y/foo', 'it')
254 self._checkLookup(
255 manifest, 'dir/x.jar!/subdir/z.jar!/foo', 'z/foo', 'it')
256
257 def test_ContainsLocales(self):
258 # Jar files need to be descended into if any locale line mentions a
259 # path inside them.
260 manifest = XpiManifest("locale in my jar:x/foo.jar!/y")
261 self.assertTrue(manifest.containsLocales("jar:x/foo.jar!/"))
262 self.assertFalse(manifest.containsLocales("jar:zzz/foo.jar!/"))
263
264 def test_NormalizeContainsLocales(self):
265 # "containsLocales" lookup is normalized, just like chrome path
266 # lookup, so it's not fazed by syntactical misspellings.
267 manifest = XpiManifest("locale main kh jar:/x/foo.jar!bar.jar!")
268 self.assertTrue(manifest.containsLocales("x/foo.jar!//bar.jar!/"))
269
270 def test_ReverseMapping(self):
271 # Test "reverse mapping" from chrome path to XPI path.
272 manifest = XpiManifest(
273 "locale browser en-US jar:locales/en-US.jar!/chrome/")
274 path = manifest.findMatchingXpiPath('browser/gui/print.dtd', 'en-US')
275 self.assertEqual(path, "jar:locales/en-US.jar!/chrome/gui/print.dtd")
276
277 def test_NoReverseMapping(self):
278 # Failed reverse lookup.
279 manifest = XpiManifest(
280 "locale browser en-US jar:locales/en-US.jar!/chrome/")
281 path = manifest.findMatchingXpiPath('manual/gui/print.dtd', 'en-US')
282 self.assertEqual(path, None)
283
284 def test_ReverseMappingWrongLocale(self):
285 # Reverse mapping fails if given the wrong locale.
286 manifest = XpiManifest(
287 "locale browser en-US jar:locales/en-US.jar!/chrome/")
288 path = manifest.findMatchingXpiPath('browser/gui/print.dtd', 'pt')
289 self.assertEqual(path, None)
290
291 def test_ReverseMappingLongestMatch(self):
292 # Reverse mapping always finds the longest match.
293 manifest = XpiManifest("""
294 locale browser en-US jar:locales/
295 locale browser en-US jar:locales/en-US.jar!/chrome/
296 locale browser en-US jar:locales/en-US.jar!/
297 """.lstrip())
298 path = manifest.findMatchingXpiPath('browser/gui/print.dtd', 'en-US')
299 self.assertEqual(path, "jar:locales/en-US.jar!/chrome/gui/print.dtd")
300
301 def test_blank_line(self):
302 # Manifests must not begin with newline.
303 self.assertRaises(
304 TranslationFormatSyntaxError,
305 XpiManifest, """
306 locale browser en-US jar:locales
307 """)
diff --git a/lib/lp/translations/utilities/tests/test_xpi_po_exporter.py b/lib/lp/translations/utilities/tests/test_xpi_po_exporter.py
index 532f9bf..ca830c7 100644
--- a/lib/lp/translations/utilities/tests/test_xpi_po_exporter.py
+++ b/lib/lp/translations/utilities/tests/test_xpi_po_exporter.py
@@ -1,21 +1,29 @@
1# -*- coding: utf-8 -*-
2# NOTE: The first line above must stay first; do not move the copyright
3# notice to the top. See http://www.python.org/dev/peps/pep-0263/.
4#
1# Copyright 2009-2017 Canonical Ltd. This software is licensed under the5# Copyright 2009-2017 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).6# GNU Affero General Public License version 3 (see the file LICENSE).
37
4__metaclass__ = type8__metaclass__ = type
59
6from textwrap import dedent10from textwrap import dedent
7import unittest
811
12from fixtures import MonkeyPatch
9import transaction13import transaction
10from zope.component import (14from zope.component import (
11 getAdapter,15 getAdapter,
12 getUtility,16 getUtility,
13 )17 )
18from zope.interface import implementer
14from zope.interface.verify import verifyObject19from zope.interface.verify import verifyObject
20from zope.security.proxy import removeSecurityProxy
1521
16from lp.app.interfaces.launchpad import ILaunchpadCelebrities22from lp.app.interfaces.launchpad import ILaunchpadCelebrities
17from lp.registry.interfaces.person import IPersonSet23from lp.registry.interfaces.person import IPersonSet
18from lp.registry.interfaces.product import IProductSet24from lp.registry.interfaces.product import IProductSet
25from lp.testing import TestCase
26from lp.testing.fixture import ZopeUtilityFixture
19from lp.testing.layers import LaunchpadZopelessLayer27from lp.testing.layers import LaunchpadZopelessLayer
20from lp.translations.enums import RosettaImportStatus28from lp.translations.enums import RosettaImportStatus
21from lp.translations.interfaces.potemplate import IPOTemplateSet29from lp.translations.interfaces.potemplate import IPOTemplateSet
@@ -25,21 +33,165 @@ from lp.translations.interfaces.translationcommonformat import (
25from lp.translations.interfaces.translationexporter import (33from lp.translations.interfaces.translationexporter import (
26 ITranslationFormatExporter,34 ITranslationFormatExporter,
27 )35 )
36from lp.translations.interfaces.translationfileformat import (
37 TranslationFileFormat,
38 )
39from lp.translations.interfaces.translationimporter import (
40 ITranslationFormatImporter,
41 ITranslationImporter,
42 )
28from lp.translations.interfaces.translationimportqueue import (43from lp.translations.interfaces.translationimportqueue import (
29 ITranslationImportQueue,44 ITranslationImportQueue,
30 )45 )
31from lp.translations.utilities.tests.test_xpi_import import (46from lp.translations.interfaces.translations import TranslationConstants
32 get_en_US_xpi_file_to_import,47from lp.translations.utilities.translation_common_format import (
48 TranslationFileData,
49 TranslationMessageData,
33 )50 )
34from lp.translations.utilities.translation_export import ExportFileStorage51from lp.translations.utilities.translation_export import ExportFileStorage
52from lp.translations.utilities.xpi_header import XpiHeader
35from lp.translations.utilities.xpi_po_exporter import XPIPOExporter53from lp.translations.utilities.xpi_po_exporter import XPIPOExporter
3654
3755
38class XPIPOExporterTestCase(unittest.TestCase):56# Hardcoded representations of what used to be found in
57# lib/lp/translations/utilities/tests/firefox-data/. We no longer have real
58# XPI import code, so we pre-parse the messages.
59test_xpi_header = dedent(u'''\
60 <?xml version="1.0"?>
61 <RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
62 xmlns:em="http://www.mozilla.org/2004/em-rdf#">
63 <Description about="urn:mozilla:install-manifest"
64 em:id="langpack-en-US@firefox.mozilla.org"
65 em:name="English U.S. (en-US) Language Pack"
66 em:version="2.0"
67 em:type="8"
68 em:creator="Danilo Å egan">
69 <em:contributor>Š”Š°Š½ŠøŠ»Š¾ ŠØŠµŠ³Š°Š½</em:contributor>
70 <em:contributor>Carlos PerellĆ³ MarĆ­n &lt;carlos@canonical.com&gt;</em:contributor>
71
72 <em:targetApplication>
73 <Description>
74 <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id><!-- firefox -->
75 <em:minVersion>2.0</em:minVersion>
76 <em:maxVersion>2.0.0.*</em:maxVersion>
77 </Description>
78 </em:targetApplication>
79 </Description>
80 </RDF>
81''')
82test_xpi_messages = [
83 (u'foozilla.menu.title', u'main/subdir/test2.dtd',
84 u'jar:chrome/en-US.jar!/subdir/test2.dtd', u'MENU',
85 u' This is a DTD file inside a subdirectory\n'),
86 (u'foozilla.menu.accesskey', u'main/subdir/test2.dtd',
87 u'jar:chrome/en-US.jar!/subdir/test2.dtd', u'M',
88 dedent(u'''\
89 Select the access key that you want to use. These have
90 to be translated in a way that the selected character is
91 present in the translated string of the label being
92 referred to, for example 'i' in 'Edit' menu item in
93 English. If a translation already exists, please don't
94 change it if you are not sure about it. Please find the
95 context of the key from the end of the 'Located in' text
96 below.
97 ''')),
98 (u'foozilla.menu.commandkey', u'main/subdir/test2.dtd',
99 u'jar:chrome/en-US.jar!/subdir/test2.dtd', u'm',
100 dedent(u'''\
101 Select the shortcut key that you want to use. It
102 should be translated, but often shortcut keys (for
103 example Ctrl + KEY) are not changed from the original. If
104 a translation already exists, please don't change it if
105 you are not sure about it. Please find the context of
106 the key from the end of the 'Located in' text below.
107 ''')),
108 (u'foozilla_something', u'main/subdir/test2.properties',
109 u'jar:chrome/en-US.jar!/subdir/test2.properties:6', u'SomeZilla',
110 dedent(u'''\
111 Translators, what you are seeing now is a lovely,
112 awesome, multiline comment aimed at you directly
113 from the streets of a .properties file
114 ''')),
115 (u'foozilla.name', u'main/test1.dtd',
116 u'jar:chrome/en-US.jar!/test1.dtd', u'FooZilla!', None),
117 (u'foozilla.play.fire', u'main/test1.dtd',
118 u'jar:chrome/en-US.jar!/test1.dtd', u'Do you want to play with fire?',
119 u" Translators, don't play with fire!\n"),
120 (u'foozilla.play.ice', u'main/test1.dtd',
121 u'jar:chrome/en-US.jar!/test1.dtd', u'Play with ice?',
122 u' This is just a comment, not a comment for translators\n'),
123 (u'foozilla.title', u'main/test1.properties',
124 u'jar:chrome/en-US.jar!/test1.properties:1', u'FooZilla Zilla Thingy',
125 None),
126 (u'foozilla.happytitle', u'main/test1.properties',
127 u'jar:chrome/en-US.jar!/test1.properties:3',
128 u'http://foozillingy.happy.net/',
129 u"Translators, if you're older than six, don't translate this\n"),
130 (u'foozilla.nocomment', u'main/test1.properties',
131 u'jar:chrome/en-US.jar!/test1.properties:4', u'No Comment',
132 u'(Except this one)\n'),
133 (u'foozilla.utf8', u'main/test1.properties',
134 u'jar:chrome/en-US.jar!/test1.properties:5', u'\u0414\u0430\u043d=Day',
135 None),
136 ]
137
138
139class FakeXPIMessage(TranslationMessageData):
140 """Simulate an XPI translation message."""
141
142 def __init__(self, key, chrome_path, file_and_line, value, last_comment):
143 super(FakeXPIMessage, self).__init__()
144 self.msgid_singular = key
145 self.context = chrome_path
146 self.file_references = '%s(%s)' % (file_and_line, key)
147 value = value.strip()
148 self.addTranslation(TranslationConstants.SINGULAR_FORM, value)
149 self.singular_text = value
150 self.source_comment = last_comment
151
152
153@implementer(ITranslationFormatImporter)
154class FakeXPIImporter:
155 """Simulate an XPI import. We no longer have real XPI import code."""
156
157 def getFormat(self, file_contents):
158 """See `ITranslationFormatImporter`."""
159 return TranslationFileFormat.XPI
160
161 priority = 0
162 content_type = 'application/zip'
163 file_extensions = ['.xpi']
164 template_suffix = 'en-US.xpi'
165 uses_source_string_msgids = True
166
167 def parse(self, translation_import_queue_entry):
168 """See `ITranslationFormatImporter`.
169
170 This takes a `TranslationImportQueueEntry` to satisfy the interface,
171 but ignores it and returns hardcoded data instead.
172 """
173 translation_file = TranslationFileData()
174 translation_file.header = XpiHeader(test_xpi_header)
175 translation_file.messages = [
176 FakeXPIMessage(*message) for message in test_xpi_messages]
177 return translation_file
178
179 def getHeaderFromString(self, header_string):
180 """See `ITranslationFormatImporter`.
181
182 This takes a header string to satisfy the interface, but ignores it
183 and returns hardcoded data instead.
184 """
185 return XpiHeader(test_xpi_header)
186
187
188class XPIPOExporterTestCase(TestCase):
39 """Class test for gettext's .po file exports"""189 """Class test for gettext's .po file exports"""
40 layer = LaunchpadZopelessLayer190 layer = LaunchpadZopelessLayer
41191
42 def setUp(self):192 def setUp(self):
193 super(XPIPOExporterTestCase, self).setUp()
194
43 self.translation_exporter = XPIPOExporter()195 self.translation_exporter = XPIPOExporter()
44196
45 # Get the importer.197 # Get the importer.
@@ -75,14 +227,24 @@ class XPIPOExporterTestCase(unittest.TestCase):
75227
76 def setUpTranslationImportQueueForTemplate(self):228 def setUpTranslationImportQueueForTemplate(self):
77 """Return an ITranslationImportQueueEntry for testing purposes."""229 """Return an ITranslationImportQueueEntry for testing purposes."""
78 # Get the file to import.230 # Install a fake XPI importer, since we no longer have real XPI
79 en_US_xpi = get_en_US_xpi_file_to_import('en-US')231 # import code.
232 fake_xpi_importer = FakeXPIImporter()
233 self.useFixture(MonkeyPatch(
234 'lp.translations.utilities.translation_import.importers',
235 {TranslationFileFormat.XPI: fake_xpi_importer}))
236 # Temporarily reinstall the translation importer without a security
237 # proxy. This avoids problems getting attributes of
238 # FakeXPIImporter, which has no Zope permissions defined.
239 self.useFixture(ZopeUtilityFixture(
240 removeSecurityProxy(getUtility(ITranslationImporter)),
241 ITranslationImporter))
80242
81 # Attach it to the import queue.243 # Attach it to the import queue.
82 translation_import_queue = getUtility(ITranslationImportQueue)244 translation_import_queue = getUtility(ITranslationImportQueue)
83 by_maintainer = True245 by_maintainer = True
84 entry = translation_import_queue.addOrUpdateEntry(246 entry = translation_import_queue.addOrUpdateEntry(
85 self.firefox_template.path, en_US_xpi, by_maintainer,247 self.firefox_template.path, b'dummy', by_maintainer,
86 self.importer, productseries=self.firefox_template.productseries,248 self.importer, productseries=self.firefox_template.productseries,
87 potemplate=self.firefox_template)249 potemplate=self.firefox_template)
88250
diff --git a/lib/lp/translations/utilities/tests/test_xpi_properties_format.py b/lib/lp/translations/utilities/tests/test_xpi_properties_format.py
89deleted file mode 100644251deleted file mode 100644
index 1d42b14..0000000
--- a/lib/lp/translations/utilities/tests/test_xpi_properties_format.py
+++ /dev/null
@@ -1,302 +0,0 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4__metaclass__ = type
5
6from textwrap import dedent
7import unittest
8
9from lp.translations.interfaces.translationimporter import (
10 TranslationFormatInvalidInputError,
11 )
12from lp.translations.utilities.mozilla_xpi_importer import PropertyFile
13from lp.translations.utilities.xpi_properties_exporter import (
14 XpiPropertiesSubExporter,
15 )
16
17
18class PropertyFileFormatTestCase(unittest.TestCase):
19 """Test class for property file format."""
20
21 def _baseContentEncodingTest(self, content):
22 """This is a base function to check different encodings."""
23 property_file = PropertyFile('test.properties', None, dedent(content))
24
25 expected = {u'default-first-title-mac': [u'Introducci\xf3n'],
26 u'default-last-title-mac': [u'Conclusi\xf3n']}
27 parsed = dict([(message.msgid_singular, message.translations)
28 for message in property_file.messages])
29 self.assertEqual(expected, parsed)
30
31 def test_UTF8PropertyFileTest(self):
32 """This test makes sure that we handle UTF-8 encoding files."""
33 content = '''
34 default-first-title-mac = Introducci\xc3\xb3n
35 default-last-title-mac = Conclusi\xc3\xb3n
36 '''
37 self._baseContentEncodingTest(content)
38
39 def test_UnicodeEscapedPropertyFileTest(self):
40 """This test makes sure that we handle unicode escaped files."""
41 content = '''
42 default-first-title-mac=Introducci\u00F3n
43 default-last-title-mac=Conclusi\u00F3n
44 '''
45 self._baseContentEncodingTest(content)
46
47 def test_InvalidPropertyFileUnicodeEscape(self):
48 # An invalid Unicode escape sequence is a
49 # TranslationFormatInvalidInputError.
50 content = '''
51 weirdness=\u1
52 '''
53 self.assertRaises(
54 TranslationFormatInvalidInputError, PropertyFile, None,
55 'test.properties', content)
56
57 def test_Latin1PropertyFileTest(self):
58 """This test makes sure that we detect bad encodings."""
59 content = '''
60 default-first-title-mac = Introducci\xf3n
61 default-last-title-mac = Conclusi\xf3n
62 '''
63 self.assertRaises(
64 TranslationFormatInvalidInputError, PropertyFile, None,
65 'test.properties', content)
66
67 def test_TrailingBackslashPropertyFileTest(self):
68 """Test whether trailing backslashes are well handled.
69
70 A trailing backslash as last char in the line continue the string in
71 the following document line.
72 """
73 content = '''
74default-first-title-mac=Introd\
75ucci\u00F3n
76'''
77 property_file = PropertyFile('test.properties', None, dedent(content))
78
79 expected = {u'default-first-title-mac': [u'Introducci\xf3n']}
80 parsed = dict([(message.msgid_singular, message.translations)
81 for message in property_file.messages])
82 self.assertEqual(expected, parsed)
83
84 def test_EscapedQuotesPropertyFileTest(self):
85 """Test whether escaped quotes are well handled.
86
87 Escaped quotes must be stored unescaped.
88 """
89 content = 'default-first-title-mac = \\\'Something\\\' \\\"more\\\"'
90
91 property_file = PropertyFile('test.properties', None, dedent(content))
92
93 expected = {u'default-first-title-mac': [u'\'Something\' \"more\"']}
94 parsed = dict([(message.msgid_singular, message.translations)
95 for message in property_file.messages])
96 self.assertEqual(expected, parsed)
97
98 def test_WholeLineCommentPropertyFileTest(self):
99 """Test whether whole line comments are well handled."""
100 content = '''
101 # Foo bar comment.
102 default-first-title-mac = blah
103
104 # This comment should be ignored.
105
106 foo = bar
107 '''
108
109 property_file = PropertyFile('test.properties', None, dedent(content))
110 expected = {u'default-first-title-mac': u'Foo bar comment.\n',
111 u'foo': None}
112 parsed = dict([(message.msgid_singular, message.source_comment)
113 for message in property_file.messages])
114 self.assertEqual(expected, parsed)
115
116 def test_EndOfLineCommentPropertyFileTest(self):
117 """Test whether end of line comments are well handled."""
118
119 content = '''
120 default-first-title-mac = blah // Foo bar comment.
121
122 # This comment should be ignored.
123 foo = bar // Something
124 '''
125
126 property_file = PropertyFile('test.properties', None, dedent(content))
127 expected_comments = {
128 u'default-first-title-mac': u'Foo bar comment.\n',
129 u'foo': u'Something\n'
130 }
131 parsed_comments = dict(
132 [(message.msgid_singular, message.source_comment)
133 for message in property_file.messages])
134
135 self.assertEqual(expected_comments, parsed_comments)
136
137 expected_translations = {
138 u'default-first-title-mac': [u'blah'],
139 u'foo': [u'bar']
140 }
141 parsed_translations = dict([(message.msgid_singular,
142 message.translations)
143 for message in property_file.messages])
144
145 self.assertEqual(expected_translations, parsed_translations)
146
147 def test_MultiLineCommentPropertyFileTest(self):
148 """Test whether multiline comments are well handled."""
149 content = '''
150 /* single line comment */
151 default-first-title-mac = blah
152
153 /* Multi line comment
154 yeah, it's multiple! */
155 foo = bar
156
157 /* Even with nested comment tags, we handle this as multiline comment:
158 # fooo
159 foos = bar
160 something = else // Comment me!
161 */
162 long_comment = foo
163 '''
164
165 property_file = PropertyFile('test.properties', None, dedent(content))
166 expected = {
167 u'default-first-title-mac': u' single line comment \n',
168 u'foo': u" Multi line comment\n yeah, it's multiple! \n",
169 u'long_comment': (
170 u' Even with nested comment tags, we handle this as' +
171 u' multiline comment:\n# fooo\nfoos = bar\n' +
172 u'something = else // Comment me!\n')
173 }
174 parsed = dict([(message.msgid_singular, message.source_comment)
175 for message in property_file.messages])
176 self.assertEqual(expected, parsed)
177
178 def test_URLNotComment(self):
179 """Double slash in a URL is not treated as end-of-line comment."""
180 content = '''
181 url = https://admin.example.com/ // Double slash in URL!
182 '''
183 property_file = PropertyFile('test.properties', None, dedent(content))
184 message = None
185 for entry in property_file.messages:
186 self.assertEqual(message, None, "More messages than expected.")
187 message = entry
188
189 self.assertEqual(message.msgid_singular, u"url")
190 self.assertEqual(message.singular_text, u"https://admin.example.com/")
191 self.assertEqual(message.source_comment, u"Double slash in URL!\n")
192
193 def test_InvalidLinePropertyFileTest(self):
194 """Test whether an invalid line is ignored."""
195 content = '''
196 # Foo bar comment.
197 default-first-title-mac = blah
198
199 # This comment should be ignored.
200 crappy-contnet
201 foo = bar
202 '''
203
204 property_file = PropertyFile('test.properties', None, dedent(content))
205 expected = {u'default-first-title-mac': u'Foo bar comment.\n',
206 u'foo': None}
207 parsed = dict([(message.msgid_singular, message.source_comment)
208 for message in property_file.messages])
209 self.assertEqual(expected, parsed)
210
211 def test_MultilinePropertyFileTest(self):
212 """Test parsing of multiline entries."""
213 content = (
214 'multiline-key = This is the first one\\nThis is the second one.')
215 property_file = PropertyFile('test.properties', None, content)
216 expected = {
217 u'multiline-key': (
218 [u'This is the first one\nThis is the second one.'])
219 }
220 parsed = dict([(message.msgid_singular, message.translations)
221 for message in property_file.messages])
222 self.assertEqual(expected, parsed)
223
224 def test_WhiteSpaceBeforeComment(self):
225 """Test that single line comment is detected even with white space."""
226 content = ' # foo = bar'
227 property_file = PropertyFile('test.properties', None, content)
228 # No message should be parsed.
229 expected = {}
230 parsed = dict([(message.msgid_singular, message.translations)
231 for message in property_file.messages])
232 self.assertEqual(expected, parsed)
233
234
235class MockFile:
236 """`TranslationFileData` boiled down to its essence for this test."""
237 def __init__(self, path='test.properties', messages=None):
238 if messages is None:
239 messages = []
240 self.path = path
241 self.messages = messages
242
243
244class MockMessage:
245 """`TranslationMessageData` boiled down to its essence for this test."""
246 def __init__(self, msgid, translation, comment=None):
247 self.msgid_singular = msgid
248 self.translations = [translation]
249 self.comment = comment
250
251
252class PropertyFileExportTest(unittest.TestCase):
253 """Test XPI `XpiPropertiesSubExporter`."""
254
255 def setUp(self):
256 self.exporter = XpiPropertiesSubExporter()
257
258 def test_properties_export(self):
259 # Test plain export of an XPI properties file.
260 file = MockFile(messages=[
261 MockMessage('foo', 'bar'),
262 MockMessage('id', 'translation', comment='comment'),
263 ])
264
265 expected = dedent("""
266 foo=bar
267
268 /* comment */
269 id=translation
270 """).strip()
271 self.assertEqual(self.exporter.export(file), expected)
272
273 def test_escape(self):
274 # Test escaping in properties files.
275 file = MockFile(messages=[
276 MockMessage("f'oo", 'b"ar', comment="Escaped quotes"),
277 MockMessage("f\\oo", "b\\ar", comment="Escaped backslashes"),
278 ])
279
280 expected = dedent("""
281 /* Escaped quotes */
282 f\\'oo=b\\"ar
283
284 /* Escaped backslashes */
285 f\\\\oo=b\\\\ar
286 """).strip()
287
288 self.assertEqual(self.exporter.export(file).strip(), expected)
289
290 def test_escape_comment(self):
291 # Test escaping of comments in properties files. Not fancy like
292 # actual translation content escaping; just making sure an
293 # ill-chosen comment does not produce wildly invalid output.
294 file = MockFile(messages=[
295 MockMessage("foo", "bar", comment="/*//*/**/ */")])
296
297 expected = dedent("""
298 /* /*X//*X/**X/ *X/ */
299 foo=bar
300 """).strip()
301
302 self.assertEqual(self.exporter.export(file).strip(), expected)
diff --git a/lib/lp/translations/utilities/tests/test_xpi_search.py b/lib/lp/translations/utilities/tests/test_xpi_search.py
303deleted file mode 1006440deleted file mode 100644
index 7bd6e74..0000000
--- a/lib/lp/translations/utilities/tests/test_xpi_search.py
+++ /dev/null
@@ -1,83 +0,0 @@
1# Copyright 2009-2017 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Functional tests for searching through XPI POTemplates"""
5__metaclass__ = type
6
7import unittest
8
9from zope.component import getUtility
10
11from lp.registry.interfaces.person import IPersonSet
12from lp.registry.interfaces.product import IProductSet
13from lp.testing.layers import LaunchpadZopelessLayer
14from lp.translations.enums import RosettaImportStatus
15from lp.translations.interfaces.potemplate import IPOTemplateSet
16from lp.translations.utilities.tests.helpers import (
17 import_pofile_or_potemplate,
18 )
19from lp.translations.utilities.tests.xpi_helpers import (
20 get_en_US_xpi_file_to_import,
21 )
22
23
24class XpiSearchTestCase(unittest.TestCase):
25 """XPI file import into Launchpad."""
26
27 layer = LaunchpadZopelessLayer
28
29 def setUp(self):
30 # Get the importer.
31 self.importer = getUtility(IPersonSet).getByName('mark')
32
33 # Get the Firefox template.
34 firefox_product = getUtility(IProductSet).getByName('firefox')
35 firefox_productseries = firefox_product.getSeries('trunk')
36 firefox_potemplate_subset = getUtility(IPOTemplateSet).getSubset(
37 productseries=firefox_productseries)
38 self.firefox_template = firefox_potemplate_subset.new(
39 name='firefox',
40 translation_domain='firefox',
41 path='en-US.xpi',
42 owner=self.importer)
43 self.spanish_firefox = self.firefox_template.newPOFile('es')
44 self.spanish_firefox.path = 'translations/es.xpi'
45
46 def setUpTranslationImportQueueForTemplate(self, subdir):
47 """Return an ITranslationImportQueueEntry for testing purposes.
48
49 :param subdir: subdirectory in firefox-data to get XPI data from.
50 """
51 # Get the file to import.
52 en_US_xpi = get_en_US_xpi_file_to_import(subdir)
53 return import_pofile_or_potemplate(
54 file_contents=en_US_xpi,
55 person=self.importer,
56 potemplate=self.firefox_template)
57
58 def test_templateSearching(self):
59 """Searching through XPI template returns English 'translations'."""
60 entry = self.setUpTranslationImportQueueForTemplate('en-US')
61
62 # The status is now IMPORTED:
63 self.assertEqual(entry.status, RosettaImportStatus.IMPORTED)
64
65 potmsgsets = self.spanish_firefox.findPOTMsgSetsContaining(
66 text='zilla')
67 message_list = [message.singular_text for message in potmsgsets]
68
69 self.assertEqual([u'SomeZilla', u'FooZilla!',
70 u'FooZilla Zilla Thingy'],
71 message_list)
72
73 def test_templateSearchingForMsgIDs(self):
74 """Searching returns no results for internal msg IDs."""
75 entry = self.setUpTranslationImportQueueForTemplate('en-US')
76
77 # The status is now IMPORTED:
78 self.assertEqual(entry.status, RosettaImportStatus.IMPORTED)
79
80 potmsgsets = list(self.spanish_firefox.findPOTMsgSetsContaining(
81 text='foozilla.title'))
82
83 self.assertEqual(potmsgsets, [])
diff --git a/lib/lp/translations/utilities/tests/xpi_helpers.py b/lib/lp/translations/utilities/tests/xpi_helpers.py
84deleted file mode 1006440deleted file mode 100644
index 338c343..0000000
--- a/lib/lp/translations/utilities/tests/xpi_helpers.py
+++ /dev/null
@@ -1,82 +0,0 @@
1# Copyright 2009-2018 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Helper methods for XPI testing"""
5__metaclass__ = type
6
7__all__ = [
8 'access_key_source_comment',
9 'command_key_source_comment',
10 'get_en_US_xpi_file_to_import',
11 ]
12
13import os.path
14import tempfile
15from textwrap import dedent
16import zipfile
17
18import scandir
19
20import lp.translations
21
22
23command_key_source_comment = dedent(u"""
24 Select the shortcut key that you want to use. It should be translated,
25 but often shortcut keys (for example Ctrl + KEY) are not changed from
26 the original. If a translation already exists, please don't change it
27 if you are not sure about it. Please find the context of the key from
28 the end of the 'Located in' text below.
29 """).strip()
30
31access_key_source_comment = dedent(u"""
32 Select the access key that you want to use. These have to be
33 translated in a way that the selected character is present in the
34 translated string of the label being referred to, for example 'i' in
35 'Edit' menu item in English. If a translation already exists, please
36 don't change it if you are not sure about it. Please find the context
37 of the key from the end of the 'Located in' text below.
38 """).strip()
39
40
41def get_en_US_xpi_file_to_import(subdir):
42 """Return an en-US.xpi file object ready to be imported.
43
44 The file is generated from utilities/tests/firefox-data/<subdir>.
45 """
46 # en-US.xpi file is a ZIP file which contains embedded JAR file (which is
47 # also a ZIP file) and a couple of other files. Embedded JAR file is
48 # named 'en-US.jar' and contains translatable resources.
49
50 # Get the root path where the data to generate .xpi file is stored.
51 test_root = os.path.join(
52 os.path.dirname(lp.translations.__file__),
53 'utilities/tests/firefox-data', subdir)
54
55 # First create a en-US.jar file to be included in XPI file.
56 jarfile = tempfile.TemporaryFile()
57 jar = zipfile.ZipFile(jarfile, 'w')
58 jarlist = []
59 data_dir = os.path.join(test_root, 'en-US-jar/')
60 for root, dirs, files in scandir.walk(data_dir):
61 for name in files:
62 relative_dir = root[len(data_dir):].strip('/')
63 jarlist.append(os.path.join(relative_dir, name))
64 for file_name in jarlist:
65 f = open(os.path.join(data_dir, file_name), 'r')
66 jar.writestr(file_name, f.read())
67 jar.close()
68 jarfile.seek(0)
69
70 # Add remaining bits and en-US.jar to en-US.xpi.
71
72 xpifile = tempfile.TemporaryFile()
73 xpi = zipfile.ZipFile(xpifile, 'w')
74 for xpi_entry in scandir.scandir(test_root):
75 if xpi_entry.name != 'en-US-jar':
76 with open(xpi_entry.path) as f:
77 xpi.writestr(xpi_entry.name, f.read())
78 xpi.writestr('chrome/en-US.jar', jarfile.read())
79 xpi.close()
80 xpifile.seek(0)
81
82 return xpifile
diff --git a/lib/lp/translations/utilities/translation_import.py b/lib/lp/translations/utilities/translation_import.py
index c08e5cd..197153d 100644
--- a/lib/lp/translations/utilities/translation_import.py
+++ b/lib/lp/translations/utilities/translation_import.py
@@ -57,7 +57,6 @@ from lp.translations.interfaces.translationmessage import (
57from lp.translations.interfaces.translations import TranslationConstants57from lp.translations.interfaces.translations import TranslationConstants
58from lp.translations.utilities.gettext_po_importer import GettextPOImporter58from lp.translations.utilities.gettext_po_importer import GettextPOImporter
59from lp.translations.utilities.kde_po_importer import KdePOImporter59from lp.translations.utilities.kde_po_importer import KdePOImporter
60from lp.translations.utilities.mozilla_xpi_importer import MozillaXpiImporter
61from lp.translations.utilities.sanitize import (60from lp.translations.utilities.sanitize import (
62 sanitize_translations_from_import,61 sanitize_translations_from_import,
63 )62 )
@@ -73,7 +72,6 @@ from lp.translations.utilities.validate import (
73importers = {72importers = {
74 TranslationFileFormat.KDEPO: KdePOImporter(),73 TranslationFileFormat.KDEPO: KdePOImporter(),
75 TranslationFileFormat.PO: GettextPOImporter(),74 TranslationFileFormat.PO: GettextPOImporter(),
76 TranslationFileFormat.XPI: MozillaXpiImporter(),
77 }75 }
7876
7977
diff --git a/lib/lp/translations/utilities/xpi_manifest.py b/lib/lp/translations/utilities/xpi_manifest.py
80deleted file mode 10064478deleted file mode 100644
index 1bbaedd..0000000
--- a/lib/lp/translations/utilities/xpi_manifest.py
+++ /dev/null
@@ -1,237 +0,0 @@
1# Copyright 2009-2014 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4__metaclass__ = type
5
6__all__ = ['make_jarpath', 'XpiManifest']
7
8
9import logging
10import re
11
12from lp.translations.interfaces.translationimporter import (
13 TranslationFormatSyntaxError,
14 )
15
16
17def normalize_path(path):
18 """Normalize filesystem path within XPI file."""
19 # Normalize "jar:" prefix. Make sure it's there when needed, not there
20 # when not needed.
21 if path.startswith('jar:'):
22 # No leading slashes please.
23 path = re.sub('^jar:/+', 'jar:', path)
24
25 if '.jar!' not in path:
26 logging.debug("Removing 'jar:' from manifest path: '%s'" % path)
27 path = path[4:]
28 else:
29 # No leading slashes please.
30 path = re.sub('^/+', '', path)
31
32 if '.jar!' in path:
33 # Path delves into a jar file, but lacks "jar:" prefix. This is
34 # really a malformed path.
35 logging.info("Adding 'jar:' to manifest path: '%s'" % path)
36 path = 'jar:' + path
37
38 # A path inside a jar file must begin with a slash.
39 path = path.replace('.jar!', '.jar!/')
40
41 # Finally, eliminate redundant slashes. The previous steps may have
42 # introduced some.
43 return re.sub('/+', '/', path)
44
45
46def is_valid_path(path):
47 """Check that path is a valid, normalized path inside an XPI file."""
48 if '//' in path:
49 return False
50 if re.search('\\.jar![^/]', path):
51 return False
52 if path.startswith('jar:'):
53 if path.startswith('jar:jar:'):
54 return False
55 if '.jar!' not in path:
56 return False
57 else:
58 if '.jar!' in path:
59 return False
60 return True
61
62
63def is_valid_dir_path(path):
64 """Check that path is a normalized directory path in an XPI file."""
65 if not is_valid_path(path):
66 return False
67 if not path.endswith('/'):
68 return False
69 return True
70
71
72def make_jarpath(path, jarname):
73 """Construct base path for files inside a jar file.
74
75 To name some translation file that's inside a jar file inside an XPI
76 file, concatenate the result of this method (for the jar file) and the
77 translation file's path within the jar file.
78
79 For example, let's say the XPI file contains foo/bar.jar. Inside
80 foo/bar.jar is a translation file locale/gui.dtd. Then
81 make_jarfile('foo', 'bar.jar') will return "jar:foo/bar.jar!/", to
82 which you can append "locale/gui.dtd" to get the full path
83 "jar:foo/bar.jar!/locale/gui.dtd" which identifies the translation
84 file within the XPI file.
85 """
86 # This function is where we drill down into a jar file, so prefix with
87 # "jar:" (unless it's already there). We carry the "jar:" prefix only
88 # for paths that drill into jar files.
89 if not path.startswith('jar:'):
90 path = 'jar:' + path
91
92 return normalize_path("%s/%s!" % (path, jarname))
93
94
95class ManifestEntry:
96 """A "locale" line in a manifest file."""
97
98 chrome = None
99 locale = None
100 path = None
101
102 def __init__(self, chrome, locale, path):
103 self.chrome = chrome
104 self.locale = locale
105
106 # Normalize path so we can do simple, reliable text matching on it.
107 # The directory paths in an XPI file should end in a single slash.
108 # Append the slash here; the normalization will take care of redundant
109 # slashes.
110 self.path = normalize_path(path + "/")
111
112 assert is_valid_dir_path(self.path), (
113 "Normalized path not valid: '%s' -> '%s'" % (path, self.path))
114
115
116def manifest_entry_sort_key(entry):
117 """We keep manifest entries sorted by path length."""
118 return len(entry.path)
119
120
121class XpiManifest:
122 """Representation of an XPI manifest file.
123
124 Does two things: parsers an XPI file; and looks up chrome paths and
125 locales for given filesystem paths inside the XPI file.
126 """
127
128 # List of locale entries, sorted by increasing path length. The sort
129 # order matters for lookup.
130 _locales = None
131
132 def __init__(self, content):
133 """Initialize: parse `content` as a manifest file."""
134 if content.startswith('\n'):
135 raise TranslationFormatSyntaxError(
136 message="Manifest begins with newline.")
137
138 locales = []
139 for line in content.splitlines():
140 words = line.split()
141 num_words = len(words)
142 if num_words == 0 or words[0] != 'locale':
143 pass
144 elif num_words < 4:
145 logging.info("Ignoring short manifest line: '%s'" % line)
146 elif num_words > 4:
147 logging.info("Ignoring long manifest line: '%s'" % line)
148 else:
149 locales.append(ManifestEntry(words[1], words[2], words[3]))
150
151 # Eliminate duplicates.
152 paths = set()
153 deletions = []
154 for index, entry in enumerate(locales):
155 assert entry.path.endswith('/'), "Manifest path lost its slash"
156
157 if entry.path in paths:
158 logging.info("Duplicate paths in manifest: '%s'" % entry.path)
159 deletions.append(index)
160
161 paths.add(entry.path)
162
163 for index in reversed(deletions):
164 del locales[index]
165
166 self._locales = sorted(locales, key=manifest_entry_sort_key)
167
168 @classmethod
169 def _normalizePath(cls, path):
170 """Normalize path. Here so it can be tested without exporting it."""
171 return normalize_path(path)
172
173 def _getMatchingEntry(self, file_path):
174 """Return longest matching entry matching file_path."""
175 assert is_valid_path(file_path), (
176 "Generated path not valid: %s" % file_path)
177
178 # Locale entries are sorted by path length. If we scan backwards, the
179 # first entry whose path is a prefix of file_path is the longest
180 # match. The fact that the entries' paths have trailing slashes
181 # guarantees that we won't match in the middle of a file or directory
182 # name.
183 for entry in reversed(self._locales):
184 if file_path.startswith(entry.path):
185 return entry
186
187 # No match found.
188 return None
189
190 def getChromePathAndLocale(self, file_path):
191 """Return chrome path and locale applying to a filesystem path.
192 """
193 assert file_path is not None, "Looking up chrome path for None"
194 file_path = self._normalizePath(file_path)
195 entry = self._getMatchingEntry(file_path)
196
197 if entry is None:
198 return None, None
199
200 assert file_path.startswith(entry.path), "Found non-matching entry"
201 replace = len(entry.path)
202 chrome_path = "%s/%s" % (entry.chrome, file_path[replace:])
203 return chrome_path, entry.locale
204
205 def containsLocales(self, file_path):
206 """Is `file_path` a prefix of any path containing locale files?
207
208 :param file_path: path of a directory or jar file inside this XPI.
209 :return: Boolean: does `file_path` contain locale files?
210 """
211 file_path = self._normalizePath(file_path)
212 for entry in self._locales:
213 if entry.path.startswith(file_path):
214 return True
215 return False
216
217 def findMatchingXpiPath(self, chrome_path, locale):
218 """Reverse-map a chrome path in a given locale to a file path.
219
220 For example, if given "browser/gui/print.dtd" for locale en-US,
221 may return "jar:locales/en-US.jar!/chrome/gui/print.dtd",
222 assuming that the file path jar:locales/en-US.jar!/chrome/
223 is associated with the chrome path browser.
224
225 If there are multiple matches, this returns the one with the
226 longest file path.
227 """
228 # Since _locales is sorted by path length, scanning it backwards
229 # finds the longest match first.
230 for entry in reversed(self._locales):
231 is_match = (chrome_path.startswith(entry.chrome + '/') and
232 entry.locale == locale)
233 if is_match:
234 return normalize_path(
235 entry.path + chrome_path[len(entry.chrome):])
236
237 return None
diff --git a/lib/lp/translations/utilities/xpi_properties_exporter.py b/lib/lp/translations/utilities/xpi_properties_exporter.py
238deleted file mode 1006440deleted file mode 100644
index 43cf397..0000000
--- a/lib/lp/translations/utilities/xpi_properties_exporter.py
+++ /dev/null
@@ -1,53 +0,0 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4__metaclass__ = type
5
6__all__ = [
7 'XpiPropertiesSubExporter'
8 ]
9
10
11import re
12
13
14def has_comment(message):
15 """Does `TranslationMessageData` contain a comment?"""
16 return message.comment is not None and message.comment.strip() != ''
17
18
19class XpiPropertiesSubExporter:
20 """Produce a properties file to go into an XPI file."""
21
22 def _escape(self, string):
23 """Escape message string for use in properties file."""
24 # Escape backslashes first, before we start inserting ones of
25 # our own. Then the other stuff. Replace newlines by \n etc.,
26 # and encode non-ASCII characters as \uXXXX.
27 string = string.replace('\\', r'\\')
28 string = re.sub('''(["'])''', r'\\\1', string)
29 # Escape newlines as \n etc, and non-ASCII as \uXXXX
30 return string.encode('ascii', 'backslashreplace')
31
32 def _escape_comment(self, comment):
33 """Escape comment string for use in properties file."""
34 # Prevent comment from breaking out of /* ... */ block.
35 comment = comment.replace('*/', '*X/')
36 return comment.encode('ascii', 'unicode-escape')
37
38 def export(self, translation_file):
39 assert translation_file.path.endswith('.properties'), (
40 "Unexpected properties file suffix: %s" % translation_file.path)
41 contents = []
42 for message in translation_file.messages:
43 if not message.translations:
44 continue
45 if has_comment(message):
46 contents.append(
47 "\n/* %s */" % self._escape_comment(message.comment))
48 msgid = self._escape(message.msgid_singular)
49 text = self._escape(message.translations[0])
50 line = "%s=%s" % (msgid, text)
51 contents.append(line)
52
53 return '\n'.join(contents)
diff --git a/utilities/sourcedeps.cache b/utilities/sourcedeps.cache
index 412ea52..fef3955 100644
--- a/utilities/sourcedeps.cache
+++ b/utilities/sourcedeps.cache
@@ -27,10 +27,6 @@
27 494,27 494,
28 "cjwatson@canonical.com-20190919081036-q1symc2h2iedtlh3"28 "cjwatson@canonical.com-20190919081036-q1symc2h2iedtlh3"
29 ],29 ],
30 "old_xmlplus": [
31 4,
32 "sinzui-20090526164636-1swugzupwvjgomo4"
33 ],
34 "pygettextpo": [30 "pygettextpo": [
35 25,31 25,
36 "launchpad@pqm.canonical.com-20140116030912-lqm1dtb6a0y4femq"32 "launchpad@pqm.canonical.com-20140116030912-lqm1dtb6a0y4femq"
diff --git a/utilities/sourcedeps.conf b/utilities/sourcedeps.conf
index b19007d..13287d4 100644
--- a/utilities/sourcedeps.conf
+++ b/utilities/sourcedeps.conf
@@ -14,5 +14,4 @@ bzr-svn lp:~launchpad-pqm/bzr-svn/devel;revno=2725
14cscvs lp:~launchpad-pqm/launchpad-cscvs/devel;revno=43314cscvs lp:~launchpad-pqm/launchpad-cscvs/devel;revno=433
15difftacular lp:~launchpad/difftacular/trunk;revno=1115difftacular lp:~launchpad/difftacular/trunk;revno=11
16loggerhead lp:~loggerhead-team/loggerhead/trunk-rich;revno=49416loggerhead lp:~loggerhead-team/loggerhead/trunk-rich;revno=494
17old_xmlplus lp:~launchpad-pqm/dtdparser/trunk;revno=4
18pygettextpo lp:~launchpad-pqm/pygettextpo/trunk;revno=2517pygettextpo lp:~launchpad-pqm/pygettextpo/trunk;revno=25

Subscribers

People subscribed via source and target branches

to status/vote changes: