Merge ~cjwatson/launchpad:remove-xpi-importer into launchpad:master

Proposed by Colin Watson
Status: Merged
Approved by: Colin Watson
Approved revision: 2dfbcd277821d0cda66f498d079b36ae523eb0aa
Merge reported by: Otto Co-Pilot
Merged at revision: not available
Proposed branch: ~cjwatson/launchpad:remove-xpi-importer
Merge into: launchpad:master
Diff against target: 3848 lines (+193/-518)
16 files modified
dev/null (+0/-53)
lib/lp/translations/configure.zcml (+0/-9)
lib/lp/translations/doc/poexport-language-pack.txt (+16/-153)
lib/lp/translations/doc/sourcepackagerelease-translations.txt (+1/-2)
lib/lp/translations/scripts/language_pack.py (+1/-25)
lib/lp/translations/scripts/tests/test_validate_translations_file.py (+0/-38)
lib/lp/translations/scripts/validate_translations_file.py (+0/-24)
lib/lp/translations/tests/test_autoapproval.py (+1/-24)
lib/lp/translations/tests/test_potmsgset.py (+3/-146)
lib/lp/translations/tests/test_translationbranchapprover.py (+0/-11)
lib/lp/translations/tests/test_translationimportqueue.py (+0/-1)
lib/lp/translations/utilities/tests/test_translation_importer.py (+2/-18)
lib/lp/translations/utilities/tests/test_xpi_po_exporter.py (+169/-7)
lib/lp/translations/utilities/translation_import.py (+0/-2)
utilities/sourcedeps.cache (+0/-4)
utilities/sourcedeps.conf (+0/-1)
Reviewer Review Type Date Requested Status
Kristian Glass (community) Approve
Review via email: mp+380059@code.launchpad.net

Commit message

Remove XPI import support

Description of the change

As far as I can tell, this hasn't been used on production since about 2011, possibly due to changes in the Firefox release model. It's sufficiently complicated that it isn't worth keeping if it isn't being used, although we can reintroduce it later if necessary.

The initial motivation for this was that it seems surprisingly difficult to parse DTDs in modern Python without the non-Python-3-friendly copy of parts of the old python-xml package that we've been keeping around in sourcecode/old_xmlplus, but it turned into a larger opportunity to prune unused code.

I had to start by rearranging test_xpi_po_exporter to avoid using the XPI importer, as it makes some sense to preserve vestigial support for exporting XPI templates in PO format in order that there's some way to extract existing information from the production database.

To post a comment you must log in.
Revision history for this message
Kristian Glass (doismellburning) wrote :

Nicely done

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/lib/lp/translations/configure.zcml b/lib/lp/translations/configure.zcml
2index 9f91581..3693a4b 100644
3--- a/lib/lp/translations/configure.zcml
4+++ b/lib/lp/translations/configure.zcml
5@@ -291,15 +291,6 @@
6 <allow
7 interface="lp.translations.interfaces.translationimporter.ITranslationFormatImporter"/>
8 </class>
9- <class
10- class="lp.translations.utilities.mozilla_xpi_importer.MozillaXpiImporter">
11- <allow
12- interface="lp.translations.interfaces.translationimporter.ITranslationFormatImporter"/>
13- </class>
14- <subscriber
15- for="lp.translations.interfaces.translationimportqueue.ITranslationImportQueueEntry"
16- provides="lp.translations.interfaces.translationimporter.ITranslationFormatImporter"
17- factory="lp.translations.utilities.mozilla_xpi_importer.MozillaXpiImporter"/>
18
19 <!-- PO File -->
20
21diff --git a/lib/lp/translations/doc/poexport-language-pack.txt b/lib/lp/translations/doc/poexport-language-pack.txt
22index 94b3624..062abc8 100644
23--- a/lib/lp/translations/doc/poexport-language-pack.txt
24+++ b/lib/lp/translations/doc/poexport-language-pack.txt
25@@ -33,9 +33,6 @@ This is handy for examining the tar files that are generated.
26 ... if not member.isreg():
27 ... # Not a regular file. No size to print.
28 ... size = '-'
29- ... elif member.name.endswith('.xpi'):
30- ... # XPI file. Binary, so don't try counting lines.
31- ... size = 'bin'
32 ... else:
33 ... size = len(tarfile.extractfile(member).readlines())
34 ... print("| %5s | %s" % (size, member.name))
35@@ -102,15 +99,17 @@ And one of the included .po files look like what we expected.
36 '# traducci\xc3\xb3n de es.po al Spanish\n'
37
38
39-Language pack with XPI translations
40------------------------------------
41+Base language pack export using Librarian with date limits
42+----------------------------------------------------------
43
44-Launchpad supports XPI file imports. However, we don't have an export
45-process ready, and thus, we do it with an external script that does that
46-last part based on .po files and the original en-US.xpi file. To achieve
47-that, we export all translations for XPI files in a special directory:
48-xpi/translation_domain/
49+Launchpad is also able to generate a tarball of all files for a
50+distribution series that only includes translation files which have been
51+changed since a certain date.
52
53+First we need to set up some data to test with, and for this we need
54+some DB classes.
55+
56+ >>> from StringIO import StringIO
57 >>> from lp.registry.interfaces.distribution import IDistributionSet
58 >>> from lp.registry.interfaces.person import IPersonSet
59 >>> from lp.registry.model.sourcepackagename import SourcePackageName
60@@ -126,122 +125,6 @@ Get the Grumpy distro series.
61
62 >>> series = getUtility(IDistributionSet)['ubuntu'].getSeries('grumpy')
63
64-
65-Sample data initialization
66-..........................
67-
68-We need to import an XPI template and a translation to see those files
69-exported as part of language packs.
70-
71- >>> from lp.translations.enums import RosettaImportStatus
72- >>> from lp.translations.interfaces.translationimportqueue import (
73- ... ITranslationImportQueue)
74- >>> from lp.translations.utilities.tests.test_xpi_import \
75- ... import get_en_US_xpi_file_to_import
76-
77-We are going to import translations for mozilla-firefox package in
78-grumpy distro series.
79-
80- >>> series = getUtility(IDistributionSet)['ubuntu'].getSeries('grumpy')
81- >>> spn = SourcePackageName.byName('mozilla-firefox')
82- >>> pot_header = 'Content-Type: text/plain; charset=UTF-8\n'
83- >>> firefox_template = POTemplate(
84- ... name='firefox', translation_domain='firefox',
85- ... distroseries=series, sourcepackagename=spn,
86- ... owner=mark, languagepack=True, path='en-US.xpi',
87- ... header=pot_header)
88-
89-Attach the en-US.xpi (the template) file.
90-
91- >>> en_US_xpi = get_en_US_xpi_file_to_import('en-US')
92- >>> translation_import_queue = getUtility(ITranslationImportQueue)
93- >>> by_maintainer = True
94- >>> template_entry = translation_import_queue.addOrUpdateEntry(
95- ... firefox_template.path, en_US_xpi, by_maintainer,
96- ... mark, distroseries=series, sourcepackagename=spn,
97- ... potemplate=firefox_template)
98-
99-Attach the es.xpi file (the translation) file.
100-
101- >>> es_xpi = get_en_US_xpi_file_to_import('en-US')
102- >>> firefox_es_translation = firefox_template.newPOFile('es')
103- >>> translation_entry = translation_import_queue.addOrUpdateEntry(
104- ... 'es.xpi', es_xpi, by_maintainer,
105- ... mark, distroseries=series, sourcepackagename=spn,
106- ... potemplate=firefox_template,
107- ... pofile=firefox_es_translation)
108-
109-Before we are ready to import the attached files, we need to approve
110-them first.
111-
112- >>> template_entry.setStatus(
113- ... RosettaImportStatus.APPROVED, rosetta_experts)
114- >>> translation_entry.setStatus(
115- ... RosettaImportStatus.APPROVED, rosetta_experts)
116-
117-Given that the files are attached to Librarian, we need to commit the
118-transaction to make sure it's stored properly and available.
119-
120- >>> transaction.commit()
121-
122-We do now the import from the queue:
123-
124- >>> (subject, body) = firefox_template.importFromQueue(template_entry)
125- >>> (subject, body) = firefox_es_translation.importFromQueue(
126- ... translation_entry)
127- >>> flush_database_caches()
128- >>> transaction.commit()
129-
130-
131-Language pack export with XPI files
132-...................................
133-
134-We are now ready to get an exported language pack with XPI files.
135-
136- >>> language_pack = export_language_pack(
137- ... distribution_name='ubuntu',
138- ... series_name='grumpy',
139- ... component=None,
140- ... force_utf8=True,
141- ... output_file=None,
142- ... logger=logger)
143- >>> transaction.commit()
144-
145-We get other entries in language pack + en-US.xpi file and the
146-translations in .po file format.
147-
148- >>> tarfile = bytes_to_tarfile(language_pack.file.read())
149- >>> examine_tarfile(tarfile)
150- | - | rosetta-grumpy
151- | 1 | rosetta-grumpy/mapping.txt
152- | 1 | rosetta-grumpy/timestamp.txt
153- | - | rosetta-grumpy/xpi
154- | - | rosetta-grumpy/xpi/firefox
155- | bin | rosetta-grumpy/xpi/firefox/en-US.xpi
156- | 94 | rosetta-grumpy/xpi/firefox/en.po
157- | 102 | rosetta-grumpy/xpi/firefox/es.po
158-
159-We got a valid en-US.xpi file.
160-
161- >>> fh = tarfile.extractfile('rosetta-grumpy/xpi/firefox/en-US.xpi')
162- >>> from zipfile import ZipFile
163- >>> zip = ZipFile(fh, 'r')
164- >>> sorted(zip.namelist())
165- ['chrome.manifest', 'chrome/en-US.jar', 'copyover3.png', 'install.rdf']
166-
167-
168-Base language pack export using Librarian with date limits
169-----------------------------------------------------------
170-
171-Launchpad is also able to generate a tarball of all files for a
172-distribution series that only includes translation files which have been
173-changed since a certain date.
174-
175-First we need to set up some data to test with, and for this we need
176-some DB classes.
177-
178- >>> from StringIO import StringIO
179-
180 Get a source package name to go with our distro series.
181
182 >>> spn = SourcePackageName.byName('evolution')
183@@ -319,12 +202,9 @@ Check that the log looks ok.
184
185 >>> print(logger.getLogBuffer())
186 DEBUG Selecting PO files for export
187- INFO Number of PO files to export: 4
188- DEBUG Exporting PO file ... (1/4)
189- DEBUG Exporting PO file ... (2/4)
190- DEBUG Exporting PO file ... (3/4)
191- DEBUG Exporting PO file ... (4/4)
192- INFO Exporting XPI template files.
193+ INFO Number of PO files to export: 2
194+ DEBUG Exporting PO file ... (1/2)
195+ DEBUG Exporting PO file ... (2/2)
196 INFO Adding timestamp file
197 INFO Adding mapping file
198 INFO Done.
199@@ -342,13 +222,8 @@ Check that the log looks ok.
200 | - | rosetta-grumpy/es
201 | - | rosetta-grumpy/es/LC_MESSAGES
202 | 21 | rosetta-grumpy/es/LC_MESSAGES/test.po
203- | 2 | rosetta-grumpy/mapping.txt
204+ | 1 | rosetta-grumpy/mapping.txt
205 | 1 | rosetta-grumpy/timestamp.txt
206- | - | rosetta-grumpy/xpi
207- | - | rosetta-grumpy/xpi/firefox
208- | bin | rosetta-grumpy/xpi/firefox/en-US.xpi
209- | 94 | rosetta-grumpy/xpi/firefox/en.po
210- | 102 | rosetta-grumpy/xpi/firefox/es.po
211
212 Check the files look OK.
213
214@@ -431,21 +306,15 @@ should get only files that were updated after 2000-01-02.
215 >>> tarfile = bytes_to_tarfile(language_pack.file.read())
216
217 Now, there is only one file exported for the 'test' domain, the one that
218-had the modification date after the last generated language pack. We
219-ignore the xpi entries because those are outside the scope of this test.
220+had the modification date after the last generated language pack.
221
222 >>> examine_tarfile(tarfile)
223 | - | rosetta-grumpy
224 | - | rosetta-grumpy/cy
225 | - | rosetta-grumpy/cy/LC_MESSAGES
226 | 21 | rosetta-grumpy/cy/LC_MESSAGES/test.po
227- | 2 | rosetta-grumpy/mapping.txt
228+ | 1 | rosetta-grumpy/mapping.txt
229 | 1 | rosetta-grumpy/timestamp.txt
230- | - | rosetta-grumpy/xpi
231- | - | rosetta-grumpy/xpi/firefox
232- | bin | rosetta-grumpy/xpi/firefox/en-US.xpi
233- | 94 | rosetta-grumpy/xpi/firefox/en.po
234- | 102 | rosetta-grumpy/xpi/firefox/es.po
235
236 There is another situation where a translation file is exported again as
237 part of a language pack update, even without being changed. It is re-
238@@ -488,13 +357,8 @@ template has. That's why we get both translations:
239 | - | rosetta-grumpy/es
240 | - | rosetta-grumpy/es/LC_MESSAGES
241 | 21 | rosetta-grumpy/es/LC_MESSAGES/test.po
242- | 2 | rosetta-grumpy/mapping.txt
243+ | 1 | rosetta-grumpy/mapping.txt
244 | 1 | rosetta-grumpy/timestamp.txt
245- | - | rosetta-grumpy/xpi
246- | - | rosetta-grumpy/xpi/firefox
247- | bin | rosetta-grumpy/xpi/firefox/en-US.xpi
248- | 94 | rosetta-grumpy/xpi/firefox/en.po
249- | 102 | rosetta-grumpy/xpi/firefox/es.po
250
251
252 Script arguments and concurrency
253@@ -543,7 +407,6 @@ different distribution and series combinations.
254 /var/lock/launchpad-language-pack-exporter__ubuntu__hoary.lock
255 INFO Exporting translations for series hoary of distribution ubuntu.
256 INFO Number of PO files to export: 12
257- INFO Exporting XPI template files.
258 INFO Adding timestamp file
259 INFO Adding mapping file
260 INFO Done.
261diff --git a/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz b/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz
262index 4473202..5a26175 100644
263Binary files a/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz and b/lib/lp/translations/doc/sourcepackagerelease-translations.tar.gz differ
264diff --git a/lib/lp/translations/doc/sourcepackagerelease-translations.txt b/lib/lp/translations/doc/sourcepackagerelease-translations.txt
265index 039d644..0a03aff 100644
266--- a/lib/lp/translations/doc/sourcepackagerelease-translations.txt
267+++ b/lib/lp/translations/doc/sourcepackagerelease-translations.txt
268@@ -83,11 +83,10 @@ And the queue should have 2 entries, with exactly the same contents.
269 >>> queue_entries = translation_import_queue.getAllEntries(target=sp_test)
270
271 >>> queue_entries.count()
272- 2
273+ 1
274
275 >>> for entry in queue_entries:
276 ... print(entry.path, entry.importer.name)
277- something/en-US.xpi maria
278 po/es.po maria
279
280 Commit, so the uploaded translations become available to the scripts.
281diff --git a/lib/lp/translations/scripts/language_pack.py b/lib/lp/translations/scripts/language_pack.py
282index 34d28c7..77fe80a 100644
283--- a/lib/lp/translations/scripts/language_pack.py
284+++ b/lib/lp/translations/scripts/language_pack.py
285@@ -32,9 +32,6 @@ from lp.services.librarian.interfaces.client import (
286 from lp.services.tarfile_helpers import LaunchpadWriteTarFile
287 from lp.translations.enums import LanguagePackType
288 from lp.translations.interfaces.languagepack import ILanguagePackSet
289-from lp.translations.interfaces.translationfileformat import (
290- TranslationFileFormat,
291- )
292 from lp.translations.interfaces.vpoexport import IVPOExportSet
293
294
295@@ -93,7 +90,6 @@ def export(distroseries, component, update, force_utf8, logger):
296
297 # XXX JeroenVermeulen 2008-02-06: Is there anything here that we can unify
298 # with the export-queue code?
299- xpi_templates_to_export = set()
300 path_prefix = 'rosetta-%s' % distroseries.name
301
302 pofiles = export_set.get_distroseries_pofiles(
303@@ -137,14 +133,7 @@ def export(distroseries, component, update, force_utf8, logger):
304
305 domain = potemplate.translation_domain.encode('ascii')
306 code = pofile.getFullLanguageCode().encode('UTF-8')
307-
308- if potemplate.source_file_format == TranslationFileFormat.XPI:
309- xpi_templates_to_export.add(potemplate)
310- path = os.path.join(
311- path_prefix, 'xpi', domain, '%s.po' % code)
312- else:
313- path = os.path.join(
314- path_prefix, code, 'LC_MESSAGES', '%s.po' % domain)
315+ path = os.path.join(path_prefix, code, 'LC_MESSAGES', '%s.po' % domain)
316
317 try:
318 # We don't want obsolete entries here, it makes no sense for a
319@@ -160,19 +149,6 @@ def export(distroseries, component, update, force_utf8, logger):
320
321 store.invalidate(pofile)
322
323- logger.info("Exporting XPI template files.")
324- librarian_client = getUtility(ILibrarianClient)
325- for template in xpi_templates_to_export:
326- if template.source_file is None:
327- logger.warning(
328- "%s doesn't have source file registered." % potemplate.title)
329- continue
330- domain = template.translation_domain.encode('ascii')
331- archive.add_file(
332- os.path.join(path_prefix, 'xpi', domain, 'en-US.xpi'),
333- librarian_client.getFileByAlias(
334- template.source_file.id).read())
335-
336 logger.info("Adding timestamp file")
337 # Is important that the timestamp contain the date when the export
338 # started, not when it finished because that notes how old is the
339diff --git a/lib/lp/translations/scripts/tests/test_validate_translations_file.py b/lib/lp/translations/scripts/tests/test_validate_translations_file.py
340index 8311718..4f1fe28 100644
341--- a/lib/lp/translations/scripts/tests/test_validate_translations_file.py
342+++ b/lib/lp/translations/scripts/tests/test_validate_translations_file.py
343@@ -16,9 +16,6 @@ from lp.translations.scripts.validate_translations_file import (
344 UnknownFileType,
345 ValidateTranslationsFile,
346 )
347-from lp.translations.utilities.tests.xpi_helpers import (
348- get_en_US_xpi_file_to_import,
349- )
350
351
352 class TestValidateTranslationsFile(TestCase):
353@@ -45,30 +42,6 @@ class TestValidateTranslationsFile(TestCase):
354 self.assertRaises(
355 UnknownFileType, validator._validateContent, 'foo.bar', 'content')
356
357- def test_validate_dtd_good(self):
358- validator = self._makeValidator()
359- result = validator._validateContent(
360- 'test.dtd', '<!ENTITY a.translatable.string "A string">\n')
361- self.assertTrue(result)
362-
363- def test_validate_dtd_bad(self):
364- validator = self._makeValidator()
365- result = validator._validateContent(
366- 'test.dtd', '<!ENTIT etc.')
367- self.assertFalse(result)
368-
369- def test_validate_xpi_manifest_good(self):
370- validator = self._makeValidator()
371- result = validator._validateContent(
372- 'chrome.manifest', 'locale foo nl jar:chrome/nl.jar!/foo/')
373- self.assertTrue(result)
374-
375- def test_validate_xpi_manifest_bad(self):
376- # XPI manifests must not begin with newline.
377- validator = self._makeValidator()
378- result = validator._validateContent('chrome.manifest', '\nlocale')
379- self.assertFalse(result)
380-
381 def test_validate_po_good(self):
382 validator = self._makeValidator()
383 result = validator._validateContent('nl.po', self._strip(r"""
384@@ -110,17 +83,6 @@ class TestValidateTranslationsFile(TestCase):
385 result = validator._validateContent('test.pot', 'garble')
386 self.assertFalse(result)
387
388- def test_validate_xpi_good(self):
389- validator = self._makeValidator()
390- xpi_content = get_en_US_xpi_file_to_import('en-US').read()
391- result = validator._validateContent('pl.xpi', xpi_content)
392- self.assertTrue(result)
393-
394- def test_validate_xpi_bad(self):
395- validator = self._makeValidator()
396- result = validator._validateContent('de.xpi', 'garble')
397- self.assertFalse(result)
398-
399 def test_script(self):
400 test_input = os.path.join(self._findTestData(), 'minimal.pot')
401 script = 'scripts/rosetta/validate-translations-file.py'
402diff --git a/lib/lp/translations/scripts/validate_translations_file.py b/lib/lp/translations/scripts/validate_translations_file.py
403index bcf2ff6..f9d9303 100644
404--- a/lib/lp/translations/scripts/validate_translations_file.py
405+++ b/lib/lp/translations/scripts/validate_translations_file.py
406@@ -8,18 +8,12 @@ __all__ = [
407 'ValidateTranslationsFile',
408 ]
409
410-from cStringIO import StringIO
411 import logging
412 from optparse import OptionParser
413 import os.path
414
415 from lp.services import scripts
416 from lp.translations.utilities.gettext_po_parser import POParser
417-from lp.translations.utilities.mozilla_dtd_parser import DtdFile
418-from lp.translations.utilities.mozilla_xpi_importer import (
419- MozillaZipImportParser,
420- )
421-from lp.translations.utilities.xpi_manifest import XpiManifest
422
423
424 class UnknownFileType(Exception):
425@@ -31,37 +25,19 @@ def validate_unknown_file_type(filename, content):
426 raise UnknownFileType("Unrecognized file type for '%s'." % filename)
427
428
429-def validate_dtd(filename, content):
430- """Validate XPI DTD file."""
431- DtdFile(filename, filename, content)
432-
433-
434 def validate_po(filename, content):
435 """Validate a gettext PO or POT file."""
436 POParser().parse(content)
437
438
439-def validate_xpi(filename, content):
440- """Validate an XPI archive."""
441- MozillaZipImportParser(filename, StringIO(content))
442-
443-
444-def validate_xpi_manifest(filename, content):
445- """Validate XPI manifest."""
446- XpiManifest(content)
447-
448-
449 class ValidateTranslationsFile:
450 """Parse translations files to see if they are well-formed."""
451
452 name = 'validate-translations-file'
453
454 validators = {
455- 'dtd': validate_dtd,
456- 'manifest': validate_xpi_manifest,
457 'po': validate_po,
458 'pot': validate_po,
459- 'xpi': validate_xpi,
460 }
461
462 def __init__(self, test_args=None):
463diff --git a/lib/lp/translations/stories/standalone/xx-translations-xpi-import.txt b/lib/lp/translations/stories/standalone/xx-translations-xpi-import.txt
464deleted file mode 100644
465index 2f0472b..0000000
466--- a/lib/lp/translations/stories/standalone/xx-translations-xpi-import.txt
467+++ /dev/null
468@@ -1,55 +0,0 @@
469-= Demonstrate import of Firefox XPI file =
470-
471-To import translations into Firefox product, we must first import en-US.xpi
472-file, which is an equivalent of a PO Template.
473-
474-Lets start with Firefox product inside trunk revision.
475-
476- >>> browser = setupBrowser(auth='Basic carlos@canonical.com:test')
477- >>> browser.open('http://translations.launchpad.test/firefox/trunk')
478-
479-Since we still don't have any POTemplates assigned, we must use the general
480-translations upload link.
481-
482- >>> browser.getLink('upload').click()
483- >>> print(browser.url)
484- http://translations.launchpad.test/firefox/trunk/+translations-upload
485-
486-Get the XPI file we are going to upload.
487-
488- >>> from lp.translations.utilities.tests import test_xpi_import
489- >>> xpifile = test_xpi_import.get_en_US_xpi_file_to_import('en-US')
490-
491-Now, lets upload this file.
492-
493- >>> browser.getControl('File:').add_file(
494- ... xpifile, 'application/zip', 'en-US.xpi')
495- >>> browser.getControl('Upload').click()
496-
497- >>> print(browser.url)
498- http://translations.launchpad.test/firefox/trunk/+translations-upload
499- >>> for tag in find_tags_by_class(browser.contents, 'message'):
500- ... print(extract_text(tag.renderContents()))
501- Thank you for your upload. It will be automatically reviewed...
502-
503-Lets check the import queue to edit this entry and set the name.
504-
505- >>> browser.getLink('Translation Import Queue').click()
506- >>> print(browser.getLink(url='en-US.xpi').url)
507- http://.../en-US.xpi
508- >>> browser.getLink(url='/+imports/').click()
509- >>> print(browser.url)
510- http://translations.launchpad.test/+imports/...
511- >>> qid = int(browser.url.rsplit('/', 1)[-1])
512-
513-All new entries need to get a template name to identify them in the context
514-where will be imported. In this case, it's 'firefox'.
515-
516- >>> browser.getControl('File Type').value = ['POT']
517- >>> browser.getControl('Name').value = 'firefox'
518- >>> browser.getControl('Translation domain').value = 'firefox'
519- >>> browser.getControl('Approve').click()
520- >>> print(browser.url)
521- http://translations.launchpad.test/firefox/trunk/+imports
522- >>> browser.getControl(name='field.status_%d' % qid).value
523- ['APPROVED']
524diff --git a/lib/lp/translations/tests/test_autoapproval.py b/lib/lp/translations/tests/test_autoapproval.py
525index a0ec5e4..03c60b3 100644
526--- a/lib/lp/translations/tests/test_autoapproval.py
527+++ b/lib/lp/translations/tests/test_autoapproval.py
528@@ -1179,10 +1179,7 @@ class TestAutoBlocking(TestCaseWithFactory):
529 translation target as `same_target_as`. This lets you create an
530 entry for the same translation target as another one.
531 """
532- if suffix == '.xpi':
533- basename = 'en-US'
534- else:
535- basename = self.factory.getUniqueString()
536+ basename = self.factory.getUniqueString()
537
538 filename = basename + suffix
539 if directory is None:
540@@ -1227,26 +1224,6 @@ class TestAutoBlocking(TestCaseWithFactory):
541
542 self.assertEqual(len(old_blocklist), len(new_blocklist))
543
544- def test_getBlockableDirectories_checks_xpi_templates(self):
545- old_blocklist = self.queue._getBlockableDirectories()
546-
547- self._makeTemplateEntry(
548- suffix='.xpi', status=RosettaImportStatus.BLOCKED)
549-
550- new_blocklist = self.queue._getBlockableDirectories()
551-
552- self.assertEqual(len(old_blocklist) + 1, len(new_blocklist))
553-
554- def test_getBlockableDirectories_ignores_xpi_translations(self):
555- old_blocklist = self.queue._getBlockableDirectories()
556-
557- self._makeTranslationEntry(
558- 'lt.xpi', status=RosettaImportStatus.BLOCKED)
559-
560- new_blocklist = self.queue._getBlockableDirectories()
561-
562- self.assertEqual(len(old_blocklist), len(new_blocklist))
563-
564 def test_isBlockable_none(self):
565 blocklist = self.queue._getBlockableDirectories()
566 entry = self._makeTranslationEntry('nl.po')
567diff --git a/lib/lp/translations/tests/test_potmsgset.py b/lib/lp/translations/tests/test_potmsgset.py
568index c5aaae5..6f0a71b 100644
569--- a/lib/lp/translations/tests/test_potmsgset.py
570+++ b/lib/lp/translations/tests/test_potmsgset.py
571@@ -15,7 +15,6 @@ from zope.security.proxy import removeSecurityProxy
572
573 from lp.app.enums import ServiceUsage
574 from lp.app.interfaces.launchpad import ILaunchpadCelebrities
575-from lp.services.propertycache import get_property_cache
576 from lp.testing import TestCaseWithFactory
577 from lp.testing.layers import (
578 DatabaseFunctionalLayer,
579@@ -82,55 +81,17 @@ class TestTranslationSharedPOTMsgSets(TestCaseWithFactory):
580 self.assertEqual(devel_potmsgsets, [self.potmsgset])
581 self.assertEqual(devel_potmsgsets, stable_potmsgsets)
582
583- def test_POTMsgSetInIncompatiblePOTemplates(self):
584- # Make sure a POTMsgSet cannot be used in two POTemplates with
585- # different incompatible source_file_format (like XPI and PO).
586- self.devel_potemplate.source_file_format = TranslationFileFormat.PO
587- self.stable_potemplate.source_file_format = TranslationFileFormat.XPI
588-
589- potmsgset = self.potmsgset
590-
591- self.assertRaises(POTMsgSetInIncompatibleTemplatesError,
592- potmsgset.setSequence, self.stable_potemplate, 1)
593-
594- # If the two file formats are compatible, it works.
595- self.stable_potemplate.source_file_format = (
596- TranslationFileFormat.KDEPO)
597- potmsgset.setSequence(self.stable_potemplate, 1)
598-
599- devel_potmsgsets = list(self.devel_potemplate.getPOTMsgSets())
600- stable_potmsgsets = list(self.stable_potemplate.getPOTMsgSets())
601- self.assertEqual(devel_potmsgsets, stable_potmsgsets)
602-
603- # We hack the POTemplate manually to make data inconsistent
604- # in database.
605- self.stable_potemplate.source_file_format = TranslationFileFormat.XPI
606- transaction.commit()
607-
608- # We remove the security proxy to be able to get a callable for
609- # properties like `uses_english_msgids` and `singular_text`.
610- naked_potmsgset = removeSecurityProxy(potmsgset)
611-
612- self.assertRaises(POTMsgSetInIncompatibleTemplatesError,
613- naked_potmsgset.__getattribute__,
614- "uses_english_msgids")
615-
616- self.assertRaises(POTMsgSetInIncompatibleTemplatesError,
617- naked_potmsgset.__getattribute__, "singular_text")
618-
619 def test_POTMsgSetUsesEnglishMsgids(self):
620 """Test that `uses_english_msgids` property works correctly."""
621+ # XXX cjwatson 2020-03-01: We currently have no file formats with
622+ # importers for which uses_english_msgids would be False (XPI used
623+ # to be such a case).
624
625 # Gettext PO format uses English strings as msgids.
626 self.devel_potemplate.source_file_format = TranslationFileFormat.PO
627 transaction.commit()
628 self.assertTrue(self.potmsgset.uses_english_msgids)
629
630- # Mozilla XPI format doesn't use English strings as msgids.
631- self.devel_potemplate.source_file_format = TranslationFileFormat.XPI
632- transaction.commit()
633- self.assertFalse(self.potmsgset.uses_english_msgids)
634-
635 def test_getCurrentTranslationMessageOrDummy_returns_upstream_tm(self):
636 pofile = self.factory.makePOFile('nl')
637 message = self.factory.makeCurrentTranslationMessage(pofile=pofile)
638@@ -1004,110 +965,6 @@ class TestPOTMsgSetText(TestCaseWithFactory):
639 english_msgid, TranslationFileFormat.PO)
640 self.assertEqual(english_msgid, potmsgset.singular_text)
641
642- def test_singular_text_xpi(self):
643- # Mozilla XPI format uses English strings as msgids if no English
644- # pofile exists.
645- symbolic_msgid = self.factory.getUniqueString()
646- potmsgset = self._makePOTMsgSet(
647- symbolic_msgid, TranslationFileFormat.XPI)
648- self.assertEqual(symbolic_msgid, potmsgset.singular_text)
649-
650- def test_singular_text_xpi_english(self):
651- # Mozilla XPI format uses English strings as msgids if no English
652- # pofile exists.
653- # POTMsgSet singular_text is read from a shared English translation.
654- symbolic_msgid = self.factory.getUniqueString()
655- english_msgid = self.factory.getUniqueString()
656- potmsgset, potemplate = self._makePOTMsgSetAndTemplate(
657- symbolic_msgid, TranslationFileFormat.XPI)
658- en_pofile = self.factory.makePOFile('en', potemplate)
659- self.factory.makeCurrentTranslationMessage(
660- pofile=en_pofile, potmsgset=potmsgset,
661- translations=[english_msgid])
662-
663- self.assertEqual(english_msgid, potmsgset.singular_text)
664-
665- def test_singular_text_xpi_english_diverged(self):
666- # A diverged (translation.potemplate != None) English translation
667- # is not used as a singular_text.
668- symbolic_msgid = self.factory.getUniqueString()
669- english_msgid = self.factory.getUniqueString()
670- diverged_msgid = self.factory.getUniqueString()
671- potmsgset, potemplate = self._makePOTMsgSetAndTemplate(
672- symbolic_msgid, TranslationFileFormat.XPI)
673- en_pofile = self.factory.makePOFile('en', potemplate)
674- self.factory.makeCurrentTranslationMessage(
675- pofile=en_pofile, potmsgset=potmsgset,
676- translations=[english_msgid])
677- self.factory.makeCurrentTranslationMessage(
678- pofile=en_pofile, potmsgset=potmsgset,
679- translations=[diverged_msgid], diverged=True)
680-
681- self.assertEqual(english_msgid, potmsgset.singular_text)
682-
683- def _setUpSharingWithUbuntu(self):
684- """Create a potmsgset shared in upstream and Ubuntu."""
685- productseries = self.factory.makeProductSeries()
686-
687- # Create the source package that this product is linked to.
688- distroseries = self.factory.makeUbuntuDistroSeries()
689- distroseries.distribution.translation_focus = distroseries
690- sourcepackagename = self.factory.makeSourcePackageName()
691- sourcepackage = self.factory.makeSourcePackage(
692- distroseries=distroseries, sourcepackagename=sourcepackagename)
693- sourcepackage.setPackaging(productseries, self.factory.makePerson())
694-
695- # Create two sharing templates.
696- self.potmsgset, upstream_potemplate = self._makePOTMsgSetAndTemplate(
697- None, TranslationFileFormat.XPI, productseries)
698- ubuntu_potemplate = self.factory.makePOTemplate(
699- distroseries=distroseries, sourcepackagename=sourcepackagename,
700- name=upstream_potemplate.name)
701- ubuntu_potemplate.source_file_format = TranslationFileFormat.XPI
702- self.potmsgset.setSequence(ubuntu_potemplate, 1)
703-
704- # The pofile is automatically created for all sharing templates.
705- self.upstream_pofile = self.factory.makePOFile(
706- 'en', upstream_potemplate, create_sharing=True)
707- self.ubuntu_pofile = ubuntu_potemplate.getPOFileByLang('en')
708- self.assertIsNot(None, self.ubuntu_pofile)
709-
710- def test_singular_text_xpi_english_uses_upstream(self):
711- # POTMsgSet singular_text is read from the upstream English
712- # translation.
713- self._setUpSharingWithUbuntu()
714- # Create different "English translations" for this potmsgset.
715- ubuntu_msgid = self.factory.getUniqueString()
716- upstream_msgid = self.factory.getUniqueString()
717-
718- self.factory.makeCurrentTranslationMessage(
719- pofile=self.upstream_pofile, potmsgset=self.potmsgset,
720- translations=[upstream_msgid])
721- self.factory.makeCurrentTranslationMessage(
722- pofile=self.ubuntu_pofile, potmsgset=self.potmsgset,
723- translations=[ubuntu_msgid])
724-
725- # makeCurrentTranslationMessage calls singular_text and caches the
726- # upstream msgid, causing the test to pass even without the
727- # Ubuntu message being present.
728- del get_property_cache(self.potmsgset).singular_text
729- self.assertEqual(upstream_msgid, self.potmsgset.singular_text)
730-
731- def test_singular_text_xpi_english_falls_back_to_ubuntu(self):
732- # POTMsgSet singular_text is read from the Ubuntu English
733- # translation if no upstream one exists. This is a safeguard against
734- # old or broken data.
735- self._setUpSharingWithUbuntu()
736-
737- # Create different "English translations" for this potmsgset.
738- ubuntu_msgid = self.factory.getUniqueString()
739-
740- self.factory.makeCurrentTranslationMessage(
741- pofile=self.ubuntu_pofile, potmsgset=self.potmsgset,
742- translations=[ubuntu_msgid])
743-
744- self.assertEqual(ubuntu_msgid, self.potmsgset.singular_text)
745-
746
747 class TestPOTMsgSetTranslationCredits(TestCaseWithFactory):
748 """Test methods related to TranslationCredits."""
749diff --git a/lib/lp/translations/tests/test_translationbranchapprover.py b/lib/lp/translations/tests/test_translationbranchapprover.py
750index 4f309c0..43e4b36 100644
751--- a/lib/lp/translations/tests/test_translationbranchapprover.py
752+++ b/lib/lp/translations/tests/test_translationbranchapprover.py
753@@ -111,17 +111,6 @@ class TestTranslationBranchApprover(TestCaseWithFactory):
754 self.assertEqual(
755 translation_domain, entry.potemplate.translation_domain)
756
757- def test_new_template_domain_with_xpi(self):
758- # For xpi files, template files are always called "en-US.xpi" so
759- # the approver won't use that string for a domain. It'll fall
760- # back to the next possibility, which is the directory.
761- translation_domain = self.factory.getUniqueString()
762- template_path = translation_domain + '/en-US.xpi'
763- entry = self._upload_file(template_path)
764- self._createApprover(template_path).approve(entry)
765- self.assertEqual(
766- translation_domain, entry.potemplate.translation_domain)
767-
768 def test_template_name(self):
769 # The name is derived from the file name and must be a valid name.
770 translation_domain = (u'Invalid-Name_with illegal#Characters')
771diff --git a/lib/lp/translations/tests/test_translationimportqueue.py b/lib/lp/translations/tests/test_translationimportqueue.py
772index e3b3e4c..c4ed52e 100644
773--- a/lib/lp/translations/tests/test_translationimportqueue.py
774+++ b/lib/lp/translations/tests/test_translationimportqueue.py
775@@ -407,7 +407,6 @@ class TestTranslationImportQueue(TestCaseWithFactory):
776 files = dict((
777 self._makeFile('pot'),
778 self._makeFile('po'),
779- self._makeFile('xpi'),
780 ))
781 tarfile_content = LaunchpadWriteTarFile.files_to_stream(files)
782 self.import_queue.addOrUpdateEntriesFromTarball(
783diff --git a/lib/lp/translations/utilities/mozilla_dtd_parser.py b/lib/lp/translations/utilities/mozilla_dtd_parser.py
784deleted file mode 100644
785index bdc4782..0000000
786--- a/lib/lp/translations/utilities/mozilla_dtd_parser.py
787+++ /dev/null
788@@ -1,150 +0,0 @@
789-# Copyright 2010 Canonical Ltd. This software is licensed under the
790-# GNU Affero General Public License version 3 (see the file LICENSE).
791-
792-"""Importer for DTD files as found in XPI archives."""
793-
794-__metaclass__ = type
795-__all__ = [
796- 'DtdFile'
797- ]
798-
799-from old_xmlplus.parsers.xmlproc import (
800- dtdparser,
801- utils,
802- xmldtd,
803- )
804-
805-from lp.translations.interfaces.translationimporter import (
806- TranslationFormatInvalidInputError,
807- TranslationFormatSyntaxError,
808- )
809-from lp.translations.interfaces.translations import TranslationConstants
810-from lp.translations.utilities.translation_common_format import (
811- TranslationMessageData,
812- )
813-
814-
815-class MozillaDtdConsumer(xmldtd.WFCDTD):
816- """Mozilla DTD translatable message parser.
817-
818- msgids are stored as entities. This class extracts it along
819- with translations, comments and source references.
820- """
821- def __init__(self, parser, filename, chrome_path, messages):
822- self.started = False
823- self.last_comment = None
824- self.chrome_path = chrome_path
825- self.messages = messages
826- self.filename = filename
827- xmldtd.WFCDTD.__init__(self, parser)
828-
829- def dtd_start(self):
830- """See `xmldtd.WFCDTD`."""
831- self.started = True
832-
833- def dtd_end(self):
834- """See `xmldtd.WFCDTD`."""
835- self.started = False
836-
837- def handle_comment(self, contents):
838- """See `xmldtd.WFCDTD`."""
839- if not self.started:
840- return
841-
842- if self.last_comment is not None:
843- self.last_comment += contents
844- elif len(contents) > 0:
845- self.last_comment = contents
846-
847- if self.last_comment and not self.last_comment.endswith('\n'):
848- # Comments must end always with a new line.
849- self.last_comment += '\n'
850-
851- def new_general_entity(self, name, value):
852- """See `xmldtd.WFCDTD`."""
853- if not self.started:
854- return
855-
856- message = TranslationMessageData()
857- message.msgid_singular = name
858- # CarlosPerelloMarin 20070326: xmldtd parser does an inline
859- # parsing which means that the content is all in a single line so we
860- # don't have a way to show the line number with the source reference.
861- message.file_references_list = ["%s(%s)" % (self.filename, name)]
862- message.addTranslation(TranslationConstants.SINGULAR_FORM, value)
863- message.singular_text = value
864- message.context = self.chrome_path
865- message.source_comment = self.last_comment
866- self.messages.append(message)
867- self.started += 1
868- self.last_comment = None
869-
870-
871-class DtdErrorHandler(utils.ErrorCounter):
872- """Error handler for the DTD parser."""
873- filename = None
874-
875- def error(self, msg):
876- raise TranslationFormatSyntaxError(
877- filename=self.filename, message=msg)
878-
879- def fatal(self, msg):
880- raise TranslationFormatInvalidInputError(
881- filename=self.filename, message=msg)
882-
883-
884-class DummyDtdFile:
885- """"File" returned when DTD SYSTEM entity tries to include a file."""
886- done = False
887-
888- def read(self, *args, **kwargs):
889- """Minimally satisfy attempt to read an included DTD file."""
890- if self.done:
891- return ''
892- else:
893- self.done = True
894- return '<!-- SYSTEM entities not supported. -->'
895-
896- def close(self):
897- """Satisfy attempt to close file."""
898- pass
899-
900-
901-class DtdInputSourceFactoryStub:
902- """Replace the class the DTD parser uses to include other DTD files."""
903-
904- def create_input_source(self, sysid):
905- """Minimally satisfy attempt to open an included DTD file.
906-
907- This is called when the DTD parser hits a SYSTEM entity.
908- """
909- return DummyDtdFile()
910-
911-
912-class DtdFile:
913- """Class for reading translatable messages from a .dtd file.
914-
915- It uses DTDParser which fills self.messages with parsed messages.
916- """
917- def __init__(self, filename, chrome_path, content):
918- self.messages = []
919- self.filename = filename
920- self.chrome_path = chrome_path
921-
922- # .dtd files are supposed to be using UTF-8 encoding, if the file is
923- # using another encoding, it's against the standard so we reject it
924- try:
925- content = content.decode('utf-8')
926- except UnicodeDecodeError:
927- raise TranslationFormatInvalidInputError(
928- 'Content is not valid UTF-8 text')
929-
930- error_handler = DtdErrorHandler()
931- error_handler.filename = filename
932-
933- parser = dtdparser.DTDParser()
934- parser.set_error_handler(error_handler)
935- parser.set_inputsource_factory(DtdInputSourceFactoryStub())
936- dtd = MozillaDtdConsumer(parser, filename, chrome_path, self.messages)
937- parser.set_dtd_consumer(dtd)
938- parser.parse_string(content)
939diff --git a/lib/lp/translations/utilities/mozilla_xpi_importer.py b/lib/lp/translations/utilities/mozilla_xpi_importer.py
940deleted file mode 100644
941index 3c76c79..0000000
942--- a/lib/lp/translations/utilities/mozilla_xpi_importer.py
943+++ /dev/null
944@@ -1,423 +0,0 @@
945-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
946-# GNU Affero General Public License version 3 (see the file LICENSE).
947-
948-__metaclass__ = type
949-
950-__all__ = [
951- 'MozillaXpiImporter',
952- 'MozillaZipImportParser',
953- ]
954-
955-from cStringIO import StringIO
956-import textwrap
957-
958-from zope.component import getUtility
959-from zope.interface import implementer
960-
961-from lp.services.librarian.interfaces.client import ILibrarianClient
962-from lp.translations.interfaces.translationfileformat import (
963- TranslationFileFormat,
964- )
965-from lp.translations.interfaces.translationimporter import (
966- ITranslationFormatImporter,
967- TranslationFormatInvalidInputError,
968- TranslationFormatSyntaxError,
969- )
970-from lp.translations.interfaces.translations import TranslationConstants
971-from lp.translations.utilities.mozilla_dtd_parser import DtdFile
972-from lp.translations.utilities.mozilla_zip import MozillaZipTraversal
973-from lp.translations.utilities.translation_common_format import (
974- TranslationFileData,
975- TranslationMessageData,
976- )
977-from lp.translations.utilities.xpi_header import XpiHeader
978-
979-
980-def add_source_comment(message, comment):
981- """Add the given comment inside message.source_comment."""
982- if message.source_comment:
983- message.source_comment += comment
984- else:
985- message.source_comment = comment
986-
987- if not message.source_comment.endswith('\n'):
988- message.source_comment += '\n'
989-
990-
991-class MozillaZipImportParser(MozillaZipTraversal):
992- """XPI and jar parser for import purposes.
993-
994- Looks for DTD and properties files, and parses them for messages.
995- All messages found are left in `self.messages`.
996- """
997-
998- # List of ITranslationMessageData representing messages found.
999- messages = None
1000-
1001- def _begin(self):
1002- """Overridable hook for `MozillaZipTraversal`."""
1003- self.messages = []
1004-
1005- def _finish(self):
1006- """Overridable hook for `MozillaZipTraversal`."""
1007- # Eliminate duplicate messages.
1008- seen_messages = set()
1009- deletions = []
1010- for index, message in enumerate(self.messages):
1011- identifier = (message.msgid_singular, message.context)
1012- if identifier in seen_messages:
1013- # This message is a duplicate. Mark it for removal.
1014- deletions.append(index)
1015- else:
1016- seen_messages.add(identifier)
1017- for index in reversed(deletions):
1018- del self.messages[index]
1019-
1020- for message in self.messages:
1021- message.file_references = ', '.join(message.file_references_list)
1022-
1023- def _processTranslatableFile(self, entry, locale_code, xpi_path,
1024- chrome_path, filename_suffix):
1025- """Overridable hook for `MozillaZipTraversal`.
1026-
1027- This implementation is only interested in DTD and properties
1028- files.
1029- """
1030- if filename_suffix == '.dtd':
1031- parser = DtdFile
1032- elif filename_suffix == '.properties':
1033- parser = PropertyFile
1034- else:
1035- # We're not interested in other file types here.
1036- return
1037-
1038- # Parse file, subsume its messages.
1039- content = self.archive.read(entry)
1040- parsed_file = parser(
1041- filename=xpi_path, chrome_path=chrome_path, content=content)
1042- if parsed_file is not None:
1043- self.extend(parsed_file.messages)
1044-
1045- def _isTemplate(self):
1046- """Is this a template?"""
1047- name = self.filename
1048- return name is not None and name.startswith('en-US.xpi')
1049-
1050- def _processNestedJar(self, zip_instance):
1051- """Overridable hook for `MozillaZipTraversal`.
1052-
1053- This implementation complements `self.messages` with those found in
1054- the jar file we just parsed.
1055- """
1056- self.extend(zip_instance.messages)
1057-
1058- def _isCommandKeyMessage(self, message):
1059- """Whether the message represents a command key shortcut."""
1060- return (
1061- self._isTemplate() and
1062- message.translations and (
1063- message.msgid_singular.endswith('.commandkey') or
1064- message.msgid_singular.endswith('.key')))
1065-
1066- def _isAccessKeyMessage(self, message):
1067- """Whether the message represents an access key shortcut."""
1068- return (
1069- self._isTemplate() and
1070- message.translations and (
1071- message.msgid_singular.endswith('.accesskey')))
1072-
1073- def extend(self, newdata):
1074- """Complement `self.messages` with messages found in contained file.
1075-
1076- :param newdata: a sequence representing the messages found in a
1077- contained file.
1078- """
1079- for message in newdata:
1080- # Special case accesskeys and commandkeys:
1081- # these are single letter messages, lets display
1082- # the value as a source comment.
1083- if self._isCommandKeyMessage(message):
1084- comment = u'\n'.join(textwrap.wrap(
1085- u"""Select the shortcut key that you want to use. It
1086- should be translated, but often shortcut keys (for
1087- example Ctrl + KEY) are not changed from the original. If
1088- a translation already exists, please don't change it if
1089- you are not sure about it. Please find the context of
1090- the key from the end of the 'Located in' text below."""))
1091- add_source_comment(message, comment)
1092- elif self._isAccessKeyMessage(message):
1093- comment = u'\n'.join(textwrap.wrap(
1094- u"""Select the access key that you want to use. These have
1095- to be translated in a way that the selected character is
1096- present in the translated string of the label being
1097- referred to, for example 'i' in 'Edit' menu item in
1098- English. If a translation already exists, please don't
1099- change it if you are not sure about it. Please find the
1100- context of the key from the end of the 'Located in' text
1101- below."""))
1102- add_source_comment(message, comment)
1103- self.messages.append(message)
1104-
1105-
1106-def valid_property_msgid(msgid):
1107- """Whether the given msgid follows the restrictions to be valid.
1108-
1109- Checks done are:
1110- - It cannot have white spaces.
1111- """
1112- return u' ' not in msgid
1113-
1114-
1115-class PropertyFile:
1116- """Class for reading translatable messages from a .properties file.
1117-
1118- The file format is described at:
1119- http://www.mozilla.org/projects/l10n/mlp_chrome.html#text
1120- """
1121-
1122- license_block_text = u'END LICENSE BLOCK'
1123-
1124- def __init__(self, filename, chrome_path, content):
1125- """Constructs a dictionary from a .properties file.
1126-
1127- :arg filename: The file name where the content came from.
1128- :arg content: The file content that we want to parse.
1129- """
1130- self.filename = filename
1131- self.chrome_path = chrome_path
1132- self.messages = []
1133-
1134- # Parse the content.
1135- self.parse(content)
1136-
1137- def parse(self, content):
1138- """Parse given content as a property file.
1139-
1140- Once the parse is done, self.messages has a list of the available
1141- `ITranslationMessageData`s.
1142- """
1143-
1144- # .properties files are supposed to be unicode-escaped, but we know
1145- # that there are some .xpi language packs that instead, use UTF-8.
1146- # That's against the specification, but Mozilla applications accept
1147- # it anyway, so we try to support it too.
1148- # To do this support, we read the text as being in UTF-8
1149- # because unicode-escaped looks like ASCII files.
1150- try:
1151- content = content.decode('utf-8')
1152- except UnicodeDecodeError:
1153- raise TranslationFormatInvalidInputError(
1154- 'Content is not valid unicode-escaped text')
1155-
1156- line_num = 0
1157- is_multi_line_comment = False
1158- last_comment = None
1159- last_comment_line_num = 0
1160- ignore_comment = False
1161- is_message = False
1162- translation = u''
1163- for line in content.splitlines():
1164- # Now, to "normalize" all to the same encoding, we encode to
1165- # unicode-escape first, and then decode it to unicode
1166- # XXX: Danilo 2006-08-01: we _might_ get performance
1167- # improvements if we reimplement this to work directly,
1168- # though, it will be hard to beat C-based de/encoder.
1169- # This call unescapes everything so we don't need to care about
1170- # quotes escaping.
1171- try:
1172- string = line.encode('raw-unicode_escape')
1173- line = string.decode('unicode_escape')
1174- except UnicodeDecodeError as exception:
1175- raise TranslationFormatInvalidInputError(
1176- filename=self.filename, line_number=line_num,
1177- message=str(exception))
1178-
1179- line_num += 1
1180- if not is_multi_line_comment:
1181- # Remove any white space before the useful data, like
1182- # ' # foo'.
1183- line = line.lstrip()
1184- if len(line) == 0:
1185- # It's an empty line. Reset any previous comment we have.
1186- last_comment = None
1187- last_comment_line_num = 0
1188- ignore_comment = False
1189- elif line.startswith(u'#') or line.startswith(u'//'):
1190- # It's a whole line comment.
1191- ignore_comment = False
1192- line = line[1:].strip()
1193- if last_comment:
1194- last_comment += line
1195- elif len(line) > 0:
1196- last_comment = line
1197-
1198- if last_comment and not last_comment.endswith('\n'):
1199- # Comments must end always with a new line.
1200- last_comment += '\n'
1201-
1202- last_comment_line_num = line_num
1203- continue
1204-
1205- # Unescaped URLs are a common mistake: the "//" starts an
1206- # end-of-line comment. To work around that, treat "://" as
1207- # a special case.
1208- just_saw_colon = False
1209-
1210- while line:
1211- if is_multi_line_comment:
1212- if line.startswith(u'*/'):
1213- # The comment ended, we jump the closing tag and
1214- # continue with the parsing.
1215- line = line[2:]
1216- is_multi_line_comment = False
1217- last_comment_line_num = line_num
1218- if ignore_comment:
1219- last_comment = None
1220- ignore_comment = False
1221-
1222- # Comments must end always with a new line.
1223- last_comment += '\n'
1224- elif line.startswith(self.license_block_text):
1225- # It's a comment with a licence notice, this
1226- # comment can be ignored.
1227- ignore_comment = True
1228- # Jump the whole tag
1229- line = line[len(self.license_block_text):]
1230- else:
1231- # Store the character.
1232- if last_comment is None:
1233- last_comment = line[0]
1234- elif last_comment_line_num == line_num:
1235- last_comment += line[0]
1236- else:
1237- last_comment = u'%s\n%s' % (last_comment, line[0])
1238- last_comment_line_num = line_num
1239- # Jump the processed char.
1240- line = line[1:]
1241- continue
1242- elif line.startswith(u'/*'):
1243- # It's a multi line comment
1244- is_multi_line_comment = True
1245- ignore_comment = False
1246- last_comment_line_num = line_num
1247- # Jump the comment starting tag
1248- line = line[2:]
1249- continue
1250- elif line.startswith(u'//') and not just_saw_colon:
1251- # End-of-line comment.
1252- last_comment = '%s\n' % line[2:].strip()
1253- last_comment_line_num = line_num
1254- # On to next line.
1255- break
1256- elif is_message:
1257- # Store the char and continue.
1258- head_char = line[0]
1259- translation += head_char
1260- line = line[1:]
1261- just_saw_colon = (head_char == ':')
1262- continue
1263- elif u'=' in line:
1264- # Looks like a message string.
1265- (key, value) = line.split('=', 1)
1266- # Remove leading and trailing white spaces.
1267- key = key.strip()
1268-
1269- if valid_property_msgid(key):
1270- is_message = True
1271- # Jump the msgid, control chars and leading white
1272- # space.
1273- line = value.lstrip()
1274- continue
1275- else:
1276- raise TranslationFormatSyntaxError(
1277- line_number=line_num,
1278- message=u"invalid msgid: '%s'" % key)
1279- else:
1280- # Got a line that is not a valid message nor a valid
1281- # comment. Ignore it because main en-US.xpi catalog from
1282- # Firefox has such line/error. We follow the 'be strict
1283- # with what you export, be permisive with what you import'
1284- # policy.
1285- break
1286- if is_message:
1287- # We just parsed a message, so we need to add it to the list
1288- # of messages.
1289- if ignore_comment or last_comment_line_num < line_num - 1:
1290- # We must ignore the comment or either the comment is not
1291- # the last thing before this message or is not in the same
1292- # line as this message.
1293- last_comment = None
1294- ignore_comment = False
1295-
1296- message = TranslationMessageData()
1297- message.msgid_singular = key
1298- message.context = self.chrome_path
1299- message.file_references_list = [
1300- "%s:%d(%s)" % (self.filename, line_num, key)]
1301- value = translation.strip()
1302- message.addTranslation(
1303- TranslationConstants.SINGULAR_FORM, value)
1304- message.singular_text = value
1305- message.source_comment = last_comment
1306- self.messages.append(message)
1307-
1308- # Reset status vars.
1309- last_comment = None
1310- last_comment_line_num = 0
1311- is_message = False
1312- translation = u''
1313-
1314-
1315-@implementer(ITranslationFormatImporter)
1316-class MozillaXpiImporter:
1317- """Support class to import Mozilla .xpi files."""
1318-
1319- def __init__(self):
1320- self.basepath = None
1321- self.productseries = None
1322- self.distroseries = None
1323- self.sourcepackagename = None
1324- self.by_maintainer = False
1325- self._translation_file = None
1326-
1327- def getFormat(self, file_contents):
1328- """See `ITranslationFormatImporter`."""
1329- return TranslationFileFormat.XPI
1330-
1331- priority = 0
1332-
1333- # using "application/x-xpinstall" would trigger installation in
1334- # firefox.
1335- content_type = 'application/zip'
1336-
1337- file_extensions = ['.xpi']
1338- template_suffix = 'en-US.xpi'
1339-
1340- uses_source_string_msgids = True
1341-
1342- def parse(self, translation_import_queue_entry):
1343- """See `ITranslationFormatImporter`."""
1344- self._translation_file = TranslationFileData()
1345- self.basepath = translation_import_queue_entry.path
1346- self.productseries = translation_import_queue_entry.productseries
1347- self.distroseries = translation_import_queue_entry.distroseries
1348- self.sourcepackagename = (
1349- translation_import_queue_entry.sourcepackagename)
1350- self.by_maintainer = translation_import_queue_entry.by_maintainer
1351-
1352- librarian_client = getUtility(ILibrarianClient)
1353- content = librarian_client.getFileByAlias(
1354- translation_import_queue_entry.content.id).read()
1355-
1356- parser = MozillaZipImportParser(self.basepath, StringIO(content))
1357- if parser.header is None:
1358- raise TranslationFormatInvalidInputError("No install.rdf found")
1359-
1360- self._translation_file.header = parser.header
1361- self._translation_file.messages = parser.messages
1362-
1363- return self._translation_file
1364-
1365- def getHeaderFromString(self, header_string):
1366- """See `ITranslationFormatImporter`."""
1367- return XpiHeader(header_string)
1368diff --git a/lib/lp/translations/utilities/mozilla_zip.py b/lib/lp/translations/utilities/mozilla_zip.py
1369deleted file mode 100644
1370index 489a6a2..0000000
1371--- a/lib/lp/translations/utilities/mozilla_zip.py
1372+++ /dev/null
1373@@ -1,170 +0,0 @@
1374-# Copyright 2009 Canonical Ltd. This software is licensed under the
1375-# GNU Affero General Public License version 3 (see the file LICENSE).
1376-
1377-__metaclass__ = type
1378-
1379-__all__ = [
1380- 'MozillaZipTraversal',
1381- ]
1382-
1383-from cStringIO import StringIO
1384-from os.path import (
1385- basename,
1386- splitext,
1387- )
1388-from zipfile import (
1389- BadZipfile,
1390- ZipFile,
1391- )
1392-
1393-from lp.translations.interfaces.translationimporter import (
1394- TranslationFormatInvalidInputError,
1395- )
1396-from lp.translations.utilities.xpi_header import XpiHeader
1397-from lp.translations.utilities.xpi_manifest import (
1398- make_jarpath,
1399- XpiManifest,
1400- )
1401-
1402-
1403-class MozillaZipTraversal:
1404- """Traversal of an XPI file, or a jar file inside an XPI file.
1405-
1406- To traverse and process an XPI file, derive a class from this one
1407- and replace any hooks that you may need.
1408-
1409- If an XPI manifest is provided, traversal will be restricted to
1410- directories it lists as containing localizable resources.
1411- """
1412-
1413- def __init__(self, filename, archive, xpi_path=None, manifest=None):
1414- """Open zip (or XPI, or jar) file and scan its contents.
1415-
1416- :param filename: Name of this zip (XPI/jar) archive.
1417- :param archive: File-like object containing this zip archive.
1418- :param xpi_path: Full path of this file inside the XPI archive.
1419- Leave out for the XPI archive itself.
1420- :param manifest: `XpiManifest` representing the XPI archive's
1421- manifest file, if any.
1422- """
1423- self.filename = filename
1424- self.header = None
1425- self.last_translator = None
1426- self.manifest = manifest
1427- try:
1428- self.archive = ZipFile(archive, 'r')
1429- except BadZipfile as exception:
1430- raise TranslationFormatInvalidInputError(
1431- filename=filename, message=str(exception))
1432-
1433- if xpi_path is None:
1434- # This is the main XPI file.
1435- xpi_path = ''
1436- contained_files = set(self.archive.namelist())
1437- if manifest is None:
1438- # Look for a manifest.
1439- for filename in ['chrome.manifest', 'en-US.manifest']:
1440- if filename in contained_files:
1441- manifest_content = self.archive.read(filename)
1442- self.manifest = XpiManifest(manifest_content)
1443- break
1444- if 'install.rdf' in contained_files:
1445- rdf_content = self.archive.read('install.rdf')
1446- self.header = XpiHeader(rdf_content)
1447-
1448- # Strip trailing newline to avoid doubling it.
1449- xpi_path = xpi_path.rstrip('/')
1450-
1451- self._begin()
1452-
1453- # Process zipped files. Sort by path to keep ordering deterministic.
1454- # Ordering matters in sequence numbering (which in turn shows up in
1455- # the UI), but also for consistency in duplicates resolution and for
1456- # automated testing.
1457- for entry in sorted(self.archive.namelist()):
1458- self._processEntry(entry, xpi_path)
1459-
1460- self._finish()
1461-
1462- def _processEntry(self, entry, xpi_path):
1463- """Read one zip archive entry, figure out what to do with it."""
1464- rootname, suffix = splitext(entry)
1465- if basename(rootname) == '':
1466- # If filename starts with a dot, that's not really a suffix.
1467- suffix = ''
1468-
1469- if suffix == '.jar':
1470- jarpath = make_jarpath(xpi_path, entry)
1471- if not self.manifest or self.manifest.containsLocales(jarpath):
1472- # If this is a jar file that may contain localizable
1473- # resources, don't process it in the normal way; recurse
1474- # by creating another parser instance.
1475- content = self.archive.read(entry)
1476- nested_instance = self.__class__(
1477- filename=entry, archive=StringIO(content),
1478- xpi_path=jarpath, manifest=self.manifest)
1479-
1480- self._processNestedJar(nested_instance)
1481- return
1482-
1483- # Construct XPI path; identical to "entry" if previous xpi_path
1484- # was empty. XPI paths use slashes as separators, regardless of
1485- # what the native filesystem uses.
1486- xpi_path = '/'.join([xpi_path, entry]).lstrip('/')
1487-
1488- if self.manifest is None:
1489- # No manifest, so we don't have chrome paths. Process
1490- # everything just to be sure.
1491- chrome_path = None
1492- locale_code = None
1493- else:
1494- chrome_path, locale_code = self.manifest.getChromePathAndLocale(
1495- xpi_path)
1496- if chrome_path is None:
1497- # Not in a directory containing localizable resources.
1498- return
1499-
1500- self._processTranslatableFile(
1501- entry, locale_code, xpi_path, chrome_path, suffix)
1502-
1503- def _begin(self):
1504- """Overridable hook: optional pre-traversal actions."""
1505-
1506- def _processTranslatableFile(self, entry, locale_code, xpi_path,
1507- chrome_path, filename_suffix):
1508- """Overridable hook: process a file entry.
1509-
1510- Called only for files that may be localizable. If there is a
1511- manifest, that means the file must be in a location (or subtree)
1512- named by a "locale" entry.
1513-
1514- :param entry: Full name of file inside this zip archive,
1515- including path relative to the archive's root.
1516- :param locale_code: Code for locale this file belongs to, e.g.
1517- "en-US".
1518- :param xpi_path: Full path of this file inside the XPI archive,
1519- e.g. "jar:locale/en-US.jar!/data/messages.dtd".
1520- :param chrome_path: File's chrome path. This is a kind of
1521- "normalized" path used in XPI to describe a virtual
1522- directory hierarchy. The zip archive's actual layout (which
1523- the XPI paths describe) may be different.
1524- :param filename_suffix: File name suffix or "extension" of the
1525- translatable file. This may be e.g. ".dtd" or ".xhtml," or
1526- the empty string if the filename does not contain a dot.
1527- """
1528- raise NotImplementedError(
1529- "XPI traversal class provides no _processTranslatableFile().")
1530-
1531- def _processNestedJar(self, zip_instance):
1532- """Overridable hook: handle a nested jar file.
1533-
1534- :param zip_instance: An instance of the same class as self, which
1535- has just parsed the nested jar file.
1536- """
1537- raise NotImplementedError(
1538- "XPI traversal class provides no _processNestedJar().")
1539-
1540- def _finish(self):
1541- """Overridable hook: post-traversal actions."""
1542- raise NotImplementedError(
1543- "XPI traversal class provides no _finish().")
1544diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/chrome.manifest b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/chrome.manifest
1545deleted file mode 100644
1546index 99c5dc3..0000000
1547--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/chrome.manifest
1548+++ /dev/null
1549@@ -1,5 +0,0 @@
1550-locale mac en-US jar:chrome/en-US.jar!/mac/
1551-locale unix en-US jar:chrome/en-US.jar!/unix/
1552-locale win en-US jar:chrome/en-US.jar!/win/
1553-override chrome://foo/bar/splat.dtd
1554-locale main en-US jar:chrome/en-US.jar!/
1555diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.dtd b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.dtd
1556deleted file mode 100644
1557index ae5af22..0000000
1558--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.dtd
1559+++ /dev/null
1560@@ -1,2 +0,0 @@
1561-<!-- This message id also occurs elsewhere in this file -->
1562-<!ENTITY foozilla.clashing.key "This message is Mac-specific, and comes from DTD.">
1563diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.properties b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.properties
1564deleted file mode 100644
1565index cb23d48..0000000
1566--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/mac/extra.properties
1567+++ /dev/null
1568@@ -1 +0,0 @@
1569-foozilla.clashing.key=This message is Mac-specific, and comes from properties.
1570diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.dtd b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.dtd
1571deleted file mode 100644
1572index 9e608fa..0000000
1573--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.dtd
1574+++ /dev/null
1575@@ -1,6 +0,0 @@
1576-<!-- This message id also occurs elsewhere in this file -->
1577-<!ENTITY foozilla.regular.message "A non-clashing message.">
1578-<!ENTITY foozilla.clashing.key "This message is in the main DTD.">
1579-
1580-<!-- Clashing msgid within same file. Should be ignored. -->
1581-<!ENTITY foozilla.regular.message "This message should be ignored.">
1582diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.properties b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.properties
1583deleted file mode 100644
1584index 0a317b1..0000000
1585--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/main.properties
1586+++ /dev/null
1587@@ -1 +0,0 @@
1588-foozilla.clashing.key=This message is in the main properties file.
1589diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.dtd b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.dtd
1590deleted file mode 100644
1591index 7388985..0000000
1592--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.dtd
1593+++ /dev/null
1594@@ -1,2 +0,0 @@
1595-<!-- This message id also occurs elsewhere in this file -->
1596-<!ENTITY foozilla.clashing.key "This message is Unix-specific, and comes from DTD.">
1597diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.properties b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.properties
1598deleted file mode 100644
1599index 6888f4d..0000000
1600--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/unix/extra.properties
1601+++ /dev/null
1602@@ -1 +0,0 @@
1603-foozilla.clashing.key=This message is Unix-specific, and comes from properties.
1604diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.dtd b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.dtd
1605deleted file mode 100644
1606index ee2dbd6..0000000
1607--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.dtd
1608+++ /dev/null
1609@@ -1,2 +0,0 @@
1610-<!-- This message id also occurs elsewhere in this file -->
1611-<!ENTITY foozilla.clashing.key "This message is Windows-specific, and comes from DTD.">
1612diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.properties b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.properties
1613deleted file mode 100644
1614index aacf3d8..0000000
1615--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/en-US-jar/win/extra.properties
1616+++ /dev/null
1617@@ -1 +0,0 @@
1618-foozilla.clashing.key=This message is Windows-specific, and comes from properties.
1619diff --git a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/install.rdf b/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/install.rdf
1620deleted file mode 100644
1621index 872390f..0000000
1622--- a/lib/lp/translations/utilities/tests/firefox-data/clashing_ids/install.rdf
1623+++ /dev/null
1624@@ -1,19 +0,0 @@
1625-<?xml version="1.0"?>
1626-<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
1627- xmlns:em="http://www.mozilla.org/2004/em-rdf#">
1628- <Description about="urn:mozilla:install-manifest"
1629- em:id="langpack-en-US@firefox.mozilla.org"
1630- em:name="English U.S. (en-US) Language Pack"
1631- em:version="2.0"
1632- em:type="8"
1633- em:creator="Jeroen Vermeulen">
1634-
1635- <em:targetApplication>
1636- <Description>
1637- <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id><!-- firefox -->
1638- <em:minVersion>2.0</em:minVersion>
1639- <em:maxVersion>2.0.0.*</em:maxVersion>
1640- </Description>
1641- </em:targetApplication>
1642- </Description>
1643-</RDF>
1644diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/chrome.manifest b/lib/lp/translations/utilities/tests/firefox-data/en-US/chrome.manifest
1645deleted file mode 100644
1646index 5c362be..0000000
1647--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/chrome.manifest
1648+++ /dev/null
1649@@ -1 +0,0 @@
1650-locale main en-US jar:chrome/en-US.jar!/
1651diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/copyover3.png b/lib/lp/translations/utilities/tests/firefox-data/en-US/copyover3.png
1652deleted file mode 100644
1653index cdac869..0000000
1654Binary files a/lib/lp/translations/utilities/tests/firefox-data/en-US/copyover3.png and /dev/null differ
1655diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/copyover1.foo b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/copyover1.foo
1656deleted file mode 100644
1657index d39d61f..0000000
1658--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/copyover1.foo
1659+++ /dev/null
1660@@ -1,3 +0,0 @@
1661-This file is copied directly over to resulting translated XPI files.
1662-
1663-We only need to make sure the contents doesn't change a bit.
1664diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/copyover2.foo b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/copyover2.foo
1665deleted file mode 100644
1666index c82082f..0000000
1667--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/copyover2.foo
1668+++ /dev/null
1669@@ -1,7 +0,0 @@
1670-This is another file which should be blindly copied without any
1671-changes in the content, no matter how much one tries.
1672-
1673-And for some binary checks, lets add some UTF-8 encoded data. For
1674-example, name of "Carlos PerellĆ³ MarĆ­n" would be written as "ŠšŠ°Ń€Š»Š¾Ń
1675-ŠŸŠµŃ€ŠµŃ™Š¾ ŠœŠ°Ń€ŠøŠ½" in Serbian (which is phonetic, and is read exactly the
1676-same minus accents).
1677diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.dtd b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.dtd
1678deleted file mode 100644
1679index 4efb0fe..0000000
1680--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.dtd
1681+++ /dev/null
1682@@ -1,5 +0,0 @@
1683-<!-- This is a DTD file inside a subdirectory -->
1684-
1685-<!ENTITY foozilla.menu.title "MENU">
1686-<!ENTITY foozilla.menu.accesskey "M">
1687-<!ENTITY foozilla.menu.commandkey "m">
1688diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.properties b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.properties
1689deleted file mode 100644
1690index e6c200b..0000000
1691--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/subdir/test2.properties
1692+++ /dev/null
1693@@ -1,6 +0,0 @@
1694-# This is a Properties file inside a subdirectory
1695-
1696-# Translators, what you are seeing now is a lovely,
1697-# awesome, multiline comment aimed at you directly
1698-# from the streets of a .properties file
1699-foozilla_something=SomeZilla
1700diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.dtd b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.dtd
1701deleted file mode 100644
1702index 0828f76..0000000
1703--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.dtd
1704+++ /dev/null
1705@@ -1,6 +0,0 @@
1706-<!ENTITY foozilla.name "FooZilla!">
1707-<!-- Translators, don't play with fire! -->
1708-<!ENTITY foozilla.play.fire "Do you want to play with fire?">
1709-<!-- This is just a comment, not a comment for translators -->
1710-
1711-<!ENTITY foozilla.play.ice "Play with ice?">
1712diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.properties b/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.properties
1713deleted file mode 100644
1714index a4b7b40..0000000
1715--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/en-US-jar/test1.properties
1716+++ /dev/null
1717@@ -1,5 +0,0 @@
1718-foozilla.title=FooZilla Zilla Thingy
1719-# Translators, if you're older than six, don't translate this
1720-foozilla.happytitle=http://foozillingy.happy.net/
1721-foozilla.nocomment=No Comment // (Except this one)
1722-foozilla.utf8=Š”Š°Š½=Day
1723diff --git a/lib/lp/translations/utilities/tests/firefox-data/en-US/install.rdf b/lib/lp/translations/utilities/tests/firefox-data/en-US/install.rdf
1724deleted file mode 100644
1725index a34cc54..0000000
1726--- a/lib/lp/translations/utilities/tests/firefox-data/en-US/install.rdf
1727+++ /dev/null
1728@@ -1,21 +0,0 @@
1729-<?xml version="1.0"?>
1730-<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
1731- xmlns:em="http://www.mozilla.org/2004/em-rdf#">
1732- <Description about="urn:mozilla:install-manifest"
1733- em:id="langpack-en-US@firefox.mozilla.org"
1734- em:name="English U.S. (en-US) Language Pack"
1735- em:version="2.0"
1736- em:type="8"
1737- em:creator="Danilo Å egan">
1738- <em:contributor>Š”Š°Š½ŠøŠ»Š¾ ŠØŠµŠ³Š°Š½</em:contributor>
1739- <em:contributor>Carlos PerellĆ³ MarĆ­n &lt;carlos@canonical.com&gt;</em:contributor>
1740-
1741- <em:targetApplication>
1742- <Description>
1743- <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id><!-- firefox -->
1744- <em:minVersion>2.0</em:minVersion>
1745- <em:maxVersion>2.0.0.*</em:maxVersion>
1746- </Description>
1747- </em:targetApplication>
1748- </Description>
1749-</RDF>
1750diff --git a/lib/lp/translations/utilities/tests/firefox-data/no-manifest/en-US-jar/file.txt b/lib/lp/translations/utilities/tests/firefox-data/no-manifest/en-US-jar/file.txt
1751deleted file mode 100644
1752index 2d4b8bd..0000000
1753--- a/lib/lp/translations/utilities/tests/firefox-data/no-manifest/en-US-jar/file.txt
1754+++ /dev/null
1755@@ -1 +0,0 @@
1756-This is a translatable text file.
1757diff --git a/lib/lp/translations/utilities/tests/firefox-data/no-manifest/no-jar.txt b/lib/lp/translations/utilities/tests/firefox-data/no-manifest/no-jar.txt
1758deleted file mode 100644
1759index 80a1d56..0000000
1760--- a/lib/lp/translations/utilities/tests/firefox-data/no-manifest/no-jar.txt
1761+++ /dev/null
1762@@ -1 +0,0 @@
1763-This is a file in an XPI but not in a jar.
1764diff --git a/lib/lp/translations/utilities/tests/firefox-data/system-entity/chrome.manifest b/lib/lp/translations/utilities/tests/firefox-data/system-entity/chrome.manifest
1765deleted file mode 100644
1766index 5c362be..0000000
1767--- a/lib/lp/translations/utilities/tests/firefox-data/system-entity/chrome.manifest
1768+++ /dev/null
1769@@ -1 +0,0 @@
1770-locale main en-US jar:chrome/en-US.jar!/
1771diff --git a/lib/lp/translations/utilities/tests/firefox-data/system-entity/en-US-jar/test.dtd b/lib/lp/translations/utilities/tests/firefox-data/system-entity/en-US-jar/test.dtd
1772deleted file mode 100644
1773index 7af5a5a..0000000
1774--- a/lib/lp/translations/utilities/tests/firefox-data/system-entity/en-US-jar/test.dtd
1775+++ /dev/null
1776@@ -1,9 +0,0 @@
1777-<!-- Test SYSTEM handling. -->
1778-
1779-<!ENTITY firststring "First translatable string">
1780-
1781-<!ENTITY % includedFile SYSTEM "chrome://includedFile.dtd">
1782-%includedFile;
1783-
1784-<!-- Parser will only get here if that last tag didn't break things. -->
1785-<!ENTITY secondstring "Second translatable string">
1786diff --git a/lib/lp/translations/utilities/tests/firefox-data/system-entity/install.rdf b/lib/lp/translations/utilities/tests/firefox-data/system-entity/install.rdf
1787deleted file mode 100644
1788index 1484c26..0000000
1789--- a/lib/lp/translations/utilities/tests/firefox-data/system-entity/install.rdf
1790+++ /dev/null
1791@@ -1,17 +0,0 @@
1792-<?xml version="1.0"?>
1793-<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
1794- xmlns:em="http://www.mozilla.org/2004/em-rdf#">
1795- <Description about="urn:mozilla:install-manifest"
1796- em:id="langpack-en-US@firefox.mozilla.org"
1797- em:name="English U.S. (en-US) Language Pack"
1798- em:version="2.0"
1799- em:type="8">
1800- <em:targetApplication>
1801- <Description>
1802- <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id><!-- firefox -->
1803- <em:minVersion>2.0</em:minVersion>
1804- <em:maxVersion>2.0.0.*</em:maxVersion>
1805- </Description>
1806- </em:targetApplication>
1807- </Description>
1808-</RDF>
1809diff --git a/lib/lp/translations/utilities/tests/test_mozilla_xpi_importer.py b/lib/lp/translations/utilities/tests/test_mozilla_xpi_importer.py
1810deleted file mode 100644
1811index ce147a7..0000000
1812--- a/lib/lp/translations/utilities/tests/test_mozilla_xpi_importer.py
1813+++ /dev/null
1814@@ -1,48 +0,0 @@
1815-# Copyright 2009-2017 Canonical Ltd. This software is licensed under the
1816-# GNU Affero General Public License version 3 (see the file LICENSE).
1817-
1818-"""Mozilla XPI importer tests."""
1819-
1820-__metaclass__ = type
1821-
1822-from io import BytesIO
1823-import unittest
1824-
1825-from zope.interface.verify import verifyObject
1826-
1827-from lp.testing.layers import LaunchpadZopelessLayer
1828-from lp.translations.interfaces.translationfileformat import (
1829- TranslationFileFormat,
1830- )
1831-from lp.translations.interfaces.translationimporter import (
1832- ITranslationFormatImporter,
1833- )
1834-from lp.translations.utilities.mozilla_xpi_importer import MozillaXpiImporter
1835-
1836-
1837-class MozillaXpiImporterTestCase(unittest.TestCase):
1838- """Class test for mozilla's .xpi file imports"""
1839-
1840- layer = LaunchpadZopelessLayer
1841-
1842- def setUp(self):
1843- self.importer = MozillaXpiImporter()
1844-
1845- def testInterface(self):
1846- """Check whether the object follows the interface."""
1847- self.assertTrue(
1848- verifyObject(ITranslationFormatImporter, self.importer))
1849-
1850- def testFormat(self):
1851- """Check that MozillaXpiImporter handles the XPI file format."""
1852- format = self.importer.getFormat(BytesIO(b''))
1853- self.assertTrue(
1854- format == TranslationFileFormat.XPI,
1855- 'MozillaXpiImporter format expected XPI but got %s' % format.name)
1856-
1857- def testHasAlternativeMsgID(self):
1858- """Check that MozillaXpiImporter has an alternative msgid."""
1859- self.assertTrue(
1860- self.importer.uses_source_string_msgids,
1861- "MozillaXpiImporter format says it's not using alternative msgid"
1862- " when it really does!")
1863diff --git a/lib/lp/translations/utilities/tests/test_mozilla_zip.py b/lib/lp/translations/utilities/tests/test_mozilla_zip.py
1864deleted file mode 100644
1865index d7eee80..0000000
1866--- a/lib/lp/translations/utilities/tests/test_mozilla_zip.py
1867+++ /dev/null
1868@@ -1,113 +0,0 @@
1869-# Copyright 2009 Canonical Ltd. This software is licensed under the
1870-# GNU Affero General Public License version 3 (see the file LICENSE).
1871-
1872-"""`MozillaZipTraversal` tests."""
1873-
1874-__metaclass__ = type
1875-
1876-import unittest
1877-
1878-from lp.testing.layers import LaunchpadZopelessLayer
1879-from lp.translations.interfaces.translationimporter import (
1880- TranslationFormatInvalidInputError,
1881- )
1882-from lp.translations.utilities.mozilla_zip import MozillaZipTraversal
1883-from lp.translations.utilities.tests.xpi_helpers import (
1884- get_en_US_xpi_file_to_import,
1885- )
1886-
1887-
1888-class TraversalRecorder(MozillaZipTraversal):
1889- """XPI "parser": records traversal of an XPI or jar file.
1890-
1891- Does nothing but keep track of the structure of nested zip files it
1892- traverses, and the various parameters for each translatable file.
1893-
1894- Produces a nice list of tuples (representing parameters for a
1895- translatable file) and lists (representing nested jar files). Each
1896- zip file's traversal, including nested ones, is concluded with a
1897- string containing a full stop (".").
1898- """
1899- traversal = None
1900-
1901- def _begin(self):
1902- self.traversal = []
1903-
1904- def _processTranslatableFile(self, entry, locale_code, xpi_path,
1905- chrome_path, filename_suffix):
1906- record = (entry, locale_code, xpi_path, chrome_path, filename_suffix)
1907- self.traversal.append(record)
1908-
1909- def _processNestedJar(self, nested_recorder):
1910- self.traversal.append(nested_recorder.traversal)
1911-
1912- def _finish(self):
1913- self.traversal.append('.')
1914-
1915-
1916-class MozillaZipTraversalTestCase(unittest.TestCase):
1917- """Test Mozilla XPI/jar traversal."""
1918-
1919- layer = LaunchpadZopelessLayer
1920-
1921- def test_InvalidXpiFile(self):
1922- # If the "XPI" file isn't really a zip file, that's a
1923- # TranslationFormatInvalidInputError.
1924- self.assertRaises(
1925- TranslationFormatInvalidInputError,
1926- TraversalRecorder,
1927- 'foo.xpi', __file__)
1928-
1929- def test_XpiTraversal(self):
1930- """Test a typical traversal of XPI file, with nested jar file."""
1931- xpi_archive = get_en_US_xpi_file_to_import('en-US')
1932- record = TraversalRecorder('', xpi_archive)
1933- self.assertEqual(record.traversal, [
1934- [
1935- ('copyover1.foo', 'en-US',
1936- 'jar:chrome/en-US.jar!/copyover1.foo',
1937- 'main/copyover1.foo', '.foo'
1938- ),
1939- ('subdir/copyover2.foo', 'en-US',
1940- 'jar:chrome/en-US.jar!/subdir/copyover2.foo',
1941- 'main/subdir/copyover2.foo', '.foo'
1942- ),
1943- ('subdir/test2.dtd', 'en-US',
1944- 'jar:chrome/en-US.jar!/subdir/test2.dtd',
1945- 'main/subdir/test2.dtd', '.dtd'
1946- ),
1947- ('subdir/test2.properties', 'en-US',
1948- 'jar:chrome/en-US.jar!/subdir/test2.properties',
1949- 'main/subdir/test2.properties', '.properties'
1950- ),
1951- ('test1.dtd', 'en-US',
1952- 'jar:chrome/en-US.jar!/test1.dtd',
1953- 'main/test1.dtd', '.dtd'
1954- ),
1955- ('test1.properties', 'en-US',
1956- 'jar:chrome/en-US.jar!/test1.properties',
1957- 'main/test1.properties', '.properties'
1958- ),
1959- '.'
1960- ],
1961- '.'
1962- ])
1963-
1964- def test_XpiTraversalWithoutManifest(self):
1965- """Test traversal of an XPI file without manifest."""
1966- xpi_archive = get_en_US_xpi_file_to_import('no-manifest')
1967- record = TraversalRecorder('', xpi_archive)
1968- # Without manifest, there is no knowledge of locale or chrome
1969- # paths, so those are None.
1970- self.assertEqual(record.traversal, [
1971- [
1972- ('file.txt', None,
1973- 'jar:chrome/en-US.jar!/file.txt', None, '.txt'
1974- ),
1975- '.'
1976- ],
1977- ('no-jar.txt', None,
1978- 'no-jar.txt', None, '.txt'
1979- ),
1980- '.'
1981- ])
1982diff --git a/lib/lp/translations/utilities/tests/test_translation_importer.py b/lib/lp/translations/utilities/tests/test_translation_importer.py
1983index ce43cd8..dcf203b 100644
1984--- a/lib/lp/translations/utilities/tests/test_translation_importer.py
1985+++ b/lib/lp/translations/utilities/tests/test_translation_importer.py
1986@@ -72,9 +72,6 @@ class TranslationImporterTestCase(TestCaseWithFactory):
1987 None,
1988 importer.getTranslationFormatImporter(
1989 TranslationFileFormat.KDEPO))
1990- self.assertIsNot(
1991- None,
1992- importer.getTranslationFormatImporter(TranslationFileFormat.XPI))
1993
1994 def testGetTranslationFileFormatByFileExtension(self):
1995 """Checked whether file format precedence works correctly."""
1996@@ -94,10 +91,6 @@ class TranslationImporterTestCase(TestCaseWithFactory):
1997 importer.getTranslationFileFormat(
1998 ".po", BytesIO(b'msgid "_: kde context\nmessage"\nmsgstr ""')))
1999
2000- self.assertEqual(
2001- TranslationFileFormat.XPI,
2002- importer.getTranslationFileFormat(".xpi", BytesIO(b"")))
2003-
2004 def testNoConflictingPriorities(self):
2005 """Check that no two importers for the same file extension have
2006 exactly the same priority."""
2007@@ -111,13 +104,12 @@ class TranslationImporterTestCase(TestCaseWithFactory):
2008 def testFileExtensionsWithImporters(self):
2009 """Check whether we get the right list of file extensions handled."""
2010 self.assertEqual(
2011- ['.po', '.pot', '.xpi'],
2012+ ['.po', '.pot'],
2013 TranslationImporter().supported_file_extensions)
2014
2015 def testTemplateSuffixes(self):
2016 """Check for changes in filename suffixes that identify templates."""
2017- self.assertEqual(
2018- ['.pot', 'en-US.xpi'], TranslationImporter().template_suffixes)
2019+ self.assertEqual(['.pot'], TranslationImporter().template_suffixes)
2020
2021 def _assertIsNotTemplate(self, path):
2022 self.assertFalse(
2023@@ -137,12 +129,8 @@ class TranslationImporterTestCase(TestCaseWithFactory):
2024 self._assertIsTemplate("bar.pot")
2025 self._assertIsTemplate("foo/bar.pot")
2026 self._assertIsTemplate("foo.bar.pot")
2027- self._assertIsTemplate("en-US.xpi")
2028- self._assertIsTemplate("translations/en-US.xpi")
2029
2030 self._assertIsNotTemplate("pt_BR.po")
2031- self._assertIsNotTemplate("pt_BR.xpi")
2032- self._assertIsNotTemplate("pt-BR.xpi")
2033
2034 def testHiddenFilesRecognition(self):
2035 # Hidden files and directories (leading dot) are recognized.
2036@@ -189,13 +177,9 @@ class TranslationImporterTestCase(TestCaseWithFactory):
2037 self._assertIsTranslation("po/el.po")
2038 self._assertIsTranslation("po/package-el.po")
2039 self._assertIsTranslation("po/package-zh_TW.po")
2040- self._assertIsTranslation("en-GB.xpi")
2041- self._assertIsTranslation("translations/en-GB.xpi")
2042
2043 self._assertIsNotTranslation("hi.pot")
2044 self._assertIsNotTranslation("po/hi.pot")
2045- self._assertIsNotTranslation("en-US.xpi")
2046- self._assertIsNotTranslation("translations/en-US.xpi")
2047
2048 def testIsIdenticalTranslation(self):
2049 """Test `is_identical_translation`."""
2050diff --git a/lib/lp/translations/utilities/tests/test_xpi_dtd_format.py b/lib/lp/translations/utilities/tests/test_xpi_dtd_format.py
2051deleted file mode 100644
2052index ddf0c70..0000000
2053--- a/lib/lp/translations/utilities/tests/test_xpi_dtd_format.py
2054+++ /dev/null
2055@@ -1,45 +0,0 @@
2056-# Copyright 2009 Canonical Ltd. This software is licensed under the
2057-# GNU Affero General Public License version 3 (see the file LICENSE).
2058-
2059-__metaclass__ = type
2060-
2061-import unittest
2062-
2063-from lp.translations.interfaces.translationimporter import (
2064- TranslationFormatInvalidInputError,
2065- )
2066-from lp.translations.utilities.mozilla_dtd_parser import DtdFile
2067-
2068-
2069-class DtdFormatTestCase(unittest.TestCase):
2070- """Test class for dtd file format."""
2071-
2072- def test_DtdSyntaxError(self):
2073- # Syntax errors in a DTD file are reported as translation format
2074- # errors.
2075- content = '<!ENTITY foo "gah"></ENTITY>'
2076- self.assertRaises(
2077- TranslationFormatInvalidInputError, DtdFile, 'test.dtd', None,
2078- content)
2079-
2080- def test_UTF8DtdFileTest(self):
2081- """This test makes sure that we handle UTF-8 encoding files."""
2082-
2083- content = (
2084- '<!ENTITY utf8.message "\xc2\xbfQuieres? \xc2\xa1S\xc3\xad!">')
2085-
2086- dtd_file = DtdFile('test.dtd', None, content)
2087-
2088- # There is a single message.
2089- self.assertEqual(len(dtd_file.messages), 1)
2090- message = dtd_file.messages[0]
2091-
2092- self.assertEqual([u'\xbfQuieres? \xa1S\xed!'], message.translations)
2093-
2094- def test_Latin1DtdFileTest(self):
2095- """This test makes sure that we detect bad encodings."""
2096-
2097- content = '<!ENTITY latin1.message "\xbfQuieres? \xa1S\xed!">\n'
2098-
2099- self.assertRaises(TranslationFormatInvalidInputError, DtdFile, None,
2100- 'test.dtd', content)
2101diff --git a/lib/lp/translations/utilities/tests/test_xpi_import.py b/lib/lp/translations/utilities/tests/test_xpi_import.py
2102deleted file mode 100644
2103index 1bbe439..0000000
2104--- a/lib/lp/translations/utilities/tests/test_xpi_import.py
2105+++ /dev/null
2106@@ -1,365 +0,0 @@
2107-# Copyright 2009-2017 Canonical Ltd. This software is licensed under the
2108-# GNU Affero General Public License version 3 (see the file LICENSE).
2109-
2110-"""Functional tests for XPI file format"""
2111-__metaclass__ = type
2112-
2113-import re
2114-import unittest
2115-
2116-from zope.component import getUtility
2117-
2118-from lp.app.interfaces.launchpad import ILaunchpadCelebrities
2119-from lp.registry.interfaces.person import IPersonSet
2120-from lp.registry.interfaces.product import IProductSet
2121-from lp.testing.layers import LaunchpadZopelessLayer
2122-from lp.translations.enums import RosettaImportStatus
2123-from lp.translations.interfaces.potemplate import IPOTemplateSet
2124-from lp.translations.utilities.mozilla_xpi_importer import MozillaXpiImporter
2125-from lp.translations.utilities.tests.helpers import (
2126- import_pofile_or_potemplate,
2127- )
2128-from lp.translations.utilities.tests.xpi_helpers import (
2129- access_key_source_comment,
2130- command_key_source_comment,
2131- get_en_US_xpi_file_to_import,
2132- )
2133-
2134-
2135-def unwrap(text):
2136- """Remove line breaks and any other wrapping artifacts from text."""
2137- return re.sub('\s+', ' ', text.strip())
2138-
2139-
2140-class XpiTestCase(unittest.TestCase):
2141- """XPI file import into Launchpad."""
2142-
2143- layer = LaunchpadZopelessLayer
2144-
2145- def setUp(self):
2146- # Get the importer.
2147- self.importer = getUtility(IPersonSet).getByName('mark')
2148-
2149- # Get the Firefox template.
2150- firefox_product = getUtility(IProductSet).getByName('firefox')
2151- firefox_productseries = firefox_product.getSeries('trunk')
2152- firefox_potemplate_subset = getUtility(IPOTemplateSet).getSubset(
2153- productseries=firefox_productseries)
2154- self.firefox_template = firefox_potemplate_subset.new(
2155- name='firefox',
2156- translation_domain='firefox',
2157- path='en-US.xpi',
2158- owner=self.importer)
2159- self.spanish_firefox = self.firefox_template.newPOFile('es')
2160- self.spanish_firefox.path = 'translations/es.xpi'
2161-
2162- def setUpTranslationImportQueueForTemplate(self, subdir):
2163- """Return an ITranslationImportQueueEntry for testing purposes.
2164-
2165- :param subdir: subdirectory in firefox-data to get XPI data from.
2166- """
2167- # Get the file to import.
2168- en_US_xpi = get_en_US_xpi_file_to_import(subdir)
2169- return import_pofile_or_potemplate(
2170- file_contents=en_US_xpi,
2171- person=self.importer,
2172- potemplate=self.firefox_template)
2173-
2174- def setUpTranslationImportQueueForTranslation(self, subdir):
2175- """Return an ITranslationImportQueueEntry for testing purposes.
2176-
2177- :param subdir: subdirectory in firefox-data to get XPI data from.
2178- """
2179- # Get the file to import. Given the way XPI file format works, we can
2180- # just use the same template file like a translation one.
2181- es_xpi = get_en_US_xpi_file_to_import(subdir)
2182- return import_pofile_or_potemplate(
2183- file_contents=es_xpi,
2184- person=self.importer,
2185- pofile=self.spanish_firefox,
2186- by_maintainer=True)
2187-
2188- def _assertXpiMessageInvariant(self, message):
2189- """Check whether invariant part of all messages are correct."""
2190- # msgid and singular_text are always different except for the keyboard
2191- # shortcuts which are the 'accesskey' and 'commandkey' ones.
2192- self.assertFalse(
2193- (message.msgid_singular.msgid == message.singular_text and
2194- message.msgid_singular.msgid not in (
2195- u'foozilla.menu.accesskey', u'foozilla.menu.commandkey')),
2196- 'msgid and singular_text should be different but both are %s' % (
2197- message.msgid_singular.msgid))
2198-
2199- # Plural forms should be None as this format is not able to handle
2200- # them.
2201- self.assertIsNone(message.msgid_plural)
2202- self.assertIsNone(message.plural_text)
2203-
2204- # There is no way to know whether a comment is from a
2205- # translator or a developer comment, so we have comenttext
2206- # always as None and store all comments as source comments.
2207- self.assertEqual(message.commenttext, u'')
2208-
2209- # This format doesn't support any functionality like .po flags.
2210- self.assertEqual(message.flagscomment, u'')
2211-
2212- def test_TemplateImport(self):
2213- """Test XPI template file import."""
2214- # Prepare the import queue to handle a new .xpi import.
2215- entry = self.setUpTranslationImportQueueForTemplate('en-US')
2216-
2217- # The status is now IMPORTED:
2218- self.assertEqual(entry.status, RosettaImportStatus.IMPORTED)
2219-
2220- # Let's validate the content of the messages.
2221- potmsgsets = list(self.firefox_template.getPOTMsgSets())
2222-
2223- messages_msgid_list = []
2224- for message in potmsgsets:
2225- messages_msgid_list.append(message.msgid_singular.msgid)
2226-
2227- # Check the common values for all messages.
2228- self._assertXpiMessageInvariant(message)
2229-
2230- if message.msgid_singular.msgid == u'foozilla.name':
2231- # It's a normal message that lacks any comment.
2232-
2233- self.assertEqual(message.singular_text, u'FooZilla!')
2234- self.assertEqual(
2235- message.filereferences,
2236- u'jar:chrome/en-US.jar!/test1.dtd(foozilla.name)')
2237- self.assertIsNone(message.sourcecomment)
2238-
2239- elif message.msgid_singular.msgid == u'foozilla.play.fire':
2240- # This one is also a normal message that has a comment.
2241-
2242- self.assertEqual(
2243- message.singular_text, u'Do you want to play with fire?')
2244- self.assertEqual(
2245- message.filereferences,
2246- u'jar:chrome/en-US.jar!/test1.dtd(foozilla.play.fire)')
2247- self.assertEqual(
2248- message.sourcecomment,
2249- u" Translators, don't play with fire! \n")
2250-
2251- elif message.msgid_singular.msgid == u'foozilla.utf8':
2252- # Now, we can see that special UTF-8 chars are extracted
2253- # correctly.
2254- self.assertEqual(
2255- message.singular_text, u'\u0414\u0430\u043d=Day')
2256- self.assertEqual(
2257- message.filereferences,
2258- u'jar:chrome/en-US.jar!/test1.properties:5' +
2259- u'(foozilla.utf8)')
2260- self.assertIsNone(message.sourcecomment)
2261- elif message.msgid_singular.msgid == u'foozilla.menu.accesskey':
2262- # access key is a special notation that is supposed to be
2263- # translated with a key shortcut.
2264- self.assertEqual(
2265- message.singular_text, u'M')
2266- self.assertEqual(
2267- message.filereferences,
2268- u'jar:chrome/en-US.jar!/subdir/test2.dtd' +
2269- u'(foozilla.menu.accesskey)')
2270- # The comment shows the key used when there is no translation,
2271- # which is noted as the en_US translation.
2272- self.assertEqual(
2273- unwrap(message.sourcecomment),
2274- unwrap(access_key_source_comment))
2275- elif message.msgid_singular.msgid == u'foozilla.menu.commandkey':
2276- # command key is a special notation that is supposed to be
2277- # translated with a key shortcut.
2278- self.assertEqual(
2279- message.singular_text, u'm')
2280- self.assertEqual(
2281- message.filereferences,
2282- u'jar:chrome/en-US.jar!/subdir/test2.dtd' +
2283- u'(foozilla.menu.commandkey)')
2284- # The comment shows the key used when there is no translation,
2285- # which is noted as the en_US translation.
2286- self.assertEqual(
2287- unwrap(message.sourcecomment),
2288- unwrap(command_key_source_comment))
2289-
2290- # Check that we got all messages.
2291- self.assertEqual(
2292- [u'foozilla.happytitle', u'foozilla.menu.accesskey',
2293- u'foozilla.menu.commandkey', u'foozilla.menu.title',
2294- u'foozilla.name', u'foozilla.nocomment', u'foozilla.play.fire',
2295- u'foozilla.play.ice', u'foozilla.title', u'foozilla.utf8',
2296- u'foozilla_something'],
2297- sorted(messages_msgid_list))
2298-
2299- def test_TwiceTemplateImport(self):
2300- """Test a template import done twice."""
2301- # Prepare the import queue to handle a new .xpi import.
2302- entry = self.setUpTranslationImportQueueForTemplate('en-US')
2303-
2304- # The status is now IMPORTED:
2305- self.assertEqual(entry.status, RosettaImportStatus.IMPORTED)
2306-
2307- # Retrieve the number of messages we got in this initial import.
2308- first_import_potmsgsets = self.firefox_template.getPOTMsgSets(
2309- ).count()
2310-
2311- # Force the entry to be imported again:
2312- entry.setStatus(RosettaImportStatus.APPROVED,
2313- getUtility(ILaunchpadCelebrities).rosetta_experts)
2314- # Now, we tell the PO template to import from the file data it has.
2315- (subject, body) = self.firefox_template.importFromQueue(entry)
2316-
2317- # Retrieve the number of messages we got in this second import.
2318- second_import_potmsgsets = self.firefox_template.getPOTMsgSets(
2319- ).count()
2320-
2321- # Both must match.
2322- self.assertEqual(first_import_potmsgsets, second_import_potmsgsets)
2323-
2324- def test_TranslationImport(self):
2325- """Test XPI translation file import."""
2326- # Prepare the import queue to handle a new .xpi import.
2327- template_entry = self.setUpTranslationImportQueueForTemplate('en-US')
2328- translation_entry = self.setUpTranslationImportQueueForTranslation(
2329- 'en-US')
2330-
2331- # The status is now IMPORTED:
2332- self.assertEqual(
2333- translation_entry.status, RosettaImportStatus.IMPORTED)
2334- self.assertEqual(template_entry.status, RosettaImportStatus.IMPORTED)
2335-
2336- # Let's validate the content of the messages.
2337- potmsgsets = list(self.firefox_template.getPOTMsgSets())
2338-
2339- messages = [message.msgid_singular.msgid for message in potmsgsets]
2340- messages.sort()
2341- self.assertEqual(
2342- [u'foozilla.happytitle',
2343- u'foozilla.menu.accesskey',
2344- u'foozilla.menu.commandkey',
2345- u'foozilla.menu.title',
2346- u'foozilla.name',
2347- u'foozilla.nocomment',
2348- u'foozilla.play.fire',
2349- u'foozilla.play.ice',
2350- u'foozilla.title',
2351- u'foozilla.utf8',
2352- u'foozilla_something'],
2353- messages)
2354-
2355- potmsgset = self.firefox_template.getPOTMsgSetByMsgIDText(
2356- u'foozilla.name', context='main/test1.dtd')
2357- translation = potmsgset.getCurrentTranslation(
2358- self.firefox_template, self.spanish_firefox.language,
2359- self.firefox_template.translation_side)
2360-
2361- # It's a normal message that lacks any comment.
2362- self.assertEqual(potmsgset.singular_text, u'FooZilla!')
2363-
2364- # With this first import, upstream and Ubuntu translations must match.
2365- self.assertEqual(
2366- translation.translations,
2367- potmsgset.getOtherTranslation(
2368- self.spanish_firefox.language,
2369- self.firefox_template.translation_side).translations)
2370-
2371- potmsgset = self.firefox_template.getPOTMsgSetByMsgIDText(
2372- u'foozilla.menu.accesskey', context='main/subdir/test2.dtd')
2373-
2374- # access key is a special notation that is supposed to be
2375- # translated with a key shortcut.
2376- self.assertEqual(potmsgset.singular_text, u'M')
2377- # The comment shows the key used when there is no translation,
2378- # which is noted as the en_US translation.
2379- self.assertEqual(
2380- unwrap(potmsgset.sourcecomment),
2381- unwrap(access_key_source_comment))
2382- # But for the translation import, we get the key directly.
2383- self.assertEqual(
2384- potmsgset.getOtherTranslation(
2385- self.spanish_firefox.language,
2386- self.firefox_template.translation_side).translations,
2387- [u'M'])
2388-
2389- potmsgset = self.firefox_template.getPOTMsgSetByMsgIDText(
2390- u'foozilla.menu.commandkey', context='main/subdir/test2.dtd')
2391- # command key is a special notation that is supposed to be
2392- # translated with a key shortcut.
2393- self.assertEqual(
2394- potmsgset.singular_text, u'm')
2395- # The comment shows the key used when there is no translation,
2396- # which is noted as the en_US translation.
2397- self.assertEqual(
2398- unwrap(potmsgset.sourcecomment),
2399- unwrap(command_key_source_comment))
2400- # But for the translation import, we get the key directly.
2401- self.assertEqual(
2402- potmsgset.getOtherTranslation(
2403- self.spanish_firefox.language,
2404- self.firefox_template.translation_side).translations,
2405- [u'm'])
2406-
2407- def test_GetLastTranslator(self):
2408- """Tests whether we extract last translator information correctly."""
2409- translation_entry = self.setUpTranslationImportQueueForTranslation(
2410- 'en-US')
2411- importer = MozillaXpiImporter()
2412- translation_file = importer.parse(translation_entry)
2413-
2414- # Let's try with the translation file, it has valid Last Translator
2415- # information.
2416- name, email = translation_file.header.getLastTranslator()
2417- self.assertEqual(name, u'Carlos Perell\xf3 Mar\xedn')
2418- self.assertEqual(email, u'carlos@canonical.com')
2419-
2420- def test_Contexts(self):
2421- """Test that message context in XPI file is set to chrome path."""
2422- queue_entry = self.setUpTranslationImportQueueForTranslation(
2423- 'clashing_ids')
2424- importer = MozillaXpiImporter()
2425- template = importer.parse(queue_entry)
2426-
2427- messages = sorted([
2428- (message.msgid_singular, message.context, message.singular_text)
2429- for message in template.messages])
2430- self.assertEqual(
2431- [
2432- (u'foozilla.clashing.key',
2433- u'mac/extra.dtd',
2434- u'This message is Mac-specific, and comes from DTD.'),
2435- (u'foozilla.clashing.key',
2436- u'mac/extra.properties',
2437- u'This message is Mac-specific, and comes from properties.'),
2438- (u'foozilla.clashing.key',
2439- u'main/main.dtd',
2440- u'This message is in the main DTD.'),
2441- (u'foozilla.clashing.key',
2442- u'main/main.properties',
2443- u'This message is in the main properties file.'),
2444- (u'foozilla.clashing.key',
2445- u'unix/extra.dtd',
2446- u'This message is Unix-specific, and comes from DTD.'),
2447- (u'foozilla.clashing.key',
2448- u'unix/extra.properties',
2449- u'This message is Unix-specific, and comes from properties.'),
2450- (u'foozilla.clashing.key',
2451- u'win/extra.dtd',
2452- u'This message is Windows-specific, and comes from DTD.'),
2453- (u'foozilla.clashing.key',
2454- u'win/extra.properties',
2455- u'This message is Windows-specific, '
2456- 'and comes from properties.'),
2457- (u'foozilla.regular.message',
2458- u'main/main.dtd',
2459- u'A non-clashing message.'),
2460- ],
2461- messages)
2462-
2463- def test_SystemEntityIsIgnored(self):
2464- """Test handling of SYSTEM entities in DTD files."""
2465- self.setUpTranslationImportQueueForTemplate('system-entity')
2466- msgids = [
2467- (potmsgset.msgid_singular.msgid, potmsgset.singular_text)
2468- for potmsgset in self.firefox_template.getPOTMsgSets()]
2469- self.assertEqual(msgids, [
2470- ('firststring', 'First translatable string'),
2471- ('secondstring', 'Second translatable string')])
2472diff --git a/lib/lp/translations/utilities/tests/test_xpi_manifest.py b/lib/lp/translations/utilities/tests/test_xpi_manifest.py
2473deleted file mode 100644
2474index b5348df..0000000
2475--- a/lib/lp/translations/utilities/tests/test_xpi_manifest.py
2476+++ /dev/null
2477@@ -1,307 +0,0 @@
2478-# Copyright 2009 Canonical Ltd. This software is licensed under the
2479-# GNU Affero General Public License version 3 (see the file LICENSE).
2480-
2481-"""Unit tests for XPI manifests."""
2482-
2483-__metaclass__ = type
2484-
2485-import unittest
2486-
2487-from lp.translations.interfaces.translationimporter import (
2488- TranslationFormatSyntaxError,
2489- )
2490-from lp.translations.utilities.xpi_manifest import XpiManifest
2491-
2492-
2493-class XpiManifestTestCase(unittest.TestCase):
2494- """Test `XpiManifest`."""
2495-
2496- def test_TrivialParse(self):
2497- # Parse and use minimal manifest.
2498- manifest = XpiManifest("locale chromepath en-US directory/")
2499- self.assertEqual(len(manifest._locales), 1)
2500- chrome_path, locale = manifest.getChromePathAndLocale(
2501- 'directory/file.dtd')
2502- self.assertIsNotNone(chrome_path, "Failed to match simple path")
2503- self.assertEqual(
2504- chrome_path, "chromepath/file.dtd", "Bad chrome path")
2505-
2506- def test_NonMatch(self):
2507- # Failure to match path.
2508- manifest = XpiManifest("locale chromepath en-US directory/")
2509- chrome_path, locale = manifest.getChromePathAndLocale(
2510- 'nonexistent/file')
2511- self.assertIsNone(chrome_path, "Unexpected path match.")
2512- self.assertIsNone(locale, "Got locale without a match.")
2513-
2514- def test_NoUsefulLines(self):
2515- # Parse manifest without useful data. Lines that don't match what
2516- # we're looking for are ignored.
2517- manifest = XpiManifest("""
2518- There are no usable
2519- locale lines
2520- in this file.
2521- """.lstrip())
2522- self.assertEqual(len(manifest._locales), 0)
2523- chrome_path, locale = manifest.getChromePathAndLocale('lines')
2524- self.assertIsNone(chrome_path, "Empty manifest matched a path.")
2525- chrome_path, locale = manifest.getChromePathAndLocale('')
2526- self.assertIsNone(chrome_path, "Matched empty path.")
2527-
2528- def _checkSortOrder(self, manifest):
2529- """Verify that manifest is sorted by increasing path length."""
2530- last_entry = None
2531- for entry in manifest._locales:
2532- if last_entry is not None:
2533- self.assertFalse(len(entry.path) < len(last_entry.path),
2534- "Manifest entries not sorted by increasing path length.")
2535- last_entry = entry
2536-
2537- def test_MultipleLines(self):
2538- # Parse manifest file with multiple entries.
2539- manifest = XpiManifest("""
2540- locale foo en-US foodir/
2541- locale bar en-US bardir/
2542- locale ixx en-US ixxdir/
2543- locale gna en-US gnadir/
2544- """.lstrip())
2545- self.assertEqual(len(manifest._locales), 4)
2546- self._checkSortOrder(manifest)
2547- for dir in ['gna', 'bar', 'ixx', 'foo']:
2548- path = "%sdir/file.html" % dir
2549- chrome_path, locale = manifest.getChromePathAndLocale(path)
2550- self.assertEqual(chrome_path, "%s/file.html" % dir,
2551- "Bad chrome path in multi-line parse.")
2552- self.assertEqual(
2553- locale, 'en-US', "Bad locale in multi-line parse.")
2554-
2555- def test_MultipleLocales(self):
2556- # Different locales.
2557- dirs = {
2558- 'foo': 'en-US',
2559- 'bar': 'es',
2560- 'ixx': 'zh_CN',
2561- 'zup': 'zh_TW',
2562- 'gna': 'pt',
2563- 'gnu': 'pt_BR'
2564- }
2565- manifest_text = '\n'.join([
2566- "locale %s %s %sdir/\n" % (dir, locale, dir)
2567- for dir, locale in dirs.iteritems()
2568- ])
2569- manifest = XpiManifest(manifest_text)
2570- self._checkSortOrder(manifest)
2571- for dir, dirlocale in dirs.iteritems():
2572- path = "%sdir/file.html" % dir
2573- chrome_path, locale = manifest.getChromePathAndLocale(path)
2574- self.assertEqual(chrome_path, "%s/file.html" % dir,
2575- "Bad chrome path in multi-line parse.")
2576- self.assertEqual(locale, dirlocale, "Locales got mixed up.")
2577-
2578- def test_IgnoredLines(self):
2579- # Ignored lines: anything that doesn't start with "locale" or doesn't
2580- # have the right number of arguments. The one correct line is picked
2581- # out though.
2582- manifest = XpiManifest("""
2583- nonlocale obsolete fr foodir/
2584- anotherline
2585-
2586- #locale obsolete fr foodir/
2587- locale okay fr foodir/
2588- locale overlong fr foordir/ etc. etc. etc.
2589- locale incomplete fr
2590- """.lstrip())
2591- self.assertEqual(len(manifest._locales), 1)
2592- chrome_path, locale = manifest.getChromePathAndLocale('foodir/x')
2593- self.assertIsNotNone(chrome_path, "Garbage lines messed up match.")
2594- self.assertEqual(chrome_path, "okay/x", "Matched wrong line.")
2595- self.assertEqual(locale, "fr", "Inexplicably mismatched locale.")
2596-
2597- def test_DuplicateLines(self):
2598- # The manifest ignores redundant lines with the same path.
2599- manifest = XpiManifest("""
2600- locale dup fy boppe
2601- locale dup fy boppe
2602- """.lstrip())
2603- self.assertEqual(len(manifest._locales), 1)
2604-
2605- def _checkLookup(self, manifest, path, chrome_path, locale):
2606- """Helper: look up `path` in `manifest`, expect given output."""
2607- found_chrome_path, found_locale = manifest.getChromePathAndLocale(
2608- path)
2609- self.assertIsNotNone(found_chrome_path, "No match found for " + path)
2610- self.assertEqual(found_chrome_path, chrome_path)
2611- self.assertEqual(found_locale, locale)
2612-
2613- def test_NormalizedLookup(self):
2614- # Both sides of a path lookup are normalized, so that a matching
2615- # prefix is recognized in a path even if the two have some meaningless
2616- # differences in their spelling.
2617- manifest = XpiManifest("locale x nn //a/dir")
2618- self._checkLookup(manifest, "a//dir///etc", 'x/etc', 'nn')
2619-
2620- def _checkNormalize(self, bad_path, good_path):
2621- """Test that `bad_path` normalizes to `good_path`."""
2622- self.assertEqual(XpiManifest._normalizePath(bad_path), good_path)
2623-
2624- def test_Normalize(self):
2625- # These paths are all wrong or difficult for one reason or another.
2626- # Check that the normalization of paths renders those little
2627- # imperfections irrelevant to path lookup.
2628- self._checkNormalize('x/', 'x/')
2629- self._checkNormalize('x', 'x')
2630- self._checkNormalize('/x', 'x')
2631- self._checkNormalize('//x', 'x')
2632- self._checkNormalize('/x/', 'x/')
2633- self._checkNormalize('x//', 'x/')
2634- self._checkNormalize('x///', 'x/')
2635- self._checkNormalize('x/y/', 'x/y/')
2636- self._checkNormalize('x/y', 'x/y')
2637- self._checkNormalize('x//y/', 'x/y/')
2638-
2639- def test_PathBoundaries(self):
2640- # Paths can only match on path boundaries, where the slashes are
2641- # supposed to be.
2642- manifest = XpiManifest("""
2643- locale short el /ploink/squit
2644- locale long he /ploink/squittle
2645- """.lstrip())
2646- self._checkSortOrder(manifest)
2647- self._checkLookup(manifest, 'ploink/squit/x', 'short/x', 'el')
2648- self._checkLookup(manifest, '/ploink/squittle/x', 'long/x', 'he')
2649-
2650- def test_Overlap(self):
2651- # Path matching looks for longest prefix. Make sure this works right,
2652- # even when nested directories are in "overlapping" manifest entries.
2653- manifest = XpiManifest("""
2654- locale foo1 ca a/
2655- locale foo2 ca a/b/
2656- locale foo3 ca a/b/c/x1
2657- locale foo4 ca a/b/c/x2
2658- """.lstrip())
2659- self._checkSortOrder(manifest)
2660- self._checkLookup(manifest, 'a/bb', 'foo1/bb', 'ca')
2661- self._checkLookup(manifest, 'a/bb/c', 'foo1/bb/c', 'ca')
2662- self._checkLookup(manifest, 'a/b/y', 'foo2/y', 'ca')
2663- self._checkLookup(manifest, 'a/b/c/', 'foo2/c/', 'ca')
2664- self._checkLookup(manifest, 'a/b/c/x12', 'foo2/c/x12', 'ca')
2665- self._checkLookup(manifest, 'a/b/c/x1/y', 'foo3/y', 'ca')
2666- self._checkLookup(manifest, 'a/b/c/x2/y', 'foo4/y', 'ca')
2667-
2668- def test_JarLookup(self):
2669- # Simple, successful lookup of a correct path inside a jar file.
2670- manifest = XpiManifest("""
2671- locale foo en_GB jar:foo.jar!/dir/
2672- locale bar id jar:bar.jar!/
2673- """.lstrip())
2674- self._checkSortOrder(manifest)
2675- self._checkLookup(
2676- manifest, 'jar:foo.jar!/dir/file', 'foo/file', 'en_GB')
2677- self._checkLookup(
2678- manifest, 'jar:bar.jar!/dir/file', 'bar/dir/file', 'id')
2679-
2680- def test_JarNormalization(self):
2681- # Various badly-formed or corner-case paths. All get normalized.
2682- self._checkNormalize('jar:jarless/path', 'jarless/path')
2683- self._checkNormalize(
2684- 'jar:foo.jar!/contained/file', 'jar:foo.jar!/contained/file')
2685- self._checkNormalize(
2686- 'foo.jar!contained/file', 'jar:foo.jar!/contained/file')
2687- self._checkNormalize(
2688- 'jar:foo.jar!//contained/file', 'jar:foo.jar!/contained/file')
2689- self._checkNormalize('splat.jar!', 'jar:splat.jar!/')
2690- self._checkNormalize('dir/x.jar!dir', 'jar:dir/x.jar!/dir')
2691-
2692- def test_NestedJarNormalization(self):
2693- # Test that paths with jars inside jars are normalized correctly.
2694- self._checkNormalize(
2695- 'jar:dir/x.jar!/y.jar!/dir', 'jar:dir/x.jar!/y.jar!/dir')
2696- self._checkNormalize(
2697- 'dir/x.jar!y.jar!dir', 'jar:dir/x.jar!/y.jar!/dir')
2698- self._checkNormalize(
2699- 'dir/x.jar!/dir/y.jar!', 'jar:dir/x.jar!/dir/y.jar!/')
2700-
2701- def test_JarMixup(self):
2702- # Two jar files can have files for the same locale. Two locales can
2703- # have files in the same jar file. Two translations in different
2704- # places can have the same chrome path.
2705- manifest = XpiManifest("""
2706- locale serbian sr jar:translations.jar!/sr/
2707- locale croatian hr jar:translations.jar!/hr/
2708- locale docs sr jar:docs.jar!/sr/
2709- locale docs hr jar:docs.jar!/hr/
2710- """.lstrip())
2711- self._checkSortOrder(manifest)
2712- self._checkLookup(
2713- manifest, 'jar:translations.jar!/sr/x', 'serbian/x', 'sr')
2714- self._checkLookup(
2715- manifest, 'jar:translations.jar!/hr/x', 'croatian/x', 'hr')
2716- self._checkLookup(manifest, 'jar:docs.jar!/sr/x', 'docs/x', 'sr')
2717- self._checkLookup(manifest, 'jar:docs.jar!/hr/x', 'docs/x', 'hr')
2718-
2719- def test_NestedJars(self):
2720- # Jar files can be contained in jar files.
2721- manifest = XpiManifest("""
2722- locale x it jar:dir/x.jar!/subdir/y.jar!/
2723- locale y it jar:dir/x.jar!/subdir/y.jar!/deep/
2724- locale z it jar:dir/x.jar!/subdir/z.jar!/
2725- """.lstrip())
2726- self._checkSortOrder(manifest)
2727- self._checkLookup(
2728- manifest, 'jar:dir/x.jar!/subdir/y.jar!/foo', 'x/foo', 'it')
2729- self._checkLookup(
2730- manifest, 'jar:dir/x.jar!/subdir/y.jar!/deep/foo', 'y/foo', 'it')
2731- self._checkLookup(
2732- manifest, 'dir/x.jar!/subdir/z.jar!/foo', 'z/foo', 'it')
2733-
2734- def test_ContainsLocales(self):
2735- # Jar files need to be descended into if any locale line mentions a
2736- # path inside them.
2737- manifest = XpiManifest("locale in my jar:x/foo.jar!/y")
2738- self.assertTrue(manifest.containsLocales("jar:x/foo.jar!/"))
2739- self.assertFalse(manifest.containsLocales("jar:zzz/foo.jar!/"))
2740-
2741- def test_NormalizeContainsLocales(self):
2742- # "containsLocales" lookup is normalized, just like chrome path
2743- # lookup, so it's not fazed by syntactical misspellings.
2744- manifest = XpiManifest("locale main kh jar:/x/foo.jar!bar.jar!")
2745- self.assertTrue(manifest.containsLocales("x/foo.jar!//bar.jar!/"))
2746-
2747- def test_ReverseMapping(self):
2748- # Test "reverse mapping" from chrome path to XPI path.
2749- manifest = XpiManifest(
2750- "locale browser en-US jar:locales/en-US.jar!/chrome/")
2751- path = manifest.findMatchingXpiPath('browser/gui/print.dtd', 'en-US')
2752- self.assertEqual(path, "jar:locales/en-US.jar!/chrome/gui/print.dtd")
2753-
2754- def test_NoReverseMapping(self):
2755- # Failed reverse lookup.
2756- manifest = XpiManifest(
2757- "locale browser en-US jar:locales/en-US.jar!/chrome/")
2758- path = manifest.findMatchingXpiPath('manual/gui/print.dtd', 'en-US')
2759- self.assertEqual(path, None)
2760-
2761- def test_ReverseMappingWrongLocale(self):
2762- # Reverse mapping fails if given the wrong locale.
2763- manifest = XpiManifest(
2764- "locale browser en-US jar:locales/en-US.jar!/chrome/")
2765- path = manifest.findMatchingXpiPath('browser/gui/print.dtd', 'pt')
2766- self.assertEqual(path, None)
2767-
2768- def test_ReverseMappingLongestMatch(self):
2769- # Reverse mapping always finds the longest match.
2770- manifest = XpiManifest("""
2771- locale browser en-US jar:locales/
2772- locale browser en-US jar:locales/en-US.jar!/chrome/
2773- locale browser en-US jar:locales/en-US.jar!/
2774- """.lstrip())
2775- path = manifest.findMatchingXpiPath('browser/gui/print.dtd', 'en-US')
2776- self.assertEqual(path, "jar:locales/en-US.jar!/chrome/gui/print.dtd")
2777-
2778- def test_blank_line(self):
2779- # Manifests must not begin with newline.
2780- self.assertRaises(
2781- TranslationFormatSyntaxError,
2782- XpiManifest, """
2783- locale browser en-US jar:locales
2784- """)
2785diff --git a/lib/lp/translations/utilities/tests/test_xpi_po_exporter.py b/lib/lp/translations/utilities/tests/test_xpi_po_exporter.py
2786index 532f9bf..ca830c7 100644
2787--- a/lib/lp/translations/utilities/tests/test_xpi_po_exporter.py
2788+++ b/lib/lp/translations/utilities/tests/test_xpi_po_exporter.py
2789@@ -1,21 +1,29 @@
2790+# -*- coding: utf-8 -*-
2791+# NOTE: The first line above must stay first; do not move the copyright
2792+# notice to the top. See http://www.python.org/dev/peps/pep-0263/.
2793+#
2794 # Copyright 2009-2017 Canonical Ltd. This software is licensed under the
2795 # GNU Affero General Public License version 3 (see the file LICENSE).
2796
2797 __metaclass__ = type
2798
2799 from textwrap import dedent
2800-import unittest
2801
2802+from fixtures import MonkeyPatch
2803 import transaction
2804 from zope.component import (
2805 getAdapter,
2806 getUtility,
2807 )
2808+from zope.interface import implementer
2809 from zope.interface.verify import verifyObject
2810+from zope.security.proxy import removeSecurityProxy
2811
2812 from lp.app.interfaces.launchpad import ILaunchpadCelebrities
2813 from lp.registry.interfaces.person import IPersonSet
2814 from lp.registry.interfaces.product import IProductSet
2815+from lp.testing import TestCase
2816+from lp.testing.fixture import ZopeUtilityFixture
2817 from lp.testing.layers import LaunchpadZopelessLayer
2818 from lp.translations.enums import RosettaImportStatus
2819 from lp.translations.interfaces.potemplate import IPOTemplateSet
2820@@ -25,21 +33,165 @@ from lp.translations.interfaces.translationcommonformat import (
2821 from lp.translations.interfaces.translationexporter import (
2822 ITranslationFormatExporter,
2823 )
2824+from lp.translations.interfaces.translationfileformat import (
2825+ TranslationFileFormat,
2826+ )
2827+from lp.translations.interfaces.translationimporter import (
2828+ ITranslationFormatImporter,
2829+ ITranslationImporter,
2830+ )
2831 from lp.translations.interfaces.translationimportqueue import (
2832 ITranslationImportQueue,
2833 )
2834-from lp.translations.utilities.tests.test_xpi_import import (
2835- get_en_US_xpi_file_to_import,
2836+from lp.translations.interfaces.translations import TranslationConstants
2837+from lp.translations.utilities.translation_common_format import (
2838+ TranslationFileData,
2839+ TranslationMessageData,
2840 )
2841 from lp.translations.utilities.translation_export import ExportFileStorage
2842+from lp.translations.utilities.xpi_header import XpiHeader
2843 from lp.translations.utilities.xpi_po_exporter import XPIPOExporter
2844
2845
2846-class XPIPOExporterTestCase(unittest.TestCase):
2847+# Hardcoded representations of what used to be found in
2848+# lib/lp/translations/utilities/tests/firefox-data/. We no longer have real
2849+# XPI import code, so we pre-parse the messages.
2850+test_xpi_header = dedent(u'''\
2851+ <?xml version="1.0"?>
2852+ <RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
2853+ xmlns:em="http://www.mozilla.org/2004/em-rdf#">
2854+ <Description about="urn:mozilla:install-manifest"
2855+ em:id="langpack-en-US@firefox.mozilla.org"
2856+ em:name="English U.S. (en-US) Language Pack"
2857+ em:version="2.0"
2858+ em:type="8"
2859+ em:creator="Danilo Å egan">
2860+ <em:contributor>Š”Š°Š½ŠøŠ»Š¾ ŠØŠµŠ³Š°Š½</em:contributor>
2861+ <em:contributor>Carlos PerellĆ³ MarĆ­n &lt;carlos@canonical.com&gt;</em:contributor>
2862+
2863+ <em:targetApplication>
2864+ <Description>
2865+ <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id><!-- firefox -->
2866+ <em:minVersion>2.0</em:minVersion>
2867+ <em:maxVersion>2.0.0.*</em:maxVersion>
2868+ </Description>
2869+ </em:targetApplication>
2870+ </Description>
2871+ </RDF>
2872+''')
2873+test_xpi_messages = [
2874+ (u'foozilla.menu.title', u'main/subdir/test2.dtd',
2875+ u'jar:chrome/en-US.jar!/subdir/test2.dtd', u'MENU',
2876+ u' This is a DTD file inside a subdirectory\n'),
2877+ (u'foozilla.menu.accesskey', u'main/subdir/test2.dtd',
2878+ u'jar:chrome/en-US.jar!/subdir/test2.dtd', u'M',
2879+ dedent(u'''\
2880+ Select the access key that you want to use. These have
2881+ to be translated in a way that the selected character is
2882+ present in the translated string of the label being
2883+ referred to, for example 'i' in 'Edit' menu item in
2884+ English. If a translation already exists, please don't
2885+ change it if you are not sure about it. Please find the
2886+ context of the key from the end of the 'Located in' text
2887+ below.
2888+ ''')),
2889+ (u'foozilla.menu.commandkey', u'main/subdir/test2.dtd',
2890+ u'jar:chrome/en-US.jar!/subdir/test2.dtd', u'm',
2891+ dedent(u'''\
2892+ Select the shortcut key that you want to use. It
2893+ should be translated, but often shortcut keys (for
2894+ example Ctrl + KEY) are not changed from the original. If
2895+ a translation already exists, please don't change it if
2896+ you are not sure about it. Please find the context of
2897+ the key from the end of the 'Located in' text below.
2898+ ''')),
2899+ (u'foozilla_something', u'main/subdir/test2.properties',
2900+ u'jar:chrome/en-US.jar!/subdir/test2.properties:6', u'SomeZilla',
2901+ dedent(u'''\
2902+ Translators, what you are seeing now is a lovely,
2903+ awesome, multiline comment aimed at you directly
2904+ from the streets of a .properties file
2905+ ''')),
2906+ (u'foozilla.name', u'main/test1.dtd',
2907+ u'jar:chrome/en-US.jar!/test1.dtd', u'FooZilla!', None),
2908+ (u'foozilla.play.fire', u'main/test1.dtd',
2909+ u'jar:chrome/en-US.jar!/test1.dtd', u'Do you want to play with fire?',
2910+ u" Translators, don't play with fire!\n"),
2911+ (u'foozilla.play.ice', u'main/test1.dtd',
2912+ u'jar:chrome/en-US.jar!/test1.dtd', u'Play with ice?',
2913+ u' This is just a comment, not a comment for translators\n'),
2914+ (u'foozilla.title', u'main/test1.properties',
2915+ u'jar:chrome/en-US.jar!/test1.properties:1', u'FooZilla Zilla Thingy',
2916+ None),
2917+ (u'foozilla.happytitle', u'main/test1.properties',
2918+ u'jar:chrome/en-US.jar!/test1.properties:3',
2919+ u'http://foozillingy.happy.net/',
2920+ u"Translators, if you're older than six, don't translate this\n"),
2921+ (u'foozilla.nocomment', u'main/test1.properties',
2922+ u'jar:chrome/en-US.jar!/test1.properties:4', u'No Comment',
2923+ u'(Except this one)\n'),
2924+ (u'foozilla.utf8', u'main/test1.properties',
2925+ u'jar:chrome/en-US.jar!/test1.properties:5', u'\u0414\u0430\u043d=Day',
2926+ None),
2927+ ]
2928+
2929+
2930+class FakeXPIMessage(TranslationMessageData):
2931+ """Simulate an XPI translation message."""
2932+
2933+ def __init__(self, key, chrome_path, file_and_line, value, last_comment):
2934+ super(FakeXPIMessage, self).__init__()
2935+ self.msgid_singular = key
2936+ self.context = chrome_path
2937+ self.file_references = '%s(%s)' % (file_and_line, key)
2938+ value = value.strip()
2939+ self.addTranslation(TranslationConstants.SINGULAR_FORM, value)
2940+ self.singular_text = value
2941+ self.source_comment = last_comment
2942+
2943+
2944+@implementer(ITranslationFormatImporter)
2945+class FakeXPIImporter:
2946+ """Simulate an XPI import. We no longer have real XPI import code."""
2947+
2948+ def getFormat(self, file_contents):
2949+ """See `ITranslationFormatImporter`."""
2950+ return TranslationFileFormat.XPI
2951+
2952+ priority = 0
2953+ content_type = 'application/zip'
2954+ file_extensions = ['.xpi']
2955+ template_suffix = 'en-US.xpi'
2956+ uses_source_string_msgids = True
2957+
2958+ def parse(self, translation_import_queue_entry):
2959+ """See `ITranslationFormatImporter`.
2960+
2961+ This takes a `TranslationImportQueueEntry` to satisfy the interface,
2962+ but ignores it and returns hardcoded data instead.
2963+ """
2964+ translation_file = TranslationFileData()
2965+ translation_file.header = XpiHeader(test_xpi_header)
2966+ translation_file.messages = [
2967+ FakeXPIMessage(*message) for message in test_xpi_messages]
2968+ return translation_file
2969+
2970+ def getHeaderFromString(self, header_string):
2971+ """See `ITranslationFormatImporter`.
2972+
2973+ This takes a header string to satisfy the interface, but ignores it
2974+ and returns hardcoded data instead.
2975+ """
2976+ return XpiHeader(test_xpi_header)
2977+
2978+
2979+class XPIPOExporterTestCase(TestCase):
2980 """Class test for gettext's .po file exports"""
2981 layer = LaunchpadZopelessLayer
2982
2983 def setUp(self):
2984+ super(XPIPOExporterTestCase, self).setUp()
2985+
2986 self.translation_exporter = XPIPOExporter()
2987
2988 # Get the importer.
2989@@ -75,14 +227,24 @@ class XPIPOExporterTestCase(unittest.TestCase):
2990
2991 def setUpTranslationImportQueueForTemplate(self):
2992 """Return an ITranslationImportQueueEntry for testing purposes."""
2993- # Get the file to import.
2994- en_US_xpi = get_en_US_xpi_file_to_import('en-US')
2995+ # Install a fake XPI importer, since we no longer have real XPI
2996+ # import code.
2997+ fake_xpi_importer = FakeXPIImporter()
2998+ self.useFixture(MonkeyPatch(
2999+ 'lp.translations.utilities.translation_import.importers',
3000+ {TranslationFileFormat.XPI: fake_xpi_importer}))
3001+ # Temporarily reinstall the translation importer without a security
3002+ # proxy. This avoids problems getting attributes of
3003+ # FakeXPIImporter, which has no Zope permissions defined.
3004+ self.useFixture(ZopeUtilityFixture(
3005+ removeSecurityProxy(getUtility(ITranslationImporter)),
3006+ ITranslationImporter))
3007
3008 # Attach it to the import queue.
3009 translation_import_queue = getUtility(ITranslationImportQueue)
3010 by_maintainer = True
3011 entry = translation_import_queue.addOrUpdateEntry(
3012- self.firefox_template.path, en_US_xpi, by_maintainer,
3013+ self.firefox_template.path, b'dummy', by_maintainer,
3014 self.importer, productseries=self.firefox_template.productseries,
3015 potemplate=self.firefox_template)
3016
3017diff --git a/lib/lp/translations/utilities/tests/test_xpi_properties_format.py b/lib/lp/translations/utilities/tests/test_xpi_properties_format.py
3018deleted file mode 100644
3019index 1d42b14..0000000
3020--- a/lib/lp/translations/utilities/tests/test_xpi_properties_format.py
3021+++ /dev/null
3022@@ -1,302 +0,0 @@
3023-# Copyright 2009 Canonical Ltd. This software is licensed under the
3024-# GNU Affero General Public License version 3 (see the file LICENSE).
3025-
3026-__metaclass__ = type
3027-
3028-from textwrap import dedent
3029-import unittest
3030-
3031-from lp.translations.interfaces.translationimporter import (
3032- TranslationFormatInvalidInputError,
3033- )
3034-from lp.translations.utilities.mozilla_xpi_importer import PropertyFile
3035-from lp.translations.utilities.xpi_properties_exporter import (
3036- XpiPropertiesSubExporter,
3037- )
3038-
3039-
3040-class PropertyFileFormatTestCase(unittest.TestCase):
3041- """Test class for property file format."""
3042-
3043- def _baseContentEncodingTest(self, content):
3044- """This is a base function to check different encodings."""
3045- property_file = PropertyFile('test.properties', None, dedent(content))
3046-
3047- expected = {u'default-first-title-mac': [u'Introducci\xf3n'],
3048- u'default-last-title-mac': [u'Conclusi\xf3n']}
3049- parsed = dict([(message.msgid_singular, message.translations)
3050- for message in property_file.messages])
3051- self.assertEqual(expected, parsed)
3052-
3053- def test_UTF8PropertyFileTest(self):
3054- """This test makes sure that we handle UTF-8 encoding files."""
3055- content = '''
3056- default-first-title-mac = Introducci\xc3\xb3n
3057- default-last-title-mac = Conclusi\xc3\xb3n
3058- '''
3059- self._baseContentEncodingTest(content)
3060-
3061- def test_UnicodeEscapedPropertyFileTest(self):
3062- """This test makes sure that we handle unicode escaped files."""
3063- content = '''
3064- default-first-title-mac=Introducci\u00F3n
3065- default-last-title-mac=Conclusi\u00F3n
3066- '''
3067- self._baseContentEncodingTest(content)
3068-
3069- def test_InvalidPropertyFileUnicodeEscape(self):
3070- # An invalid Unicode escape sequence is a
3071- # TranslationFormatInvalidInputError.
3072- content = '''
3073- weirdness=\u1
3074- '''
3075- self.assertRaises(
3076- TranslationFormatInvalidInputError, PropertyFile, None,
3077- 'test.properties', content)
3078-
3079- def test_Latin1PropertyFileTest(self):
3080- """This test makes sure that we detect bad encodings."""
3081- content = '''
3082- default-first-title-mac = Introducci\xf3n
3083- default-last-title-mac = Conclusi\xf3n
3084- '''
3085- self.assertRaises(
3086- TranslationFormatInvalidInputError, PropertyFile, None,
3087- 'test.properties', content)
3088-
3089- def test_TrailingBackslashPropertyFileTest(self):
3090- """Test whether trailing backslashes are well handled.
3091-
3092- A trailing backslash as last char in the line continue the string in
3093- the following document line.
3094- """
3095- content = '''
3096-default-first-title-mac=Introd\
3097-ucci\u00F3n
3098-'''
3099- property_file = PropertyFile('test.properties', None, dedent(content))
3100-
3101- expected = {u'default-first-title-mac': [u'Introducci\xf3n']}
3102- parsed = dict([(message.msgid_singular, message.translations)
3103- for message in property_file.messages])
3104- self.assertEqual(expected, parsed)
3105-
3106- def test_EscapedQuotesPropertyFileTest(self):
3107- """Test whether escaped quotes are well handled.
3108-
3109- Escaped quotes must be stored unescaped.
3110- """
3111- content = 'default-first-title-mac = \\\'Something\\\' \\\"more\\\"'
3112-
3113- property_file = PropertyFile('test.properties', None, dedent(content))
3114-
3115- expected = {u'default-first-title-mac': [u'\'Something\' \"more\"']}
3116- parsed = dict([(message.msgid_singular, message.translations)
3117- for message in property_file.messages])
3118- self.assertEqual(expected, parsed)
3119-
3120- def test_WholeLineCommentPropertyFileTest(self):
3121- """Test whether whole line comments are well handled."""
3122- content = '''
3123- # Foo bar comment.
3124- default-first-title-mac = blah
3125-
3126- # This comment should be ignored.
3127-
3128- foo = bar
3129- '''
3130-
3131- property_file = PropertyFile('test.properties', None, dedent(content))
3132- expected = {u'default-first-title-mac': u'Foo bar comment.\n',
3133- u'foo': None}
3134- parsed = dict([(message.msgid_singular, message.source_comment)
3135- for message in property_file.messages])
3136- self.assertEqual(expected, parsed)
3137-
3138- def test_EndOfLineCommentPropertyFileTest(self):
3139- """Test whether end of line comments are well handled."""
3140-
3141- content = '''
3142- default-first-title-mac = blah // Foo bar comment.
3143-
3144- # This comment should be ignored.
3145- foo = bar // Something
3146- '''
3147-
3148- property_file = PropertyFile('test.properties', None, dedent(content))
3149- expected_comments = {
3150- u'default-first-title-mac': u'Foo bar comment.\n',
3151- u'foo': u'Something\n'
3152- }
3153- parsed_comments = dict(
3154- [(message.msgid_singular, message.source_comment)
3155- for message in property_file.messages])
3156-
3157- self.assertEqual(expected_comments, parsed_comments)
3158-
3159- expected_translations = {
3160- u'default-first-title-mac': [u'blah'],
3161- u'foo': [u'bar']
3162- }
3163- parsed_translations = dict([(message.msgid_singular,
3164- message.translations)
3165- for message in property_file.messages])
3166-
3167- self.assertEqual(expected_translations, parsed_translations)
3168-
3169- def test_MultiLineCommentPropertyFileTest(self):
3170- """Test whether multiline comments are well handled."""
3171- content = '''
3172- /* single line comment */
3173- default-first-title-mac = blah
3174-
3175- /* Multi line comment
3176- yeah, it's multiple! */
3177- foo = bar
3178-
3179- /* Even with nested comment tags, we handle this as multiline comment:
3180- # fooo
3181- foos = bar
3182- something = else // Comment me!
3183- */
3184- long_comment = foo
3185- '''
3186-
3187- property_file = PropertyFile('test.properties', None, dedent(content))
3188- expected = {
3189- u'default-first-title-mac': u' single line comment \n',
3190- u'foo': u" Multi line comment\n yeah, it's multiple! \n",
3191- u'long_comment': (
3192- u' Even with nested comment tags, we handle this as' +
3193- u' multiline comment:\n# fooo\nfoos = bar\n' +
3194- u'something = else // Comment me!\n')
3195- }
3196- parsed = dict([(message.msgid_singular, message.source_comment)
3197- for message in property_file.messages])
3198- self.assertEqual(expected, parsed)
3199-
3200- def test_URLNotComment(self):
3201- """Double slash in a URL is not treated as end-of-line comment."""
3202- content = '''
3203- url = https://admin.example.com/ // Double slash in URL!
3204- '''
3205- property_file = PropertyFile('test.properties', None, dedent(content))
3206- message = None
3207- for entry in property_file.messages:
3208- self.assertEqual(message, None, "More messages than expected.")
3209- message = entry
3210-
3211- self.assertEqual(message.msgid_singular, u"url")
3212- self.assertEqual(message.singular_text, u"https://admin.example.com/")
3213- self.assertEqual(message.source_comment, u"Double slash in URL!\n")
3214-
3215- def test_InvalidLinePropertyFileTest(self):
3216- """Test whether an invalid line is ignored."""
3217- content = '''
3218- # Foo bar comment.
3219- default-first-title-mac = blah
3220-
3221- # This comment should be ignored.
3222- crappy-contnet
3223- foo = bar
3224- '''
3225-
3226- property_file = PropertyFile('test.properties', None, dedent(content))
3227- expected = {u'default-first-title-mac': u'Foo bar comment.\n',
3228- u'foo': None}
3229- parsed = dict([(message.msgid_singular, message.source_comment)
3230- for message in property_file.messages])
3231- self.assertEqual(expected, parsed)
3232-
3233- def test_MultilinePropertyFileTest(self):
3234- """Test parsing of multiline entries."""
3235- content = (
3236- 'multiline-key = This is the first one\\nThis is the second one.')
3237- property_file = PropertyFile('test.properties', None, content)
3238- expected = {
3239- u'multiline-key': (
3240- [u'This is the first one\nThis is the second one.'])
3241- }
3242- parsed = dict([(message.msgid_singular, message.translations)
3243- for message in property_file.messages])
3244- self.assertEqual(expected, parsed)
3245-
3246- def test_WhiteSpaceBeforeComment(self):
3247- """Test that single line comment is detected even with white space."""
3248- content = ' # foo = bar'
3249- property_file = PropertyFile('test.properties', None, content)
3250- # No message should be parsed.
3251- expected = {}
3252- parsed = dict([(message.msgid_singular, message.translations)
3253- for message in property_file.messages])
3254- self.assertEqual(expected, parsed)
3255-
3256-
3257-class MockFile:
3258- """`TranslationFileData` boiled down to its essence for this test."""
3259- def __init__(self, path='test.properties', messages=None):
3260- if messages is None:
3261- messages = []
3262- self.path = path
3263- self.messages = messages
3264-
3265-
3266-class MockMessage:
3267- """`TranslationMessageData` boiled down to its essence for this test."""
3268- def __init__(self, msgid, translation, comment=None):
3269- self.msgid_singular = msgid
3270- self.translations = [translation]
3271- self.comment = comment
3272-
3273-
3274-class PropertyFileExportTest(unittest.TestCase):
3275- """Test XPI `XpiPropertiesSubExporter`."""
3276-
3277- def setUp(self):
3278- self.exporter = XpiPropertiesSubExporter()
3279-
3280- def test_properties_export(self):
3281- # Test plain export of an XPI properties file.
3282- file = MockFile(messages=[
3283- MockMessage('foo', 'bar'),
3284- MockMessage('id', 'translation', comment='comment'),
3285- ])
3286-
3287- expected = dedent("""
3288- foo=bar
3289-
3290- /* comment */
3291- id=translation
3292- """).strip()
3293- self.assertEqual(self.exporter.export(file), expected)
3294-
3295- def test_escape(self):
3296- # Test escaping in properties files.
3297- file = MockFile(messages=[
3298- MockMessage("f'oo", 'b"ar', comment="Escaped quotes"),
3299- MockMessage("f\\oo", "b\\ar", comment="Escaped backslashes"),
3300- ])
3301-
3302- expected = dedent("""
3303- /* Escaped quotes */
3304- f\\'oo=b\\"ar
3305-
3306- /* Escaped backslashes */
3307- f\\\\oo=b\\\\ar
3308- """).strip()
3309-
3310- self.assertEqual(self.exporter.export(file).strip(), expected)
3311-
3312- def test_escape_comment(self):
3313- # Test escaping of comments in properties files. Not fancy like
3314- # actual translation content escaping; just making sure an
3315- # ill-chosen comment does not produce wildly invalid output.
3316- file = MockFile(messages=[
3317- MockMessage("foo", "bar", comment="/*//*/**/ */")])
3318-
3319- expected = dedent("""
3320- /* /*X//*X/**X/ *X/ */
3321- foo=bar
3322- """).strip()
3323-
3324- self.assertEqual(self.exporter.export(file).strip(), expected)
3325diff --git a/lib/lp/translations/utilities/tests/test_xpi_search.py b/lib/lp/translations/utilities/tests/test_xpi_search.py
3326deleted file mode 100644
3327index 7bd6e74..0000000
3328--- a/lib/lp/translations/utilities/tests/test_xpi_search.py
3329+++ /dev/null
3330@@ -1,83 +0,0 @@
3331-# Copyright 2009-2017 Canonical Ltd. This software is licensed under the
3332-# GNU Affero General Public License version 3 (see the file LICENSE).
3333-
3334-"""Functional tests for searching through XPI POTemplates"""
3335-__metaclass__ = type
3336-
3337-import unittest
3338-
3339-from zope.component import getUtility
3340-
3341-from lp.registry.interfaces.person import IPersonSet
3342-from lp.registry.interfaces.product import IProductSet
3343-from lp.testing.layers import LaunchpadZopelessLayer
3344-from lp.translations.enums import RosettaImportStatus
3345-from lp.translations.interfaces.potemplate import IPOTemplateSet
3346-from lp.translations.utilities.tests.helpers import (
3347- import_pofile_or_potemplate,
3348- )
3349-from lp.translations.utilities.tests.xpi_helpers import (
3350- get_en_US_xpi_file_to_import,
3351- )
3352-
3353-
3354-class XpiSearchTestCase(unittest.TestCase):
3355- """XPI file import into Launchpad."""
3356-
3357- layer = LaunchpadZopelessLayer
3358-
3359- def setUp(self):
3360- # Get the importer.
3361- self.importer = getUtility(IPersonSet).getByName('mark')
3362-
3363- # Get the Firefox template.
3364- firefox_product = getUtility(IProductSet).getByName('firefox')
3365- firefox_productseries = firefox_product.getSeries('trunk')
3366- firefox_potemplate_subset = getUtility(IPOTemplateSet).getSubset(
3367- productseries=firefox_productseries)
3368- self.firefox_template = firefox_potemplate_subset.new(
3369- name='firefox',
3370- translation_domain='firefox',
3371- path='en-US.xpi',
3372- owner=self.importer)
3373- self.spanish_firefox = self.firefox_template.newPOFile('es')
3374- self.spanish_firefox.path = 'translations/es.xpi'
3375-
3376- def setUpTranslationImportQueueForTemplate(self, subdir):
3377- """Return an ITranslationImportQueueEntry for testing purposes.
3378-
3379- :param subdir: subdirectory in firefox-data to get XPI data from.
3380- """
3381- # Get the file to import.
3382- en_US_xpi = get_en_US_xpi_file_to_import(subdir)
3383- return import_pofile_or_potemplate(
3384- file_contents=en_US_xpi,
3385- person=self.importer,
3386- potemplate=self.firefox_template)
3387-
3388- def test_templateSearching(self):
3389- """Searching through XPI template returns English 'translations'."""
3390- entry = self.setUpTranslationImportQueueForTemplate('en-US')
3391-
3392- # The status is now IMPORTED:
3393- self.assertEqual(entry.status, RosettaImportStatus.IMPORTED)
3394-
3395- potmsgsets = self.spanish_firefox.findPOTMsgSetsContaining(
3396- text='zilla')
3397- message_list = [message.singular_text for message in potmsgsets]
3398-
3399- self.assertEqual([u'SomeZilla', u'FooZilla!',
3400- u'FooZilla Zilla Thingy'],
3401- message_list)
3402-
3403- def test_templateSearchingForMsgIDs(self):
3404- """Searching returns no results for internal msg IDs."""
3405- entry = self.setUpTranslationImportQueueForTemplate('en-US')
3406-
3407- # The status is now IMPORTED:
3408- self.assertEqual(entry.status, RosettaImportStatus.IMPORTED)
3409-
3410- potmsgsets = list(self.spanish_firefox.findPOTMsgSetsContaining(
3411- text='foozilla.title'))
3412-
3413- self.assertEqual(potmsgsets, [])
3414diff --git a/lib/lp/translations/utilities/tests/xpi_helpers.py b/lib/lp/translations/utilities/tests/xpi_helpers.py
3415deleted file mode 100644
3416index 338c343..0000000
3417--- a/lib/lp/translations/utilities/tests/xpi_helpers.py
3418+++ /dev/null
3419@@ -1,82 +0,0 @@
3420-# Copyright 2009-2018 Canonical Ltd. This software is licensed under the
3421-# GNU Affero General Public License version 3 (see the file LICENSE).
3422-
3423-"""Helper methods for XPI testing"""
3424-__metaclass__ = type
3425-
3426-__all__ = [
3427- 'access_key_source_comment',
3428- 'command_key_source_comment',
3429- 'get_en_US_xpi_file_to_import',
3430- ]
3431-
3432-import os.path
3433-import tempfile
3434-from textwrap import dedent
3435-import zipfile
3436-
3437-import scandir
3438-
3439-import lp.translations
3440-
3441-
3442-command_key_source_comment = dedent(u"""
3443- Select the shortcut key that you want to use. It should be translated,
3444- but often shortcut keys (for example Ctrl + KEY) are not changed from
3445- the original. If a translation already exists, please don't change it
3446- if you are not sure about it. Please find the context of the key from
3447- the end of the 'Located in' text below.
3448- """).strip()
3449-
3450-access_key_source_comment = dedent(u"""
3451- Select the access key that you want to use. These have to be
3452- translated in a way that the selected character is present in the
3453- translated string of the label being referred to, for example 'i' in
3454- 'Edit' menu item in English. If a translation already exists, please
3455- don't change it if you are not sure about it. Please find the context
3456- of the key from the end of the 'Located in' text below.
3457- """).strip()
3458-
3459-
3460-def get_en_US_xpi_file_to_import(subdir):
3461- """Return an en-US.xpi file object ready to be imported.
3462-
3463- The file is generated from utilities/tests/firefox-data/<subdir>.
3464- """
3465- # en-US.xpi file is a ZIP file which contains embedded JAR file (which is
3466- # also a ZIP file) and a couple of other files. Embedded JAR file is
3467- # named 'en-US.jar' and contains translatable resources.
3468-
3469- # Get the root path where the data to generate .xpi file is stored.
3470- test_root = os.path.join(
3471- os.path.dirname(lp.translations.__file__),
3472- 'utilities/tests/firefox-data', subdir)
3473-
3474- # First create a en-US.jar file to be included in XPI file.
3475- jarfile = tempfile.TemporaryFile()
3476- jar = zipfile.ZipFile(jarfile, 'w')
3477- jarlist = []
3478- data_dir = os.path.join(test_root, 'en-US-jar/')
3479- for root, dirs, files in scandir.walk(data_dir):
3480- for name in files:
3481- relative_dir = root[len(data_dir):].strip('/')
3482- jarlist.append(os.path.join(relative_dir, name))
3483- for file_name in jarlist:
3484- f = open(os.path.join(data_dir, file_name), 'r')
3485- jar.writestr(file_name, f.read())
3486- jar.close()
3487- jarfile.seek(0)
3488-
3489- # Add remaining bits and en-US.jar to en-US.xpi.
3490-
3491- xpifile = tempfile.TemporaryFile()
3492- xpi = zipfile.ZipFile(xpifile, 'w')
3493- for xpi_entry in scandir.scandir(test_root):
3494- if xpi_entry.name != 'en-US-jar':
3495- with open(xpi_entry.path) as f:
3496- xpi.writestr(xpi_entry.name, f.read())
3497- xpi.writestr('chrome/en-US.jar', jarfile.read())
3498- xpi.close()
3499- xpifile.seek(0)
3500-
3501- return xpifile
3502diff --git a/lib/lp/translations/utilities/translation_import.py b/lib/lp/translations/utilities/translation_import.py
3503index c08e5cd..197153d 100644
3504--- a/lib/lp/translations/utilities/translation_import.py
3505+++ b/lib/lp/translations/utilities/translation_import.py
3506@@ -57,7 +57,6 @@ from lp.translations.interfaces.translationmessage import (
3507 from lp.translations.interfaces.translations import TranslationConstants
3508 from lp.translations.utilities.gettext_po_importer import GettextPOImporter
3509 from lp.translations.utilities.kde_po_importer import KdePOImporter
3510-from lp.translations.utilities.mozilla_xpi_importer import MozillaXpiImporter
3511 from lp.translations.utilities.sanitize import (
3512 sanitize_translations_from_import,
3513 )
3514@@ -73,7 +72,6 @@ from lp.translations.utilities.validate import (
3515 importers = {
3516 TranslationFileFormat.KDEPO: KdePOImporter(),
3517 TranslationFileFormat.PO: GettextPOImporter(),
3518- TranslationFileFormat.XPI: MozillaXpiImporter(),
3519 }
3520
3521
3522diff --git a/lib/lp/translations/utilities/xpi_manifest.py b/lib/lp/translations/utilities/xpi_manifest.py
3523deleted file mode 100644
3524index 1bbaedd..0000000
3525--- a/lib/lp/translations/utilities/xpi_manifest.py
3526+++ /dev/null
3527@@ -1,237 +0,0 @@
3528-# Copyright 2009-2014 Canonical Ltd. This software is licensed under the
3529-# GNU Affero General Public License version 3 (see the file LICENSE).
3530-
3531-__metaclass__ = type
3532-
3533-__all__ = ['make_jarpath', 'XpiManifest']
3534-
3535-
3536-import logging
3537-import re
3538-
3539-from lp.translations.interfaces.translationimporter import (
3540- TranslationFormatSyntaxError,
3541- )
3542-
3543-
3544-def normalize_path(path):
3545- """Normalize filesystem path within XPI file."""
3546- # Normalize "jar:" prefix. Make sure it's there when needed, not there
3547- # when not needed.
3548- if path.startswith('jar:'):
3549- # No leading slashes please.
3550- path = re.sub('^jar:/+', 'jar:', path)
3551-
3552- if '.jar!' not in path:
3553- logging.debug("Removing 'jar:' from manifest path: '%s'" % path)
3554- path = path[4:]
3555- else:
3556- # No leading slashes please.
3557- path = re.sub('^/+', '', path)
3558-
3559- if '.jar!' in path:
3560- # Path delves into a jar file, but lacks "jar:" prefix. This is
3561- # really a malformed path.
3562- logging.info("Adding 'jar:' to manifest path: '%s'" % path)
3563- path = 'jar:' + path
3564-
3565- # A path inside a jar file must begin with a slash.
3566- path = path.replace('.jar!', '.jar!/')
3567-
3568- # Finally, eliminate redundant slashes. The previous steps may have
3569- # introduced some.
3570- return re.sub('/+', '/', path)
3571-
3572-
3573-def is_valid_path(path):
3574- """Check that path is a valid, normalized path inside an XPI file."""
3575- if '//' in path:
3576- return False
3577- if re.search('\\.jar![^/]', path):
3578- return False
3579- if path.startswith('jar:'):
3580- if path.startswith('jar:jar:'):
3581- return False
3582- if '.jar!' not in path:
3583- return False
3584- else:
3585- if '.jar!' in path:
3586- return False
3587- return True
3588-
3589-
3590-def is_valid_dir_path(path):
3591- """Check that path is a normalized directory path in an XPI file."""
3592- if not is_valid_path(path):
3593- return False
3594- if not path.endswith('/'):
3595- return False
3596- return True
3597-
3598-
3599-def make_jarpath(path, jarname):
3600- """Construct base path for files inside a jar file.
3601-
3602- To name some translation file that's inside a jar file inside an XPI
3603- file, concatenate the result of this method (for the jar file) and the
3604- translation file's path within the jar file.
3605-
3606- For example, let's say the XPI file contains foo/bar.jar. Inside
3607- foo/bar.jar is a translation file locale/gui.dtd. Then
3608- make_jarfile('foo', 'bar.jar') will return "jar:foo/bar.jar!/", to
3609- which you can append "locale/gui.dtd" to get the full path
3610- "jar:foo/bar.jar!/locale/gui.dtd" which identifies the translation
3611- file within the XPI file.
3612- """
3613- # This function is where we drill down into a jar file, so prefix with
3614- # "jar:" (unless it's already there). We carry the "jar:" prefix only
3615- # for paths that drill into jar files.
3616- if not path.startswith('jar:'):
3617- path = 'jar:' + path
3618-
3619- return normalize_path("%s/%s!" % (path, jarname))
3620-
3621-
3622-class ManifestEntry:
3623- """A "locale" line in a manifest file."""
3624-
3625- chrome = None
3626- locale = None
3627- path = None
3628-
3629- def __init__(self, chrome, locale, path):
3630- self.chrome = chrome
3631- self.locale = locale
3632-
3633- # Normalize path so we can do simple, reliable text matching on it.
3634- # The directory paths in an XPI file should end in a single slash.
3635- # Append the slash here; the normalization will take care of redundant
3636- # slashes.
3637- self.path = normalize_path(path + "/")
3638-
3639- assert is_valid_dir_path(self.path), (
3640- "Normalized path not valid: '%s' -> '%s'" % (path, self.path))
3641-
3642-
3643-def manifest_entry_sort_key(entry):
3644- """We keep manifest entries sorted by path length."""
3645- return len(entry.path)
3646-
3647-
3648-class XpiManifest:
3649- """Representation of an XPI manifest file.
3650-
3651- Does two things: parsers an XPI file; and looks up chrome paths and
3652- locales for given filesystem paths inside the XPI file.
3653- """
3654-
3655- # List of locale entries, sorted by increasing path length. The sort
3656- # order matters for lookup.
3657- _locales = None
3658-
3659- def __init__(self, content):
3660- """Initialize: parse `content` as a manifest file."""
3661- if content.startswith('\n'):
3662- raise TranslationFormatSyntaxError(
3663- message="Manifest begins with newline.")
3664-
3665- locales = []
3666- for line in content.splitlines():
3667- words = line.split()
3668- num_words = len(words)
3669- if num_words == 0 or words[0] != 'locale':
3670- pass
3671- elif num_words < 4:
3672- logging.info("Ignoring short manifest line: '%s'" % line)
3673- elif num_words > 4:
3674- logging.info("Ignoring long manifest line: '%s'" % line)
3675- else:
3676- locales.append(ManifestEntry(words[1], words[2], words[3]))
3677-
3678- # Eliminate duplicates.
3679- paths = set()
3680- deletions = []
3681- for index, entry in enumerate(locales):
3682- assert entry.path.endswith('/'), "Manifest path lost its slash"
3683-
3684- if entry.path in paths:
3685- logging.info("Duplicate paths in manifest: '%s'" % entry.path)
3686- deletions.append(index)
3687-
3688- paths.add(entry.path)
3689-
3690- for index in reversed(deletions):
3691- del locales[index]
3692-
3693- self._locales = sorted(locales, key=manifest_entry_sort_key)
3694-
3695- @classmethod
3696- def _normalizePath(cls, path):
3697- """Normalize path. Here so it can be tested without exporting it."""
3698- return normalize_path(path)
3699-
3700- def _getMatchingEntry(self, file_path):
3701- """Return longest matching entry matching file_path."""
3702- assert is_valid_path(file_path), (
3703- "Generated path not valid: %s" % file_path)
3704-
3705- # Locale entries are sorted by path length. If we scan backwards, the
3706- # first entry whose path is a prefix of file_path is the longest
3707- # match. The fact that the entries' paths have trailing slashes
3708- # guarantees that we won't match in the middle of a file or directory
3709- # name.
3710- for entry in reversed(self._locales):
3711- if file_path.startswith(entry.path):
3712- return entry
3713-
3714- # No match found.
3715- return None
3716-
3717- def getChromePathAndLocale(self, file_path):
3718- """Return chrome path and locale applying to a filesystem path.
3719- """
3720- assert file_path is not None, "Looking up chrome path for None"
3721- file_path = self._normalizePath(file_path)
3722- entry = self._getMatchingEntry(file_path)
3723-
3724- if entry is None:
3725- return None, None
3726-
3727- assert file_path.startswith(entry.path), "Found non-matching entry"
3728- replace = len(entry.path)
3729- chrome_path = "%s/%s" % (entry.chrome, file_path[replace:])
3730- return chrome_path, entry.locale
3731-
3732- def containsLocales(self, file_path):
3733- """Is `file_path` a prefix of any path containing locale files?
3734-
3735- :param file_path: path of a directory or jar file inside this XPI.
3736- :return: Boolean: does `file_path` contain locale files?
3737- """
3738- file_path = self._normalizePath(file_path)
3739- for entry in self._locales:
3740- if entry.path.startswith(file_path):
3741- return True
3742- return False
3743-
3744- def findMatchingXpiPath(self, chrome_path, locale):
3745- """Reverse-map a chrome path in a given locale to a file path.
3746-
3747- For example, if given "browser/gui/print.dtd" for locale en-US,
3748- may return "jar:locales/en-US.jar!/chrome/gui/print.dtd",
3749- assuming that the file path jar:locales/en-US.jar!/chrome/
3750- is associated with the chrome path browser.
3751-
3752- If there are multiple matches, this returns the one with the
3753- longest file path.
3754- """
3755- # Since _locales is sorted by path length, scanning it backwards
3756- # finds the longest match first.
3757- for entry in reversed(self._locales):
3758- is_match = (chrome_path.startswith(entry.chrome + '/') and
3759- entry.locale == locale)
3760- if is_match:
3761- return normalize_path(
3762- entry.path + chrome_path[len(entry.chrome):])
3763-
3764- return None
3765diff --git a/lib/lp/translations/utilities/xpi_properties_exporter.py b/lib/lp/translations/utilities/xpi_properties_exporter.py
3766deleted file mode 100644
3767index 43cf397..0000000
3768--- a/lib/lp/translations/utilities/xpi_properties_exporter.py
3769+++ /dev/null
3770@@ -1,53 +0,0 @@
3771-# Copyright 2009 Canonical Ltd. This software is licensed under the
3772-# GNU Affero General Public License version 3 (see the file LICENSE).
3773-
3774-__metaclass__ = type
3775-
3776-__all__ = [
3777- 'XpiPropertiesSubExporter'
3778- ]
3779-
3780-
3781-import re
3782-
3783-
3784-def has_comment(message):
3785- """Does `TranslationMessageData` contain a comment?"""
3786- return message.comment is not None and message.comment.strip() != ''
3787-
3788-
3789-class XpiPropertiesSubExporter:
3790- """Produce a properties file to go into an XPI file."""
3791-
3792- def _escape(self, string):
3793- """Escape message string for use in properties file."""
3794- # Escape backslashes first, before we start inserting ones of
3795- # our own. Then the other stuff. Replace newlines by \n etc.,
3796- # and encode non-ASCII characters as \uXXXX.
3797- string = string.replace('\\', r'\\')
3798- string = re.sub('''(["'])''', r'\\\1', string)
3799- # Escape newlines as \n etc, and non-ASCII as \uXXXX
3800- return string.encode('ascii', 'backslashreplace')
3801-
3802- def _escape_comment(self, comment):
3803- """Escape comment string for use in properties file."""
3804- # Prevent comment from breaking out of /* ... */ block.
3805- comment = comment.replace('*/', '*X/')
3806- return comment.encode('ascii', 'unicode-escape')
3807-
3808- def export(self, translation_file):
3809- assert translation_file.path.endswith('.properties'), (
3810- "Unexpected properties file suffix: %s" % translation_file.path)
3811- contents = []
3812- for message in translation_file.messages:
3813- if not message.translations:
3814- continue
3815- if has_comment(message):
3816- contents.append(
3817- "\n/* %s */" % self._escape_comment(message.comment))
3818- msgid = self._escape(message.msgid_singular)
3819- text = self._escape(message.translations[0])
3820- line = "%s=%s" % (msgid, text)
3821- contents.append(line)
3822-
3823- return '\n'.join(contents)
3824diff --git a/utilities/sourcedeps.cache b/utilities/sourcedeps.cache
3825index 412ea52..fef3955 100644
3826--- a/utilities/sourcedeps.cache
3827+++ b/utilities/sourcedeps.cache
3828@@ -27,10 +27,6 @@
3829 494,
3830 "cjwatson@canonical.com-20190919081036-q1symc2h2iedtlh3"
3831 ],
3832- "old_xmlplus": [
3833- 4,
3834- "sinzui-20090526164636-1swugzupwvjgomo4"
3835- ],
3836 "pygettextpo": [
3837 25,
3838 "launchpad@pqm.canonical.com-20140116030912-lqm1dtb6a0y4femq"
3839diff --git a/utilities/sourcedeps.conf b/utilities/sourcedeps.conf
3840index b19007d..13287d4 100644
3841--- a/utilities/sourcedeps.conf
3842+++ b/utilities/sourcedeps.conf
3843@@ -14,5 +14,4 @@ bzr-svn lp:~launchpad-pqm/bzr-svn/devel;revno=2725
3844 cscvs lp:~launchpad-pqm/launchpad-cscvs/devel;revno=433
3845 difftacular lp:~launchpad/difftacular/trunk;revno=11
3846 loggerhead lp:~loggerhead-team/loggerhead/trunk-rich;revno=494
3847-old_xmlplus lp:~launchpad-pqm/dtdparser/trunk;revno=4
3848 pygettextpo lp:~launchpad-pqm/pygettextpo/trunk;revno=25

Subscribers

People subscribed via source and target branches

to status/vote changes: