Merge lp:~abompard/mailman/mailman-templates-utf8 into lp:mailman

Proposed by Aurélien Bompard on 2015-02-11
Status: Merged
Merge reported by: Barry Warsaw
Merged at revision: not available
Proposed branch: lp:~abompard/mailman/mailman-templates-utf8
Merge into: lp:mailman
Diff against target: 45 lines (+16/-1)
2 files modified
src/mailman/utilities/i18n.py (+1/-1)
src/mailman/utilities/tests/test_templates.py (+15/-0)
To merge this branch: bzr merge lp:~abompard/mailman/mailman-templates-utf8
Reviewer Review Type Date Requested Status
Barry Warsaw 2015-02-11 Approve on 2015-04-15
Review via email: mp+249358@code.launchpad.net

Description of the change

As discussed on the mailing-list in October, the templates should be encoded in UTF-8.

(https://mail.python.org/pipermail/mailman-developers/2013-October/023347.html)

However, the current code does not specify this and relies on locale.getpreferredencoding() returning UTF-8 to read the templates.

You can verify this by running the test suite with LANG=C.

This branch forces the templates encoding to UTF-8 on read().

To post a comment you must log in.
Barry Warsaw (barry) :
review: Needs Fixing
Barry Warsaw (barry) wrote :

Chatting about this at Pycon 2015, we decided to apply the patch, which seems reasonable on the face of it. I won't apply the test; eventually CI should probably run them under different locales, or we should use some subunit or other subprocess runner.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'src/mailman/utilities/i18n.py'
--- src/mailman/utilities/i18n.py 2015-01-05 01:40:47 +0000
+++ src/mailman/utilities/i18n.py 2015-02-11 16:56:51 +0000
@@ -150,7 +150,7 @@
150 try:150 try:
151 if _trace:151 if _trace:
152 print('@@@', path, end='', file=sys.stderr)152 print('@@@', path, end='', file=sys.stderr)
153 fp = open(path)153 fp = open(path, encoding="utf-8")
154 except IOError as error:154 except IOError as error:
155 if error.errno == errno.ENOENT:155 if error.errno == errno.ENOENT:
156 if _trace:156 if _trace:
157157
=== modified file 'src/mailman/utilities/tests/test_templates.py'
--- src/mailman/utilities/tests/test_templates.py 2015-01-05 01:22:39 +0000
+++ src/mailman/utilities/tests/test_templates.py 2015-02-11 16:56:51 +0000
@@ -25,6 +25,7 @@
2525
2626
27import os27import os
28import locale
28import shutil29import shutil
29import tempfile30import tempfile
30import unittest31import unittest
@@ -227,6 +228,20 @@
227 find('missing.txt', self.mlist)228 find('missing.txt', self.mlist)
228 self.assertEqual(cm.exception.template_file, 'missing.txt')229 self.assertEqual(cm.exception.template_file, 'missing.txt')
229230
231 def test_encoding(self):
232 with open(self.xxsite, 'w', encoding="utf-8") as fp:
233 fp.write('Ol\ufffd!')
234 # Settings LC_ALL to 'C' will clear locale.getpreferredencoding() from
235 # references to UTF-8 that it would have caught up reading the
236 # environment.
237 locale.setlocale(locale.LC_ALL, 'C')
238 filename, self.fp = find('site.txt', language='xx')
239 try:
240 content = self.fp.read()
241 except UnicodeDecodeError:
242 self.fail("Templates should be considered UTF-8 by default")
243 self.assertEqual(content, 'Ol\ufffd!')
244
230245
231246
232247
233class TestMake(unittest.TestCase):248class TestMake(unittest.TestCase):