Merge lp:~abompard/mailman/mailman-templates-utf8 into lp:mailman

Proposed by Aurélien Bompard
Status: Merged
Merge reported by: Barry Warsaw
Merged at revision: not available
Proposed branch: lp:~abompard/mailman/mailman-templates-utf8
Merge into: lp:mailman
Diff against target: 45 lines (+16/-1)
2 files modified
src/mailman/utilities/i18n.py (+1/-1)
src/mailman/utilities/tests/test_templates.py (+15/-0)
To merge this branch: bzr merge lp:~abompard/mailman/mailman-templates-utf8
Reviewer Review Type Date Requested Status
Barry Warsaw Approve
Review via email: mp+249358@code.launchpad.net

Description of the change

As discussed on the mailing-list in October, the templates should be encoded in UTF-8.

(https://mail.python.org/pipermail/mailman-developers/2013-October/023347.html)

However, the current code does not specify this and relies on locale.getpreferredencoding() returning UTF-8 to read the templates.

You can verify this by running the test suite with LANG=C.

This branch forces the templates encoding to UTF-8 on read().

To post a comment you must log in.
Revision history for this message
Barry Warsaw (barry) :
review: Needs Fixing
Revision history for this message
Aurélien Bompard (abompard) :
Revision history for this message
Barry Warsaw (barry) wrote :

Chatting about this at Pycon 2015, we decided to apply the patch, which seems reasonable on the face of it. I won't apply the test; eventually CI should probably run them under different locales, or we should use some subunit or other subprocess runner.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'src/mailman/utilities/i18n.py'
2--- src/mailman/utilities/i18n.py 2015-01-05 01:40:47 +0000
3+++ src/mailman/utilities/i18n.py 2015-02-11 16:56:51 +0000
4@@ -150,7 +150,7 @@
5 try:
6 if _trace:
7 print('@@@', path, end='', file=sys.stderr)
8- fp = open(path)
9+ fp = open(path, encoding="utf-8")
10 except IOError as error:
11 if error.errno == errno.ENOENT:
12 if _trace:
13
14=== modified file 'src/mailman/utilities/tests/test_templates.py'
15--- src/mailman/utilities/tests/test_templates.py 2015-01-05 01:22:39 +0000
16+++ src/mailman/utilities/tests/test_templates.py 2015-02-11 16:56:51 +0000
17@@ -25,6 +25,7 @@
18
19
20 import os
21+import locale
22 import shutil
23 import tempfile
24 import unittest
25@@ -227,6 +228,20 @@
26 find('missing.txt', self.mlist)
27 self.assertEqual(cm.exception.template_file, 'missing.txt')
28
29+ def test_encoding(self):
30+ with open(self.xxsite, 'w', encoding="utf-8") as fp:
31+ fp.write('Ol\ufffd!')
32+ # Settings LC_ALL to 'C' will clear locale.getpreferredencoding() from
33+ # references to UTF-8 that it would have caught up reading the
34+ # environment.
35+ locale.setlocale(locale.LC_ALL, 'C')
36+ filename, self.fp = find('site.txt', language='xx')
37+ try:
38+ content = self.fp.read()
39+ except UnicodeDecodeError:
40+ self.fail("Templates should be considered UTF-8 by default")
41+ self.assertEqual(content, 'Ol\ufffd!')
42+
43
44
45
46 class TestMake(unittest.TestCase):