Merge lp:~abompard/mailman/mailman-templates-utf8 into lp:mailman

[Aurélien Bompard]

As discussed on the mailing-list in October, the templates should be encoded in UTF-8.

(https://mail.python.org/pipermail/mailman-developers/2013-October/023347.html)

However, the current code does not specify this and relies on locale.getpreferredencoding() returning UTF-8 to read the templates.

You can verify this by running the test suite with LANG=C.

This branch forces the templates encoding to UTF-8 on read().

[Barry Warsaw (barry)]

review: Needs Fixing

[Barry Warsaw (barry)]

Chatting about this at Pycon 2015, we decided to apply the patch, which seems reasonable on the face of it. I won't apply the test; eventually CI should probably run them under different locales, or we should use some subunit or other subprocess runner.

review: Approve

=== modified file 'src/mailman/utilities/i18n.py'
--- src/mailman/utilities/i18n.py	2015-01-05 01:40:47 +0000
+++ src/mailman/utilities/i18n.py	2015-02-11 16:56:51 +0000
@@ -150,7 +150,7 @@
         try:
             if _trace:
                 print('@@@', path, end='', file=sys.stderr)
-            fp = open(path)
+            fp = open(path, encoding="utf-8")
         except IOError as error:
             if error.errno == errno.ENOENT:
                 if _trace:
 
=== modified file 'src/mailman/utilities/tests/test_templates.py'
--- src/mailman/utilities/tests/test_templates.py	2015-01-05 01:22:39 +0000
+++ src/mailman/utilities/tests/test_templates.py	2015-02-11 16:56:51 +0000
@@ -25,6 +25,7 @@
 
 
 import os
+import locale
 import shutil
 import tempfile
 import unittest
@@ -227,6 +228,20 @@
             find('missing.txt', self.mlist)
         self.assertEqual(cm.exception.template_file, 'missing.txt')
 
+    def test_encoding(self):
+        with open(self.xxsite, 'w', encoding="utf-8") as fp:
+            fp.write('Ol\ufffd!')
+        # Settings LC_ALL to 'C' will clear locale.getpreferredencoding() from
+        # references to UTF-8 that it would have caught up reading the
+        # environment.
+        locale.setlocale(locale.LC_ALL, 'C')
+        filename, self.fp = find('site.txt', language='xx')
+        try:
+            content = self.fp.read()
+        except UnicodeDecodeError:
+            self.fail("Templates should be considered UTF-8 by default")
+        self.assertEqual(content, 'Ol\ufffd!')
+
 
 
 
 class TestMake(unittest.TestCase):