Merge lp:~futatuki/mailman/enhance-i18n-list-overview into lp:mailman/2.1

[Yasuhito FUTATSUKI at POEM]

enhance i18n of listinfo/admin overview page

On listinfo/admin overview page, lists' description may be stored in different charset/encoding from what the page uses. However current implementation of listinfo/admin overview page uses those regardlessly of the difference, so those cannot be displayed correctly.

To avoid this, (1) I've implemented a new function to get description as string of specified charset (with XML character reference) in MailList class, MailList.GetDescription(), and use it instead of accessing .description attribute directly in overview functions.

This function uses a precondition that .description attribute is stored in charset of list's preferred_language. This is fundamentally true, but in case list's preferred_language has been changed and of the description hasn't, it is not true(Suppose changing preferred_language temporary to edit template files via Web UI, for example). To help to ensure the precondition, (2) I've added hook to change charset of list's description when preferred_language is changed via Web UI. (This is not always preserve identity of description string when preferred_languages is changed one of other charset/encoding and then changed back, because of multiple mapping of character between charset, and intentional use of XML character reference)

[Mark Sapiro (msapiro)]

In your change to Mailman/Cgi/admin.py, mlist.GetDescription() should be Utils.websafe(mlist.GetDescription()). I know it isn't currently websafed, but it should be.

I understand the need for the changes to admin.py and listinfo.py because the overview page is in the charset of the site's preferred language and the description may not be.

There is also another issue in the listinfo for a specific list. Here the <mm-list-description> replacement puts the description at the top of the page and the page can be viewed in a language other than the list default, so I think Mailman/HTMLFormatter.py needs to also be changed to use GetDescription() for the <mm-list-description> replacement.

Finally though, I don't understand the need for the change to Mailman/Gui/GUIBase.py since there we are dealing with changes via the admin CGI for a list and the admin UI should be in the list's preferred language charset always. Am I missing something here?

review: Needs Information (code)

[Mark Sapiro (msapiro)]

I've looked a bit more carefully and I now see that the change to Mailman/Gui/GUIBase.py is to recode mlist.description if the list's preferred language is changed. This is probably useful, but it is incomplete as there are several other strings encoded in the charset of the list's preferred language such as mlist.info and the message and digest headers and footers.

This is what makes it difficult to change Mailman's character set for a language. See the recode(), doitem() and convert() functions in Mailman/versions.py for more on this.

I'm not sure it's worth the effort to try to do this automatically if the list's preferred language is changed.

[Mark Sapiro (msapiro)]

It would have helped if I had actually fully read your description before commenting based on my reading of the diff.

In any case, at this point, I agree with the changes to admin.py and listinfo.py and the new GetDescription() method, and I think a change to HTMLFormatter.py would also be good, but I don't think I want to do the recoding on preferred language change.

Also, I had planned on a 2.1.27 release tomorrow, and I don't want to delay that too much.

[Yasuhito FUTATSUKI at POEM (futatuki)]

I've also thought the change of Gui/GUIBase.py is optional, and I understand the dificulty. So I agree with drop it.

[lp:~futatuki/mailman/enhance-i18n-list-overview]

1783. By Yasuhito FUTATSUKI at POEM on 2018-06-22

* apply Utils.websafe() to description string in admin.py
* Use GetDescription() in HTMLFormatter.py

1784. By Yasuhito FUTATSUKI at POEM on 2018-06-22

revert Mailman/Gui/GUIBase.py to drop change of this branch

[Yasuhito FUTATSUKI at POEM (futatuki)]

Now I've fixed the matter you pointed out.

=== modified file 'Mailman/Cgi/admin.py'
--- Mailman/Cgi/admin.py	2018-06-18 11:35:51 +0000
+++ Mailman/Cgi/admin.py	2018-06-22 00:40:28 +0000
@@ -295,7 +295,7 @@
             else:
                 advertised.append((mlist.GetScriptURL('admin'),
                                    mlist.real_name,
-                                   mlist.description))
+                                   Utils.websafe(mlist.GetDescription())))
     # Greeting depends on whether there was an error or not
     if msg:
         greeting = FontAttr(msg, color="ff5060", size="+1")

=== modified file 'Mailman/Cgi/listinfo.py'
--- Mailman/Cgi/listinfo.py	2018-06-03 20:19:49 +0000
+++ Mailman/Cgi/listinfo.py	2018-06-22 00:40:28 +0000
@@ -114,7 +114,7 @@
             else:
                 advertised.append((mlist.GetScriptURL('listinfo'),
                                    mlist.real_name,
-                                   Utils.websafe(mlist.description)))
+                                   Utils.websafe(mlist.GetDescription())))
     if msg:
         greeting = FontAttr(msg, color="ff5060", size="+1")
     else:

=== modified file 'Mailman/HTMLFormatter.py'
--- Mailman/HTMLFormatter.py	2018-06-17 23:47:34 +0000
+++ Mailman/HTMLFormatter.py	2018-06-22 00:40:28 +0000
@@ -389,11 +389,16 @@
             listlangs = _(Utils.GetLanguageDescr(self.preferred_language))
         else:
             listlangs = self.GetLangSelectBox(lang).Format()
+        if lang:
+            cset = Utils.GetCharSet(lang) or 'us-ascii'
+        else:
+            cset = Utils.GetCharSet(self.preferred_language) or 'us-ascii'
         d = {
             '<mm-mailman-footer>' : self.GetMailmanFooter(),
             '<mm-list-name>' : self.real_name,
             '<mm-email-user>' : self._internal_name,
-            '<mm-list-description>' : Utils.websafe(self.description),
+            '<mm-list-description>' :
+                Utils.websafe(self.GetDescription(cset)),
             '<mm-list-info>' : 
                 '<!---->' + BR.join(self.info.split(NL)) + '<!---->',
             '<mm-form-end>'  : self.FormatFormEnd(),

=== modified file 'Mailman/MailList.py'
--- Mailman/MailList.py	2018-06-18 11:35:51 +0000
+++ Mailman/MailList.py	2018-06-22 00:40:28 +0000
@@ -262,6 +262,27 @@
             user = Utils.ObscureEmail(user)
         return '%s/%s' % (url, urllib.quote(user.lower()))
 
+    def GetDescription(self, cset=None, errors='xmlcharrefreplace'):
+        # Get list's description in charset specified by cset.
+        # If cset is None, it uses charset of context language.
+        mcset = Utils.GetCharSet(self.preferred_language)
+        if cset is None:
+            # translation context may not be initialized
+            trns = i18n.get_translation()
+            if trns is None:
+                ccset = 'us-ascii'
+            else:
+                ccset = i18n.get_translation().charset() or 'us-ascii'
+        else:
+            ccset = cset
+        if isinstance(self.description, unicode):
+            return self.description.encode(ccset, errors)
+        if mcset == ccset:
+            return self.description
+        return Utils.xml_to_unicode(self.description, mcset).encode(ccset,
+                                                                    errors)
+
+
 

     #
     # Instance and subcomponent initialization