Merge lp:~openerp-dev/openobject-server/6.0-bug-732512-xrg into lp:openobject-server/6.0

[xrg]

[OpenERP buildbot (openerp-buildbot)]

review: Approve

Unmerged revisions

3363. By xrg on 2011-03-10

tools/misc: try harder to encode smtp address headers

Problem is that encoding the full header (comma-separated list of
email addresses) in one =?c?...?= chunk is not properly handled by
some mail transport agents[1]. Even worse, when parts cannot fit
into one 76-char line (note that utf-8 over base64 tends to be rather
long), Postfix thinks each line is another address, that it decides
to trivial-rewrite and append domain name etc.
In another case noted, invalid chars (not properly encoded) or half-
addresses were adding to the spam-score of mails.

Solution: according to RFC 2047, we can have multiple encoded-words
(the =?.?..?= part) as any whitespace-delimited part of the header.
We /cannot/ have them inside quoted strings like "=?..?=", but need
to include the quotes inside the encoded-word.
The patch splits the address tokens to their smallest allowed "words"
and then tries to encode them one by one. Hopefully, names may need the
encoding, while emails will be pure us-ascii and will be appended
verbatim. It also includes support for one 'hint' encoding (iso8859-7 in
my case, for example) that would have shorter representation than utf-8.
Saves the day for some long names.

Note: a side-effect is that we add one space /before/ the comma when
we separate multiple addresses, like ' "Me" <me@here> , "Him" <him@there>'
It should be tolerated by the protocol, AFAICT.

[1] I've experienced it all these years we've been sending Greek names
in our From/To addresses. My maildirs are full of examples.
(cherry picked from commit 7fb0827e94b50b88d003c7c0b37abd918eed98ec)

3362. By xrg on 2011-03-10

misc/email: improve, utf-8 aware headers, better logging.

Some trivial improvements to email_send():
 - use pythonic logger
 - have one debug() line and one info() line before and after sending
   a message, respectively
 - if message is plain, don't do multipart, but simple email
 - if any header is non-latin, encode it with email.Header()
 - otoh, if mail is latin, don't encode it.
(cherry picked from commit 9c025f545a086e80c2258c35abada8084ccabf2f)

=== modified file 'bin/tools/misc.py'
--- bin/tools/misc.py	2011-01-20 12:40:20 +0000
+++ bin/tools/misc.py	2011-03-10 11:32:35 +0000
@@ -43,6 +43,7 @@
 from email.Header import Header
 from email.Utils import formatdate, COMMASPACE
 from email import Encoders
+from email import Charset
 from itertools import islice, izip
 from lxml import etree
 from which import which
@@ -432,12 +433,14 @@
         :return: True if the mail was delivered successfully to the smtp,
                  else False (+ exception logged)
     """
+    logger = logging.getLogger('email_send')
+    
     class WriteToLogger(object):
-        def __init__(self):
-            self.logger = netsvc.Logger()
+        def __init__(self, logger):
+            self.logger = logger
 
         def write(self, s):
-            self.logger.notifyChannel('email_send', netsvc.LOG_DEBUG, s)
+            self.logger.debug(s)
 
     if openobject_id:
         message['Message-Id'] = generate_tracking_message_id(openobject_id)
@@ -448,19 +451,22 @@
         if smtp_server.startswith('maildir:/'):
             from mailbox import Maildir
             maildir_path = smtp_server[8:]
-            mdir = Maildir(maildir_path,factory=None, create = True)
+            mdir = Maildir(maildir_path, factory=None, create = True)
             mdir.add(message.as_string(True))
+            logger.info("1 message to %d recepients saved at %s. Dry run.", len(smtp_to_list), maildir_path)
             return True
 
         oldstderr = smtplib.stderr
-        if not ssl: ssl = config.get('smtp_ssl', False)
+        ssl = ssl or config.get('smtp_ssl', False)
         s = smtplib.SMTP()
         try:
             # in case of debug, the messages are printed to stderr.
             if debug:
-                smtplib.stderr = WriteToLogger()
+                smtplib.stderr = WriteToLogger(logger)
 
+            logger.debug("Sending Message to %s through %s", ','.join(smtp_to_list), smtp_server)
             s.set_debuglevel(int(bool(debug)))  # 0 or 1
+            
             s.connect(smtp_server, config['smtp_port'])
             if ssl:
                 s.ehlo()
@@ -471,6 +477,7 @@
                 s.login(config['smtp_user'], config['smtp_password'])
 
             s.sendmail(smtp_from, smtp_to_list, message.as_string())
+            logger.info("1 message sent to %d recepients through %s", len(smtp_to_list), smtp_server)
         finally:
             try:
                 s.quit()
@@ -481,7 +488,7 @@
                 pass
 
     except Exception:
-        _logger.error('could not deliver email', exc_info=True)
+        logger.error('could not deliver email', exc_info=True)
         return False
 
     return True
@@ -503,50 +510,109 @@
     if x_headers is None:
         x_headers = {}
 
+    def Header_Encoded(hstr):
+        """Format hstr for an email header, possibly through utf8 encoding
+        """
+
+        if not isinstance(hstr, unicode):
+            hstr = ustr(hstr)
+
+        try:
+            return hstr.encode('us-ascii')
+        except UnicodeError:
+            return Header(hstr, 'utf-8')
+
+    email_adre = re.compile(r'^((?:"[^"]+?")|(?:[^,<]+?)|\A)\s*<([\w\.-~]+?(?:@[\w\.-~]+)?)>$')
+    email_charset = config.get_misc('smtp', 'charset', None)
+
+    def Address_Encoded(inp, header_name=None):
+        """ Encode the inp address into a valid RFC 2047 header
+        The problem is, that some mail transports (like Postfix <= 2.5) don't
+        like all the header encoded into one string (that spans lines) and the
+        protocol doesn't provide a line-continuation mark around that case.
+        So, we have to split tokens and encode them separately.
+        """
+        if isinstance(inp, basestring):
+            inp = [inp,]
+
+        tokens = []
+        for itok in inp:
+            if not isinstance(itok, unicode):
+                itok = ustr(itok)
+            itok = itok.strip()
+            if not itok: # skip empty ones
+                continue
+            if tokens:
+                tokens.append(',')
+            m = email_adre.match(itok)
+            if m:
+                if m.group(1):
+                    tokens.append(m.group(1))
+                tokens.append('<' + m.group(2) + '>')
+            else:
+                tokens.append(itok)
+        
+        # Now, encode them one by one
+        ret = Header(header_name=header_name)
+        for t in tokens:
+            # this method will try 'us-ascii', email_charset and 'utf-8'
+            ret.append(t, charset=email_charset)
+        
+        return ret
 
     if not (email_from or config['email_from']):
         raise ValueError("Sending an email requires either providing a sender "
                          "address or having configured one")
 
     if not email_from: email_from = config.get('email_from', False)
-    email_from = ustr(email_from).encode('utf-8')
 
     if not email_cc: email_cc = []
     if not email_bcc: email_bcc = []
-    if not body: body = u''
-
-    email_body = ustr(body).encode('utf-8')
-    email_text = MIMEText(email_body or '',_subtype=subtype,_charset='utf-8')
-
-    msg = MIMEMultipart()
-
-    msg['Subject'] = Header(ustr(subject), 'utf-8')
-    msg['From'] = email_from
+    body_charset = 'us-ascii'
+    if not body:
+        body = u''
+    else:
+        body = ustr(body)
+        try:
+            txt_body = body.encode('us-ascii')
+        except UnicodeError:
+            txt_body = body.encode('utf-8')
+            body_charset = 'utf-8'
+
+    email_text = MIMEText(txt_body,_subtype=subtype,_charset=body_charset)
+
+    if attach or (html2text and subtype == 'html'):
+        msg = MIMEMultipart()
+    else:
+        msg = email_text
+
+    msg['Subject'] = Header_Encoded(subject)
+    msg['From'] = Address_Encoded(email_from, 'From')
     del msg['Reply-To']
     if reply_to:
-        msg['Reply-To'] = reply_to
+        msg['Reply-To'] = Address_Encoded(reply_to, 'Reply-To')
     else:
         msg['Reply-To'] = msg['From']
-    msg['To'] = COMMASPACE.join(email_to)
+    msg['To'] = Address_Encoded(email_to, 'To')
     if email_cc:
-        msg['Cc'] = COMMASPACE.join(email_cc)
+        msg['Cc'] = Address_Encoded(email_cc, 'Cc')
     if email_bcc:
-        msg['Bcc'] = COMMASPACE.join(email_bcc)
+        msg['Bcc'] = Address_Encoded(email_bcc, 'Bcc')
     msg['Date'] = formatdate(localtime=True)
 
     msg['X-Priority'] = priorities.get(priority, '3 (Normal)')
 
     # Add dynamic X Header
     for key, value in x_headers.iteritems():
-        msg['%s' % key] = str(value)
+        msg['%s' % key] =  Header_Encoded(value)
 
     if html2text and subtype == 'html':
-        text = html2text(email_body.decode('utf-8')).encode('utf-8')
+        text = html2text(body).encode(body_charset)
         alternative_part = MIMEMultipart(_subtype="alternative")
-        alternative_part.attach(MIMEText(text, _charset='utf-8', _subtype='plain'))
+        alternative_part.attach(MIMEText(text, _charset=body_charset, _subtype='plain'))
         alternative_part.attach(email_text)
         msg.attach(alternative_part)
-    else:
+    elif msg is not email_text:
         msg.attach(email_text)
 
     if attach: