Merge ~cjwatson/launchpad:email-utf8-inline-comments into launchpad:master

[Colin Watson]

Fix emailing of non-ASCII inline comments

I think this regressed as part of my port of codehosting to Breezy. The tests didn't catch it because they were using \u escapes in non-Unicode string literals, which meant that e.g. \u03b4 was interpreted as the byte sequence '\', 'u', '0', '3', 'b', '4' rather than U+03B4.

[Kristian Glass (doismellburning)]

Cripes, that test-data issue is disconcertingly easy to run into!

LGTM

review: Approve

[Colin Watson (cjwatson)]

Indeed it is! pyflakes didn't spot it. I grepped for other occurrences and didn't find any though.

diff --git a/lib/lp/code/mail/codereviewcomment.py b/lib/lp/code/mail/codereviewcomment.py
index 760d923..953c839 100644
--- a/lib/lp/code/mail/codereviewcomment.py
+++ b/lib/lp/code/mail/codereviewcomment.py
@@ -1,4 +1,4 @@
-# Copyright 2009-2015 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2020 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Email notifications for code review comments."""
@@ -175,7 +175,8 @@ def format_comment(comment):
     comment_lines = []
     if comment is not None:
         comment_lines.append(b'')
-        comment_lines.extend(comment.splitlines())
+        comment_lines.extend(
+            [line.encode('UTF-8') for line in comment.splitlines()])
         comment_lines.append(b'')
     return comment_lines
 
diff --git a/lib/lp/code/mail/tests/test_codereviewcomment.py b/lib/lp/code/mail/tests/test_codereviewcomment.py
index 2ea2595..930753e 100644
--- a/lib/lp/code/mail/tests/test_codereviewcomment.py
+++ b/lib/lp/code/mail/tests/test_codereviewcomment.py
@@ -1,4 +1,4 @@
-# Copyright 2009-2015 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2020 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Test CodeReviewComment emailing functionality."""
@@ -278,7 +278,7 @@ class TestCodeReviewComment(TestCaseWithFactory):
         See `build_inline_comments_section` tests for formatting details.
         """
         comment = self.makeCommentWithInlineComments(
-            inline_comments={'3': 'Is this from Pl\u0060net Earth ?'})
+            inline_comments={'3': u'Is this from Pl\u00e4net Earth ?'})
         switch_dbuser(config.IBranchMergeProposalJobSource.dbuser)
         mailer = CodeReviewCommentMailer.forCreation(comment)
         commenter = comment.branch_merge_proposal.registrant
@@ -295,7 +295,7 @@ class TestCodeReviewComment(TestCaseWithFactory):
             ('> +++ yvo/yc/pbqr/vagresnprf/qvss.cl      '
              '2010-02-02 15:48:56 +0000'),
             '',
-            'Is this from Pl\u0060net Earth ?',
+            u'Is this from Pl\u00e4net Earth ?',
             '',
         ]
         self.assertEqual(expected_lines, ctrl.body.splitlines()[1:10])
@@ -548,12 +548,12 @@ class TestInlineCommentsSection(testtools.TestCase):
     def test_single_line_comment(self):
         # The inline comments are correctly contextualized in the diff.
         # and prefixed with '>>> '
-        comments = {'4': '\u03b4\u03bf\u03ba\u03b9\u03bc\u03ae'}
+        comments = {'4': u'\u03b4\u03bf\u03ba\u03b9\u03bc\u03ae'}
         self.assertEqual(
             map(unicode, [
                 '> +++ bar.py\t1969-12-31 19:00:00.000000000 -0500',
                 '',
-                '\u03b4\u03bf\u03ba\u03b9\u03bc\u03ae',
+                u'\u03b4\u03bf\u03ba\u03b9\u03bc\u03ae',
                 '']),
             self.getSection(comments).splitlines()[7:11])