Merge lp:~jml/pkgme/extended-characters-in-description into lp:pkgme

[Jonathan Lange]

Support extended characters in the description

We had a case where John submitted a package that had pretty quotes in the description field. 'json' correctly serializes and deserializes these values as unicode, so the unicode made its way over the HTTP request, into the devportal-metadata.json file and straight into the Description info element.

http://www.debian.org/doc/debian-policy/ch-controlfields.html states that 'all control files must be encoded in UTF-8', which is very sensible of them. The implication for us is that every InfoElement's get_value must return a UTF-8 encoded bytestring. Well, if that's where we want to draw the line. We could also change PackageFile to encode the values it receives before serializing them.

Anyway, I only did this for Description, since that was the presenting case. I've added a test, and have also build & installed a package that used smart quotes.

[James Westby (james-w)]

Hi,

Looks good.

I wonder if it should be applied more widely, but this is a fine start.

Thanks,

James

review: Approve

=== modified file 'pkgme/info_elements.py'
--- pkgme/info_elements.py	2012-02-04 18:29:54 +0000
+++ pkgme/info_elements.py	2012-03-26 11:41:21 +0000
@@ -236,6 +236,8 @@
             formatted but actually is not.
         :return: A correctly-formatted version of that string.
         """
+        if isinstance(value, unicode):
+            value = value.encode('utf-8')
         if not value.strip():
             return ''
         return '\n'.join(cls._format_lines(value.strip().splitlines()))

=== modified file 'pkgme/tests/test_info_elements.py'
--- pkgme/tests/test_info_elements.py	2012-02-04 18:29:54 +0000
+++ pkgme/tests/test_info_elements.py	2012-03-26 11:41:21 +0000
@@ -264,6 +264,15 @@
  follow-up line
  more information""", cleaned)
 
+    def test_extended_characters_in_descriptions(self):
+        # Some people like putting extended characters in their descriptions.
+        # Let's see what happens.
+        description = (
+            u'\u201cPretty \u2018speech\u2019 marks\u201d\u202a \u2013'
+            u'what fun!')
+        cleaned = Description.clean(description)
+        self.assertEqual(description.encode('utf-8'), cleaned)
+
 
 class TestPackageName(TestCase):