Merge ~chad.smith/cloud-init:fix/1768600-utf8-in-user-data into cloud-init:master

[Chad Smith]

Be more safe on string/bytes when writing multipart user-data to disk.

When creating the multipart mime message that is written as
user-data.txt.i, cloud-init losing data on conversion to some things
as a string.

LP: #1768600

Author: Scott Moser <email address hidden>
Co-Authored-By: Chad Smith <email address hidden>

Took over Scott's branch https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/347727 and added/tweaked a couple of unit tests to show failure path on master without this branch.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:d6a4ebb6cc4ab55debe0caea4ebaf28178717bac
https://jenkins.ubuntu.com/server/job/cloud-init-ci/67/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/67/rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

Whitespace question/nit. I'm fine either way.

review: Approve

867db9c... by Chad Smith on 2018-06-12

Drop unicode character from docstr in unit test.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:e6233f16e6ea8b78b7ed1446cee88b1c9277608f
https://jenkins.ubuntu.com/server/job/cloud-init-ci/68/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/68/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:867db9cb78bddd3bed70146fd0d2c67c49bb1628
https://jenkins.ubuntu.com/server/job/cloud-init-ci/69/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/69/rebuild

review: Needs Fixing (continuous-integration)

d6262a2... by Chad Smith on 2018-06-13

pycodestyle spacing fix

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:d6262a2e83ecc5277eab30fad374aaa7602b41c1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/79/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/79/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

review: Approve

[Scott Moser (smoser)]

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=faa6f07e

diff --git a/cloudinit/user_data.py b/cloudinit/user_data.py
index 8f6aba1..ed83d2d 100644
--- a/cloudinit/user_data.py
+++ b/cloudinit/user_data.py
@@ -337,8 +337,10 @@ def is_skippable(part):
 
 # Coverts a raw string into a mime message
 def convert_string(raw_data, content_type=NOT_MULTIPART_TYPE):
+    """convert a string (more likely bytes) or a message into
+    a mime message."""
     if not raw_data:
-        raw_data = ''
+        raw_data = b''
 
     def create_binmsg(data, content_type):
         maintype, subtype = content_type.split("/", 1)
@@ -346,15 +348,17 @@ def convert_string(raw_data, content_type=NOT_MULTIPART_TYPE):
         msg.set_payload(data)
         return msg
 
-    try:
-        data = util.decode_binary(util.decomp_gzip(raw_data))
-        if "mime-version:" in data[0:4096].lower():
-            msg = util.message_from_string(data)
-        else:
-            msg = create_binmsg(data, content_type)
-    except UnicodeDecodeError:
-        msg = create_binmsg(raw_data, content_type)
+    if isinstance(raw_data, six.text_type):
+        bdata = raw_data.encode('utf-8')
+    else:
+        bdata = raw_data
+    bdata = util.decomp_gzip(bdata, decode=False)
+    if b"mime-version:" in bdata[0:4096].lower():
+        msg = util.message_from_string(bdata.decode('utf-8'))
+    else:
+        msg = create_binmsg(bdata, content_type)
 
     return msg
 
+
 # vi: ts=4 expandtab
diff --git a/tests/unittests/test_data.py b/tests/unittests/test_data.py
index 91d35cb..3efe7ad 100644
--- a/tests/unittests/test_data.py
+++ b/tests/unittests/test_data.py
@@ -606,8 +606,10 @@ class TestUDProcess(helpers.ResourceUsingTestCase):
 
 
 class TestConvertString(helpers.TestCase):
+
     def test_handles_binary_non_utf8_decodable(self):
-        blob = b'\x32\x99'
+        """Printable unicode (not utf8-decodable) is safely converted."""
+        blob = b'#!/bin/bash\necho \xc3\x84\n'
         msg = ud.convert_string(blob)
         self.assertEqual(blob, msg.get_payload(decode=True))
 
@@ -621,6 +623,13 @@ class TestConvertString(helpers.TestCase):
         msg = ud.convert_string(text)
         self.assertEqual(text, msg.get_payload(decode=False))
 
+    def test_handle_mime_parts(self):
+        """Mime parts are properly returned as a mime message."""
+        message = MIMEBase("text", "plain")
+        message.set_payload("Just text")
+        msg = ud.convert_string(str(message))
+        self.assertEqual("Just text", msg.get_payload(decode=False))
+
 
 class TestFetchBaseConfig(helpers.TestCase):
     def test_only_builtin_gets_builtin(self):