UTF-8 support in User Data (text/x-shellscript) is broken

Also reproducible with boot commands as well,

`/var/lib/cloud/instances/i-0123456789abc/user-data.txt`
        #cloud-config

        # boot commands
        # default: none
        # this is very similar to runcmd, but commands run very early
        # in the boot process, only slightly after a 'boothook' would run.
        # bootcmd should really only be used for things that could not be
        # done later in the boot process. bootcmd is very much like
        # boothook, but possibly with more friendly.
        # - bootcmd will run on every boot
        # - the INSTANCE_ID variable will be set to the current instance id.
        # - you can use 'cloud-init-per' command to help only run once Ä
        bootcmd:
         - echo Reprod | tee -a /home/ubuntu/tee
         - echo Ä

$ sudo grep -iE 'err|warn|shells|v. 1' /var/log/cloud-init.log
        2018-05-03 12:00:55,484 - util.py[DEBUG]: Cloud-init v. 17.2 running 'init-local' at Thu, 03 May 2018 12:00:55 +0000. Up 6.24 seconds.
        2018-05-03 12:00:56,435 - DataSourceEc2.py[DEBUG]: strict_mode: warn, cloud_platform=AWS
        2018-05-03 12:00:56,526 - DataSourceEc2.py[DEBUG]: Found preferred metadata version 2016-09-02
        2018-05-03 12:00:58,401 - util.py[DEBUG]: Cloud-init v. 17.2 running 'init' at Thu, 03 May 2018 12:00:58 +0000. Up 9.23 seconds.
        2018-05-03 12:00:58,473 - stages.py[DEBUG]: Added default handler for {'text/x-shellscript'} from ShellScriptPartHandler: [['text/x-shellscript']]
        2018-05-03 12:00:58,473 - __init__.py[DEBUG]: Calling handler ShellScriptPartHandler: [['text/x-shellscript']] (__begin__, None, 2) with frequency once-per-instance
        2018-05-03 12:00:58,474 - util.py[WARNING]: Failed loading yaml blob
        yaml.reader.ReaderError: unacceptable character #xdcc4: special characters are not allowed
        2018-05-03 12:00:58,483 - util.py[WARNING]: Failed at merging in cloud config part from part-001
        AttributeError: 'NoneType' object has no attribute 'pop'
        2018-05-03 12:00:58,488 - __init__.py[DEBUG]: Calling handler ShellScriptPartHandler: [['text/x-shellscript']] (__end__, None, 2) with frequency once-per-instance
        2018-05-03 12:01:00,631 - util.py[DEBUG]: Cloud-init v. 17.2 running 'modules:config' at Thu, 03 May 2018 12:01:00 +0000. Up 11.43 seconds.
        2018-05-03 12:01:02,193 - util.py[DEBUG]: Cloud-init v. 17.2 running 'modules:final' at Thu, 03 May 2018 12:01:02 +0000. Up 12.99 seconds.
        2018-05-03 12:01:02,371 - util.py[DEBUG]: Cloud-init v. 17.2 finished at Thu, 03 May 2018 12:01:02 +0000. Datasource DataSourceEc2Local. Up 13.22 seconds

Logs are attached as well (filename: 2018-05-03T13-05_0100_boot_commands_cloud-init.tar.gz).

Thanks for this good bug filing against cloud-init 17.2 and for making cloud-init better.

I validated that the traceback is still seen in 18.2.4

steps to reproduce with lxc

cat > runcmd.yaml <<EOF
#cloud-config
runcmd:
 - echo Ä
EOF

lxc init ubuntu-daily:bionic myb1
lxc config set myb1 user.user-data - < runcmd.yaml
lxc start myb1;
lxc exec myb1 cloud-init status --wait --long;
lxc exec myb1 grep yaml.reader /var/log/cloud-init.log

@chad.smith,
Hi Chad,

Q. I validated that the traceback is still seen in 18.2.4
A. Yep, correct, the issue is reproducible on cloud-init v. 18.2.4 as well (I just forgot to update this bug (apologize)).

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=faa6f07e

This bug is believed to be fixed in cloud-init in version 18.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3170