Merge ~smoser/cloud-init:bug/1717598-fix-gce-user-data into cloud-init:master

[Scott Moser]

GCE: Fix usage of user-data.

This regressed in the rework of GCE datasource to have a main.
The fix really just stores the user-data read as userdata_raw, which is
correct and consistent with other datasources.

The main is updated to address the fact that user-data is binary.

LP: #1717598

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:d8a6ff62e6c33443da69ff2f53acd4823445ef62
https://jenkins.ubuntu.com/server/job/cloud-init-ci/317/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/317/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

addressed Chad's comment. (thanks chad).

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:ea3818a26e866cf12d9b64dbd405de1aa12eec31
https://jenkins.ubuntu.com/server/job/cloud-init-ci/318/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/318/rebuild

review: Approve (continuous-integration)

[Dan Watkins (daniel-thewatkins)]

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:dbfa3e879044329797fbb20fb329144e225f6456
https://jenkins.ubuntu.com/server/job/cloud-init-ci/320/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/320/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:bc110e1ca1f4505cbb636121f02d7f2fc26105ee
https://jenkins.ubuntu.com/server/job/cloud-init-ci/321/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/321/rebuild

review: Needs Fixing (continuous-integration)

[Chad Smith (chad.smith)]

You've got a couple of flake/pylint issues w/ the b64encode import

Also you can apply this patch to the existing unit tests which would have raised a unit test error on the branch previous to this due to decoding issues.
http://paste.ubuntu.com/25567020/

[Chad Smith (chad.smith)]

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:7376ff391ecfab182c0405a81b4369b92c661add
https://jenkins.ubuntu.com/server/job/cloud-init-ci/322/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/322/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:c174e78699f883d96189fe449041c6d76d846d96
https://jenkins.ubuntu.com/server/job/cloud-init-ci/324/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    FAILED: Ubuntu LTS: Build

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/324/rebuild

review: Needs Fixing (continuous-integration)

diff --git a/cloudinit/sources/DataSourceGCE.py b/cloudinit/sources/DataSourceGCE.py
index 94484d6..ccae420 100644
--- a/cloudinit/sources/DataSourceGCE.py
+++ b/cloudinit/sources/DataSourceGCE.py
@@ -62,7 +62,7 @@ class DataSourceGCE(sources.DataSource):
                 LOG.debug(ret['reason'])
             return False
         self.metadata = ret['meta-data']
-        self.userdata = ret['user-data']
+        self.userdata_raw = ret['user-data']
         return True
 
     @property
@@ -80,9 +80,6 @@ class DataSourceGCE(sources.DataSource):
         # GCE has long FDQN's and has asked for short hostnames
         return self.metadata['local-hostname'].split('.')[0]
 
-    def get_userdata_raw(self):
-        return self.userdata
-
     @property
     def availability_zone(self):
         return self.metadata['availability-zone']
@@ -202,6 +199,9 @@ def get_datasource_list(depends):
 if __name__ == "__main__":
     import argparse
     import json
+    import sys
+
+    from base64 import b64encode
 
     parser = argparse.ArgumentParser(description='Query GCE Metadata Service')
     parser.add_argument("--endpoint", metavar="URL",
@@ -211,8 +211,20 @@ if __name__ == "__main__":
                         help="Ignore smbios platform check",
                         action='store_false', default=True)
     args = parser.parse_args()
-    print(json.dumps(
-        read_md(address=args.endpoint, platform_check=args.platform_check),
-        indent=1, sort_keys=True, separators=(',', ': ')))
+    data = read_md(address=args.endpoint, platform_check=args.platform_check)
+    if 'user-data' in data:
+        # user-data is bytes not string like other things. Handle it specially.
+        # if it can be represented as utf-8 then do so.  Otherwise print base64
+        # encoded value in the key user-data-b64.
+        try:
+            data['user-data'] = data['user-data'].decode()
+        except UnicodeDecodeError:
+            sys.stderr.write("User-data cannot be decoded. "
+                             "Writing as base64\n")
+            del data['user-data']
+            # b64encode returns a bytes value. decode to get the string.
+            data['user-data-b64'] = b64encode(data['user-data']).decode()
+
+    print(json.dumps(data, indent=1, sort_keys=True, separators=(',', ': ')))
 
 # vi: ts=4 expandtab
diff --git a/tests/unittests/test_datasource/test_gce.py b/tests/unittests/test_datasource/test_gce.py
index 50e49a1..d399ae7 100644
--- a/tests/unittests/test_datasource/test_gce.py
+++ b/tests/unittests/test_datasource/test_gce.py
@@ -23,7 +23,8 @@ GCE_META = {
     'instance/zone': 'foo/bar',
     'project/attributes/sshKeys': 'user:ssh-rsa AA2..+aRD0fyVw== root@server',
     'instance/hostname': 'server.project-foo.local',
-    'instance/attributes/user-data': b'/bin/echo foo\n',
+    # UnicodeDecodeError below if set to ds.userdata instead of userdata_raw
+    'instance/attributes/user-data': b'/bin/echo \xff\n',
 }
 
 GCE_META_PARTIAL = {