Merge ~smoser/cloud-init:bug/1742479-no-manual_cache_clean-warning into cloud-init:master

[Scott Moser]

Do not log warning on config files that represent None.

This issue was first identified when manual_cache_clean was set, as
ds-identify would write /run/cloud-init/cloud.cfg with
  # manual_cache_clean
that would generate a warning as cloud-init expected to load a dict.
Any other "empty" config would also log such a warning.

Also fix reading of di_report to allow it to be None, as ds-identify
would write:
  di_report:
    # manual_cache_clean
which reads as 'di_report: None' rather than di_report: {}.

LP: #1742479

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:73f1dc2b0217dd967485dbbb46baf6e489cc769e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/686/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/686/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:6f59e49b4ece8eceb43b01a3d7c063ee7899a051
https://jenkins.ubuntu.com/server/job/cloud-init-ci/690/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/690/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:5a6ed582ed7577a884a9b628d59fd26f1b1316d1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/691/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/691/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Good fix, LGTM!

review: Approve

diff --git a/cloudinit/cmd/main.py b/cloudinit/cmd/main.py
index 30b37fe..d2f1b77 100644
--- a/cloudinit/cmd/main.py
+++ b/cloudinit/cmd/main.py
@@ -421,7 +421,13 @@ def di_report_warn(datasource, cfg):
         LOG.debug("no di_report found in config.")
         return
 
-    dicfg = cfg.get('di_report', {})
+    dicfg = cfg['di_report']
+    if dicfg is None:
+        # ds-identify may write 'di_report:\n #comment\n'
+        # which reads as {'di_report': None}
+        LOG.debug("di_report was None.")
+        return
+
     if not isinstance(dicfg, dict):
         LOG.warning("di_report config not a dictionary: %s", dicfg)
         return
diff --git a/cloudinit/util.py b/cloudinit/util.py
index 8a9f1ab..e42498d 100644
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -891,17 +891,17 @@ def load_yaml(blob, default=None, allowed=(dict,)):
                   "of length %s with allowed root types %s",
                   len(blob), allowed)
         converted = safeyaml.load(blob)
-        if not isinstance(converted, allowed):
+        if converted is None:
+            LOG.debug("loaded blob returned None, returning default.")
+            converted = default
+        elif not isinstance(converted, allowed):
             # Yes this will just be caught, but thats ok for now...
             raise TypeError(("Yaml load allows %s root types,"
                              " but got %s instead") %
                             (allowed, type_utils.obj_name(converted)))
         loaded = converted
     except (yaml.YAMLError, TypeError, ValueError):
-        if len(blob) == 0:
-            LOG.debug("load_yaml given empty string, returning default")
-        else:
-            logexc(LOG, "Failed loading yaml blob")
+        logexc(LOG, "Failed loading yaml blob")
     return loaded
 
 
diff --git a/tests/unittests/test_util.py b/tests/unittests/test_util.py
index 787ca20..d63b760 100644
--- a/tests/unittests/test_util.py
+++ b/tests/unittests/test_util.py
@@ -299,6 +299,14 @@ class TestLoadYaml(helpers.TestCase):
                                         default=self.mydefault),
                          myobj)
 
+    def test_none_returns_default(self):
+        """If yaml.load returns None, then default should be returned."""
+        blobs = ("", " ", "# foo\n", "#")
+        mdef = self.mydefault
+        self.assertEqual(
+            [(b, self.mydefault) for b in blobs],
+            [(b, util.load_yaml(blob=b, default=mdef)) for b in blobs])
+
 
 class TestMountinfoParsing(helpers.ResourceUsingTestCase):
     def test_invalid_mountinfo(self):