Merge ~mgerdts/cloud-init:update_metadata into cloud-init:master

[Mike Gerdts]

update_metadata re-config on every boot comments and tests not quite right

The comment in update_metadata() that explains how a datasource should
enable network reconfig on every boot presumes that
EventType.BOOT_NEW_INSTANCE is a subset of EventType.BOOT. That's not
the case, and as such a datasource that needs to configure networking
when it is a new instance and every boot needs to include both event
types.

To make the situation above easier to debug, update_metadata() now
logs when it returns false.

To make it so that datasources do not need to test before appending to
the update_events['network'], it is changed from a list to a set.

test_update_metadata_only_acts_on_supported_update_events is updated
to allow datasources to support EventType.BOOT.

[Scott Moser (smoser)]

that looks good. thanks.

review: Approve

[Server Team CI bot (server-team-bot)]

FAILED: Autolanding.
More details in the following jenkins job:
https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/11/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: MAAS Compatability Testing

review: Needs Fixing (continuous-integration)

b7b135b... by Mike Gerdts on 2018-07-20

fix python 2.6

[Scott Moser (smoser)]

I fixed the last 2.6 issue and pushed an MP up at
 https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/351355

We'll just move this over there.

Unmerged commits

b7b135b... by Mike Gerdts on 2018-07-20

fix python 2.6

7f602de... by Mike Gerdts on 2018-07-20

update_metadata re-config on every boot comments and tests not quite right

The comment in update_metadata() that explains how a datasource should enable
network reconfig on every boot presumes that EventType.BOOT_NEW_INSTANCE is a
subset of EventType.BOOT. That's not the case, and as such a datasource that
needs to configure networking when it is a new instance and every boot needs to
include both event types.

To make the situation above easier to debug, update_metadata() now logs when it
returns false.

To make it so that datasources do not need to test before appending to
the update_events['network'], it is changed from a list to a set.

test_update_metadata_only_acts_on_supported_update_events is updated to allow
datasources to support EventType.BOOT.

diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
index f424316..06e613f 100644
--- a/cloudinit/sources/__init__.py
+++ b/cloudinit/sources/__init__.py
@@ -103,14 +103,14 @@ class DataSource(object):
     url_timeout = 10    # timeout for each metadata url read attempt
     url_retries = 5     # number of times to retry url upon 404
 
-    # The datasource defines a list of supported EventTypes during which
+    # The datasource defines a set of supported EventTypes during which
     # the datasource can react to changes in metadata and regenerate
     # network configuration on metadata changes.
     # A datasource which supports writing network config on each system boot
-    # would set update_events = {'network': [EventType.BOOT]}
+    # would call update_events['network'].add(EventType.BOOT).
 
     # Default: generate network config on new instance id (first boot).
-    update_events = {'network': [EventType.BOOT_NEW_INSTANCE]}
+    update_events = {'network': set([EventType.BOOT_NEW_INSTANCE])}
 
     # N-tuple listing default values for any metadata-related class
     # attributes cached on an instance by a process_data runs. These attribute
@@ -475,8 +475,8 @@ class DataSource(object):
             for update_scope, update_events in self.update_events.items():
                 if event in update_events:
                     if not supported_events.get(update_scope):
-                        supported_events[update_scope] = []
-                    supported_events[update_scope].append(event)
+                        supported_events[update_scope] = set()
+                    supported_events[update_scope].add(event)
         for scope, matched_events in supported_events.items():
             LOG.debug(
                 "Update datasource metadata and %s config due to events: %s",
@@ -490,6 +490,8 @@ class DataSource(object):
             result = self.get_data()
             if result:
                 return True
+        LOG.debug("Datasource %s not updated for events: %s", self,
+                  ', '.join(source_event_types))
         return False
 
     def check_instance_id(self, sys_cfg):
diff --git a/cloudinit/sources/tests/test_init.py b/cloudinit/sources/tests/test_init.py
index dcd221b..762ec17 100644
--- a/cloudinit/sources/tests/test_init.py
+++ b/cloudinit/sources/tests/test_init.py
@@ -429,8 +429,9 @@ class TestDataSource(CiTestCase):
 
     def test_update_metadata_only_acts_on_supported_update_events(self):
         """update_metadata won't get_data on unsupported update events."""
+        self.datasource.update_events['network'].discard(EventType.BOOT)
         self.assertEqual(
-            {'network': [EventType.BOOT_NEW_INSTANCE]},
+            {'network': {EventType.BOOT_NEW_INSTANCE}},
             self.datasource.update_events)
 
         def fake_get_data():