Merge ~chad.smith/cloud-init:ubuntu/xenial into cloud-init:ubuntu/xenial

[Chad Smith]

new upstream snapshot for SRU into xenial..

QUESTION: we changed ubuntu-advantage cloud config module in an incompatible way on expectation that new ubuntu-advantage-tools would be released before cloud-init SRU completes. How do we want to resolve this, I've suggested reverting all upstream ubuntu advantage changes with a reverse patch (just cc_ubuntu_advantage.py and it's test file)

That is contained in this branch with
debian/patches/ubuntu-advantage-revert-tip.patch

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:40bd980d11dc1da884ce6e55c60518c3c4ebe106
https://jenkins.ubuntu.com/server/job/cloud-init-ci/718/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/718/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:421a036c999c3c1ad12872e6509315472089d53d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/719/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/719/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

LGTM. Verified I get the same branch as what's proposed.

review: Approve

diff --git a/ChangeLog b/ChangeLog
index 8fa6fdd..bf48fd4 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,120 @@
+19.1:
+  - freebsd: add chpasswd pkg in the image [Gonéri Le Bouder]
+  - tests: add Eoan release [Paride Legovini]
+  - cc_mounts: check if mount -a on no-change fstab path
+    [Jason Zions (MSFT)] (LP: #1825596)
+  - replace remaining occurrences of LOG.warn [Daniel Watkins]
+  - DataSourceAzure: Adjust timeout for polling IMDS [Anh Vo]
+  - Azure: Changes to the Hyper-V KVP Reporter [Anh Vo]
+  - git tests: no longer show warning about safe yaml.
+  - tools/read-version: handle errors [Chad Miller]
+  - net/sysconfig: only indicate available on known sysconfig distros
+    (LP: #1819994)
+  - packages: update rpm specs for new bash completion path
+    [Daniel Watkins] (LP: #1825444)
+  - test_azure: mock util.SeLinuxGuard where needed
+    [Jason Zions (MSFT)] (LP: #1825253)
+  - setup.py: install bash completion script in new location [Daniel Watkins]
+  - mount_cb: do not pass sync and rw options to mount
+    [Gonéri Le Bouder] (LP: #1645824)
+  - cc_apt_configure: fix typo in apt documentation [Dominic Schlegel]
+  - Revert "DataSource: move update_events from a class to an instance..."
+    [Daniel Watkins]
+  - Change DataSourceNoCloud to ignore file system label's case.
+    [Risto Oikarinen]
+  - cmd:main.py: Fix missing 'modules-init' key in modes dict
+    [Antonio Romito] (LP: #1815109)
+  - ubuntu_advantage: rewrite cloud-config module
+  - Azure: Treat _unset network configuration as if it were absent
+    [Jason Zions (MSFT)] (LP: #1823084)
+  - DatasourceAzure: add additional logging for azure datasource [Anh Vo]
+  - cloud_tests: fix apt_pipelining test-cases
+  - Azure: Ensure platform random_seed is always serializable as JSON.
+    [Jason Zions (MSFT)]
+  - net/sysconfig: write out SUSE-compatible IPv6 config [Robert Schweikert]
+  - tox: Update testenv for openSUSE Leap to 15.0 [Thomas Bechtold]
+  - net: Fix ipv6 static routes when using eni renderer
+    [Raphael Glon] (LP: #1818669)
+  - Add ubuntu_drivers config module [Daniel Watkins]
+  - doc: Refresh Azure walinuxagent docs [Daniel Watkins]
+  - tox: bump pylint version to latest (2.3.1) [Daniel Watkins]
+  - DataSource: move update_events from a class to an instance attribute
+    [Daniel Watkins] (LP: #1819913)
+  - net/sysconfig: Handle default route setup for dhcp configured NICs
+    [Robert Schweikert] (LP: #1812117)
+  - DataSourceEc2: update RELEASE_BLOCKER to be more accurate
+    [Daniel Watkins]
+  - cloud-init-per: POSIX sh does not support string subst, use sed
+    (LP: #1819222)
+  - Support locking user with usermod if passwd is not available.
+  - Example for Microsoft Azure data disk added. [Anton Olifir]
+  - clean: correctly determine the path for excluding seed directory
+    [Daniel Watkins] (LP: #1818571)
+  - helpers/openstack: Treat unknown link types as physical
+    [Daniel Watkins] (LP: #1639263)
+  - drop Python 2.6 support and our NIH version detection [Daniel Watkins]
+  - tip-pylint: Fix assignment-from-return-none errors
+  - net: append type:dhcp[46] only if dhcp[46] is True in v2 netconfig
+    [Kurt Stieger] (LP: #1818032)
+  - cc_apt_pipelining: stop disabling pipelining by default
+    [Daniel Watkins] (LP: #1794982)
+  - tests: fix some slow tests and some leaking state [Daniel Watkins]
+  - util: don't determine string_types ourselves [Daniel Watkins]
+  - cc_rsyslog: Escape possible nested set [Daniel Watkins] (LP: #1816967)
+  - Enable encrypted_data_bag_secret support for Chef
+    [Eric Williams] (LP: #1817082)
+  - azure: Filter list of ssh keys pulled from fabric [Jason Zions (MSFT)]
+  - doc: update merging doc with fixes and some additional details/examples
+  - tests: integration test failure summary to use traceback if empty error
+  - This is to fix https://bugs.launchpad.net/cloud-init/+bug/1812676
+    [Vitaly Kuznetsov]
+  - EC2: Rewrite network config on AWS Classic instances every boot
+    [Guilherme G. Piccoli] (LP: #1802073)
+  - netinfo: Adjust ifconfig output parsing for FreeBSD ipv6 entries
+    (LP: #1779672)
+  - netplan: Don't render yaml aliases when dumping netplan (LP: #1815051)
+  - add PyCharm IDE .idea/ path to .gitignore [Dominic Schlegel]
+  - correct grammar issue in instance metadata documentation
+    [Dominic Schlegel] (LP: #1802188)
+  - clean: cloud-init clean should not trace when run from within cloud_dir
+    (LP: #1795508)
+  - Resolve flake8 comparison and pycodestyle over-ident issues
+    [Paride Legovini]
+  - opennebula: also exclude epochseconds from changed environment vars
+    (LP: #1813641)
+  - systemd: Render generator from template to account for system
+    differences. [Robert Schweikert]
+  - sysconfig: On SUSE, use STARTMODE instead of ONBOOT
+    [Robert Schweikert] (LP: #1799540)
+  - flake8: use ==/!= to compare str, bytes, and int literals
+    [Paride Legovini]
+  - opennebula: exclude EPOCHREALTIME as known bash env variable with a
+    delta (LP: #1813383)
+  - tox: fix disco httpretty dependencies for py37 (LP: #1813361)
+  - run-container: uncomment baseurl in yum.repos.d/*.repo when using a
+    proxy [Paride Legovini]
+  - lxd: install zfs-linux instead of zfs meta package
+    [Johnson Shi] (LP: #1799779)
+  - net/sysconfig: do not write a resolv.conf file with only the header.
+    [Robert Schweikert]
+  - net: Make sysconfig renderer compatible with Network Manager.
+    [Eduardo Otubo]
+  - cc_set_passwords: Fix regex when parsing hashed passwords
+    [Marlin Cremers] (LP: #1811446)
+  - net: Wait for dhclient to daemonize before reading lease file
+    [Jason Zions] (LP: #1794399)
+  - [Azure] Increase retries when talking to Wireserver during metadata walk
+    [Jason Zions]
+  - Add documentation on adding a datasource.
+  - doc: clean up some datasource documentation.
+  - ds-identify: fix wrong variable name in ovf_vmware_transport_guestinfo.
+  - Scaleway: Support ssh keys provided inside an instance tag. [PORTE Loïc]
+  - OVF: simplify expected return values of transport functions.
+  - Vmware: Add support for the com.vmware.guestInfo OVF transport.
+    (LP: #1807466)
+  - HACKING.rst: change contact info to Josh Powers
+  - Update to pylint 2.2.2.
+
 18.5:
  - tests: add Disco release [Joshua Powers]
  - net: render 'metric' values in per-subnet routes (LP: #1805871)
diff --git a/cloudinit/config/cc_apt_configure.py b/cloudinit/config/cc_apt_configure.py
index e18944e..919d199 100644
--- a/cloudinit/config/cc_apt_configure.py
+++ b/cloudinit/config/cc_apt_configure.py
@@ -127,7 +127,7 @@ to ``^[\\w-]+:\\w``
 
 Source list entries can be specified as a dictionary under the ``sources``
 config key, with key in the dict representing a different source file. The key
-The key of each source entry will be used as an id that can be referenced in
+of each source entry will be used as an id that can be referenced in
 other config entries, as well as the filename for the source's configuration
 under ``/etc/apt/sources.list.d``. If the name does not end with ``.list``,
 it will be appended. If there is no configuration for a key in ``sources``, no
diff --git a/cloudinit/config/cc_mounts.py b/cloudinit/config/cc_mounts.py
index 339baba..123ffb8 100644
--- a/cloudinit/config/cc_mounts.py
+++ b/cloudinit/config/cc_mounts.py
@@ -439,6 +439,7 @@ def handle(_name, cfg, cloud, log, _args):
 
     cc_lines = []
     needswap = False
+    need_mount_all = False
     dirs = []
     for line in actlist:
         # write 'comment' in the fs_mntops, entry,  claiming this
@@ -449,11 +450,18 @@ def handle(_name, cfg, cloud, log, _args):
             dirs.append(line[1])
         cc_lines.append('\t'.join(line))
 
+    mount_points = [v['mountpoint'] for k, v in util.mounts().items()
+                    if 'mountpoint' in v]
     for d in dirs:
         try:
             util.ensure_dir(d)
         except Exception:
             util.logexc(log, "Failed to make '%s' config-mount", d)
+        # dirs is list of directories on which a volume should be mounted.
+        # If any of them does not already show up in the list of current
+        # mount points, we will definitely need to do mount -a.
+        if not need_mount_all and d not in mount_points:
+            need_mount_all = True
 
     sadds = [WS.sub(" ", n) for n in cc_lines]
     sdrops = [WS.sub(" ", n) for n in fstab_removed]
@@ -473,6 +481,9 @@ def handle(_name, cfg, cloud, log, _args):
         log.debug("No changes to /etc/fstab made.")
     else:
         log.debug("Changes to fstab: %s", sops)
+        need_mount_all = True
+
+    if need_mount_all:
         activate_cmds.append(["mount", "-a"])
         if uses_systemd:
             activate_cmds.append(["systemctl", "daemon-reload"])
diff --git a/cloudinit/net/sysconfig.py b/cloudinit/net/sysconfig.py
index 0998392..a47da0a 100644
--- a/cloudinit/net/sysconfig.py
+++ b/cloudinit/net/sysconfig.py
@@ -18,6 +18,8 @@ from .network_state import (
 
 LOG = logging.getLogger(__name__)
 NM_CFG_FILE = "/etc/NetworkManager/NetworkManager.conf"
+KNOWN_DISTROS = [
+    'opensuse', 'sles', 'suse', 'redhat', 'fedora', 'centos']
 
 
 def _make_header(sep='#'):
@@ -717,8 +719,8 @@ class Renderer(renderer.Renderer):
 def available(target=None):
     sysconfig = available_sysconfig(target=target)
     nm = available_nm(target=target)
-
-    return any([nm, sysconfig])
+    return (util.get_linux_distro()[0] in KNOWN_DISTROS
+            and any([nm, sysconfig]))
 
 
 def available_sysconfig(target=None):
diff --git a/cloudinit/net/tests/test_init.py b/cloudinit/net/tests/test_init.py
index f55c31e..6d2affe 100644
--- a/cloudinit/net/tests/test_init.py
+++ b/cloudinit/net/tests/test_init.py
@@ -7,11 +7,11 @@ import mock
 import os
 import requests
 import textwrap
-import yaml
 
 import cloudinit.net as net
 from cloudinit.util import ensure_file, write_file, ProcessExecutionError
 from cloudinit.tests.helpers import CiTestCase, HttprettyTestCase
+from cloudinit import safeyaml as yaml
 
 
 class TestSysDevPath(CiTestCase):
diff --git a/cloudinit/reporting/handlers.py b/cloudinit/reporting/handlers.py
 old mode 100644
 new mode 100755
index 6d23558..10165ae
--- a/cloudinit/reporting/handlers.py
+++ b/cloudinit/reporting/handlers.py
@@ -5,7 +5,6 @@ import fcntl
 import json
 import six
 import os
-import re
 import struct
 import threading
 import time
@@ -14,6 +13,7 @@ from cloudinit import log as logging
 from cloudinit.registry import DictRegistry
 from cloudinit import (url_helper, util)
 from datetime import datetime
+from six.moves.queue import Empty as QueueEmptyError
 
 if six.PY2:
     from multiprocessing.queues import JoinableQueue as JQueue
@@ -129,24 +129,50 @@ class HyperVKvpReportingHandler(ReportingHandler):
     DESC_IDX_KEY = 'msg_i'
     JSON_SEPARATORS = (',', ':')
     KVP_POOL_FILE_GUEST = '/var/lib/hyperv/.kvp_pool_1'
+    _already_truncated_pool_file = False
 
     def __init__(self,
                  kvp_file_path=KVP_POOL_FILE_GUEST,
                  event_types=None):
         super(HyperVKvpReportingHandler, self).__init__()
         self._kvp_file_path = kvp_file_path
+        HyperVKvpReportingHandler._truncate_guest_pool_file(
+            self._kvp_file_path)
+
         self._event_types = event_types
         self.q = JQueue()
-        self.kvp_file = None
         self.incarnation_no = self._get_incarnation_no()
         self.event_key_prefix = u"{0}|{1}".format(self.EVENT_PREFIX,
                                                   self.incarnation_no)
-        self._current_offset = 0
         self.publish_thread = threading.Thread(
                 target=self._publish_event_routine)
         self.publish_thread.daemon = True
         self.publish_thread.start()
 
+    @classmethod
+    def _truncate_guest_pool_file(cls, kvp_file):
+        """
+        Truncate the pool file if it has not been truncated since boot.
+        This should be done exactly once for the file indicated by
+        KVP_POOL_FILE_GUEST constant above. This method takes a filename
+        so that we can use an arbitrary file during unit testing.
+        Since KVP is a best-effort telemetry channel we only attempt to
+        truncate the file once and only if the file has not been modified
+        since boot. Additional truncation can lead to loss of existing
+        KVPs.
+        """
+        if cls._already_truncated_pool_file:
+            return
+        boot_time = time.time() - float(util.uptime())
+        try:
+            if os.path.getmtime(kvp_file) < boot_time:
+                with open(kvp_file, "w"):
+                    pass
+        except (OSError, IOError) as e:
+            LOG.warning("failed to truncate kvp pool file, %s", e)
+        finally:
+            cls._already_truncated_pool_file = True
+
     def _get_incarnation_no(self):
         """
         use the time passed as the incarnation number.
@@ -162,20 +188,15 @@ class HyperVKvpReportingHandler(ReportingHandler):
 
     def _iterate_kvps(self, offset):
         """iterate the kvp file from the current offset."""
-        try:
-            with open(self._kvp_file_path, 'rb+') as f:
-                self.kvp_file = f
-                fcntl.flock(f, fcntl.LOCK_EX)
-                f.seek(offset)
+        with open(self._kvp_file_path, 'rb') as f:
+            fcntl.flock(f, fcntl.LOCK_EX)
+            f.seek(offset)
+            record_data = f.read(self.HV_KVP_RECORD_SIZE)
+            while len(record_data) == self.HV_KVP_RECORD_SIZE:
+                kvp_item = self._decode_kvp_item(record_data)
+                yield kvp_item
                 record_data = f.read(self.HV_KVP_RECORD_SIZE)
-                while len(record_data) == self.HV_KVP_RECORD_SIZE:
-                    self._current_offset += self.HV_KVP_RECORD_SIZE
-                    kvp_item = self._decode_kvp_item(record_data)
-                    yield kvp_item
-                    record_data = f.read(self.HV_KVP_RECORD_SIZE)
-                fcntl.flock(f, fcntl.LOCK_UN)
-        finally:
-            self.kvp_file = None
+            fcntl.flock(f, fcntl.LOCK_UN)
 
     def _event_key(self, event):
         """
@@ -207,23 +228,13 @@ class HyperVKvpReportingHandler(ReportingHandler):
 
         return {'key': k, 'value': v}
 
-    def _update_kvp_item(self, record_data):
-        if self.kvp_file is None:
-            raise ReportException(
-                "kvp file '{0}' not opened."
-                .format(self._kvp_file_path))
-        self.kvp_file.seek(-self.HV_KVP_RECORD_SIZE, 1)
-        self.kvp_file.write(record_data)
-
     def _append_kvp_item(self, record_data):
-        with open(self._kvp_file_path, 'rb+') as f:
+        with open(self._kvp_file_path, 'ab') as f:
             fcntl.flock(f, fcntl.LOCK_EX)
-            # seek to end of the file
-            f.seek(0, 2)
-            f.write(record_data)
+            for data in record_data:
+                f.write(data)
             f.flush()
             fcntl.flock(f, fcntl.LOCK_UN)
-            self._current_offset = f.tell()
 
     def _break_down(self, key, meta_data, description):
         del meta_data[self.MSG_KEY]
@@ -279,40 +290,26 @@ class HyperVKvpReportingHandler(ReportingHandler):
 
     def _publish_event_routine(self):
         while True:
+            items_from_queue = 0
             try:
                 event = self.q.get(block=True)
-                need_append = True
+                items_from_queue += 1
+                encoded_data = []
+                while event is not None:
+                    encoded_data += self._encode_event(event)
+                    try:
+                        # get all the rest of the events in the queue
+                        event = self.q.get(block=False)
+                        items_from_queue += 1
+                    except QueueEmptyError:
+                        event = None
                 try:
-                    if not os.path.exists(self._kvp_file_path):
-                        LOG.warning(
-                            "skip writing events %s to %s. file not present.",
-                            event.as_string(),
-                            self._kvp_file_path)
-                    encoded_event = self._encode_event(event)
-                    # for each encoded_event
-                    for encoded_data in (encoded_event):
-                        for kvp in self._iterate_kvps(self._current_offset):
-                            match = (
-                                re.match(
-                                    r"^{0}\|(\d+)\|.+"
-                                    .format(self.EVENT_PREFIX),
-                                    kvp['key']
-                                ))
-                            if match:
-                                match_groups = match.groups(0)
-                                if int(match_groups[0]) < self.incarnation_no:
-                                    need_append = False
-                                    self._update_kvp_item(encoded_data)
-                                    continue
-                        if need_append:
-                            self._append_kvp_item(encoded_data)
-                except IOError as e:
-                    LOG.warning(
-                        "failed posting event to kvp: %s e:%s",
-                        event.as_string(), e)
+                    self._append_kvp_item(encoded_data)
+                except (OSError, IOError) as e:
+                    LOG.warning("failed posting events to kvp, %s", e)
                 finally:
-                    self.q.task_done()
-
+                    for _ in range(items_from_queue):
+                        self.q.task_done()
             # when main process exits, q.get() will through EOFError
             # indicating we should exit this thread.
             except EOFError:
@@ -322,7 +319,7 @@ class HyperVKvpReportingHandler(ReportingHandler):
     # if the kvp pool already contains a chunk of data,
     # so defer it to another thread.
     def publish_event(self, event):
-        if (not self._event_types or event.event_type in self._event_types):
+        if not self._event_types or event.event_type in self._event_types:
             self.q.put(event)
 
     def flush(self):
diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py
index 76b1661..b7440c1 100755
--- a/cloudinit/sources/DataSourceAzure.py
+++ b/cloudinit/sources/DataSourceAzure.py
@@ -57,7 +57,12 @@ AZURE_CHASSIS_ASSET_TAG = '7783-7084-3265-9085-8269-3286-77'
 REPROVISION_MARKER_FILE = "/var/lib/cloud/data/poll_imds"
 REPORTED_READY_MARKER_FILE = "/var/lib/cloud/data/reported_ready"
 AGENT_SEED_DIR = '/var/lib/waagent'
+
+# In the event where the IMDS primary server is not
+# available, it takes 1s to fallback to the secondary one
+IMDS_TIMEOUT_IN_SECONDS = 2
 IMDS_URL = "http://169.254.169.254/metadata/"
+
 PLATFORM_ENTROPY_SOURCE = "/sys/firmware/acpi/tables/OEM0"
 
 # List of static scripts and network config artifacts created by
@@ -407,7 +412,7 @@ class DataSourceAzure(sources.DataSource):
                 elif cdev.startswith("/dev/"):
                     if util.is_FreeBSD():
                         ret = util.mount_cb(cdev, load_azure_ds_dir,
-                                            mtype="udf", sync=False)
+                                            mtype="udf")
                     else:
                         ret = util.mount_cb(cdev, load_azure_ds_dir)
                 else:
@@ -582,9 +587,9 @@ class DataSourceAzure(sources.DataSource):
                         return
                     self._ephemeral_dhcp_ctx.clean_network()
                 else:
-                    return readurl(url, timeout=1, headers=headers,
-                                   exception_cb=exc_cb, infinite=True,
-                                   log_req_resp=False).contents
+                    return readurl(url, timeout=IMDS_TIMEOUT_IN_SECONDS,
+                                   headers=headers, exception_cb=exc_cb,
+                                   infinite=True, log_req_resp=False).contents
             except UrlError:
                 # Teardown our EphemeralDHCPv4 context on failure as we retry
                 self._ephemeral_dhcp_ctx.clean_network()
@@ -1291,8 +1296,8 @@ def _get_metadata_from_imds(retries):
     headers = {"Metadata": "true"}
     try:
         response = readurl(
-            url, timeout=1, headers=headers, retries=retries,
-            exception_cb=retry_on_url_exc)
+            url, timeout=IMDS_TIMEOUT_IN_SECONDS, headers=headers,
+            retries=retries, exception_cb=retry_on_url_exc)
     except Exception as e:
         LOG.debug('Ignoring IMDS instance metadata: %s', e)
         return {}
diff --git a/cloudinit/sources/DataSourceCloudStack.py b/cloudinit/sources/DataSourceCloudStack.py
index d4b758f..f185dc7 100644
--- a/cloudinit/sources/DataSourceCloudStack.py
+++ b/cloudinit/sources/DataSourceCloudStack.py
@@ -95,7 +95,7 @@ class DataSourceCloudStack(sources.DataSource):
         start_time = time.time()
         url = uhelp.wait_for_url(
             urls=urls, max_wait=url_params.max_wait_seconds,
-            timeout=url_params.timeout_seconds, status_cb=LOG.warn)
+            timeout=url_params.timeout_seconds, status_cb=LOG.warning)
 
         if url:
             LOG.debug("Using metadata source: '%s'", url)
diff --git a/cloudinit/sources/DataSourceConfigDrive.py b/cloudinit/sources/DataSourceConfigDrive.py
index 564e3eb..571d30d 100644
--- a/cloudinit/sources/DataSourceConfigDrive.py
+++ b/cloudinit/sources/DataSourceConfigDrive.py
@@ -72,15 +72,12 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource):
             dslist = self.sys_cfg.get('datasource_list')
             for dev in find_candidate_devs(dslist=dslist):
                 try:
-                    # Set mtype if freebsd and turn off sync
-                    if dev.startswith("/dev/cd"):
+                    if util.is_FreeBSD() and dev.startswith("/dev/cd"):
                         mtype = "cd9660"
-                        sync = False
                     else:
                         mtype = None
-                        sync = True
                     results = util.mount_cb(dev, read_config_drive,
-                                            mtype=mtype, sync=sync)
+                                            mtype=mtype)
                     found = dev
                 except openstack.NonReadable:
                     pass
diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourceEc2.py
index ac28f1d..5c017bf 100644
--- a/cloudinit/sources/DataSourceEc2.py
+++ b/cloudinit/sources/DataSourceEc2.py
@@ -208,7 +208,7 @@ class DataSourceEc2(sources.DataSource):
         start_time = time.time()
         url = uhelp.wait_for_url(
             urls=urls, max_wait=url_params.max_wait_seconds,
-            timeout=url_params.timeout_seconds, status_cb=LOG.warn)
+            timeout=url_params.timeout_seconds, status_cb=LOG.warning)
 
         if url:
             self.metadata_address = url2base[url]
diff --git a/cloudinit/sources/DataSourceNoCloud.py b/cloudinit/sources/DataSourceNoCloud.py
index 6860f0c..fcf5d58 100644
--- a/cloudinit/sources/DataSourceNoCloud.py
+++ b/cloudinit/sources/DataSourceNoCloud.py
@@ -106,7 +106,9 @@ class DataSourceNoCloud(sources.DataSource):
             fslist = util.find_devs_with("TYPE=vfat")
             fslist.extend(util.find_devs_with("TYPE=iso9660"))
 
-            label_list = util.find_devs_with("LABEL=%s" % label)
+            label_list = util.find_devs_with("LABEL=%s" % label.upper())
+            label_list.extend(util.find_devs_with("LABEL=%s" % label.lower()))
+
             devlist = list(set(fslist) & set(label_list))
             devlist.sort(reverse=True)
 
diff --git a/cloudinit/sources/DataSourceScaleway.py b/cloudinit/sources/DataSourceScaleway.py
index 54bfc1f..b573b38 100644
--- a/cloudinit/sources/DataSourceScaleway.py
+++ b/cloudinit/sources/DataSourceScaleway.py
@@ -171,11 +171,10 @@ def query_data_api(api_type, api_address, retries, timeout):
 
 class DataSourceScaleway(sources.DataSource):
     dsname = "Scaleway"
+    update_events = {'network': [EventType.BOOT_NEW_INSTANCE, EventType.BOOT]}
 
     def __init__(self, sys_cfg, distro, paths):
         super(DataSourceScaleway, self).__init__(sys_cfg, distro, paths)
-        self.update_events = {
-            'network': {EventType.BOOT_NEW_INSTANCE, EventType.BOOT}}
 
         self.ds_cfg = util.mergemanydict([
             util.get_cfg_by_path(sys_cfg, ["datasource", "Scaleway"], {}),
diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
index 1604932..e6966b3 100644
--- a/cloudinit/sources/__init__.py
+++ b/cloudinit/sources/__init__.py
@@ -164,6 +164,9 @@ class DataSource(object):
     # A datasource which supports writing network config on each system boot
     # would call update_events['network'].add(EventType.BOOT).
 
+    # Default: generate network config on new instance id (first boot).
+    update_events = {'network': set([EventType.BOOT_NEW_INSTANCE])}
+
     # N-tuple listing default values for any metadata-related class
     # attributes cached on an instance by a process_data runs. These attribute
     # values are reset via clear_cached_attrs during any update_metadata call.
@@ -188,9 +191,6 @@ class DataSource(object):
         self.vendordata = None
         self.vendordata_raw = None
 
-        # Default: generate network config on new instance id (first boot).
-        self.update_events = {'network': {EventType.BOOT_NEW_INSTANCE}}
-
         self.ds_cfg = util.get_cfg_by_path(
             self.sys_cfg, ("datasource", self.dsname), {})
         if not self.ds_cfg:
diff --git a/cloudinit/sources/helpers/azure.py b/cloudinit/sources/helpers/azure.py
index d3af05e..82c4c8c 100755
--- a/cloudinit/sources/helpers/azure.py
+++ b/cloudinit/sources/helpers/azure.py
@@ -20,6 +20,9 @@ from cloudinit.reporting import events
 
 LOG = logging.getLogger(__name__)
 
+# This endpoint matches the format as found in dhcp lease files, since this
+# value is applied if the endpoint can't be found within a lease file
+DEFAULT_WIRESERVER_ENDPOINT = "a8:3f:81:10"
 
 azure_ds_reporter = events.ReportEventStack(
     name="azure-ds",
@@ -297,7 +300,12 @@ class WALinuxAgentShim(object):
     @azure_ds_telemetry_reporter
     def _get_value_from_leases_file(fallback_lease_file):
         leases = []
-        content = util.load_file(fallback_lease_file)
+        try:
+            content = util.load_file(fallback_lease_file)
+        except IOError as ex:
+            LOG.error("Failed to read %s: %s", fallback_lease_file, ex)
+            return None
+
         LOG.debug("content is %s", content)
         option_name = _get_dhcp_endpoint_option_name()
         for line in content.splitlines():
@@ -372,9 +380,9 @@ class WALinuxAgentShim(object):
                           fallback_lease_file)
                 value = WALinuxAgentShim._get_value_from_leases_file(
                     fallback_lease_file)
-
         if value is None:
-            raise ValueError('No endpoint found.')
+            LOG.warning("No lease found; using default endpoint")
+            value = DEFAULT_WIRESERVER_ENDPOINT
 
         endpoint_ip_address = WALinuxAgentShim.get_ip_from_lease_value(value)
         LOG.debug('Azure endpoint found at %s', endpoint_ip_address)
diff --git a/cloudinit/sources/tests/test_init.py b/cloudinit/sources/tests/test_init.py
index cb1912b..6378e98 100644
--- a/cloudinit/sources/tests/test_init.py
+++ b/cloudinit/sources/tests/test_init.py
@@ -575,21 +575,6 @@ class TestDataSource(CiTestCase):
             " events: New instance first boot",
             self.logs.getvalue())
 
-    def test_data_sources_cant_mutate_update_events_for_others(self):
-        """update_events shouldn't be changed for other DSes (LP: #1819913)"""
-
-        class ModifyingDS(DataSource):
-
-            def __init__(self, sys_cfg, distro, paths):
-                # This mirrors what DataSourceAzure does which causes LP:
-                # #1819913
-                DataSource.__init__(self, sys_cfg, distro, paths)
-                self.update_events['network'].add(EventType.BOOT)
-
-        before_update_events = copy.deepcopy(self.datasource.update_events)
-        ModifyingDS(self.sys_cfg, self.distro, self.paths)
-        self.assertEqual(before_update_events, self.datasource.update_events)
-
 
 class TestRedactSensitiveData(CiTestCase):
 
diff --git a/cloudinit/util.py b/cloudinit/util.py
index 385f231..ea4199c 100644
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -1679,7 +1679,7 @@ def mounts():
     return mounted
 
 
-def mount_cb(device, callback, data=None, rw=False, mtype=None, sync=True,
+def mount_cb(device, callback, data=None, mtype=None,
              update_env_for_mount=None):
     """
     Mount the device, call method 'callback' passing the directory
@@ -1726,18 +1726,7 @@ def mount_cb(device, callback, data=None, rw=False, mtype=None, sync=True,
             for mtype in mtypes:
                 mountpoint = None
                 try:
-                    mountcmd = ['mount']
-                    mountopts = []
-                    if rw:
-                        mountopts.append('rw')
-                    else:
-                        mountopts.append('ro')
-                    if sync:
-                        # This seems like the safe approach to do
-                        # (ie where this is on by default)
-                        mountopts.append("sync")
-                    if mountopts:
-                        mountcmd.extend(["-o", ",".join(mountopts)])
+                    mountcmd = ['mount', '-o', 'ro']
                     if mtype:
                         mountcmd.extend(['-t', mtype])
                     mountcmd.append(device)
diff --git a/cloudinit/version.py b/cloudinit/version.py
index a2c5d43..ddcd436 100644
--- a/cloudinit/version.py
+++ b/cloudinit/version.py
@@ -4,7 +4,7 @@
 #
 # This file is part of cloud-init. See LICENSE file for license information.
 
-__VERSION__ = "18.5"
+__VERSION__ = "19.1"
 _PACKAGED_VERSION = '@@PACKAGED_VERSION@@'
 
 FEATURES = [
diff --git a/debian/changelog b/debian/changelog
index d21167b..270b0f3 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,11 +1,57 @@
-cloud-init (18.5-45-g3554ffe8-0ubuntu1~16.04.2) UNRELEASED; urgency=medium
+cloud-init (19.1-1-gbaa47854-0ubuntu1~16.04.1) xenial; urgency=medium
 
+  * debian/patches/ubuntu-advantage-revert-tip.patch
+    Revert ubuntu-advantage config module changes until ubuntu-advantage-tools
+    19.1 publishes to Xenial (LP: #1828641)
   * refresh patches:
    + debian/patches/azure-apply-network-config-false.patch
    + debian/patches/azure-use-walinux-agent.patch
    + debian/patches/ec2-classic-dont-reapply-networking.patch
+  * refresh patches:
+   + debian/patches/azure-apply-network-config-false.patch
+   + debian/patches/azure-use-walinux-agent.patch
+  * New upstream snapshot. (LP: #1828637)
+    - Azure: Return static fallback address as if failed to find endpoint
+      [Jason Zions (MSFT)]
+    - release 19.1
+    - freebsd: add chpasswd pkg in the image [Gonéri Le Bouder]
+    - tests: add Eoan release [Paride Legovini]
+    - cc_mounts: check if mount -a on no-change fstab path [Jason Zions (MSFT)]
+    - replace remaining occurrences of LOG.warn
+    - DataSourceAzure: Adjust timeout for polling IMDS [Anh Vo]
+    - Azure: Changes to the Hyper-V KVP Reporter [Anh Vo]
+    - git tests: no longer show warning about safe yaml. [Scott Moser]
+    - tools/read-version: handle errors [Chad Miller]
+    - net/sysconfig: only indicate available on known sysconfig distros
+    - packages: update rpm specs for new bash completion path
+    - test_azure: mock util.SeLinuxGuard where needed [Jason Zions (MSFT)]
+    - setup.py: install bash completion script in new location
+    - mount_cb: do not pass sync and rw options to mount [Gonéri Le Bouder]
+    - cc_apt_configure: fix typo in apt documentation [Dominic Schlegel]
+    - Revert "DataSource: move update_events from a class to an instance..."
+    - Change DataSourceNoCloud to ignore file system label's case.
+      [Risto Oikarinen]
+    - cmd:main.py: Fix missing 'modules-init' key in modes dict
+      [Antonio Romito]
+    - ubuntu_advantage: rewrite cloud-config module
+    - Azure: Treat _unset network configuration as if it were absent
+      [Jason Zions (MSFT)]
+    - DatasourceAzure: add additional logging for azure datasource [Anh Vo]
+    - cloud_tests: fix apt_pipelining test-cases
+    - Azure: Ensure platform random_seed is always serializable as JSON.
+      [Jason Zions (MSFT)]
+    - net/sysconfig: write out SUSE-compatible IPv6 config [Robert Schweikert]
+    - tox: Update testenv for openSUSE Leap to 15.0 [Thomas Bechtold]
+    - net: Fix ipv6 static routes when using eni renderer [Raphael Glon]
+    - Add ubuntu_drivers config module
+    - doc: Refresh Azure walinuxagent docs
+    - tox: bump pylint version to latest (2.3.1)
+    - DataSource: move update_events from a class to an instance attribute
+    - net/sysconfig: Handle default route setup for dhcp configured NICs
+      [Robert Schweikert]
+    - DataSourceEc2: update RELEASE_BLOCKER to be more accurate
 
- -- Ryan Harper <[e-mail]>  Tue, 09 Apr 2019 11:20:17 -0500
+ -- Chad Smith <[e-mail]>  Fri, 10 May 2019 16:26:48 -0600
 
 cloud-init (18.5-45-g3554ffe8-0ubuntu1~16.04.1) xenial; urgency=medium
 
diff --git a/debian/patches/azure-apply-network-config-false.patch b/debian/patches/azure-apply-network-config-false.patch
index e16ad64..f0c2fcf 100644
--- a/debian/patches/azure-apply-network-config-false.patch
+++ b/debian/patches/azure-apply-network-config-false.patch
@@ -10,7 +10,7 @@ Forwarded: not-needed
 Last-Update: 2018-10-17
 --- a/cloudinit/sources/DataSourceAzure.py
 +++ b/cloudinit/sources/DataSourceAzure.py
-@@ -215,7 +215,7 @@ BUILTIN_DS_CONFIG = {
+@@ -220,7 +220,7 @@ BUILTIN_DS_CONFIG = {
      },
      'disk_aliases': {'ephemeral0': RESOURCE_DISK_PATH},
      'dhclient_lease_file': LEASE_FILE,
diff --git a/debian/patches/azure-use-walinux-agent.patch b/debian/patches/azure-use-walinux-agent.patch
index 3f60dfd..b4ad76c 100644
--- a/debian/patches/azure-use-walinux-agent.patch
+++ b/debian/patches/azure-use-walinux-agent.patch
@@ -6,7 +6,7 @@ Forwarded: not-needed
 Author: Scott Moser <[e-mail]>
 --- a/cloudinit/sources/DataSourceAzure.py
 +++ b/cloudinit/sources/DataSourceAzure.py
-@@ -204,7 +204,7 @@ if util.is_FreeBSD():
+@@ -209,7 +209,7 @@ if util.is_FreeBSD():
      PLATFORM_ENTROPY_SOURCE = None
  
  BUILTIN_DS_CONFIG = {
diff --git a/debian/patches/series b/debian/patches/series
index d37ae8a..5d6995e 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -4,3 +4,4 @@ stable-release-no-jsonschema-dep.patch
 openstack-no-network-config.patch
 azure-apply-network-config-false.patch
 ec2-classic-dont-reapply-networking.patch
+ubuntu-advantage-revert-tip.patch
diff --git a/debian/patches/ubuntu-advantage-revert-tip.patch b/debian/patches/ubuntu-advantage-revert-tip.patch
 new file mode 100644
index 0000000..08bdc81
--- /dev/null
+++ b/debian/patches/ubuntu-advantage-revert-tip.patch
@@ -0,0 +1,735 @@
+Description: Revert upstream changes for ubuntu-advantage-tools v 19.1
+ ubuntu-advantage-tools v. 19.1 or later is required for the newcw
+ cloud-config module becaues the two command lines are incompatible.
+ Xenial can drop this patch once ubuntu-advantage-tools has been SRU'd >= 19.1
+Author: Chad Smith <[e-mail]>
+Origin: backport
+Bug: https://bugs.launchpad.net/cloud-init/+bug/1828641
+Forwarded: not-needed
+Last-Update: 2019-05-10
+---
+This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
+Index: cloud-init/cloudinit/config/cc_ubuntu_advantage.py
+===================================================================
+--- cloud-init.orig/cloudinit/config/cc_ubuntu_advantage.py
++++ cloud-init/cloudinit/config/cc_ubuntu_advantage.py
+@@ -1,143 +1,150 @@
++# Copyright (C) 2018 Canonical Ltd.
++#
+ # This file is part of cloud-init. See LICENSE file for license information.
+ 
+-"""ubuntu_advantage: Configure Ubuntu Advantage support services"""
++"""Ubuntu advantage: manage ubuntu-advantage offerings from Canonical."""
+ 
++import sys
+ from textwrap import dedent
+ 
+-import six
+-
++from cloudinit import log as logging
+ from cloudinit.config.schema import (
+     get_schema_doc, validate_cloudconfig_schema)
+-from cloudinit import log as logging
+ from cloudinit.settings import PER_INSTANCE
++from cloudinit.subp import prepend_base_command
+ from cloudinit import util
+ 
+ 
+-UA_URL = 'https://ubuntu.com/advantage'
+-
+ distros = ['ubuntu']
++frequency = PER_INSTANCE
++
++LOG = logging.getLogger(__name__)
+ 
+ schema = {
+     'id': 'cc_ubuntu_advantage',
+     'name': 'Ubuntu Advantage',
+-    'title': 'Configure Ubuntu Advantage support services',
++    'title': 'Install, configure and manage ubuntu-advantage offerings',
+     'description': dedent("""\
+-        Attach machine to an existing Ubuntu Advantage support contract and
+-        enable or disable support services such as Livepatch, ESM,
+-        FIPS and FIPS Updates. When attaching a machine to Ubuntu Advantage,
+-        one can also specify services to enable.  When the 'enable'
+-        list is present, any named service will be enabled and all absent
+-        services will remain disabled.
+-
+-        Note that when enabling FIPS or FIPS updates you will need to schedule
+-        a reboot to ensure the machine is running the FIPS-compliant kernel.
+-        See :ref:`Power State Change` for information on how to configure
+-        cloud-init to perform this reboot.
++        This module provides configuration options to setup ubuntu-advantage
++        subscriptions.
++
++        .. note::
++            Both ``commands`` value can be either a dictionary or a list. If
++            the configuration provided is a dictionary, the keys are only used
++            to order the execution of the commands and the dictionary is
++            merged with any vendor-data ubuntu-advantage configuration
++            provided. If a ``commands`` is provided as a list, any vendor-data
++            ubuntu-advantage ``commands`` are ignored.
++
++        Ubuntu-advantage ``commands`` is a dictionary or list of
++        ubuntu-advantage commands to run on the deployed machine.
++        These commands can be used to enable or disable subscriptions to
++        various ubuntu-advantage products. See 'man ubuntu-advantage' for more
++        information on supported subcommands.
++
++        .. note::
++           Each command item can be a string or list. If the item is a list,
++           'ubuntu-advantage' can be omitted and it will automatically be
++           inserted as part of the command.
+         """),
+     'distros': distros,
+     'examples': [dedent("""\
+-        # Attach the machine to a Ubuntu Advantage support contract with a
+-        # UA contract token obtained from %s.
+-        ubuntu_advantage:
+-          token: <ua_contract_token>
+-    """ % UA_URL), dedent("""\
+-        # Attach the machine to an Ubuntu Advantage support contract enabling
+-        # only fips and esm services. Services will only be enabled if
+-        # the environment supports said service. Otherwise warnings will
+-        # be logged for incompatible services specified.
++        # Enable Extended Security Maintenance using your service auth token
++        ubuntu-advantage:
++            commands:
++              00: ubuntu-advantage enable-esm <token>
++    """), dedent("""\
++        # Enable livepatch by providing your livepatch token
+         ubuntu-advantage:
+-          token: <ua_contract_token>
+-          enable:
+-          - fips
+-          - esm
++            commands:
++                00: ubuntu-advantage enable-livepatch <livepatch-token>
++
+     """), dedent("""\
+-        # Attach the machine to an Ubuntu Advantage support contract and enable
+-        # the FIPS service.  Perform a reboot once cloud-init has
+-        # completed.
+-        power_state:
+-          mode: reboot
++        # Convenience: the ubuntu-advantage command can be omitted when
++        # specifying commands as a list and 'ubuntu-advantage' will
++        # automatically be prepended.
++        # The following commands are equivalent
+         ubuntu-advantage:
+-          token: <ua_contract_token>
+-          enable:
+-          - fips
+-        """)],
++            commands:
++                00: ['enable-livepatch', 'my-token']
++                01: ['ubuntu-advantage', 'enable-livepatch', 'my-token']
++                02: ubuntu-advantage enable-livepatch my-token
++                03: 'ubuntu-advantage enable-livepatch my-token'
++    """)],
+     'frequency': PER_INSTANCE,
+     'type': 'object',
+     'properties': {
+-        'ubuntu_advantage': {
++        'ubuntu-advantage': {
+             'type': 'object',
+             'properties': {
+-                'enable': {
+-                    'type': 'array',
+-                    'items': {'type': 'string'},
+-                },
+-                'token': {
+-                    'type': 'string',
+-                    'description': (
+-                        'A contract token obtained from %s.' % UA_URL)
++                'commands': {
++                    'type': ['object', 'array'],  # Array of strings or dict
++                    'items': {
++                        'oneOf': [
++                            {'type': 'array', 'items': {'type': 'string'}},
++                            {'type': 'string'}]
++                    },
++                    'additionalItems': False,  # Reject non-string & non-list
++                    'minItems': 1,
++                    'minProperties': 1,
+                 }
+             },
+-            'required': ['token'],
+-            'additionalProperties': False
++            'additionalProperties': False,  # Reject keys not in schema
++            'required': ['commands']
+         }
+     }
+ }
+ 
++# TODO schema for 'assertions' and 'commands' are too permissive at the moment.
++# Once python-jsonschema supports schema draft 6 add support for arbitrary
++# object keys with 'patternProperties' constraint to validate string values.
++
+ __doc__ = get_schema_doc(schema)  # Supplement python help()
+ 
+-LOG = logging.getLogger(__name__)
++UA_CMD = "ubuntu-advantage"
+ 
+ 
+-def configure_ua(token=None, enable=None):
+-    """Call ua commandline client to attach or enable services."""
+-    error = None
+-    if not token:
+-        error = ('ubuntu_advantage: token must be provided')
+-        LOG.error(error)
+-        raise RuntimeError(error)
+-
+-    if enable is None:
+-        enable = []
+-    elif isinstance(enable, six.string_types):
+-        LOG.warning('ubuntu_advantage: enable should be a list, not'
+-                    ' a string; treating as a single enable')
+-        enable = [enable]
+-    elif not isinstance(enable, list):
+-        LOG.warning('ubuntu_advantage: enable should be a list, not'
+-                    ' a %s; skipping enabling services',
+-                    type(enable).__name__)
+-        enable = []
++def run_commands(commands):
++    """Run the commands provided in ubuntu-advantage:commands config.
+ 
+-    attach_cmd = ['ua', 'attach', token]
+-    LOG.debug('Attaching to Ubuntu Advantage. %s', ' '.join(attach_cmd))
+-    try:
+-        util.subp(attach_cmd)
+-    except util.ProcessExecutionError as e:
+-        msg = 'Failure attaching Ubuntu Advantage:\n{error}'.format(
+-            error=str(e))
+-        util.logexc(LOG, msg)
+-        raise RuntimeError(msg)
+-    enable_errors = []
+-    for service in enable:
++     Commands are run individually. Any errors are collected and reported
++     after attempting all commands.
++
++     @param commands: A list or dict containing commands to run. Keys of a
++         dict will be used to order the commands provided as dict values.
++     """
++    if not commands:
++        return
++    LOG.debug('Running user-provided ubuntu-advantage commands')
++    if isinstance(commands, dict):
++        # Sort commands based on dictionary key
++        commands = [v for _, v in sorted(commands.items())]
++    elif not isinstance(commands, list):
++        raise TypeError(
++            'commands parameter was not a list or dict: {commands}'.format(
++                commands=commands))
++
++    fixed_ua_commands = prepend_base_command('ubuntu-advantage', commands)
++
++    cmd_failures = []
++    for command in fixed_ua_commands:
++        shell = isinstance(command, str)
+         try:
+-            cmd = ['ua', 'enable', service]
+-            util.subp(cmd, capture=True)
++            util.subp(command, shell=shell, status_cb=sys.stderr.write)
+         except util.ProcessExecutionError as e:
+-            enable_errors.append((service, e))
+-    if enable_errors:
+-        for service, error in enable_errors:
+-            msg = 'Failure enabling "{service}":\n{error}'.format(
+-                service=service, error=str(error))
+-            util.logexc(LOG, msg)
+-        raise RuntimeError(
+-            'Failure enabling Ubuntu Advantage service(s): {}'.format(
+-                ', '.join('"{}"'.format(service)
+-                          for service, _ in enable_errors)))
++            cmd_failures.append(str(e))
++    if cmd_failures:
++        msg = (
++            'Failures running ubuntu-advantage commands:\n'
++            '{cmd_failures}'.format(
++                cmd_failures=cmd_failures))
++        util.logexc(LOG, msg)
++        raise RuntimeError(msg)
+ 
+ 
+ def maybe_install_ua_tools(cloud):
+     """Install ubuntu-advantage-tools if not present."""
+-    if util.which('ua'):
++    if util.which('ubuntu-advantage'):
+         return
+     try:
+         cloud.distro.update_package_sources()
+@@ -152,28 +159,14 @@ def maybe_install_ua_tools(cloud):
+ 
+ 
+ def handle(name, cfg, cloud, log, args):
+-    ua_section = None
+-    if 'ubuntu-advantage' in cfg:
+-        LOG.warning('Deprecated configuration key "ubuntu-advantage" provided.'
+-                    ' Expected underscore delimited "ubuntu_advantage"; will'
+-                    ' attempt to continue.')
+-        ua_section = cfg['ubuntu-advantage']
+-    if 'ubuntu_advantage' in cfg:
+-        ua_section = cfg['ubuntu_advantage']
+-    if ua_section is None:
+-        LOG.debug("Skipping module named %s,"
+-                  " no 'ubuntu_advantage' configuration found", name)
++    cfgin = cfg.get('ubuntu-advantage')
++    if cfgin is None:
++        LOG.debug(("Skipping module named %s,"
++                   " no 'ubuntu-advantage' key in configuration"), name)
+         return
+-    validate_cloudconfig_schema(cfg, schema)
+-    if 'commands' in ua_section:
+-        msg = (
+-            'Deprecated configuration "ubuntu-advantage: commands" provided.'
+-            ' Expected "token"')
+-        LOG.error(msg)
+-        raise RuntimeError(msg)
+ 
++    validate_cloudconfig_schema(cfg, schema)
+     maybe_install_ua_tools(cloud)
+-    configure_ua(token=ua_section.get('token'),
+-                 enable=ua_section.get('enable'))
++    run_commands(cfgin.get('commands', []))
+ 
+ # vi: ts=4 expandtab
+Index: cloud-init/cloudinit/config/tests/test_ubuntu_advantage.py
+===================================================================
+--- cloud-init.orig/cloudinit/config/tests/test_ubuntu_advantage.py
++++ cloud-init/cloudinit/config/tests/test_ubuntu_advantage.py
+@@ -1,7 +1,10 @@
+ # This file is part of cloud-init. See LICENSE file for license information.
+ 
++import re
++from six import StringIO
++
+ from cloudinit.config.cc_ubuntu_advantage import (
+-    configure_ua, handle, maybe_install_ua_tools, schema)
++    handle, maybe_install_ua_tools, run_commands, schema)
+ from cloudinit.config.schema import validate_cloudconfig_schema
+ from cloudinit import util
+ from cloudinit.tests.helpers import (
+@@ -17,120 +20,90 @@ class FakeCloud(object):
+         self.distro = distro
+ 
+ 
+-class TestConfigureUA(CiTestCase):
++class TestRunCommands(CiTestCase):
+ 
+     with_logs = True
+     allowed_subp = [CiTestCase.SUBP_SHELL_TRUE]
+ 
+     def setUp(self):
+-        super(TestConfigureUA, self).setUp()
++        super(TestRunCommands, self).setUp()
+         self.tmp = self.tmp_dir()
+ 
+     @mock.patch('%s.util.subp' % MPATH)
+-    def test_configure_ua_attach_error(self, m_subp):
+-        """Errors from ua attach command are raised."""
+-        m_subp.side_effect = util.ProcessExecutionError(
+-            'Invalid token SomeToken')
+-        with self.assertRaises(RuntimeError) as context_manager:
+-            configure_ua(token='SomeToken')
++    def test_run_commands_on_empty_list(self, m_subp):
++        """When provided with an empty list, run_commands does nothing."""
++        run_commands([])
++        self.assertEqual('', self.logs.getvalue())
++        m_subp.assert_not_called()
++
++    def test_run_commands_on_non_list_or_dict(self):
++        """When provided an invalid type, run_commands raises an error."""
++        with self.assertRaises(TypeError) as context_manager:
++            run_commands(commands="I'm Not Valid")
+         self.assertEqual(
+-            'Failure attaching Ubuntu Advantage:\nUnexpected error while'
+-            ' running command.\nCommand: -\nExit code: -\nReason: -\n'
+-            'Stdout: Invalid token SomeToken\nStderr: -',
++            "commands parameter was not a list or dict: I'm Not Valid",
+             str(context_manager.exception))
+ 
+-    @mock.patch('%s.util.subp' % MPATH)
+-    def test_configure_ua_attach_with_token(self, m_subp):
+-        """When token is provided, attach the machine to ua using the token."""
+-        configure_ua(token='SomeToken')
+-        m_subp.assert_called_once_with(['ua', 'attach', 'SomeToken'])
+-        self.assertEqual(
+-            'DEBUG: Attaching to Ubuntu Advantage. ua attach SomeToken\n',
+-            self.logs.getvalue())
+-
+-    @mock.patch('%s.util.subp' % MPATH)
+-    def test_configure_ua_attach_on_service_error(self, m_subp):
+-        """all services should be enabled and then any failures raised"""
+-
+-        def fake_subp(cmd, capture=None):
+-            fail_cmds = [['ua', 'enable', svc] for svc in ['esm', 'cc']]
+-            if cmd in fail_cmds and capture:
+-                svc = cmd[-1]
+-                raise util.ProcessExecutionError(
+-                    'Invalid {} credentials'.format(svc.upper()))
++    def test_run_command_logs_commands_and_exit_codes_to_stderr(self):
++        """All exit codes are logged to stderr."""
++        outfile = self.tmp_path('output.log', dir=self.tmp)
++
++        cmd1 = 'echo "HI" >> %s' % outfile
++        cmd2 = 'bogus command'
++        cmd3 = 'echo "MOM" >> %s' % outfile
++        commands = [cmd1, cmd2, cmd3]
++
++        mock_path = '%s.sys.stderr' % MPATH
++        with mock.patch(mock_path, new_callable=StringIO) as m_stderr:
++            with self.assertRaises(RuntimeError) as context_manager:
++                run_commands(commands=commands)
++
++        self.assertIsNotNone(
++            re.search(r'bogus: (command )?not found',
++                      str(context_manager.exception)),
++            msg='Expected bogus command not found')
++        expected_stderr_log = '\n'.join([
++            'Begin run command: {cmd}'.format(cmd=cmd1),
++            'End run command: exit(0)',
++            'Begin run command: {cmd}'.format(cmd=cmd2),
++            'ERROR: End run command: exit(127)',
++            'Begin run command: {cmd}'.format(cmd=cmd3),
++            'End run command: exit(0)\n'])
++        self.assertEqual(expected_stderr_log, m_stderr.getvalue())
++
++    def test_run_command_as_lists(self):
++        """When commands are specified as a list, run them in order."""
++        outfile = self.tmp_path('output.log', dir=self.tmp)
++
++        cmd1 = 'echo "HI" >> %s' % outfile
++        cmd2 = 'echo "MOM" >> %s' % outfile
++        commands = [cmd1, cmd2]
++        with mock.patch('%s.sys.stderr' % MPATH, new_callable=StringIO):
++            run_commands(commands=commands)
+ 
+-        m_subp.side_effect = fake_subp
+-
+-        with self.assertRaises(RuntimeError) as context_manager:
+-            configure_ua(token='SomeToken', enable=['esm', 'cc', 'fips'])
+-        self.assertEqual(
+-            m_subp.call_args_list,
+-            [mock.call(['ua', 'attach', 'SomeToken']),
+-             mock.call(['ua', 'enable', 'esm'], capture=True),
+-             mock.call(['ua', 'enable', 'cc'], capture=True),
+-             mock.call(['ua', 'enable', 'fips'], capture=True)])
+         self.assertIn(
+-            'WARNING: Failure enabling "esm":\nUnexpected error'
+-            ' while running command.\nCommand: -\nExit code: -\nReason: -\n'
+-            'Stdout: Invalid ESM credentials\nStderr: -\n',
++            'DEBUG: Running user-provided ubuntu-advantage commands',
+             self.logs.getvalue())
++        self.assertEqual('HI\nMOM\n', util.load_file(outfile))
+         self.assertIn(
+-            'WARNING: Failure enabling "cc":\nUnexpected error'
+-            ' while running command.\nCommand: -\nExit code: -\nReason: -\n'
+-            'Stdout: Invalid CC credentials\nStderr: -\n',
+-            self.logs.getvalue())
+-        self.assertEqual(
+-            'Failure enabling Ubuntu Advantage service(s): "esm", "cc"',
+-            str(context_manager.exception))
+-
+-    @mock.patch('%s.util.subp' % MPATH)
+-    def test_configure_ua_attach_with_empty_services(self, m_subp):
+-        """When services is an empty list, do not auto-enable attach."""
+-        configure_ua(token='SomeToken', enable=[])
+-        m_subp.assert_called_once_with(['ua', 'attach', 'SomeToken'])
+-        self.assertEqual(
+-            'DEBUG: Attaching to Ubuntu Advantage. ua attach SomeToken\n',
+-            self.logs.getvalue())
+-
+-    @mock.patch('%s.util.subp' % MPATH)
+-    def test_configure_ua_attach_with_specific_services(self, m_subp):
+-        """When services a list, only enable specific services."""
+-        configure_ua(token='SomeToken', enable=['fips'])
+-        self.assertEqual(
+-            m_subp.call_args_list,
+-            [mock.call(['ua', 'attach', 'SomeToken']),
+-             mock.call(['ua', 'enable', 'fips'], capture=True)])
+-        self.assertEqual(
+-            'DEBUG: Attaching to Ubuntu Advantage. ua attach SomeToken\n',
+-            self.logs.getvalue())
+-
+-    @mock.patch('%s.maybe_install_ua_tools' % MPATH, mock.MagicMock())
+-    @mock.patch('%s.util.subp' % MPATH)
+-    def test_configure_ua_attach_with_string_services(self, m_subp):
+-        """When services a string, treat as singleton list and warn"""
+-        configure_ua(token='SomeToken', enable='fips')
+-        self.assertEqual(
+-            m_subp.call_args_list,
+-            [mock.call(['ua', 'attach', 'SomeToken']),
+-             mock.call(['ua', 'enable', 'fips'], capture=True)])
+-        self.assertEqual(
+-            'WARNING: ubuntu_advantage: enable should be a list, not a'
+-            ' string; treating as a single enable\n'
+-            'DEBUG: Attaching to Ubuntu Advantage. ua attach SomeToken\n',
++            'WARNING: Non-ubuntu-advantage commands in ubuntu-advantage'
++            ' config:',
+             self.logs.getvalue())
+ 
+-    @mock.patch('%s.util.subp' % MPATH)
+-    def test_configure_ua_attach_with_weird_services(self, m_subp):
+-        """When services not string or list, warn but still attach"""
+-        configure_ua(token='SomeToken', enable={'deffo': 'wont work'})
+-        self.assertEqual(
+-            m_subp.call_args_list,
+-            [mock.call(['ua', 'attach', 'SomeToken'])])
+-        self.assertEqual(
+-            'WARNING: ubuntu_advantage: enable should be a list, not a'
+-            ' dict; skipping enabling services\n'
+-            'DEBUG: Attaching to Ubuntu Advantage. ua attach SomeToken\n',
+-            self.logs.getvalue())
++    def test_run_command_dict_sorted_as_command_script(self):
++        """When commands are a dict, sort them and run."""
++        outfile = self.tmp_path('output.log', dir=self.tmp)
++        cmd1 = 'echo "HI" >> %s' % outfile
++        cmd2 = 'echo "MOM" >> %s' % outfile
++        commands = {'02': cmd1, '01': cmd2}
++        with mock.patch('%s.sys.stderr' % MPATH, new_callable=StringIO):
++            run_commands(commands=commands)
++
++        expected_messages = [
++            'DEBUG: Running user-provided ubuntu-advantage commands']
++        for message in expected_messages:
++            self.assertIn(message, self.logs.getvalue())
++        self.assertEqual('MOM\nHI\n', util.load_file(outfile))
+ 
+ 
+ @skipUnlessJsonSchema()
+@@ -139,50 +112,90 @@ class TestSchema(CiTestCase, SchemaTestC
+     with_logs = True
+     schema = schema
+ 
+-    @mock.patch('%s.maybe_install_ua_tools' % MPATH)
+-    @mock.patch('%s.configure_ua' % MPATH)
+-    def test_schema_warns_on_ubuntu_advantage_not_dict(self, _cfg, _):
+-        """If ubuntu_advantage configuration is not a dict, emit a warning."""
+-        validate_cloudconfig_schema({'ubuntu_advantage': 'wrong type'}, schema)
++    def test_schema_warns_on_ubuntu_advantage_not_as_dict(self):
++        """If ubuntu-advantage configuration is not a dict, emit a warning."""
++        validate_cloudconfig_schema({'ubuntu-advantage': 'wrong type'}, schema)
+         self.assertEqual(
+-            "WARNING: Invalid config:\nubuntu_advantage: 'wrong type' is not"
++            "WARNING: Invalid config:\nubuntu-advantage: 'wrong type' is not"
+             " of type 'object'\n",
+             self.logs.getvalue())
+ 
+-    @mock.patch('%s.maybe_install_ua_tools' % MPATH)
+-    @mock.patch('%s.configure_ua' % MPATH)
+-    def test_schema_disallows_unknown_keys(self, _cfg, _):
+-        """Unknown keys in ubuntu_advantage configuration emit warnings."""
++    @mock.patch('%s.run_commands' % MPATH)
++    def test_schema_disallows_unknown_keys(self, _):
++        """Unknown keys in ubuntu-advantage configuration emit warnings."""
+         validate_cloudconfig_schema(
+-            {'ubuntu_advantage': {'token': 'winner', 'invalid-key': ''}},
++            {'ubuntu-advantage': {'commands': ['ls'], 'invalid-key': ''}},
+             schema)
+         self.assertIn(
+-            'WARNING: Invalid config:\nubuntu_advantage: Additional properties'
++            'WARNING: Invalid config:\nubuntu-advantage: Additional properties'
+             " are not allowed ('invalid-key' was unexpected)",
+             self.logs.getvalue())
+ 
+-    @mock.patch('%s.maybe_install_ua_tools' % MPATH)
+-    @mock.patch('%s.configure_ua' % MPATH)
+-    def test_warn_schema_requires_token(self, _cfg, _):
+-        """Warn if ubuntu_advantage configuration lacks token."""
++    def test_warn_schema_requires_commands(self):
++        """Warn when ubuntu-advantage configuration lacks commands."""
+         validate_cloudconfig_schema(
+-            {'ubuntu_advantage': {'enable': ['esm']}}, schema)
++            {'ubuntu-advantage': {}}, schema)
+         self.assertEqual(
+-            "WARNING: Invalid config:\nubuntu_advantage:"
+-            " 'token' is a required property\n", self.logs.getvalue())
++            "WARNING: Invalid config:\nubuntu-advantage: 'commands' is a"
++            " required property\n",
++            self.logs.getvalue())
+ 
+-    @mock.patch('%s.maybe_install_ua_tools' % MPATH)
+-    @mock.patch('%s.configure_ua' % MPATH)
+-    def test_warn_schema_services_is_not_list_or_dict(self, _cfg, _):
+-        """Warn when ubuntu_advantage:enable config is not a list."""
++    @mock.patch('%s.run_commands' % MPATH)
++    def test_warn_schema_commands_is_not_list_or_dict(self, _):
++        """Warn when ubuntu-advantage:commands config is not a list or dict."""
+         validate_cloudconfig_schema(
+-            {'ubuntu_advantage': {'enable': 'needslist'}}, schema)
++            {'ubuntu-advantage': {'commands': 'broken'}}, schema)
+         self.assertEqual(
+-            "WARNING: Invalid config:\nubuntu_advantage: 'token' is a"
+-            " required property\nubuntu_advantage.enable: 'needslist'"
+-            " is not of type 'array'\n",
++            "WARNING: Invalid config:\nubuntu-advantage.commands: 'broken' is"
++            " not of type 'object', 'array'\n",
+             self.logs.getvalue())
+ 
++    @mock.patch('%s.run_commands' % MPATH)
++    def test_warn_schema_when_commands_is_empty(self, _):
++        """Emit warnings when ubuntu-advantage:commands is empty."""
++        validate_cloudconfig_schema(
++            {'ubuntu-advantage': {'commands': []}}, schema)
++        validate_cloudconfig_schema(
++            {'ubuntu-advantage': {'commands': {}}}, schema)
++        self.assertEqual(
++            "WARNING: Invalid config:\nubuntu-advantage.commands: [] is too"
++            " short\nWARNING: Invalid config:\nubuntu-advantage.commands: {}"
++            " does not have enough properties\n",
++            self.logs.getvalue())
++
++    @mock.patch('%s.run_commands' % MPATH)
++    def test_schema_when_commands_are_list_or_dict(self, _):
++        """No warnings when ubuntu-advantage:commands are a list or dict."""
++        validate_cloudconfig_schema(
++            {'ubuntu-advantage': {'commands': ['valid']}}, schema)
++        validate_cloudconfig_schema(
++            {'ubuntu-advantage': {'commands': {'01': 'also valid'}}}, schema)
++        self.assertEqual('', self.logs.getvalue())
++
++    def test_duplicates_are_fine_array_array(self):
++        """Duplicated commands array/array entries are allowed."""
++        self.assertSchemaValid(
++            {'commands': [["echo", "bye"], ["echo" "bye"]]},
++            "command entries can be duplicate.")
++
++    def test_duplicates_are_fine_array_string(self):
++        """Duplicated commands array/string entries are allowed."""
++        self.assertSchemaValid(
++            {'commands': ["echo bye", "echo bye"]},
++            "command entries can be duplicate.")
++
++    def test_duplicates_are_fine_dict_array(self):
++        """Duplicated commands dict/array entries are allowed."""
++        self.assertSchemaValid(
++            {'commands': {'00': ["echo", "bye"], '01': ["echo", "bye"]}},
++            "command entries can be duplicate.")
++
++    def test_duplicates_are_fine_dict_string(self):
++        """Duplicated commands dict/string entries are allowed."""
++        self.assertSchemaValid(
++            {'commands': {'00': "echo bye", '01': "echo bye"}},
++            "command entries can be duplicate.")
++
+ 
+ class TestHandle(CiTestCase):
+ 
+@@ -192,89 +205,41 @@ class TestHandle(CiTestCase):
+         super(TestHandle, self).setUp()
+         self.tmp = self.tmp_dir()
+ 
++    @mock.patch('%s.run_commands' % MPATH)
+     @mock.patch('%s.validate_cloudconfig_schema' % MPATH)
+-    def test_handle_no_config(self, m_schema):
++    def test_handle_no_config(self, m_schema, m_run):
+         """When no ua-related configuration is provided, nothing happens."""
+         cfg = {}
+         handle('ua-test', cfg=cfg, cloud=None, log=self.logger, args=None)
+         self.assertIn(
+-            "DEBUG: Skipping module named ua-test, no 'ubuntu_advantage'"
+-            ' configuration found',
++            "DEBUG: Skipping module named ua-test, no 'ubuntu-advantage' key"
++            " in config",
+             self.logs.getvalue())
+         m_schema.assert_not_called()
++        m_run.assert_not_called()
+ 
+-    @mock.patch('%s.configure_ua' % MPATH)
+     @mock.patch('%s.maybe_install_ua_tools' % MPATH)
+-    def test_handle_tries_to_install_ubuntu_advantage_tools(
+-            self, m_install, m_cfg):
++    def test_handle_tries_to_install_ubuntu_advantage_tools(self, m_install):
+         """If ubuntu_advantage is provided, try installing ua-tools package."""
+-        cfg = {'ubuntu_advantage': {'token': 'valid'}}
++        cfg = {'ubuntu-advantage': {}}
+         mycloud = FakeCloud(None)
+         handle('nomatter', cfg=cfg, cloud=mycloud, log=self.logger, args=None)
+         m_install.assert_called_once_with(mycloud)
+ 
+-    @mock.patch('%s.configure_ua' % MPATH)
+     @mock.patch('%s.maybe_install_ua_tools' % MPATH)
+-    def test_handle_passes_credentials_and_services_to_configure_ua(
+-            self, m_install, m_configure_ua):
+-        """All ubuntu_advantage config keys are passed to configure_ua."""
+-        cfg = {'ubuntu_advantage': {'token': 'token', 'enable': ['esm']}}
+-        handle('nomatter', cfg=cfg, cloud=None, log=self.logger, args=None)
+-        m_configure_ua.assert_called_once_with(
+-            token='token', enable=['esm'])
+-
+-    @mock.patch('%s.maybe_install_ua_tools' % MPATH, mock.MagicMock())
+-    @mock.patch('%s.configure_ua' % MPATH)
+-    def test_handle_warns_on_deprecated_ubuntu_advantage_key_w_config(
+-            self, m_configure_ua):
+-        """Warning when ubuntu-advantage key is present with new config"""
+-        cfg = {'ubuntu-advantage': {'token': 'token', 'enable': ['esm']}}
+-        handle('nomatter', cfg=cfg, cloud=None, log=self.logger, args=None)
+-        self.assertEqual(
+-            'WARNING: Deprecated configuration key "ubuntu-advantage"'
+-            ' provided. Expected underscore delimited "ubuntu_advantage";'
+-            ' will attempt to continue.',
+-            self.logs.getvalue().splitlines()[0])
+-        m_configure_ua.assert_called_once_with(
+-            token='token', enable=['esm'])
+-
+-    def test_handle_error_on_deprecated_commands_key_dashed(self):
+-        """Error when commands is present in ubuntu-advantage key."""
+-        cfg = {'ubuntu-advantage': {'commands': 'nogo'}}
+-        with self.assertRaises(RuntimeError) as context_manager:
+-            handle('nomatter', cfg=cfg, cloud=None, log=self.logger, args=None)
+-        self.assertEqual(
+-            'Deprecated configuration "ubuntu-advantage: commands" provided.'
+-            ' Expected "token"',
+-            str(context_manager.exception))
+-
+-    def test_handle_error_on_deprecated_commands_key_underscored(self):
+-        """Error when commands is present in ubuntu_advantage key."""
+-        cfg = {'ubuntu_advantage': {'commands': 'nogo'}}
+-        with self.assertRaises(RuntimeError) as context_manager:
+-            handle('nomatter', cfg=cfg, cloud=None, log=self.logger, args=None)
+-        self.assertEqual(
+-            'Deprecated configuration "ubuntu-advantage: commands" provided.'
+-            ' Expected "token"',
+-            str(context_manager.exception))
++    def test_handle_runs_commands_provided(self, m_install):
++        """When commands are specified as a list, run them."""
++        outfile = self.tmp_path('output.log', dir=self.tmp)
+ 
+-    @mock.patch('%s.maybe_install_ua_tools' % MPATH, mock.MagicMock())
+-    @mock.patch('%s.configure_ua' % MPATH)
+-    def test_handle_prefers_new_style_config(
+-            self, m_configure_ua):
+-        """ubuntu_advantage should be preferred over ubuntu-advantage"""
+         cfg = {
+-            'ubuntu-advantage': {'token': 'nope', 'enable': ['wrong']},
+-            'ubuntu_advantage': {'token': 'token', 'enable': ['esm']},
+-        }
+-        handle('nomatter', cfg=cfg, cloud=None, log=self.logger, args=None)
+-        self.assertEqual(
+-            'WARNING: Deprecated configuration key "ubuntu-advantage"'
+-            ' provided. Expected underscore delimited "ubuntu_advantage";'
+-            ' will attempt to continue.',
+-            self.logs.getvalue().splitlines()[0])
+-        m_configure_ua.assert_called_once_with(
+-            token='token', enable=['esm'])
++            'ubuntu-advantage': {'commands': ['echo "HI" >> %s' % outfile,
++                                              'echo "MOM" >> %s' % outfile]}}
++        mock_path = '%s.sys.stderr' % MPATH
++        with self.allow_subp([CiTestCase.SUBP_SHELL_TRUE]):
++            with mock.patch(mock_path, new_callable=StringIO):
++                handle('nomatter', cfg=cfg, cloud=None, log=self.logger,
++                       args=None)
++        self.assertEqual('HI\nMOM\n', util.load_file(outfile))
+ 
+ 
+ class TestMaybeInstallUATools(CiTestCase):
+@@ -288,7 +253,7 @@ class TestMaybeInstallUATools(CiTestCase
+     @mock.patch('%s.util.which' % MPATH)
+     def test_maybe_install_ua_tools_noop_when_ua_tools_present(self, m_which):
+         """Do nothing if ubuntu-advantage-tools already exists."""
+-        m_which.return_value = '/usr/bin/ua'  # already installed
++        m_which.return_value = '/usr/bin/ubuntu-advantage'  # already installed
+         distro = mock.MagicMock()
+         distro.update_package_sources.side_effect = RuntimeError(
+             'Some apt error')
diff --git a/doc/rtd/topics/datasources/nocloud.rst b/doc/rtd/topics/datasources/nocloud.rst
index 08578e8..1c5cf96 100644
--- a/doc/rtd/topics/datasources/nocloud.rst
+++ b/doc/rtd/topics/datasources/nocloud.rst
@@ -9,7 +9,7 @@ network at all).
 
 You can provide meta-data and user-data to a local vm boot via files on a
 `vfat`_ or `iso9660`_ filesystem. The filesystem volume label must be
-``cidata``.
+``cidata`` or ``CIDATA``.
 
 Alternatively, you can provide meta-data via kernel command line or SMBIOS
 "serial number" option. The data must be passed in the form of a string:
diff --git a/packages/redhat/cloud-init.spec.in b/packages/redhat/cloud-init.spec.in
index 6b2022b..057a578 100644
--- a/packages/redhat/cloud-init.spec.in
+++ b/packages/redhat/cloud-init.spec.in
@@ -205,7 +205,9 @@ fi
 %dir                    %{_sysconfdir}/cloud/templates
 %config(noreplace)      %{_sysconfdir}/cloud/templates/*
 %config(noreplace) %{_sysconfdir}/rsyslog.d/21-cloudinit.conf
-%{_sysconfdir}/bash_completion.d/cloud-init
+
+# Bash completion script
+%{_datadir}/bash-completion/completions/cloud-init
 
 %{_libexecdir}/%{name}
 %dir %{_sharedstatedir}/cloud
diff --git a/packages/suse/cloud-init.spec.in b/packages/suse/cloud-init.spec.in
index 26894b3..004b875 100644
--- a/packages/suse/cloud-init.spec.in
+++ b/packages/suse/cloud-init.spec.in
@@ -120,7 +120,9 @@ version_pys=$(cd "%{buildroot}" && find . -name version.py -type f)
 %config(noreplace) %{_sysconfdir}/cloud/cloud.cfg.d/README
 %dir               %{_sysconfdir}/cloud/templates
 %config(noreplace) %{_sysconfdir}/cloud/templates/*
-%{_sysconfdir}/bash_completion.d/cloud-init
+
+# Bash completion script
+%{_datadir}/bash-completion/completions/cloud-init
 
 %{_sysconfdir}/dhcp/dhclient-exit-hooks.d/hook-dhclient
 %{_sysconfdir}/NetworkManager/dispatcher.d/hook-network-manager
diff --git a/setup.py b/setup.py
index 186e215..fcaf26f 100755
--- a/setup.py
+++ b/setup.py
@@ -245,13 +245,14 @@ if not in_virtualenv():
         INITSYS_ROOTS[k] = "/" + INITSYS_ROOTS[k]
 
 data_files = [
-    (ETC + '/bash_completion.d', ['bash_completion/cloud-init']),
     (ETC + '/cloud', [render_tmpl("config/cloud.cfg.tmpl")]),
     (ETC + '/cloud/cloud.cfg.d', glob('config/cloud.cfg.d/*')),
     (ETC + '/cloud/templates', glob('templates/*')),
     (USR_LIB_EXEC + '/cloud-init', ['tools/ds-identify',
                                     'tools/uncloud-init',
                                     'tools/write-ssh-key-fingerprints']),
+    (USR + '/share/bash-completion/completions',
+     ['bash_completion/cloud-init']),
     (USR + '/share/doc/cloud-init', [f for f in glob('doc/*') if is_f(f)]),
     (USR + '/share/doc/cloud-init/examples',
         [f for f in glob('doc/examples/*') if is_f(f)]),
diff --git a/tests/cloud_tests/releases.yaml b/tests/cloud_tests/releases.yaml
index ec5da72..924ad95 100644
--- a/tests/cloud_tests/releases.yaml
+++ b/tests/cloud_tests/releases.yaml
@@ -129,6 +129,22 @@ features:
 
 releases:
     # UBUNTU =================================================================
+    eoan:
+        # EOL: Jul 2020
+        default:
+            enabled: true
+            release: eoan
+            version: 19.10
+            os: ubuntu
+            feature_groups:
+                - base
+                - debian_base
+                - ubuntu_specific
+        lxd:
+            sstreams_server: https://cloud-images.ubuntu.com/daily
+            alias: eoan
+            setup_overrides: null
+            override_templates: false
     disco:
         # EOL: Jan 2020
         default:
diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py
index 53c56cd..427ab7e 100644
--- a/tests/unittests/test_datasource/test_azure.py
+++ b/tests/unittests/test_datasource/test_azure.py
@@ -163,7 +163,8 @@ class TestGetMetadataFromIMDS(HttprettyTestCase):
 
         m_readurl.assert_called_with(
             self.network_md_url, exception_cb=mock.ANY,
-            headers={'Metadata': 'true'}, retries=2, timeout=1)
+            headers={'Metadata': 'true'}, retries=2,
+            timeout=dsaz.IMDS_TIMEOUT_IN_SECONDS)
 
     @mock.patch('cloudinit.url_helper.time.sleep')
     @mock.patch(MOCKPATH + 'net.is_up')
@@ -1375,12 +1376,15 @@ class TestCanDevBeReformatted(CiTestCase):
         self._domock(p + "util.mount_cb", 'm_mount_cb')
         self._domock(p + "os.path.realpath", 'm_realpath')
         self._domock(p + "os.path.exists", 'm_exists')
+        self._domock(p + "util.SeLinuxGuard", 'm_selguard')
 
         self.m_exists.side_effect = lambda p: p in bypath
         self.m_realpath.side_effect = realpath
         self.m_has_ntfs_filesystem.side_effect = has_ntfs_fs
         self.m_mount_cb.side_effect = mount_cb
         self.m_partitions_on_device.side_effect = partitions_on_device
+        self.m_selguard.__enter__ = mock.Mock(return_value=False)
+        self.m_selguard.__exit__ = mock.Mock()
 
     def test_three_partitions_is_false(self):
         """A disk with 3 partitions can not be formatted."""
@@ -1788,7 +1792,8 @@ class TestAzureDataSourcePreprovisioning(CiTestCase):
                                     headers={'Metadata': 'true',
                                              'User-Agent':
                                              'Cloud-Init/%s' % vs()
-                                             }, method='GET', timeout=1,
+                                             }, method='GET',
+                                    timeout=dsaz.IMDS_TIMEOUT_IN_SECONDS,
                                     url=full_url)])
         self.assertEqual(m_dhcp.call_count, 2)
         m_net.assert_any_call(
@@ -1825,7 +1830,9 @@ class TestAzureDataSourcePreprovisioning(CiTestCase):
                                     headers={'Metadata': 'true',
                                              'User-Agent':
                                              'Cloud-Init/%s' % vs()},
-                                    method='GET', timeout=1, url=full_url)])
+                                    method='GET',
+                                    timeout=dsaz.IMDS_TIMEOUT_IN_SECONDS,
+                                    url=full_url)])
         self.assertEqual(m_dhcp.call_count, 2)
         m_net.assert_any_call(
             broadcast='192.168.2.255', interface='eth9', ip='192.168.2.9',
diff --git a/tests/unittests/test_datasource/test_azure_helper.py b/tests/unittests/test_datasource/test_azure_helper.py
index 0255616..bd006ab 100644
--- a/tests/unittests/test_datasource/test_azure_helper.py
+++ b/tests/unittests/test_datasource/test_azure_helper.py
@@ -67,12 +67,17 @@ class TestFindEndpoint(CiTestCase):
         self.networkd_leases.return_value = None
 
     def test_missing_file(self):
-        self.assertRaises(ValueError, wa_shim.find_endpoint)
+        """wa_shim find_endpoint uses default endpoint if leasefile not found
+        """
+        self.assertEqual(wa_shim.find_endpoint(), "168.63.129.16")
 
     def test_missing_special_azure_line(self):
+        """wa_shim find_endpoint uses default endpoint if leasefile is found
+        but does not contain DHCP Option 245 (whose value is the endpoint)
+        """
         self.load_file.return_value = ''
         self.dhcp_options.return_value = {'eth0': {'key': 'value'}}
-        self.assertRaises(ValueError, wa_shim.find_endpoint)
+        self.assertEqual(wa_shim.find_endpoint(), "168.63.129.16")
 
     @staticmethod
     def _build_lease_content(encoded_address):
diff --git a/tests/unittests/test_datasource/test_nocloud.py b/tests/unittests/test_datasource/test_nocloud.py
index 3429272..b785362 100644
--- a/tests/unittests/test_datasource/test_nocloud.py
+++ b/tests/unittests/test_datasource/test_nocloud.py
@@ -32,6 +32,36 @@ class TestNoCloudDataSource(CiTestCase):
         self.mocks.enter_context(
             mock.patch.object(util, 'read_dmi_data', return_value=None))
 
+    def _test_fs_config_is_read(self, fs_label, fs_label_to_search):
+        vfat_device = 'device-1'
+
+        def m_mount_cb(device, callback, mtype):
+            if (device == vfat_device):
+                return {'meta-data': yaml.dump({'instance-id': 'IID'})}
+            else:
+                return {}
+
+        def m_find_devs_with(query='', path=''):
+            if 'TYPE=vfat' == query:
+                return [vfat_device]
+            elif 'LABEL={}'.format(fs_label) == query:
+                return [vfat_device]
+            else:
+                return []
+
+        self.mocks.enter_context(
+            mock.patch.object(util, 'find_devs_with',
+                              side_effect=m_find_devs_with))
+        self.mocks.enter_context(
+            mock.patch.object(util, 'mount_cb',
+                              side_effect=m_mount_cb))
+        sys_cfg = {'datasource': {'NoCloud': {'fs_label': fs_label_to_search}}}
+        dsrc = dsNoCloud(sys_cfg=sys_cfg, distro=None, paths=self.paths)
+        ret = dsrc.get_data()
+
+        self.assertEqual(dsrc.metadata.get('instance-id'), 'IID')
+        self.assertTrue(ret)
+
     def test_nocloud_seed_dir_on_lxd(self, m_is_lxd):
         md = {'instance-id': 'IID', 'dsmode': 'local'}
         ud = b"USER_DATA_HERE"
@@ -90,6 +120,18 @@ class TestNoCloudDataSource(CiTestCase):
         ret = dsrc.get_data()
         self.assertFalse(ret)
 
+    def test_fs_config_lowercase_label(self, m_is_lxd):
+        self._test_fs_config_is_read('cidata', 'cidata')
+
+    def test_fs_config_uppercase_label(self, m_is_lxd):
+        self._test_fs_config_is_read('CIDATA', 'cidata')
+
+    def test_fs_config_lowercase_label_search_uppercase(self, m_is_lxd):
+        self._test_fs_config_is_read('cidata', 'CIDATA')
+
+    def test_fs_config_uppercase_label_search_uppercase(self, m_is_lxd):
+        self._test_fs_config_is_read('CIDATA', 'CIDATA')
+
     def test_no_datasource_expected(self, m_is_lxd):
         # no source should be found if no cmdline, config, and fs_label=None
         sys_cfg = {'datasource': {'NoCloud': {'fs_label': None}}}
diff --git a/tests/unittests/test_datasource/test_scaleway.py b/tests/unittests/test_datasource/test_scaleway.py
index 3bfd752..f96bf0a 100644
--- a/tests/unittests/test_datasource/test_scaleway.py
+++ b/tests/unittests/test_datasource/test_scaleway.py
@@ -7,7 +7,6 @@ import requests
 
 from cloudinit import helpers
 from cloudinit import settings
-from cloudinit.event import EventType
 from cloudinit.sources import DataSourceScaleway
 
 from cloudinit.tests.helpers import mock, HttprettyTestCase, CiTestCase
@@ -404,9 +403,3 @@ class TestDataSourceScaleway(HttprettyTestCase):
 
         netcfg = self.datasource.network_config
         self.assertEqual(netcfg, '0xdeadbeef')
-
-    def test_update_events_is_correct(self):
-        """ensure update_events contains correct data"""
-        self.assertEqual(
-            {'network': {EventType.BOOT_NEW_INSTANCE, EventType.BOOT}},
-            self.datasource.update_events)
diff --git a/tests/unittests/test_ds_identify.py b/tests/unittests/test_ds_identify.py
index d00c1b4..8c18aa1 100644
--- a/tests/unittests/test_ds_identify.py
+++ b/tests/unittests/test_ds_identify.py
@@ -520,6 +520,10 @@ class TestDsIdentify(DsIdentifyBase):
         """NoCloud is found with iso9660 filesystem on non-cdrom disk."""
         self._test_ds_found('NoCloud')
 
+    def test_nocloud_upper(self):
+        """NoCloud is found with uppercase filesystem label."""
+        self._test_ds_found('NoCloudUpper')
+
     def test_nocloud_seed(self):
         """Nocloud seed directory."""
         self._test_ds_found('NoCloud-seed')
@@ -713,6 +717,19 @@ VALID_CFG = {
             'dev/vdb': 'pretend iso content for cidata\n',
         }
     },
+    'NoCloudUpper': {
+        'ds': 'NoCloud',
+        'mocks': [
+            MOCK_VIRT_IS_KVM,
+            {'name': 'blkid', 'ret': 0,
+             'out': blkid_out(
+                 BLKID_UEFI_UBUNTU +
+                 [{'DEVNAME': 'vdb', 'TYPE': 'iso9660', 'LABEL': 'CIDATA'}])},
+        ],
+        'files': {
+            'dev/vdb': 'pretend iso content for cidata\n',
+        }
+    },
     'NoCloud-seed': {
         'ds': 'NoCloud',
         'files': {
diff --git a/tests/unittests/test_handler/test_handler_mounts.py b/tests/unittests/test_handler/test_handler_mounts.py
index 8fea6c2..0fb160b 100644
--- a/tests/unittests/test_handler/test_handler_mounts.py
+++ b/tests/unittests/test_handler/test_handler_mounts.py
@@ -154,7 +154,15 @@ class TestFstabHandling(test_helpers.FilesystemMockingTestCase):
                        return_value=True)
 
         self.add_patch('cloudinit.config.cc_mounts.util.subp',
-                       'mock_util_subp')
+                       'm_util_subp')
+
+        self.add_patch('cloudinit.config.cc_mounts.util.mounts',
+                       'mock_util_mounts',
+                       return_value={
+                           '/dev/sda1': {'fstype': 'ext4',
+                                         'mountpoint': '/',
+                                         'opts': 'rw,relatime,discard'
+                                         }})
 
         self.mock_cloud = mock.Mock()
         self.mock_log = mock.Mock()
@@ -230,4 +238,24 @@ class TestFstabHandling(test_helpers.FilesystemMockingTestCase):
             fstab_new_content = fd.read()
             self.assertEqual(fstab_expected_content, fstab_new_content)
 
+    def test_no_change_fstab_sets_needs_mount_all(self):
+        '''verify unchanged fstab entries are mounted if not call mount -a'''
+        fstab_original_content = (
+            'LABEL=cloudimg-rootfs / ext4 defaults 0 0\n'
+            'LABEL=UEFI /boot/efi vfat defaults 0 0\n'
+            '/dev/vdb /mnt auto defaults,noexec,comment=cloudconfig 0 2\n'
+        )
+        fstab_expected_content = fstab_original_content
+        cc = {'mounts': [
+                 ['/dev/vdb', '/mnt', 'auto', 'defaults,noexec']]}
+        with open(cc_mounts.FSTAB_PATH, 'w') as fd:
+            fd.write(fstab_original_content)
+        with open(cc_mounts.FSTAB_PATH, 'r') as fd:
+            fstab_new_content = fd.read()
+            self.assertEqual(fstab_expected_content, fstab_new_content)
+        cc_mounts.handle(None, cc, self.mock_cloud, self.mock_log, [])
+        self.m_util_subp.assert_has_calls([
+            mock.call(['mount', '-a']),
+            mock.call(['systemctl', 'daemon-reload'])])
+
 # vi: ts=4 expandtab
diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py
index fd03deb..e85e964 100644
--- a/tests/unittests/test_net.py
+++ b/tests/unittests/test_net.py
@@ -9,6 +9,7 @@ from cloudinit.net import (
 from cloudinit.sources.helpers import openstack
 from cloudinit import temp_utils
 from cloudinit import util
+from cloudinit import safeyaml as yaml
 
 from cloudinit.tests.helpers import (
     CiTestCase, FilesystemMockingTestCase, dir2dict, mock, populate_dir)
@@ -21,7 +22,7 @@ import json
 import os
 import re
 import textwrap
-import yaml
+from yaml.serializer import Serializer
 
 
 DHCP_CONTENT_1 = """
@@ -3269,9 +3270,12 @@ class TestNetplanPostcommands(CiTestCase):
         mock_netplan_generate.assert_called_with(run=True)
         mock_net_setup_link.assert_called_with(run=True)
 
+    @mock.patch('cloudinit.util.SeLinuxGuard')
     @mock.patch.object(netplan, "get_devicelist")
     @mock.patch('cloudinit.util.subp')
-    def test_netplan_postcmds(self, mock_subp, mock_devlist):
+    def test_netplan_postcmds(self, mock_subp, mock_devlist, mock_sel):
+        mock_sel.__enter__ = mock.Mock(return_value=False)
+        mock_sel.__exit__ = mock.Mock()
         mock_devlist.side_effect = [['lo']]
         tmp_dir = self.tmp_dir()
         ns = network_state.parse_net_config_data(self.mycfg,
@@ -3572,7 +3576,7 @@ class TestNetplanRoundTrip(CiTestCase):
         # now look for any alias, avoid rendering them entirely
         # generate the first anchor string using the template
         # as of this writing, looks like "&id001"
-        anchor = r'&' + yaml.serializer.Serializer.ANCHOR_TEMPLATE % 1
+        anchor = r'&' + Serializer.ANCHOR_TEMPLATE % 1
         found_alias = re.search(anchor, content, re.MULTILINE)
         if found_alias:
             msg = "Error at: %s\nContent:\n%s" % (found_alias, content)
@@ -3826,6 +3830,41 @@ class TestNetRenderers(CiTestCase):
         self.assertRaises(net.RendererNotFoundError, renderers.select,
                           priority=['sysconfig', 'eni'])
 
+    @mock.patch("cloudinit.net.renderers.netplan.available")
+    @mock.patch("cloudinit.net.renderers.sysconfig.available_sysconfig")
+    @mock.patch("cloudinit.net.renderers.sysconfig.available_nm")
+    @mock.patch("cloudinit.net.renderers.eni.available")
+    @mock.patch("cloudinit.net.renderers.sysconfig.util.get_linux_distro")
+    def test_sysconfig_selected_on_sysconfig_enabled_distros(self, m_distro,
+                                                             m_eni, m_sys_nm,
+                                                             m_sys_scfg,
+                                                             m_netplan):
+        """sysconfig only selected on specific distros (rhel/sles)."""
+
+        # Ubuntu with Network-Manager installed
+        m_eni.return_value = False       # no ifupdown (ifquery)
+        m_sys_scfg.return_value = False  # no sysconfig/ifup/ifdown
+        m_sys_nm.return_value = True     # network-manager is installed
+        m_netplan.return_value = True    # netplan is installed
+        m_distro.return_value = ('ubuntu', None, None)
+        self.assertEqual('netplan', renderers.select(priority=None)[0])
+
+        # Centos with Network-Manager installed
+        m_eni.return_value = False       # no ifupdown (ifquery)
+        m_sys_scfg.return_value = False  # no sysconfig/ifup/ifdown
+        m_sys_nm.return_value = True     # network-manager is installed
+        m_netplan.return_value = False    # netplan is not installed
+        m_distro.return_value = ('centos', None, None)
+        self.assertEqual('sysconfig', renderers.select(priority=None)[0])
+
+        # OpenSuse with Network-Manager installed
+        m_eni.return_value = False       # no ifupdown (ifquery)
+        m_sys_scfg.return_value = False  # no sysconfig/ifup/ifdown
+        m_sys_nm.return_value = True     # network-manager is installed
+        m_netplan.return_value = False    # netplan is not installed
+        m_distro.return_value = ('opensuse', None, None)
+        self.assertEqual('sysconfig', renderers.select(priority=None)[0])
+
 
 class TestGetInterfaces(CiTestCase):
     _data = {'bonds': ['bond1'],
diff --git a/tests/unittests/test_reporting_hyperv.py b/tests/unittests/test_reporting_hyperv.py
 old mode 100644
 new mode 100755
index 2e64c6c..d01ed5b
--- a/tests/unittests/test_reporting_hyperv.py
+++ b/tests/unittests/test_reporting_hyperv.py
@@ -1,10 +1,12 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
 from cloudinit.reporting import events
-from cloudinit.reporting import handlers
+from cloudinit.reporting.handlers import HyperVKvpReportingHandler
 
 import json
 import os
+import struct
+import time
 
 from cloudinit import util
 from cloudinit.tests.helpers import CiTestCase
@@ -13,7 +15,7 @@ from cloudinit.tests.helpers import CiTestCase
 class TestKvpEncoding(CiTestCase):
     def test_encode_decode(self):
         kvp = {'key': 'key1', 'value': 'value1'}
-        kvp_reporting = handlers.HyperVKvpReportingHandler()
+        kvp_reporting = HyperVKvpReportingHandler()
         data = kvp_reporting._encode_kvp_item(kvp['key'], kvp['value'])
         self.assertEqual(len(data), kvp_reporting.HV_KVP_RECORD_SIZE)
         decoded_kvp = kvp_reporting._decode_kvp_item(data)
@@ -26,57 +28,9 @@ class TextKvpReporter(CiTestCase):
         self.tmp_file_path = self.tmp_path('kvp_pool_file')
         util.ensure_file(self.tmp_file_path)
 
-    def test_event_type_can_be_filtered(self):
-        reporter = handlers.HyperVKvpReportingHandler(
-            kvp_file_path=self.tmp_file_path,
-            event_types=['foo', 'bar'])
-
-        reporter.publish_event(
-            events.ReportingEvent('foo', 'name', 'description'))
-        reporter.publish_event(
-            events.ReportingEvent('some_other', 'name', 'description3'))
-        reporter.q.join()
-
-        kvps = list(reporter._iterate_kvps(0))
-        self.assertEqual(1, len(kvps))
-
-        reporter.publish_event(
-            events.ReportingEvent('bar', 'name', 'description2'))
-        reporter.q.join()
-        kvps = list(reporter._iterate_kvps(0))
-        self.assertEqual(2, len(kvps))
-
-        self.assertIn('foo', kvps[0]['key'])
-        self.assertIn('bar', kvps[1]['key'])
-        self.assertNotIn('some_other', kvps[0]['key'])
-        self.assertNotIn('some_other', kvps[1]['key'])
-
-    def test_events_are_over_written(self):
-        reporter = handlers.HyperVKvpReportingHandler(
-            kvp_file_path=self.tmp_file_path)
-
-        self.assertEqual(0, len(list(reporter._iterate_kvps(0))))
-
-        reporter.publish_event(
-            events.ReportingEvent('foo', 'name1', 'description'))
-        reporter.publish_event(
-            events.ReportingEvent('foo', 'name2', 'description'))
-        reporter.q.join()
-        self.assertEqual(2, len(list(reporter._iterate_kvps(0))))
-
-        reporter2 = handlers.HyperVKvpReportingHandler(
-            kvp_file_path=self.tmp_file_path)
-        reporter2.incarnation_no = reporter.incarnation_no + 1
-        reporter2.publish_event(
-            events.ReportingEvent('foo', 'name3', 'description'))
-        reporter2.q.join()
-
-        self.assertEqual(2, len(list(reporter2._iterate_kvps(0))))
-
     def test_events_with_higher_incarnation_not_over_written(self):
-        reporter = handlers.HyperVKvpReportingHandler(
+        reporter = HyperVKvpReportingHandler(
             kvp_file_path=self.tmp_file_path)
-
         self.assertEqual(0, len(list(reporter._iterate_kvps(0))))
 
         reporter.publish_event(
@@ -86,7 +40,7 @@ class TextKvpReporter(CiTestCase):
         reporter.q.join()
         self.assertEqual(2, len(list(reporter._iterate_kvps(0))))
 
-        reporter3 = handlers.HyperVKvpReportingHandler(
+        reporter3 = HyperVKvpReportingHandler(
             kvp_file_path=self.tmp_file_path)
         reporter3.incarnation_no = reporter.incarnation_no - 1
         reporter3.publish_event(
@@ -95,7 +49,7 @@ class TextKvpReporter(CiTestCase):
         self.assertEqual(3, len(list(reporter3._iterate_kvps(0))))
 
     def test_finish_event_result_is_logged(self):
-        reporter = handlers.HyperVKvpReportingHandler(
+        reporter = HyperVKvpReportingHandler(
             kvp_file_path=self.tmp_file_path)
         reporter.publish_event(
             events.FinishReportingEvent('name2', 'description1',
@@ -105,7 +59,7 @@ class TextKvpReporter(CiTestCase):
 
     def test_file_operation_issue(self):
         os.remove(self.tmp_file_path)
-        reporter = handlers.HyperVKvpReportingHandler(
+        reporter = HyperVKvpReportingHandler(
             kvp_file_path=self.tmp_file_path)
         reporter.publish_event(
             events.FinishReportingEvent('name2', 'description1',
@@ -113,7 +67,7 @@ class TextKvpReporter(CiTestCase):
         reporter.q.join()
 
     def test_event_very_long(self):
-        reporter = handlers.HyperVKvpReportingHandler(
+        reporter = HyperVKvpReportingHandler(
             kvp_file_path=self.tmp_file_path)
         description = 'ab' * reporter.HV_KVP_EXCHANGE_MAX_VALUE_SIZE
         long_event = events.FinishReportingEvent(
@@ -132,3 +86,43 @@ class TextKvpReporter(CiTestCase):
             self.assertEqual(msg_slice['msg_i'], i)
             full_description += msg_slice['msg']
         self.assertEqual(description, full_description)
+
+    def test_not_truncate_kvp_file_modified_after_boot(self):
+        with open(self.tmp_file_path, "wb+") as f:
+            kvp = {'key': 'key1', 'value': 'value1'}
+            data = (struct.pack("%ds%ds" % (
+                    HyperVKvpReportingHandler.HV_KVP_EXCHANGE_MAX_KEY_SIZE,
+                    HyperVKvpReportingHandler.HV_KVP_EXCHANGE_MAX_VALUE_SIZE),
+                    kvp['key'].encode('utf-8'), kvp['value'].encode('utf-8')))
+            f.write(data)
+        cur_time = time.time()
+        os.utime(self.tmp_file_path, (cur_time, cur_time))
+
+        # reset this because the unit test framework
+        # has already polluted the class variable
+        HyperVKvpReportingHandler._already_truncated_pool_file = False
+
+        reporter = HyperVKvpReportingHandler(kvp_file_path=self.tmp_file_path)
+        kvps = list(reporter._iterate_kvps(0))
+        self.assertEqual(1, len(kvps))
+
+    def test_truncate_stale_kvp_file(self):
+        with open(self.tmp_file_path, "wb+") as f:
+            kvp = {'key': 'key1', 'value': 'value1'}
+            data = (struct.pack("%ds%ds" % (
+                HyperVKvpReportingHandler.HV_KVP_EXCHANGE_MAX_KEY_SIZE,
+                HyperVKvpReportingHandler.HV_KVP_EXCHANGE_MAX_VALUE_SIZE),
+                kvp['key'].encode('utf-8'), kvp['value'].encode('utf-8')))
+            f.write(data)
+
+        # set the time ways back to make it look like
+        # we had an old kvp file
+        os.utime(self.tmp_file_path, (1000000, 1000000))
+
+        # reset this because the unit test framework
+        # has already polluted the class variable
+        HyperVKvpReportingHandler._already_truncated_pool_file = False
+
+        reporter = HyperVKvpReportingHandler(kvp_file_path=self.tmp_file_path)
+        kvps = list(reporter._iterate_kvps(0))
+        self.assertEqual(0, len(kvps))
diff --git a/tools/build-on-freebsd b/tools/build-on-freebsd
index d23fde2..dc3b974 100755
--- a/tools/build-on-freebsd
+++ b/tools/build-on-freebsd
@@ -9,6 +9,7 @@ fail() { echo "FAILED:" "$@" 1>&2; exit 1; }
 depschecked=/tmp/c-i.dependencieschecked
 pkgs="
    bash
+   chpasswd
    dmidecode
    e2fsprogs
    py27-Jinja2
@@ -17,6 +18,7 @@ pkgs="
    py27-configobj
    py27-jsonpatch
    py27-jsonpointer
+   py27-jsonschema
    py27-oauthlib
    py27-requests
    py27-serial
@@ -28,12 +30,9 @@ pkgs="
 [ -f "$depschecked" ] || pkg install ${pkgs} || fail "install packages"
 touch $depschecked
 
-# Required but unavailable port/pkg: py27-jsonpatch py27-jsonpointer
-# Luckily, the install step will take care of this by installing it from pypi...
-
 # Build the code and install in /usr/local/:
-python setup.py build
-python setup.py install -O1 --skip-build --prefix /usr/local/ --init-system sysvinit_freebsd
+python2.7 setup.py build
+python2.7 setup.py install -O1 --skip-build --prefix /usr/local/ --init-system sysvinit_freebsd
 
 # Enable cloud-init in /etc/rc.conf:
 sed -i.bak -e "/cloudinit_enable=.*/d" /etc/rc.conf
diff --git a/tools/ds-identify b/tools/ds-identify
index b78b273..6518901 100755
--- a/tools/ds-identify
+++ b/tools/ds-identify
@@ -620,7 +620,7 @@ dscheck_MAAS() {
 }
 
 dscheck_NoCloud() {
-    local fslabel="cidata" d=""
+    local fslabel="cidata CIDATA" d=""
     case " ${DI_KERNEL_CMDLINE} " in
         *\ ds=nocloud*) return ${DS_FOUND};;
     esac
@@ -632,9 +632,10 @@ dscheck_NoCloud() {
         check_seed_dir "$d" meta-data user-data && return ${DS_FOUND}
         check_writable_seed_dir "$d" meta-data user-data && return ${DS_FOUND}
     done
-    if has_fs_with_label "${fslabel}"; then
+    if has_fs_with_label $fslabel; then
         return ${DS_FOUND}
     fi
+
     return ${DS_NOT_FOUND}
 }
 
@@ -762,7 +763,7 @@ is_cdrom_ovf() {
 
     # explicitly skip known labels of other types. rd_rdfe is azure.
     case "$label" in
-        config-2|CONFIG-2|rd_rdfe_stable*|cidata) return 1;;
+        config-2|CONFIG-2|rd_rdfe_stable*|cidata|CIDATA) return 1;;
     esac
 
     local idstr="http://schemas.dmtf.org/ovf/environment/1"
diff --git a/tools/read-version b/tools/read-version
index e69c2ce..6dca659 100755
--- a/tools/read-version
+++ b/tools/read-version
@@ -71,9 +71,12 @@ if is_gitdir(_tdir) and which("git"):
         flags = ['--tags']
     cmd = ['git', 'describe', '--abbrev=8', '--match=[0-9]*'] + flags
 
-    version = tiny_p(cmd).strip()
+    try:
+        version = tiny_p(cmd).strip()
+    except RuntimeError:
+        version = None
 
-    if not version.startswith(src_version):
+    if version is None or not version.startswith(src_version):
         sys.stderr.write("git describe version (%s) differs from "
                          "cloudinit.version (%s)\n" % (version, src_version))
         sys.stderr.write(