Merge ~chad.smith/cloud-init:datasource-ibmcloud into cloud-init:master

[Chad Smith]

IBMCloud: Initial IBM Cloud datasource.

This adds a specific IBM Cloud datasource.
IBM Cloud is identified by:
 a.) running on xen
 b.) one of a LABEL=METADATA disk or a LABEL=config-2 disk with
     UUID=9796-932E

The datasource contains its own config-drive reader that reads
only the currently supported portion of config-drive needed for
ibm cloud.

During the provisioning boot, cloud-init is disabled.

See the docstring in DataSourceIBMCloud.py for more more information.

see commit message. Shepherd smoser's branch into tip

[Chad Smith (chad.smith)]

Addressed minor review comments to land this branch.

review: Approve

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:4641cef07c60683296f596d395d3a4d2430b177b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/921/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/921/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:cca002cca7855f3123c3e2d0d57a43c213305ec7
https://jenkins.ubuntu.com/server/job/cloud-init-ci/922/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/922/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=e0f644b7

diff --git a/cloudinit/sources/DataSourceConfigDrive.py b/cloudinit/sources/DataSourceConfigDrive.py
index b8db626..c7b5fe5 100644
--- a/cloudinit/sources/DataSourceConfigDrive.py
+++ b/cloudinit/sources/DataSourceConfigDrive.py
@@ -14,6 +14,7 @@ from cloudinit import util
 
 from cloudinit.net import eni
 
+from cloudinit.sources.DataSourceIBMCloud import get_ibm_platform
 from cloudinit.sources.helpers import openstack
 
 LOG = logging.getLogger(__name__)
@@ -255,6 +256,15 @@ def find_candidate_devs(probe_optical=True):
     # an unpartitioned block device (ex sda, not sda1)
     devices = [d for d in candidates
                if d in by_label or not util.is_partition(d)]
+
+    if devices:
+        # IBMCloud uses config-2 label, but limited to a single UUID.
+        ibm_platform, ibm_path = get_ibm_platform()
+        if ibm_path in devices:
+            devices.remove(ibm_path)
+            LOG.debug("IBMCloud device '%s' (%s) removed from candidate list",
+                      ibm_path, ibm_platform)
+
     return devices
 
 
diff --git a/cloudinit/sources/DataSourceIBMCloud.py b/cloudinit/sources/DataSourceIBMCloud.py
 new file mode 100644
index 0000000..02b3d56
--- /dev/null
+++ b/cloudinit/sources/DataSourceIBMCloud.py
@@ -0,0 +1,325 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+"""Datasource for IBMCloud.
+
+IBMCloud is also know as SoftLayer or BlueMix.
+IBMCloud hypervisor is xen (2018-03-10).
+
+There are 2 different api exposed launch methods.
+ * template: This is the legacy method of launching instances.
+   When booting from an image template, the system boots first into
+   a "provisioning" mode.  There, host <-> guest mechanisms are utilized
+   to execute code in the guest and provision it.
+
+   Cloud-init will disable itself when it detects that it is in the
+   provisioning mode.  It detects this by the presence of
+   a file '/root/provisioningConfiguration.cfg'.
+
+   When provided with user-data, the "first boot" will contain a
+   ConfigDrive-like disk labeled with 'METADATA'.  If there is no user-data
+   provided, then there is no data-source.
+
+   Cloud-init never does any network configuration in this mode.
+
+ * os_code: Essentially "launch by OS Code" (Operating System Code).
+   This is a more modern approach.  There is no specific "provisioning" boot.
+   Instead, cloud-init does all the customization.  With or without
+   user-data provided, an OpenStack ConfigDrive like disk is attached.
+
+   Only disks with label 'config-2' and UUID '9796-932E' are considered.
+   This is to avoid this datasource claiming ConfigDrive.  This does
+   mean that 1 in 8^16 (~4 billion) Xen ConfigDrive systems will be
+   incorrectly identified as IBMCloud.
+
+TODO:
+ * is uuid (/sys/hypervisor/uuid) stable for life of an instance?
+   it seems it is not the same as data's uuid in the os_code case
+   but is in the template case.
+
+"""
+import base64
+import json
+import os
+
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit.sources.helpers import openstack
+from cloudinit import util
+
+LOG = logging.getLogger(__name__)
+
+IBM_CONFIG_UUID = "9796-932E"
+
+
+class Platforms(object):
+    TEMPLATE_LIVE_METADATA = "Template/Live/Metadata"
+    TEMPLATE_LIVE_NODATA = "UNABLE TO BE IDENTIFIED."
+    TEMPLATE_PROVISIONING_METADATA = "Template/Provisioning/Metadata"
+    TEMPLATE_PROVISIONING_NODATA = "Template/Provisioning/No-Metadata"
+    OS_CODE = "OS-Code/Live"
+
+
+PROVISIONING = (
+    Platforms.TEMPLATE_PROVISIONING_METADATA,
+    Platforms.TEMPLATE_PROVISIONING_NODATA)
+
+
+class DataSourceIBMCloud(sources.DataSource):
+
+    dsname = 'IBMCloud'
+    system_uuid = None
+
+    def __init__(self, sys_cfg, distro, paths):
+        super(DataSourceIBMCloud, self).__init__(sys_cfg, distro, paths)
+        self.source = None
+        self._network_config = None
+        self.network_json = None
+        self.platform = None
+
+    def __str__(self):
+        root = super(DataSourceIBMCloud, self).__str__()
+        mstr = "%s [%s %s]" % (root, self.platform, self.source)
+        return mstr
+
+    def _get_data(self):
+        results = read_md()
+        if results is None:
+            return False
+
+        self.source = results['source']
+        self.platform = results['platform']
+        self.metadata = results['metadata']
+        self.userdata_raw = results.get('userdata')
+        self.network_json = results.get('networkdata')
+        vd = results.get('vendordata')
+        self.vendordata_pure = vd
+        self.system_uuid = results['system-uuid']
+        try:
+            self.vendordata_raw = sources.convert_vendordata(vd)
+        except ValueError as e:
+            LOG.warning("Invalid content in vendor-data: %s", e)
+            self.vendordata_raw = None
+
+        return True
+
+    def check_instance_id(self, sys_cfg):
+        """quickly (local check only) if self.instance_id is still valid
+
+        in Template mode, the system uuid (/sys/hypervisor/uuid) is the
+        same as found in the METADATA disk.  But that is not true in OS_CODE
+        mode.  So we read the system_uuid and keep that for later compare."""
+        if self.system_uuid is None:
+            return False
+        return self.system_uuid == _read_system_uuid()
+
+    @property
+    def network_config(self):
+        if self.platform != Platforms.OS_CODE:
+            # If deployed from template, an agent in the provisioning
+            # environment handles networking configuration. Not cloud-init.
+            return {'config': 'disabled', 'version': 1}
+        if self._network_config is None:
+            if self.network_json is not None:
+                LOG.debug("network config provided via network_json")
+                self._network_config = openstack.convert_net_json(
+                    self.network_json, known_macs=None)
+            else:
+                LOG.debug("no network configuration available.")
+        return self._network_config
+
+
+def _read_system_uuid():
+    uuid_path = "/sys/hypervisor/uuid"
+    if not os.path.isfile(uuid_path):
+        return None
+    return util.load_file(uuid_path).strip().lower()
+
+
+def _is_xen():
+    return os.path.exists("/proc/xen")
+
+
+def _is_ibm_provisioning():
+    return os.path.exists("/root/provisioningConfiguration.cfg")
+
+
+def get_ibm_platform():
+    """Return a tuple (Platform, path)
+
+    If this is Not IBM cloud, then the return value is (None, None).
+    An instance in provisioning mode is considered running on IBM cloud."""
+    label_mdata = "METADATA"
+    label_cfg2 = "CONFIG-2"
+    not_found = (None, None)
+
+    if not _is_xen():
+        return not_found
+
+    # fslabels contains only the first entry with a given label.
+    fslabels = {}
+    try:
+        devs = util.blkid()
+    except util.ProcessExecutionError as e:
+        LOG.warning("Failed to run blkid: %s", e)
+        return (None, None)
+
+    for dev in sorted(devs.keys()):
+        data = devs[dev]
+        label = data.get("LABEL", "").upper()
+        uuid = data.get("UUID", "").upper()
+        if label not in (label_mdata, label_cfg2):
+            continue
+        if label in fslabels:
+            LOG.warning("Duplicate fslabel '%s'. existing=%s current=%s",
+                        label, fslabels[label], data)
+            continue
+        if label == label_cfg2 and uuid != IBM_CONFIG_UUID:
+            LOG.debug("Skipping %s with LABEL=%s due to uuid != %s: %s",
+                      dev, label, uuid, data)
+            continue
+        fslabels[label] = data
+
+    metadata_path = fslabels.get(label_mdata, {}).get('DEVNAME')
+    cfg2_path = fslabels.get(label_cfg2, {}).get('DEVNAME')
+
+    if cfg2_path:
+        return (Platforms.OS_CODE, cfg2_path)
+    elif metadata_path:
+        if _is_ibm_provisioning():
+            return (Platforms.TEMPLATE_PROVISIONING_METADATA, metadata_path)
+        else:
+            return (Platforms.TEMPLATE_LIVE_METADATA, metadata_path)
+    elif _is_ibm_provisioning():
+            return (Platforms.TEMPLATE_PROVISIONING_NODATA, None)
+    return not_found
+
+
+def read_md():
+    """Read data from IBM Cloud.
+
+    @return: None if not running on IBM Cloud.
+             dictionary with guaranteed fields: metadata, version
+             and optional fields: userdata, vendordata, networkdata.
+             Also includes the system uuid from /sys/hypervisor/uuid."""
+    platform, path = get_ibm_platform()
+    if platform is None:
+        LOG.debug("This is not an IBMCloud platform.")
+        return None
+    elif platform in PROVISIONING:
+        LOG.debug("Cloud-init is disabled during provisioning: %s.",
+                  platform)
+        return None
+
+    ret = {'platform': platform, 'source': path,
+           'system-uuid': _read_system_uuid()}
+
+    try:
+        if os.path.isdir(path):
+            results = metadata_from_dir(path)
+        else:
+            results = util.mount_cb(path, metadata_from_dir)
+    except BrokenMetadata as e:
+        raise RuntimeError(
+            "Failed reading IBM config disk (platform=%s path=%s): %s" %
+            (platform, path, e))
+
+    ret.update(results)
+    return ret
+
+
+class BrokenMetadata(IOError):
+    pass
+
+
+def metadata_from_dir(source_dir):
+    """Walk source_dir extracting standardized metadata.
+
+    Certain metadata keys are renamed to present a standardized set of metadata
+    keys.
+
+    This function has a lot in common with ConfigDriveReader.read_v2 but
+    there are a number of inconsistencies, such key renames and as only
+    presenting a 'latest' version which make it an unlikely candidate to share
+    code.
+
+    @return: Dict containing translated metadata, userdata, vendordata,
+        networkdata as present.
+    """
+
+    def opath(fname):
+        return os.path.join("openstack", "latest", fname)
+
+    def load_json_bytes(blob):
+        return json.loads(blob.decode('utf-8'))
+
+    files = [
+        # tuples of (results_name, path, translator)
+        ('metadata_raw', opath('meta_data.json'), load_json_bytes),
+        ('userdata', opath('user_data'), None),
+        ('vendordata', opath('vendor_data.json'), load_json_bytes),
+        ('networkdata', opath('network_data.json'), load_json_bytes),
+    ]
+
+    results = {}
+    for (name, path, transl) in files:
+        fpath = os.path.join(source_dir, path)
+        raw = None
+        try:
+            raw = util.load_file(fpath, decode=False)
+        except IOError as e:
+            LOG.debug("Failed reading path '%s': %s", fpath, e)
+
+        if raw is None or transl is None:
+            data = raw
+        else:
+            try:
+                data = transl(raw)
+            except Exception as e:
+                raise BrokenMetadata("Failed decoding %s: %s" % (path, e))
+
+        results[name] = data
+
+    if results.get('metadata_raw') is None:
+        raise BrokenMetadata(
+            "%s missing required file 'meta_data.json'" % source_dir)
+
+    results['metadata'] = {}
+
+    md_raw = results['metadata_raw']
+    md = results['metadata']
+    if 'random_seed' in md_raw:
+        try:
+            md['random_seed'] = base64.b64decode(md_raw['random_seed'])
+        except (ValueError, TypeError) as e:
+            raise BrokenMetadata(
+                "Badly formatted metadata random_seed entry: %s" % e)
+
+    renames = (
+        ('public_keys', 'public-keys'), ('hostname', 'local-hostname'),
+        ('uuid', 'instance-id'))
+    for mdname, newname in renames:
+        if mdname in md_raw:
+            md[newname] = md_raw[mdname]
+
+    return results
+
+
+# Used to match classes to dependencies
+datasources = [
+    (DataSourceIBMCloud, (sources.DEP_FILESYSTEM,)),
+]
+
+
+# Return a list of data sources that match this set of dependencies
+def get_datasource_list(depends):
+    return sources.list_from_depends(depends, datasources)
+
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description='Query IBM Cloud Metadata')
+    args = parser.parse_args()
+    data = read_md()
+    print(util.json_dumps(data))
+
+# vi: ts=4 expandtab
diff --git a/cloudinit/tests/test_util.py b/cloudinit/tests/test_util.py
index d30643d..3f37dbb 100644
--- a/cloudinit/tests/test_util.py
+++ b/cloudinit/tests/test_util.py
@@ -3,6 +3,7 @@
 """Tests for cloudinit.util"""
 
 import logging
+from textwrap import dedent
 
 import cloudinit.util as util
 
@@ -140,4 +141,75 @@ class TestGetHostnameFqdn(CiTestCase):
             [{'fqdn': True, 'metadata_only': True},
              {'metadata_only': True}], mycloud.calls)
 
+
+class TestBlkid(CiTestCase):
+    ids = {
+        "id01": "1111-1111",
+        "id02": "22222222-2222",
+        "id03": "33333333-3333",
+        "id04": "44444444-4444",
+        "id05": "55555555-5555-5555-5555-555555555555",
+        "id06": "66666666-6666-6666-6666-666666666666",
+        "id07": "52894610484658920398",
+        "id08": "86753098675309867530",
+        "id09": "99999999-9999-9999-9999-999999999999",
+    }
+
+    blkid_out = dedent("""\
+        /dev/loop0: TYPE="squashfs"
+        /dev/loop1: TYPE="squashfs"
+        /dev/loop2: TYPE="squashfs"
+        /dev/loop3: TYPE="squashfs"
+        /dev/sda1: UUID="{id01}" TYPE="vfat" PARTUUID="{id02}"
+        /dev/sda2: UUID="{id03}" TYPE="ext4" PARTUUID="{id04}"
+        /dev/sda3: UUID="{id05}" TYPE="ext4" PARTUUID="{id06}"
+        /dev/sda4: LABEL="default" UUID="{id07}" UUID_SUB="{id08}" """
+                       """TYPE="zfs_member" PARTUUID="{id09}"
+        /dev/loop4: TYPE="squashfs"
+      """)
+
+    maxDiff = None
+
+    def _get_expected(self):
+        return ({
+            "/dev/loop0": {"DEVNAME": "/dev/loop0", "TYPE": "squashfs"},
+            "/dev/loop1": {"DEVNAME": "/dev/loop1", "TYPE": "squashfs"},
+            "/dev/loop2": {"DEVNAME": "/dev/loop2", "TYPE": "squashfs"},
+            "/dev/loop3": {"DEVNAME": "/dev/loop3", "TYPE": "squashfs"},
+            "/dev/loop4": {"DEVNAME": "/dev/loop4", "TYPE": "squashfs"},
+            "/dev/sda1": {"DEVNAME": "/dev/sda1", "TYPE": "vfat",
+                          "UUID": self.ids["id01"],
+                          "PARTUUID": self.ids["id02"]},
+            "/dev/sda2": {"DEVNAME": "/dev/sda2", "TYPE": "ext4",
+                          "UUID": self.ids["id03"],
+                          "PARTUUID": self.ids["id04"]},
+            "/dev/sda3": {"DEVNAME": "/dev/sda3", "TYPE": "ext4",
+                          "UUID": self.ids["id05"],
+                          "PARTUUID": self.ids["id06"]},
+            "/dev/sda4": {"DEVNAME": "/dev/sda4", "TYPE": "zfs_member",
+                          "LABEL": "default",
+                          "UUID": self.ids["id07"],
+                          "UUID_SUB": self.ids["id08"],
+                          "PARTUUID": self.ids["id09"]},
+        })
+
+    @mock.patch("cloudinit.util.subp")
+    def test_functional_blkid(self, m_subp):
+        m_subp.return_value = (
+            self.blkid_out.format(**self.ids), "")
+        self.assertEqual(self._get_expected(), util.blkid())
+        m_subp.assert_called_with(["blkid", "-o", "full"], capture=True,
+                                  decode="replace")
+
+    @mock.patch("cloudinit.util.subp")
+    def test_blkid_no_cache_uses_no_cache(self, m_subp):
+        """blkid should turn off cache if disable_cache is true."""
+        m_subp.return_value = (
+            self.blkid_out.format(**self.ids), "")
+        self.assertEqual(self._get_expected(),
+                         util.blkid(disable_cache=True))
+        m_subp.assert_called_with(["blkid", "-o", "full", "-c", "/dev/null"],
+                                  capture=True, decode="replace")
+
+
 # vi: ts=4 expandtab
diff --git a/cloudinit/util.py b/cloudinit/util.py
index cae8b19..fb4ee5f 100644
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -1237,6 +1237,37 @@ def find_devs_with(criteria=None, oformat='device',
     return entries
 
 
+def blkid(devs=None, disable_cache=False):
+    """Get all device tags details from blkid.
+
+    @param devs: Optional list of device paths you wish to query.
+    @param disable_cache: Bool, set True to start with clean cache.
+
+    @return: Dict of key value pairs of info for the device.
+    """
+    if devs is None:
+        devs = []
+    else:
+        devs = list(devs)
+
+    cmd = ['blkid', '-o', 'full']
+    if disable_cache:
+        cmd.extend(['-c', '/dev/null'])
+    cmd.extend(devs)
+
+    # we have to decode with 'replace' as shelx.split (called by
+    # load_shell_content) can't take bytes.  So this is potentially
+    # lossy of non-utf-8 chars in blkid output.
+    out, _ = subp(cmd, capture=True, decode="replace")
+    ret = {}
+    for line in out.splitlines():
+        dev, _, data = line.partition(":")
+        ret[dev] = load_shell_content(data)
+        ret[dev]["DEVNAME"] = dev
+
+    return ret
+
+
 def peek_file(fname, max_bytes):
     LOG.debug("Peeking at %s (max_bytes=%s)", fname, max_bytes)
     with open(fname, 'rb') as ifh:
diff --git a/tests/unittests/test_datasource/test_ibmcloud.py b/tests/unittests/test_datasource/test_ibmcloud.py
 new file mode 100644
index 0000000..621cfe4
--- /dev/null
+++ b/tests/unittests/test_datasource/test_ibmcloud.py
@@ -0,0 +1,262 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+from cloudinit.sources import DataSourceIBMCloud as ibm
+from cloudinit.tests import helpers as test_helpers
+
+import base64
+import copy
+import json
+import mock
+from textwrap import dedent
+
+D_PATH = "cloudinit.sources.DataSourceIBMCloud."
+
+
+class TestIBMCloud(test_helpers.CiTestCase):
+    """Test the datasource."""
+    def setUp(self):
+        super(TestIBMCloud, self).setUp()
+        pass
+
+
[e-mail](D_PATH + "_is_xen", return_value=True)
[e-mail](D_PATH + "_is_ibm_provisioning")
[e-mail](D_PATH + "util.blkid")
+class TestGetIBMPlatform(test_helpers.CiTestCase):
+    """Test the get_ibm_platform helper."""
+
+    blkid_base = {
+        "/dev/xvda1": {
+            "DEVNAME": "/dev/xvda1", "LABEL": "cloudimg-bootfs",
+            "TYPE": "ext3"},
+        "/dev/xvda2": {
+            "DEVNAME": "/dev/xvda2", "LABEL": "cloudimg-rootfs",
+            "TYPE": "ext4"},
+    }
+
+    blkid_metadata_disk = {
+        "/dev/xvdh1": {
+            "DEVNAME": "/dev/xvdh1", "LABEL": "METADATA", "TYPE": "vfat",
+            "SEC_TYPE": "msdos", "UUID": "681B-8C5D",
+            "PARTUUID": "3d631e09-01"},
+    }
+
+    blkid_oscode_disk = {
+        "/dev/xvdh": {
+            "DEVNAME": "/dev/xvdh", "LABEL": "config-2", "TYPE": "vfat",
+            "SEC_TYPE": "msdos", "UUID": ibm.IBM_CONFIG_UUID}
+    }
+
+    def setUp(self):
+        self.blkid_metadata = copy.deepcopy(self.blkid_base)
+        self.blkid_metadata.update(copy.deepcopy(self.blkid_metadata_disk))
+
+        self.blkid_oscode = copy.deepcopy(self.blkid_base)
+        self.blkid_oscode.update(copy.deepcopy(self.blkid_oscode_disk))
+
+    def test_id_template_live_metadata(self, m_blkid, m_is_prov, _m_xen):
+        """identify TEMPLATE_LIVE_METADATA."""
+        m_blkid.return_value = self.blkid_metadata
+        m_is_prov.return_value = False
+        self.assertEqual(
+            (ibm.Platforms.TEMPLATE_LIVE_METADATA, "/dev/xvdh1"),
+            ibm.get_ibm_platform())
+
+    def test_id_template_prov_metadata(self, m_blkid, m_is_prov, _m_xen):
+        """identify TEMPLATE_PROVISIONING_METADATA."""
+        m_blkid.return_value = self.blkid_metadata
+        m_is_prov.return_value = True
+        self.assertEqual(
+            (ibm.Platforms.TEMPLATE_PROVISIONING_METADATA, "/dev/xvdh1"),
+            ibm.get_ibm_platform())
+
+    def test_id_template_prov_nodata(self, m_blkid, m_is_prov, _m_xen):
+        """identify TEMPLATE_PROVISIONING_NODATA."""
+        m_blkid.return_value = self.blkid_base
+        m_is_prov.return_value = True
+        self.assertEqual(
+            (ibm.Platforms.TEMPLATE_PROVISIONING_NODATA, None),
+            ibm.get_ibm_platform())
+
+    def test_id_os_code(self, m_blkid, m_is_prov, _m_xen):
+        """Identify OS_CODE."""
+        m_blkid.return_value = self.blkid_oscode
+        m_is_prov.return_value = False
+        self.assertEqual((ibm.Platforms.OS_CODE, "/dev/xvdh"),
+                         ibm.get_ibm_platform())
+
+    def test_id_os_code_must_match_uuid(self, m_blkid, m_is_prov, _m_xen):
+        """Test against false positive on openstack with non-ibm UUID."""
+        blkid = self.blkid_oscode
+        blkid["/dev/xvdh"]["UUID"] = "9999-9999"
+        m_blkid.return_value = blkid
+        m_is_prov.return_value = False
+        self.assertEqual((None, None), ibm.get_ibm_platform())
+
+
[e-mail](D_PATH + "_read_system_uuid", return_value=None)
[e-mail](D_PATH + "get_ibm_platform")
+class TestReadMD(test_helpers.CiTestCase):
+    """Test the read_datasource helper."""
+
+    template_md = {
+        "files": [],
+        "network_config": {"content_path": "/content/interfaces"},
+        "hostname": "ci-fond-ram",
+        "name": "ci-fond-ram",
+        "domain": "testing.ci.cloud-init.org",
+        "meta": {"dsmode": "net"},
+        "uuid": "8e636730-9f5d-c4a5-327c-d7123c46e82f",
+        "public_keys": {"1091307": "ssh-rsa AAAAB3NzaC1...Hw== ci-pubkey"},
+    }
+
+    oscode_md = {
+        "hostname": "ci-grand-gannet.testing.ci.cloud-init.org",
+        "name": "ci-grand-gannet",
+        "uuid": "2f266908-8e6c-4818-9b5c-42e9cc66a785",
+        "random_seed": "bm90LXJhbmRvbQo=",
+        "crypt_key": "ssh-rsa AAAAB3NzaC1yc2..n6z/",
+        "configuration_token": "eyJhbGciOi..M3ZA",
+        "public_keys": {"1091307": "ssh-rsa AAAAB3N..Hw== ci-pubkey"},
+    }
+
+    content_interfaces = dedent("""\
+        auto lo
+        iface lo inet loopback
+
+        auto eth0
+        allow-hotplug eth0
+        iface eth0 inet static
+        address 10.82.43.5
+        netmask 255.255.255.192
+        """)
+
+    userdata = b"#!/bin/sh\necho hi mom\n"
+    # meta.js file gets json encoded userdata as a list.
+    meta_js = '["#!/bin/sh\necho hi mom\n"]'
+    vendor_data = {
+        "cloud-init": "#!/bin/bash\necho 'root:$6$5ab01p1m1' | chpasswd -e"}
+
+    network_data = {
+        "links": [
+            {"id": "interface_29402281", "name": "eth0", "mtu": None,
+             "type": "phy", "ethernet_mac_address": "06:00:f1:bd:da:25"},
+            {"id": "interface_29402279", "name": "eth1", "mtu": None,
+             "type": "phy", "ethernet_mac_address": "06:98:5e:d0:7f:86"}
+        ],
+        "networks": [
+            {"id": "network_109887563", "link": "interface_29402281",
+             "type": "ipv4", "ip_address": "10.82.43.2",
+             "netmask": "255.255.255.192",
+             "routes": [
+                 {"network": "10.0.0.0", "netmask": "255.0.0.0",
+                  "gateway": "10.82.43.1"},
+                 {"network": "161.26.0.0", "netmask": "255.255.0.0",
+                  "gateway": "10.82.43.1"}]},
+            {"id": "network_109887551", "link": "interface_29402279",
+             "type": "ipv4", "ip_address": "108.168.194.252",
+             "netmask": "255.255.255.248",
+             "routes": [
+                 {"network": "0.0.0.0", "netmask": "0.0.0.0",
+                  "gateway": "108.168.194.249"}]}
+        ],
+        "services": [
+            {"type": "dns", "address": "10.0.80.11"},
+            {"type": "dns", "address": "10.0.80.12"}
+        ],
+    }
+
+    sysuuid = '7f79ebf5-d791-43c3-a723-854e8389d59f'
+
+    def _get_expected_metadata(self, os_md):
+        """return expected 'metadata' for data loaded from meta_data.json."""
+        os_md = copy.deepcopy(os_md)
+        renames = (
+            ('hostname', 'local-hostname'),
+            ('uuid', 'instance-id'),
+            ('public_keys', 'public-keys'))
+        ret = {}
+        for osname, mdname in renames:
+            if osname in os_md:
+                ret[mdname] = os_md[osname]
+        if 'random_seed' in os_md:
+            ret['random_seed'] = base64.b64decode(os_md['random_seed'])
+
+        return ret
+
+    def test_provisioning_md(self, m_platform, m_sysuuid):
+        """Provisioning env with a metadata disk should return None."""
+        m_platform.return_value = (
+            ibm.Platforms.TEMPLATE_PROVISIONING_METADATA, "/dev/xvdh")
+        self.assertIsNone(ibm.read_md())
+
+    def test_provisioning_no_metadata(self, m_platform, m_sysuuid):
+        """Provisioning env with no metadata disk should return None."""
+        m_platform.return_value = (
+            ibm.Platforms.TEMPLATE_PROVISIONING_NODATA, None)
+        self.assertIsNone(ibm.read_md())
+
+    def test_provisioning_not_ibm(self, m_platform, m_sysuuid):
+        """Provisioning env but not identified as IBM should return None."""
+        m_platform.return_value = (None, None)
+        self.assertIsNone(ibm.read_md())
+
+    def test_template_live(self, m_platform, m_sysuuid):
+        """Template live environment should be identified."""
+        tmpdir = self.tmp_dir()
+        m_platform.return_value = (
+            ibm.Platforms.TEMPLATE_LIVE_METADATA, tmpdir)
+        m_sysuuid.return_value = self.sysuuid
+
+        test_helpers.populate_dir(tmpdir, {
+            'openstack/latest/meta_data.json': json.dumps(self.template_md),
+            'openstack/latest/user_data': self.userdata,
+            'openstack/content/interfaces': self.content_interfaces,
+            'meta.js': self.meta_js})
+
+        ret = ibm.read_md()
+        self.assertEqual(ibm.Platforms.TEMPLATE_LIVE_METADATA,
+                         ret['platform'])
+        self.assertEqual(tmpdir, ret['source'])
+        self.assertEqual(self.userdata, ret['userdata'])
+        self.assertEqual(self._get_expected_metadata(self.template_md),
+                         ret['metadata'])
+        self.assertEqual(self.sysuuid, ret['system-uuid'])
+
+    def test_os_code_live(self, m_platform, m_sysuuid):
+        """Verify an os_code metadata path."""
+        tmpdir = self.tmp_dir()
+        m_platform.return_value = (ibm.Platforms.OS_CODE, tmpdir)
+        netdata = json.dumps(self.network_data)
+        test_helpers.populate_dir(tmpdir, {
+            'openstack/latest/meta_data.json': json.dumps(self.oscode_md),
+            'openstack/latest/user_data': self.userdata,
+            'openstack/latest/vendor_data.json': json.dumps(self.vendor_data),
+            'openstack/latest/network_data.json': netdata,
+        })
+
+        ret = ibm.read_md()
+        self.assertEqual(ibm.Platforms.OS_CODE, ret['platform'])
+        self.assertEqual(tmpdir, ret['source'])
+        self.assertEqual(self.userdata, ret['userdata'])
+        self.assertEqual(self._get_expected_metadata(self.oscode_md),
+                         ret['metadata'])
+
+    def test_os_code_live_no_userdata(self, m_platform, m_sysuuid):
+        """Verify os_code without user-data."""
+        tmpdir = self.tmp_dir()
+        m_platform.return_value = (ibm.Platforms.OS_CODE, tmpdir)
+        test_helpers.populate_dir(tmpdir, {
+            'openstack/latest/meta_data.json': json.dumps(self.oscode_md),
+            'openstack/latest/vendor_data.json': json.dumps(self.vendor_data),
+        })
+
+        ret = ibm.read_md()
+        self.assertEqual(ibm.Platforms.OS_CODE, ret['platform'])
+        self.assertEqual(tmpdir, ret['source'])
+        self.assertIsNone(ret['userdata'])
+        self.assertEqual(self._get_expected_metadata(self.oscode_md),
+                         ret['metadata'])
+
+
+# vi: ts=4 expandtab
diff --git a/tests/unittests/test_ds_identify.py b/tests/unittests/test_ds_identify.py
index 85999b7..5364398 100644
--- a/tests/unittests/test_ds_identify.py
+++ b/tests/unittests/test_ds_identify.py
@@ -9,6 +9,8 @@ from cloudinit import util
 from cloudinit.tests.helpers import (
     CiTestCase, dir2dict, populate_dir)
 
+from cloudinit.sources import DataSourceIBMCloud as dsibm
+
 UNAME_MYSYS = ("Linux bart 4.4.0-62-generic #83-Ubuntu "
                "SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 GNU/Linux")
 UNAME_PPC64EL = ("Linux diamond 4.4.0-83-generic #106-Ubuntu SMP "
@@ -37,8 +39,8 @@ BLKID_UEFI_UBUNTU = [
 
 POLICY_FOUND_ONLY = "search,found=all,maybe=none,notfound=disabled"
 POLICY_FOUND_OR_MAYBE = "search,found=all,maybe=all,notfound=disabled"
-DI_DEFAULT_POLICY = "search,found=all,maybe=all,notfound=enabled"
-DI_DEFAULT_POLICY_NO_DMI = "search,found=all,maybe=all,notfound=disabled"
+DI_DEFAULT_POLICY = "search,found=all,maybe=all,notfound=disabled"
+DI_DEFAULT_POLICY_NO_DMI = "search,found=all,maybe=all,notfound=enabled"
 DI_EC2_STRICT_ID_DEFAULT = "true"
 OVF_MATCH_STRING = 'http://schemas.dmtf.org/ovf/environment/1'
 
@@ -64,6 +66,9 @@ P_SYS_VENDOR = "sys/class/dmi/id/sys_vendor"
 P_SEED_DIR = "var/lib/cloud/seed"
 P_DSID_CFG = "etc/cloud/ds-identify.cfg"
 
+IBM_PROVISIONING_CHECK_PATH = "/root/provisioningConfiguration.cfg"
+IBM_CONFIG_UUID = "9796-932E"
+
 MOCK_VIRT_IS_KVM = {'name': 'detect_virt', 'RET': 'kvm', 'ret': 0}
 MOCK_VIRT_IS_VMWARE = {'name': 'detect_virt', 'RET': 'vmware', 'ret': 0}
 MOCK_VIRT_IS_XEN = {'name': 'detect_virt', 'RET': 'xen', 'ret': 0}
@@ -239,6 +244,57 @@ class TestDsIdentify(CiTestCase):
         self._test_ds_found('ConfigDriveUpper')
         return
 
+    def test_ibmcloud_template_userdata_in_provisioning(self):
+        """Template provisioned with user-data during provisioning stage.
+
+        Template provisioning with user-data has METADATA disk,
+        datasource should return not found."""
+        data = copy.deepcopy(VALID_CFG['IBMCloud-metadata'])
+        data['files'] = {IBM_PROVISIONING_CHECK_PATH: 'xxx'}
+        return self._check_via_dict(data, RC_NOT_FOUND)
+
+    def test_ibmcloud_template_userdata(self):
+        """Template provisioned with user-data first boot.
+
+        Template provisioning with user-data has METADATA disk.
+        datasource should return found."""
+        self._test_ds_found('IBMCloud-metadata')
+
+    def test_ibmcloud_template_no_userdata_in_provisioning(self):
+        """Template provisioned with no user-data during provisioning.
+
+        no disks attached.  Datasource should return not found."""
+        data = copy.deepcopy(VALID_CFG['IBMCloud-nodisks'])
+        data['files'] = {IBM_PROVISIONING_CHECK_PATH: 'xxx'}
+        return self._check_via_dict(data, RC_NOT_FOUND)
+
+    def test_ibmcloud_template_no_userdata(self):
+        """Template provisioned with no user-data first boot.
+
+        no disks attached.  Datasource should return found."""
+        self._check_via_dict(VALID_CFG['IBMCloud-nodisks'], RC_NOT_FOUND)
+
+    def test_ibmcloud_os_code(self):
+        """Launched by os code always has config-2 disk."""
+        self._test_ds_found('IBMCloud-config-2')
+
+    def test_ibmcloud_os_code_different_uuid(self):
+        """IBM cloud config-2 disks must be explicit match on UUID.
+
+        If the UUID is not 9796-932E then we actually expect ConfigDrive."""
+        data = copy.deepcopy(VALID_CFG['IBMCloud-config-2'])
+        offset = None
+        for m, d in enumerate(data['mocks']):
+            if d.get('name') == "blkid":
+                offset = m
+                break
+        if not offset:
+            raise ValueError("Expected to find 'blkid' mock, but did not.")
+        data['mocks'][offset]['out'] = d['out'].replace(dsibm.IBM_CONFIG_UUID,
+                                                        "DEAD-BEEF")
+        self._check_via_dict(
+            data, rc=RC_FOUND, dslist=['ConfigDrive', DS_NONE])
+
     def test_policy_disabled(self):
         """A Builtin policy of 'disabled' should return not found.
 
@@ -452,7 +508,7 @@ VALID_CFG = {
     },
     'Ec2-xen': {
         'ds': 'Ec2',
-        'mocks': [{'name': 'detect_virt', 'RET': 'xen', 'ret': 0}],
+        'mocks': [MOCK_VIRT_IS_XEN],
         'files': {
             'sys/hypervisor/uuid': 'ec2c6e2f-5fac-4fc7-9c82-74127ec14bbb\n'
         },
@@ -579,6 +635,48 @@ VALID_CFG = {
         'ds': 'Hetzner',
         'files': {P_SYS_VENDOR: 'Hetzner\n'},
     },
+    'IBMCloud-metadata': {
+        'ds': 'IBMCloud',
+        'mocks': [
+            MOCK_VIRT_IS_XEN,
+            {'name': 'blkid', 'ret': 0,
+             'out': blkid_out(
+                 [{'DEVNAME': 'xvda1', 'TYPE': 'vfat', 'PARTUUID': uuid4()},
+                  {'DEVNAME': 'xvda2', 'TYPE': 'ext4',
+                   'LABEL': 'cloudimg-rootfs', 'PARTUUID': uuid4()},
+                  {'DEVNAME': 'xvdb', 'TYPE': 'vfat', 'LABEL': 'METADATA'}]),
+             },
+        ],
+    },
+    'IBMCloud-config-2': {
+        'ds': 'IBMCloud',
+        'mocks': [
+            MOCK_VIRT_IS_XEN,
+            {'name': 'blkid', 'ret': 0,
+             'out': blkid_out(
+                 [{'DEVNAME': 'xvda1', 'TYPE': 'ext3', 'PARTUUID': uuid4(),
+                   'UUID': uuid4(), 'LABEL': 'cloudimg-bootfs'},
+                  {'DEVNAME': 'xvdb', 'TYPE': 'vfat', 'LABEL': 'config-2',
+                   'UUID': dsibm.IBM_CONFIG_UUID},
+                  {'DEVNAME': 'xvda2', 'TYPE': 'ext4',
+                   'LABEL': 'cloudimg-rootfs', 'PARTUUID': uuid4(),
+                   'UUID': uuid4()},
+                  ]),
+             },
+        ],
+    },
+    'IBMCloud-nodisks': {
+        'ds': 'IBMCloud',
+        'mocks': [
+            MOCK_VIRT_IS_XEN,
+            {'name': 'blkid', 'ret': 0,
+             'out': blkid_out(
+                 [{'DEVNAME': 'xvda1', 'TYPE': 'vfat', 'PARTUUID': uuid4()},
+                  {'DEVNAME': 'xvda2', 'TYPE': 'ext4',
+                   'LABEL': 'cloudimg-rootfs', 'PARTUUID': uuid4()}]),
+             },
+        ],
+    },
 }
 
 # vi: ts=4 expandtab
diff --git a/tools/ds-identify b/tools/ds-identify
index e2552c8..9a2db5c 100755
--- a/tools/ds-identify
+++ b/tools/ds-identify
@@ -92,6 +92,7 @@ DI_DMI_SYS_VENDOR=""
 DI_DMI_PRODUCT_SERIAL=""
 DI_DMI_PRODUCT_UUID=""
 DI_FS_LABELS=""
+DI_FS_UUIDS=""
 DI_ISO9660_DEVS=""
 DI_KERNEL_CMDLINE=""
 DI_VIRT=""
@@ -114,7 +115,7 @@ DI_DSNAME=""
 # be searched if there is no setting found in config.
 DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \
 CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack \
-OVF SmartOS Scaleway Hetzner"
+OVF SmartOS Scaleway Hetzner IBMCloud"
 DI_DSLIST=""
 DI_MODE=""
 DI_ON_FOUND=""
@@ -123,6 +124,8 @@ DI_ON_NOTFOUND=""
 
 DI_EC2_STRICT_ID_DEFAULT="true"
 
+_IS_IBM_CLOUD=""
+
 error() {
     set -- "ERROR:" "$@";
     debug 0 "$@"
@@ -196,7 +199,7 @@ read_fs_info() {
         return
     fi
     local oifs="$IFS" line="" delim=","
-    local ret=0 out="" labels="" dev="" label="" ftype="" isodevs=""
+    local ret=0 out="" labels="" dev="" label="" ftype="" isodevs="" uuids=""
     out=$(blkid -c /dev/null -o export) || {
         ret=$?
         error "failed running [$ret]: blkid -c /dev/null -o export"
@@ -219,12 +222,14 @@ read_fs_info() {
             LABEL=*) label="${line#LABEL=}";
                      labels="${labels}${line#LABEL=}${delim}";;
             TYPE=*) ftype=${line#TYPE=};;
+            UUID=*) uuids="${uuids}${line#UUID=}$delim";;
         esac
     done
     [ -n "$dev" -a "$ftype" = "iso9660" ] &&
         isodevs="${isodevs} ${dev}=$label"
 
     DI_FS_LABELS="${labels%${delim}}"
+    DI_FS_UUIDS="${uuids%${delim}}"
     DI_ISO9660_DEVS="${isodevs# }"
 }
 
@@ -437,14 +442,25 @@ dmi_sys_vendor_is() {
     [ "${DI_DMI_SYS_VENDOR}" = "$1" ]
 }
 
-has_fs_with_label() {
-    local label="$1"
-    case ",${DI_FS_LABELS}," in
-        *,$label,*) return 0;;
+has_fs_with_uuid() {
+    case ",${DI_FS_UUIDS}," in
+        *,$1,*) return 0;;
     esac
     return 1
 }
 
+has_fs_with_label() {
+    # has_fs_with_label(label1[ ,label2 ..])
+    # return 0 if a there is a filesystem that matches any of the labels.
+    local label=""
+    for label in "$@"; do
+        case ",${DI_FS_LABELS}," in
+            *,$label,*) return 0;;
+        esac
+    done
+    return 1
+}
+
 nocase_equal() {
     # nocase_equal(a, b)
     # return 0 if case insenstive comparision a.lower() == b.lower()
@@ -583,6 +599,8 @@ dscheck_NoCloud() {
     case " ${DI_DMI_PRODUCT_SERIAL} " in
         *\ ds=nocloud*) return ${DS_FOUND};;
     esac
+
+    is_ibm_cloud && return ${DS_NOT_FOUND}
     for d in nocloud nocloud-net; do
         check_seed_dir "$d" meta-data user-data && return ${DS_FOUND}
         check_writable_seed_dir "$d" meta-data user-data && return ${DS_FOUND}
@@ -594,9 +612,8 @@ dscheck_NoCloud() {
 }
 
 check_configdrive_v2() {
-    if has_fs_with_label "config-2"; then
-        return ${DS_FOUND}
-    elif has_fs_with_label "CONFIG-2"; then
+    is_ibm_cloud && return ${DS_NOT_FOUND}
+    if has_fs_with_label CONFIG-2 config-2; then
         return ${DS_FOUND}
     fi
     # look in /config-drive <vlc>/seed/config_drive for a directory
@@ -988,6 +1005,36 @@ dscheck_Hetzner() {
     return ${DS_NOT_FOUND}
 }
 
+is_ibm_provisioning() {
+    [ -f "${PATH_ROOT}/root/provisioningConfiguration.cfg" ]
+}
+
+is_ibm_cloud() {
+    cached "${_IS_IBM_CLOUD}" && return ${_IS_IBM_CLOUD}
+    local ret=1
+    if [ "$DI_VIRT" = "xen" ]; then
+        if is_ibm_provisioning; then
+            ret=0
+        elif has_fs_with_label METADATA metadata; then
+            ret=0
+        elif has_fs_with_uuid 9796-932E &&
+            has_fs_with_label CONFIG-2 config-2; then
+            ret=0
+        fi
+    fi
+    _IS_IBM_CLOUD=$ret
+    return $ret
+}
+
+dscheck_IBMCloud() {
+    if is_ibm_provisioning; then
+        debug 1 "cloud-init disabled during provisioning on IBMCloud"
+        return ${DS_NOT_FOUND}
+    fi
+    is_ibm_cloud && return ${DS_FOUND}
+    return ${DS_NOT_FOUND}
+}
+
 collect_info() {
     read_virt
     read_pid1_product_name