Merge ~chad.smith/cloud-init:azure-di-id-asset-tag into cloud-init:master

[Chad Smith]

azure: ds-identify and Datasource.get_data checks DMI chassis-asset-tag

Azure sets a known chassis asset tag to 7783-7084-3265-9085-8269-3286-77. We can inspect this in both ds-identify and DataSource.get_data to determine whether we are on Azure. Added unit tests to cover these changes and some minor tweaks to Exception error message content to give more context on malformed or missing ovf-env.xml files.

LP: #1693939

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:bd9b28377d36741e31ea5ce843e62e85a6c96c61
https://jenkins.ubuntu.com/server/job/cloud-init-ci/432/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/432/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/432/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/432/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/432/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/432/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/432/rebuild

review: Needs Fixing (continuous-integration)

[Scott Moser (smoser)]

https://gist.github.com/smoser/a5435f49b61e2ae45c367bbb266bfac3

Add DI_DMI_CHASSIS_ASSET_TAG to _print_info in tools/ds-identify.

fc29533... by Chad Smith on 2017-06-01

dmi_product_name_is is dropped from ds-identify, fix it for AliYun datasource. unit test mock fixes for read_dmi_data

4ce9ca1... by Chad Smith on 2017-06-01

resolve merge conflicts

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:4ce9ca112fbaa94f8df9005d1e82469ec2bd7065
https://jenkins.ubuntu.com/server/job/cloud-init-ci/439/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/439/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/439/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/439/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/439/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/439/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/439/rebuild

review: Needs Fixing (continuous-integration)

d2abad2... by Chad Smith on 2017-06-01

add unit test for variables printed by ds-identify's _print_info

2275666... by Chad Smith on 2017-06-01

lint and drop custom NonAzureDataSource message

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:d2abad2ecf83e73590991f0bd004d8d4747b4fdb
https://jenkins.ubuntu.com/server/job/cloud-init-ci/440/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/440/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/440/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/440/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/440/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/440/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/440/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:227566642f525130d1dc1ac05c9bb3b56043cfa2
https://jenkins.ubuntu.com/server/job/cloud-init-ci/441/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/441
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/441
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/441
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/441
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/441

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/441/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

I approve with some minor changes.
http://paste.ubuntu.com/24738693/

if you want to ACK those i can just integrate themm, squash and merge.

review: Approve

54480ea... by Chad Smith on 2017-06-01

    smoser fixups

     - fix up comment spelling
     - drop unused change.
     - capital letters are for global vars

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:54480ea3ce7d5ad4d7d56ea9e8bb2be43d15ff89
https://jenkins.ubuntu.com/server/job/cloud-init-ci/442/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/442
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/442
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/442
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/442
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/442

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/442/rebuild

review: Approve (continuous-integration)

diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py
index b9458ff..a0b9eae 100644
--- a/cloudinit/sources/DataSourceAzure.py
+++ b/cloudinit/sources/DataSourceAzure.py
@@ -36,6 +36,8 @@ RESOURCE_DISK_PATH = '/dev/disk/cloud/azure_resource'
 DEFAULT_PRIMARY_NIC = 'eth0'
 LEASE_FILE = '/var/lib/dhcp/dhclient.eth0.leases'
 DEFAULT_FS = 'ext4'
+# DMI chassis-asset-tag is set static for all azure instances
+AZURE_CHASSIS_ASSET_TAG = '7783-7084-3265-9085-8269-3286-77'
 
 
 def find_storvscid_from_sysctl_pnpinfo(sysctl_out, deviceid):
@@ -320,6 +322,10 @@ class DataSourceAzureNet(sources.DataSource):
         # azure removes/ejects the cdrom containing the ovf-env.xml
         # file on reboot.  So, in order to successfully reboot we
         # need to look in the datadir and consider that valid
+        asset_tag = util.read_dmi_data('chassis-asset-tag')
+        if asset_tag != AZURE_CHASSIS_ASSET_TAG:
+            LOG.debug("Non-Azure DMI asset tag '%s' discovered.", asset_tag)
+            return False
         ddir = self.ds_cfg['data_dir']
 
         candidates = [self.seed_dir]
@@ -694,7 +700,7 @@ def read_azure_ovf(contents):
     try:
         dom = minidom.parseString(contents)
     except Exception as e:
-        raise BrokenAzureDataSource("invalid xml: %s" % e)
+        raise BrokenAzureDataSource("Invalid ovf-env.xml: %s" % e)
 
     results = find_child(dom.documentElement,
                          lambda n: n.localName == "ProvisioningSection")
diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py
index 852ec70..42f49e0 100644
--- a/tests/unittests/test_datasource/test_azure.py
+++ b/tests/unittests/test_datasource/test_azure.py
@@ -76,7 +76,9 @@ def construct_valid_ovf_env(data=None, pubkeys=None, userdata=None):
     return content
 
 
-class TestAzureDataSource(TestCase):
+class TestAzureDataSource(CiTestCase):
+
+    with_logs = True
 
     def setUp(self):
         super(TestAzureDataSource, self).setUp()
@@ -160,6 +162,12 @@ scbus-1 on xpt0 bus 0
 
         self.instance_id = 'test-instance-id'
 
+        def _dmi_mocks(key):
+            if key == 'system-uuid':
+                return self.instance_id
+            elif key == 'chassis-asset-tag':
+                return '7783-7084-3265-9085-8269-3286-77'
+
         self.apply_patches([
             (dsaz, 'list_possible_azure_ds_devs', dsdevs),
             (dsaz, 'invoke_agent', _invoke_agent),
@@ -170,7 +178,7 @@ scbus-1 on xpt0 bus 0
             (dsaz, 'set_hostname', mock.MagicMock()),
             (dsaz, 'get_metadata_from_fabric', self.get_metadata_from_fabric),
             (dsaz.util, 'read_dmi_data', mock.MagicMock(
-                return_value=self.instance_id)),
+                side_effect=_dmi_mocks)),
         ])
 
         dsrc = dsaz.DataSourceAzureNet(
@@ -241,6 +249,23 @@ fdescfs            /dev/fd          fdescfs rw              0 0
             res = get_path_dev_freebsd('/etc', mnt_list)
             self.assertIsNotNone(res)
 
+    @mock.patch('cloudinit.sources.DataSourceAzure.util.read_dmi_data')
+    def test_non_azure_dmi_chassis_asset_tag(self, m_read_dmi_data):
+        """Report non-azure when DMI's chassis asset tag doesn't match.
+
+        Return False when the asset tag doesn't match Azure's static
+        AZURE_CHASSIS_ASSET_TAG.
+        """
+        # Return a non-matching asset tag value
+        nonazure_tag = dsaz.AZURE_CHASSIS_ASSET_TAG + 'X'
+        m_read_dmi_data.return_value = nonazure_tag
+        dsrc = dsaz.DataSourceAzureNet(
+            {}, distro=None, paths=self.paths)
+        self.assertFalse(dsrc.get_data())
+        self.assertEqual(
+            "Non-Azure DMI asset tag '{0}' discovered.\n".format(nonazure_tag),
+            self.logs.getvalue())
+
     def test_basic_seed_dir(self):
         odata = {'HostName': "myhost", 'UserName': "myuser"}
         data = {'ovfcontent': construct_valid_ovf_env(data=odata),
@@ -531,9 +556,17 @@ class TestAzureBounce(TestCase):
         self.patches.enter_context(
             mock.patch.object(dsaz, 'get_metadata_from_fabric',
                               mock.MagicMock(return_value={})))
+
+        def _dmi_mocks(key):
+            if key == 'system-uuid':
+                return 'test-instance-id'
+            elif key == 'chassis-asset-tag':
+                return '7783-7084-3265-9085-8269-3286-77'
+            raise RuntimeError('should not get here')
+
         self.patches.enter_context(
             mock.patch.object(dsaz.util, 'read_dmi_data',
-                              mock.MagicMock(return_value='test-instance-id')))
+                              mock.MagicMock(side_effect=_dmi_mocks)))
 
     def setUp(self):
         super(TestAzureBounce, self).setUp()
@@ -696,6 +729,33 @@ class TestAzureBounce(TestCase):
         self.assertEqual(0, self.set_hostname.call_count)
 
 
+class TestLoadAzureDsDir(CiTestCase):
+    """Tests for load_azure_ds_dir."""
+
+    def setUp(self):
+        self.source_dir = self.tmp_dir()
+        super(TestLoadAzureDsDir, self).setUp()
+
+    def test_missing_ovf_env_xml_raises_non_azure_datasource_error(self):
+        """load_azure_ds_dir raises an error When ovf-env.xml doesn't exit."""
+        with self.assertRaises(dsaz.NonAzureDataSource) as context_manager:
+            dsaz.load_azure_ds_dir(self.source_dir)
+        self.assertEqual(
+            'No ovf-env file found',
+            str(context_manager.exception))
+
+    def test_wb_invalid_ovf_env_xml_calls_read_azure_ovf(self):
+        """load_azure_ds_dir calls read_azure_ovf to parse the xml."""
+        ovf_path = os.path.join(self.source_dir, 'ovf-env.xml')
+        with open(ovf_path, 'wb') as stream:
+            stream.write(b'invalid xml')
+        with self.assertRaises(dsaz.BrokenAzureDataSource) as context_manager:
+            dsaz.load_azure_ds_dir(self.source_dir)
+        self.assertEqual(
+            'Invalid ovf-env.xml: syntax error: line 1, column 0',
+            str(context_manager.exception))
+
+
 class TestReadAzureOvf(TestCase):
     def test_invalid_xml_raises_non_azure_ds(self):
         invalid_xml = "<foo>" + construct_valid_ovf_env(data={})
diff --git a/tests/unittests/test_ds_identify.py b/tests/unittests/test_ds_identify.py
index 5c26e65..8ccfe55 100644
--- a/tests/unittests/test_ds_identify.py
+++ b/tests/unittests/test_ds_identify.py
@@ -39,9 +39,11 @@ RC_FOUND = 0
 RC_NOT_FOUND = 1
 DS_NONE = 'None'
 
+P_CHASSIS_ASSET_TAG = "sys/class/dmi/id/chassis_asset_tag"
 P_PRODUCT_NAME = "sys/class/dmi/id/product_name"
 P_PRODUCT_SERIAL = "sys/class/dmi/id/product_serial"
 P_PRODUCT_UUID = "sys/class/dmi/id/product_uuid"
+P_SEED_DIR = "var/lib/cloud/seed"
 P_DSID_CFG = "etc/cloud/ds-identify.cfg"
 
 MOCK_VIRT_IS_KVM = {'name': 'detect_virt', 'RET': 'kvm', 'ret': 0}
@@ -160,6 +162,30 @@ class TestDsIdentify(CiTestCase):
                 _print_run_output(rc, out, err, cfg, files)
         return rc, out, err, cfg, files
 
+    def test_wb_print_variables(self):
+        """_print_info reports an array of discovered variables to stderr."""
+        data = VALID_CFG['Azure-dmi-detection']
+        _, _, err, _, _ = self._call_via_dict(data)
+        expected_vars = [
+            'DMI_PRODUCT_NAME', 'DMI_SYS_VENDOR', 'DMI_PRODUCT_SERIAL',
+            'DMI_PRODUCT_UUID', 'PID_1_PRODUCT_NAME', 'DMI_CHASSIS_ASSET_TAG',
+            'FS_LABELS', 'KERNEL_CMDLINE', 'VIRT', 'UNAME_KERNEL_NAME',
+            'UNAME_KERNEL_RELEASE', 'UNAME_KERNEL_VERSION', 'UNAME_MACHINE',
+            'UNAME_NODENAME', 'UNAME_OPERATING_SYSTEM', 'DSNAME', 'DSLIST',
+            'MODE', 'ON_FOUND', 'ON_MAYBE', 'ON_NOTFOUND']
+        for var in expected_vars:
+            self.assertIn('{0}='.format(var), err)
+
+    def test_azure_dmi_detection_from_chassis_asset_tag(self):
+        """Azure datasource is detected from DMI chassis-asset-tag"""
+        self._test_ds_found('Azure-dmi-detection')
+
+    def test_azure_seed_file_detection(self):
+        """Azure datasource is detected due to presence of a seed file.
+
+        The seed file tested  is /var/lib/cloud/seed/azure/ovf-env.xml."""
+        self._test_ds_found('Azure-seed-detection')
+
     def test_aws_ec2_hvm(self):
         """EC2: hvm instances use dmi serial and uuid starting with 'ec2'."""
         self._test_ds_found('Ec2-hvm')
@@ -272,6 +298,19 @@ VALID_CFG = {
         'ds': 'AliYun',
         'files': {P_PRODUCT_NAME: 'Alibaba Cloud ECS\n'},
     },
+    'Azure-dmi-detection': {
+        'ds': 'Azure',
+        'files': {
+            P_CHASSIS_ASSET_TAG: '7783-7084-3265-9085-8269-3286-77\n',
+        }
+    },
+    'Azure-seed-detection': {
+        'ds': 'Azure',
+        'files': {
+            P_CHASSIS_ASSET_TAG: 'No-match\n',
+            os.path.join(P_SEED_DIR, 'azure', 'ovf-env.xml'): 'present\n',
+        }
+    },
     'Ec2-hvm': {
         'ds': 'Ec2',
         'mocks': [{'name': 'detect_virt', 'RET': 'kvm', 'ret': 0}],
diff --git a/tools/ds-identify b/tools/ds-identify
index 5fc500b..546e0f5 100755
--- a/tools/ds-identify
+++ b/tools/ds-identify
@@ -85,6 +85,7 @@ DI_MAIN=${DI_MAIN:-main}
 
 DI_DEFAULT_POLICY="search,found=all,maybe=all,notfound=${DI_DISABLED}"
 DI_DEFAULT_POLICY_NO_DMI="search,found=all,maybe=all,notfound=${DI_ENABLED}"
+DI_DMI_CHASSIS_ASSET_TAG=""
 DI_DMI_PRODUCT_NAME=""
 DI_DMI_SYS_VENDOR=""
 DI_DMI_PRODUCT_SERIAL=""
@@ -259,6 +260,12 @@ read_kernel_cmdline() {
     DI_KERNEL_CMDLINE="$cmdline"
 }
 
+read_dmi_chassis_asset_tag() {
+    cached "${DI_DMI_CHASSIS_ASSET_TAG}" && return
+    get_dmi_field chassis_asset_tag
+    DI_DMI_CHASSIS_ASSET_TAG="$_RET"
+}
+
 read_dmi_sys_vendor() {
     cached "${DI_DMI_SYS_VENDOR}" && return
     get_dmi_field sys_vendor
@@ -386,6 +393,14 @@ read_pid1_product_name() {
     DI_PID_1_PRODUCT_NAME="$product_name"
 }
 
+dmi_chassis_asset_tag_matches() {
+    is_container && return 1
+    case "${DI_DMI_CHASSIS_ASSET_TAG}" in
+        $1) return 0;;
+    esac
+    return 1
+}
+
 dmi_product_name_matches() {
     is_container && return 1
     case "${DI_DMI_PRODUCT_NAME}" in
@@ -402,11 +417,6 @@ dmi_product_serial_matches() {
     return 1
 }
 
-dmi_product_name_is() {
-    is_container && return 1
-    [ "${DI_DMI_PRODUCT_NAME}" = "$1" ]
-}
-
 dmi_sys_vendor_is() {
     is_container && return 1
     [ "${DI_DMI_SYS_VENDOR}" = "$1" ]
@@ -478,7 +488,7 @@ dscheck_CloudStack() {
 
 dscheck_CloudSigma() {
     # http://paste.ubuntu.com/23624795/
-    dmi_product_name_is "CloudSigma" && return $DS_FOUND
+    dmi_product_name_matches "CloudSigma" && return $DS_FOUND
     return $DS_NOT_FOUND
 }
 
@@ -654,6 +664,8 @@ dscheck_Azure() {
     #  UUID="112D211272645f72" LABEL="rd_rdfe_stable.161212-1209"
     #  TYPE="udf">/dev/sr0</device>
     #
+    local azure_chassis="7783-7084-3265-9085-8269-3286-77"
+    dmi_chassis_asset_tag_matches "${azure_chassis}" && return $DS_FOUND
     check_seed_dir azure ovf-env.xml && return ${DS_FOUND}
 
     [ "${DI_VIRT}" = "microsoft" ] || return ${DS_NOT_FOUND}
@@ -786,7 +798,7 @@ dscheck_Ec2() {
 }
 
 dscheck_GCE() {
-    if dmi_product_name_is "Google Compute Engine"; then
+    if dmi_product_name_matches "Google Compute Engine"; then
         return ${DS_FOUND}
     fi
     # product name is not guaranteed (LP: #1674861)
@@ -807,10 +819,10 @@ dscheck_OpenStack() {
         return ${DS_NOT_FOUND}
     fi
     local nova="OpenStack Nova" compute="OpenStack Compute"
-    if dmi_product_name_is "$nova"; then
+    if dmi_product_name_matches "$nova"; then
         return ${DS_FOUND}
     fi
-    if dmi_product_name_is "$compute"; then
+    if dmi_product_name_matches "$compute"; then
         # RDO installed nova (LP: #1675349).
         return ${DS_FOUND}
     fi
@@ -823,7 +835,7 @@ dscheck_OpenStack() {
 
 dscheck_AliYun() {
     check_seed_dir "AliYun" meta-data user-data && return ${DS_FOUND}
-    if dmi_product_name_is "Alibaba Cloud ECS"; then
+    if dmi_product_name_matches "Alibaba Cloud ECS"; then
         return $DS_FOUND
     fi
     return $DS_NOT_FOUND
@@ -889,6 +901,7 @@ collect_info() {
     read_config
     read_datasource_list
     read_dmi_sys_vendor
+    read_dmi_chassis_asset_tag
     read_dmi_product_name
     read_dmi_product_serial
     read_dmi_product_uuid
@@ -903,7 +916,7 @@ print_info() {
 _print_info() {
     local n="" v="" vars=""
     vars="DMI_PRODUCT_NAME DMI_SYS_VENDOR DMI_PRODUCT_SERIAL"
-    vars="$vars DMI_PRODUCT_UUID PID_1_PRODUCT_NAME"
+    vars="$vars DMI_PRODUCT_UUID PID_1_PRODUCT_NAME DMI_CHASSIS_ASSET_TAG"
     vars="$vars FS_LABELS KERNEL_CMDLINE VIRT"
     vars="$vars UNAME_KERNEL_NAME UNAME_KERNEL_RELEASE UNAME_KERNEL_VERSION"
     vars="$vars UNAME_MACHINE UNAME_NODENAME UNAME_OPERATING_SYSTEM"