Merge ~ruansx/cloud-init:add-zstack-datasource into cloud-init:master

[Steve Ruan]

Add datasource for ZStack platform.

Zstack platform provides a AWS Ec2 metadata service, and
identifies their platform to the guest by setting the 'chassis asset tag'
to a string that ends with '.zstack.io'.

LP: #1841181

[Scott Moser (smoser)]

Quick review things:
 * make a better commit message (the 'Commit message` above will be be used when this is ultimately squashed and merged)
 * add 'LP: #1841181' to your commit message. See git log for examples. Thats how MPs get associated with bugs.
 * Tests needed.
    * tests/unittests/test_ds_identify.py should be fairly to understand and update for Zstack
    * Probably add a tests/unittests/test_datasource/test_zstack.py
 * Add doc/rtd/topics/datasources/zstack.rst and update doc/rtd/topics/datasources.rst to reference it. See Exoscale for a recently added datasource example.

  * Did you check to see if simply adding Zstack in the same way as BrightBox is added would work? That could be even less code.

[Steve Ruan (ruansx)]

Thanks.
It's more clean to add ZStack in same way as BrightBox，and I changed it.

[Scott Moser (smoser)]

Some comments inline. I also updated your commit message a bit.

[Scott Moser (smoser)]

Also, this request is still relevant.

 * Add doc/rtd/topics/datasources/zstack.rst and update doc/rtd/topics/datasources.rst to reference it. See Exoscale for a recently added datasource example.

[Scott Moser (smoser)]

Opened https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/372622
to address the brightbox issue.

[Steve Ruan (ruansx)]

Thanks, Scott.
All comments are fixed.

[Scott Moser (smoser)]

This looks good to me now, thanks.

Steve, one style comment in-line though.

I'll get someone to point the c-i bot at this too.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:c8d47870a851f3a5b7a96fabb1442ab0224b9388
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1138/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1138//rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

@Steve,

Just for your ease, I put together a change that updates the Zstack/ZSTACK and adds a test case also.

http://paste.ubuntu.com/p/KZWYKZCmcd/

Please grab that and update.

[Ryan Harper (raharper)]

Suggest a different word in the documentation, see inline comment.

[Steve Ruan (ruansx)]

Sorry for the late response because there is a short vacation.

Thanks for the comment. @Scott and @Ryan.

[Scott Moser (smoser)]

I'm +1 now. But you do have to fix a merge issue (look for <<<<).

[Steve Ruan (ruansx)]

doc/rtd/topics/datasources.rst is changed in master branch. Someone move the section "datasource/xx" from the buttom to the middle, and my commit also changed this part.

The conflict is fixed now.

[Scott Moser (smoser)]

review: Approve

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
More details in the following jenkins job:
https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/333/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
More details in the following jenkins job:
https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/334/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
More details in the following jenkins job:
https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/335/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

review: Needs Fixing (continuous-integration)

[Scott Moser (smoser)]

Steve,
I'm sorry for not having run pycodestyle on my suggested changes..
the Lander rejected this due to some code formatting.
Fix at http://paste.ubuntu.com/p/9st9YCstxf/

you can run those tests locally with:
 tox
or specifically the pycodestyle with
  tox -e pycodestyle

Fix that and we can land. Sorry for the loss of another day.

[Steve Ruan (ruansx)]

Thanks Scott. It's my fault, I should run tox before pushing to the branch.

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

diff --git a/cloudinit/apport.py b/cloudinit/apport.py
index 003ff1f..fde1f75 100644
--- a/cloudinit/apport.py
+++ b/cloudinit/apport.py
@@ -37,6 +37,7 @@ KNOWN_CLOUD_NAMES = [
     'Scaleway',
     'SmartOS',
     'VMware',
+    'ZStack',
     'Other']
 
 # Potentially clear text collected logs
diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourceEc2.py
index 1010745..6c72ace 100644
--- a/cloudinit/sources/DataSourceEc2.py
+++ b/cloudinit/sources/DataSourceEc2.py
@@ -33,6 +33,7 @@ class CloudNames(object):
     ALIYUN = "aliyun"
     AWS = "aws"
     BRIGHTBOX = "brightbox"
+    ZSTACK = "zstack"
     # UNKNOWN indicates no positive id.  If strict_id is 'warn' or 'false',
     # then an attempt at the Ec2 Metadata service will be made.
     UNKNOWN = "unknown"
@@ -477,10 +478,16 @@ def identify_brightbox(data):
         return CloudNames.BRIGHTBOX
 
 
+def identify_zstack(data):
+    if data['asset_tag'].endswith('.zstack.io'):
+        return CloudNames.ZSTACK
+
+
 def identify_platform():
     # identify the platform and return an entry in CloudNames.
     data = _collect_platform_data()
-    checks = (identify_aws, identify_brightbox, lambda x: CloudNames.UNKNOWN)
+    checks = (identify_aws, identify_brightbox, identify_zstack,
+              lambda x: CloudNames.UNKNOWN)
     for checker in checks:
         try:
             result = checker(data)
@@ -498,6 +505,7 @@ def _collect_platform_data():
        uuid: system-uuid from dmi or /sys/hypervisor
        uuid_source: 'hypervisor' (/sys/hypervisor/uuid) or 'dmi'
        serial: dmi 'system-serial-number' (/sys/.../product_serial)
+       asset_tag: 'dmidecode -s chassis-asset-tag'
 
     On Ec2 instances experimentation is that product_serial is upper case,
     and product_uuid is lower case.  This returns lower case values for both.
@@ -520,6 +528,12 @@ def _collect_platform_data():
 
     data['serial'] = serial.lower()
 
+    asset_tag = util.read_dmi_data('chassis-asset-tag')
+    if asset_tag is None:
+        asset_tag = ''
+
+    data['asset_tag'] = asset_tag.lower()
+
     return data
 
 
diff --git a/doc/rtd/topics/datasources.rst b/doc/rtd/topics/datasources.rst
index 8e58be9..a337c08 100644
--- a/doc/rtd/topics/datasources.rst
+++ b/doc/rtd/topics/datasources.rst
@@ -45,6 +45,7 @@ The following is a list of documents for each supported datasource:
    datasources/oracle.rst
    datasources/ovf.rst
    datasources/smartos.rst
+   datasources/zstack.rst
 
 
 Creation
diff --git a/doc/rtd/topics/datasources/zstack.rst b/doc/rtd/topics/datasources/zstack.rst
new file mode 100644
index 0000000..36e60ff
--- /dev/null
+++ b/doc/rtd/topics/datasources/zstack.rst
@@ -0,0 +1,36 @@
+.. _datasource_zstack:
+
+ZStack
+======
+ZStack platform provides a AWS Ec2 metadata service, but with different datasource identity.
+More information about ZStack can be found at `ZStack <https://www.zstack.io>`__.
+
+Discovery
+---------
+To determine whether a vm running on ZStack platform, cloud-init checks DMI information
+by 'dmidecode -s chassis-asset-tag', if the output ends with '.zstack.io', it's running
+on ZStack platform:
+
+
+Metadata
+^^^^^^^^
+Same as EC2, instance metadata can be queried at
+
+::
+
+    GET http://169.254.169.254/2009-04-04/meta-data/
+    instance-id
+    local-hostname
+
+Userdata
+^^^^^^^^
+Same as EC2, instance userdata can be queried at
+
+::
+
+    GET http://169.254.169.254/2009-04-04/user-data/
+    meta_data.json
+    user_data
+    password
+
+.. vi: textwidth=78
diff --git a/tests/unittests/test_datasource/test_ec2.py b/tests/unittests/test_datasource/test_ec2.py
index 1ec8e00..6fabf25 100644
--- a/tests/unittests/test_datasource/test_ec2.py
+++ b/tests/unittests/test_datasource/test_ec2.py
@@ -662,4 +662,32 @@ class TestConvertEc2MetadataNetworkConfig(test_helpers.CiTestCase):
                 expected,
                 ec2.convert_ec2_metadata_network_config(self.network_metadata))
 
+
+class TesIdentifyPlatform(test_helpers.CiTestCase):
+
+    def collmock(self, **kwargs):
+        """return non-special _collect_platform_data updated with changes."""
+        unspecial = {
+            'asset_tag': '3857-0037-2746-7462-1818-3997-77',
+            'serial': 'H23-C4J3JV-R6',
+            'uuid': '81c7e555-6471-4833-9551-1ab366c4cfd2',
+            'uuid_source': 'dmi',
+        }
+        unspecial.update(**kwargs)
+        return unspecial
+
+    @mock.patch('cloudinit.sources.DataSourceEc2._collect_platform_data')
+    def test_identify_zstack(self, m_collect):
+        """zstack should be identified if cassis-asset-tag ends in .zstack.io
+        """
+        m_collect.return_value = self.collmock(asset_tag='123456.zstack.io')
+        self.assertEqual(ec2.CloudNames.ZSTACK, ec2.identify_platform())
+
+    @mock.patch('cloudinit.sources.DataSourceEc2._collect_platform_data')
+    def test_identify_zstack_full_domain_only(self, m_collect):
+        """zstack asset-tag matching should match only on full domain boundary.
+        """
+        m_collect.return_value = self.collmock(asset_tag='123456.buzzstack.io')
+        self.assertEqual(ec2.CloudNames.UNKNOWN, ec2.identify_platform())
+
 # vi: ts=4 expandtab
diff --git a/tests/unittests/test_ds_identify.py b/tests/unittests/test_ds_identify.py
index de87be2..7aeeb91 100644
--- a/tests/unittests/test_ds_identify.py
+++ b/tests/unittests/test_ds_identify.py
@@ -609,6 +609,10 @@ class TestDsIdentify(DsIdentifyBase):
         self.assertEqual(expected, [p for p in expected if p in toks],
                          "path did not have expected tokens")
 
+    def test_zstack_is_ec2(self):
+        """EC2: chassis asset tag ends with 'zstack.io'"""
+        self._test_ds_found('Ec2-ZStack')
+
 
 class TestIsIBMProvisioning(DsIdentifyBase):
     """Test the is_ibm_provisioning method in ds-identify."""
@@ -971,8 +975,11 @@ VALID_CFG = {
             {'name': 'blkid', 'ret': 2, 'out': ''},
         ],
         'files': {ds_smartos.METADATA_SOCKFILE: 'would be a socket\n'},
+    },
+    'Ec2-ZStack': {
+        'ds': 'Ec2',
+        'files': {P_CHASSIS_ASSET_TAG: '123456.zstack.io\n'},
     }
-
 }
 
 # vi: ts=4 expandtab
diff --git a/tools/ds-identify b/tools/ds-identify
index 2447d14..f76f2a6 100755
--- a/tools/ds-identify
+++ b/tools/ds-identify
@@ -895,6 +895,11 @@ ec2_identify_platform() {
         *.brightbox.com) _RET="Brightbox"; return 0;;
     esac
 
+    local asset_tag="${DI_DMI_CHASSIS_ASSET_TAG}"
+    case "$asset_tag" in
+        *.zstack.io) _RET="ZStack"; return 0;;
+    esac
+
     # AWS http://docs.aws.amazon.com/AWSEC2/
     #     latest/UserGuide/identify_ec2_instances.html
     local uuid="" hvuuid="${PATH_SYS_HYPERVISOR}/uuid"