Merge ~msaikia/cloud-init:topic-msaikia-vmware into cloud-init:master

[Maitreyee Saikia]

Guest Customization support for product vcloud director - Password Configuration.
This feature enables the following VMware VCloud Director functionalities:
1. Setting admin password
2. Expire password.
3. Set admin password and expire.

1. Retrieve values from configuration file.
2. Parse data and call password customization.
3. Unit tests for password configurator

3c065e8... by Maitreyee Saikia on 2016-10-25

Addressed cloud-init review comments: 1.Changed property names to not have get. 2.No CamelCase for method name.

5b39e5a... by Maitreyee Saikia on 2016-10-25

Addressed cloud-init review comments: 1.Changed property names to not have get. 2.No CamelCase for method name.

9666992... by Maitreyee Saikia on 2016-10-25

Addressed cloud-init review comments: 1.Changed property names to not have get. 2.No CamelCase for method name.

[Maitreyee Saikia (msaikia)]

Thanks for the comments Joshua.
Updated the diff.

9c2b189... by Maitreyee Saikia on 2016-11-21

Fixed formatting for vmware config file

fe0e369... by Maitreyee Saikia on 2016-11-21

Fixed formatting for vmware config file

5c2f89e... by Maitreyee Saikia on 2016-11-21

Fixed formatting for vmware config file

be30849... by Maitreyee Saikia on 2016-11-21

Merge branch 'topic-msaikia-vmware' of git.launchpad.net:~msaikia/cloud-init into topic-msaikia-vmware

[Maitreyee Saikia (msaikia)]

Updated some formatting.

[Scott Moser (smoser)]

Please set a combined commit message ('Set commit message' above.

Can you explain where you're going also?
I'm concerned about adding more "vmware" paths for configuring things that are done elsewhere (or not elsewhere) in cloud-init. We'd like have consistent paths for doing things as much as possible.

I understand that you're trying to have cloud-init take over another more solution, and I'm not entirely opposed to that, but I want to integrate as much as possible rather than having specific paths and function on vmware.

[Maitreyee Saikia (msaikia)]

> Please set a combined commit message ('Set commit message' above.
>
> Can you explain where you're going also?
> I'm concerned about adding more "vmware" paths for configuring things that are
> done elsewhere (or not elsewhere) in cloud-init. We'd like have consistent
> paths for doing things as much as possible.
>
> I understand that you're trying to have cloud-init take over another more
> solution, and I'm not entirely opposed to that, but I want to integrate as
> much as possible rather than having specific paths and function on vmware.

Thanks for your input Scott. I have updated the commit message.
This changeset is just to retrieve data from customization spec, for implementing some other customization functionalities like setting password, running post customization scripts etc. I shall post another changeset with the password configurator, which manipulates the values retrieved from these methods to be used by cloud-init's cc_set_password. We will be using as much cloud-init code as possible. For other functionalities like running pre and post customization specs that will be uploaded by the user, I dont think there is existing support. Please let me know if that sounds ok. Also for functions that are "vmware" specific, shall we still keep it in the helper path that we have created for vmware, or do we want to move them to some common code area?
Thanks
Maitreyee

3128086... by Maitreyee Saikia on 2017-01-24

Merge conflicts

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:312808680eb5484b112ef874bf13fe2fdac8515e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/87/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/87
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/87/console
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/87
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-amd64/87
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/87

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/87/rebuild

review: Needs Fixing (continuous-integration)

7c480f7... by Maitreyee Saikia on 2017-03-09

Merge from main

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:7c480f71a2016f9a727479a54a002d03ec24453f
https://jenkins.ubuntu.com/server/job/cloud-init-ci/94/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/94
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/94
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/94
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/94
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/94

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/94/rebuild

review: Approve (continuous-integration)

f71def6... by Maitreyee Saikia on 2017-03-09

Added password configurator

[Scott Moser (smoser)]

There is one small nitpick in line.
There isn't anything wrong with this merge proposal, but it
a.) doesn't really do anything
  I realize you're planning on using these things going forward, and thats fine, but there is no reason to accept this MP until we also see that further work. So just add it here. You can rebase/squahs your commits into single function items in a "series" if that helps you.

b.) should add some unit tests.
unit tests will help stop others from breaking your code path inadvertently.

review: Needs Fixing

f482c10... by Maitreyee Saikia on 2017-03-09

Call to Password Configurator

[Scott Moser (smoser)]

Hi. I'm going to put this in "work in progress".
See the comments above. When youv'e addressed them, move it back to Needs Review.
Thanks
Scott

50e9a02... by Maitreyee Saikia on 2017-04-04

merge conflicts

4580c7b... by Maitreyee Saikia on 2017-04-04

typo fix

1a801a4... by Maitreyee Saikia on 2017-04-04

Return bool for reset_passwd configuration

c886370... by Maitreyee Saikia on 2017-04-07

remove regex check for post-gc-statusWq

[Maitreyee Saikia (msaikia)]

Hi Scott
I have resubmitted my review request after addressing your comments.
https://code.launchpad.net/~msaikia/cloud-init/+git/cloud-init/+merge/322991
Please let me know if you get a chance to review the new changeset.

Thanks
Maitreyee

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Tuesday, March 14, 2017 7:25 AM
To: Maitreyee Saikia <email address hidden>
Subject: Re: [Merge] ~msaikia/cloud-init:topic-msaikia-vmware into cloud-init:master

Hi. I'm going to put this in "work in progress".
See the comments above. When youv'e addressed them, move it back to Needs Review.
Thanks
Scott
--
https://code.launchpad.net/~msaikia/cloud-init/+git/cloud-init/+merge/305427
You are the owner of ~msaikia/cloud-init:topic-msaikia-vmware.

Unmerged commits

c886370... by Maitreyee Saikia on 2017-04-07

remove regex check for post-gc-statusWq

1a801a4... by Maitreyee Saikia on 2017-04-04

Return bool for reset_passwd configuration

4580c7b... by Maitreyee Saikia on 2017-04-04

typo fix

50e9a02... by Maitreyee Saikia on 2017-04-04

merge conflicts

7c480f7... by Maitreyee Saikia on 2017-03-09

Merge from main

3128086... by Maitreyee Saikia on 2017-01-24

Merge conflicts

be30849... by Maitreyee Saikia on 2016-11-21

Merge branch 'topic-msaikia-vmware' of git.launchpad.net:~msaikia/cloud-init into topic-msaikia-vmware

9666992... by Maitreyee Saikia on 2016-10-25

Addressed cloud-init review comments: 1.Changed property names to not have get. 2.No CamelCase for method name.

0e9f7cc... by Maitreyee Saikia on 2016-09-12

Added methods to retrieve new fields from vmware customization spec file

5c2f89e... by Maitreyee Saikia on 2016-11-21

Fixed formatting for vmware config file

diff --git a/cloudinit/sources/DataSourceOVF.py b/cloudinit/sources/DataSourceOVF.py
index d70784a..e4decaa 100644
--- a/cloudinit/sources/DataSourceOVF.py
+++ b/cloudinit/sources/DataSourceOVF.py
@@ -25,6 +25,8 @@ from cloudinit.sources.helpers.vmware.imc.config_file \
     import ConfigFile
 from cloudinit.sources.helpers.vmware.imc.config_nic \
     import NicConfigurator
+from cloudinit.sources.helpers.vmware.imc.config_passwd \
+    import PasswordConfigurator
 from cloudinit.sources.helpers.vmware.imc.guestcust_error \
     import GuestCustErrorEnum
 from cloudinit.sources.helpers.vmware.imc.guestcust_event \
@@ -117,6 +119,8 @@ class DataSourceOVF(sources.DataSource):
                 (md, ud, cfg) = read_vmware_imc(conf)
                 dirpath = os.path.dirname(vmwareImcConfigFilePath)
                 nics = get_nics_to_enable(dirpath)
+                markerid = conf.marker_id
+                markerexists = check_marker_exists(markerid)
             except Exception as e:
                 LOG.debug("Error parsing the customization Config File")
                 LOG.exception(e)
@@ -127,7 +131,6 @@ class DataSourceOVF(sources.DataSource):
                 return False
             finally:
                 util.del_dir(os.path.dirname(vmwareImcConfigFilePath))
-
             try:
                 LOG.debug("Applying the Network customization")
                 nicConfigurator = NicConfigurator(conf.nics)
@@ -140,6 +143,37 @@ class DataSourceOVF(sources.DataSource):
                     GuestCustEventEnum.GUESTCUST_EVENT_NETWORK_SETUP_FAILED)
                 enable_nics(nics)
                 return False
+            if markerid and not markerexists:
+                try:
+                    LOG.debug("Applying password customization")
+                    pwdConfigurator = PasswordConfigurator()
+                    adminpwd = conf.admin_password
+                    resetpwd = conf.reset_password
+                    if adminpwd or resetpwd:
+                        cfg_new = pwdConfigurator.configure(adminpwd, resetpwd)
+                        cfg = util.mergemanydict([cfg, cfg_new])
+                    else:
+                        LOG.debug("Changing password is not needed")
+                except Exception as e:
+                    LOG.debug("Error applying the Password Configuration")
+                    LOG.exception(e)
+                    set_customization_status(
+                        GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
+                        GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)
+                    enable_nics(nics)
+                    return False
+            if markerid:
+                LOG.debug("Handle marker creation")
+                try:
+                    setup_marker_files(markerid)
+                except Exception as e:
+                    LOG.debug("Error creating marker files")
+                    LOG.exception(e)
+                    set_customization_status(
+                        GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
+                        GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)
+                    enable_nics(nics)
+                    return False
 
             vmwarePlatformFound = True
             set_customization_status(
@@ -445,4 +479,23 @@ datasources = (
 def get_datasource_list(depends):
     return sources.list_from_depends(depends, datasources)
 
+
+# To check if marker file exists
+def check_marker_exists(markerid):
+    if not markerid:
+        return False
+    markerfile = "/.markerfile-" + markerid
+    if os.path.exists(markerfile):
+        return True
+    return False
+
+
+# Create a marker file
+def setup_marker_files(markerid):
+    if not markerid:
+        return
+    markerfile = "/.markerfile-" + markerid
+    util.subp(["/bin/rm", "-f", "/.markerfile-*.txt"])
+    util.subp(["/bin/touch", markerfile])
+
 # vi: ts=4 expandtab
diff --git a/cloudinit/sources/helpers/vmware/imc/config.py b/cloudinit/sources/helpers/vmware/imc/config.py
index 9a5e3a8..54e26ae 100644
--- a/cloudinit/sources/helpers/vmware/imc/config.py
+++ b/cloudinit/sources/helpers/vmware/imc/config.py
@@ -5,6 +5,7 @@
 #
 # This file is part of cloud-init. See LICENSE file for license information.
 
+
 from .nic import Nic
 
 
@@ -14,13 +15,17 @@ class Config(object):
     Specification file.
     """
 
+    CUSTOM_SCRIPT = 'CUSTOM-SCRIPT|SCRIPT-NAME'
     DNS = 'DNS|NAMESERVER|'
-    SUFFIX = 'DNS|SUFFIX|'
+    DOMAINNAME = 'NETWORK|DOMAINNAME'
+    HOSTNAME = 'NETWORK|HOSTNAME'
+    MARKERID = 'MISC|MARKER-ID'
     PASS = 'PASSWORD|-PASS'
+    POST_GC = 'MISC|POST-GC-STATUS'
+    RESETPASS = 'PASSWORD|RESET'
+    SUFFIX = 'DNS|SUFFIX|'
     TIMEZONE = 'DATETIME|TIMEZONE'
     UTC = 'DATETIME|UTC'
-    HOSTNAME = 'NETWORK|HOSTNAME'
-    DOMAINNAME = 'NETWORK|DOMAINNAME'
 
     def __init__(self, configFile):
         self._configFile = configFile
@@ -82,4 +87,30 @@ class Config(object):
 
         return res
 
+    @property
+    def reset_password(self):
+        """Retreives if the root password needs to be reset."""
+        resetPass = self._configFile.get(Config.RESETPASS, None)
+        if resetPass and not resetPass.lower() in ('yes', 'no'):
+            raise ValueError('ResetPassword value should be yes/no')
+        return True if resetPass.lower() == 'yes' else False
+
+    @property
+    def marker_id(self):
+        """Returns marker id."""
+        return self._configFile.get(Config.MARKERID, None)
+
+    @property
+    def post_gc_status(self):
+        """Retreives if customization status needs to be posted."""
+        postGcStatus = self._configFile.get(Config.POST_GC, None)
+        if postGcStatus and not postGcStatus.lower() in ('yes', 'no'):
+            raise ValueError('GC status value should be yes/no')
+        return postGcStatus
+
+    @property
+    def custom_script_name(self):
+        """Return the name of custom (pre/post) script."""
+        return self._configFile.get(Config.CUSTOM_SCRIPT, None)
+
 # vi: ts=4 expandtab
diff --git a/cloudinit/sources/helpers/vmware/imc/config_passwd.py b/cloudinit/sources/helpers/vmware/imc/config_passwd.py
new file mode 100644
index 0000000..3812eb0
--- /dev/null
+++ b/cloudinit/sources/helpers/vmware/imc/config_passwd.py
@@ -0,0 +1,140 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2016 Canonical Ltd.
+#    Copyright (C) 2016 VMware INC.
+#
+#    Author: Maitreyee Saikia <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import logging
+import os
+import subprocess
+
+from cloudinit import util
+
+logger = logging.getLogger(__name__)
+
+
+class ProgramNotFound(Exception):
+    pass
+
+
+class Apps(object):
+    """
+    Stores the path of applications needed to expire or set admin passwords.
+    """
+    BASE64 = '/usr/bin/base64'
+    CHAGE = '/usr/bin/chage'
+    CHPASSWD = '/usr/sbin/chpasswd'
+    PASSWD = '/usr/bin/passwd'
+
+
+class PasswordConfigurator(object):
+    """
+    Class for changing configurations related to passwords in a VM. Includes
+    setting and expiring passwords.
+    """
+    def configure(self, passwd, resetPasswd):
+        """
+        Main method to perform all functionalities based on configuration file
+        inputs.
+        @param passwd: encoded admin password.
+        @param resetPasswd: boolean to determine if password needs to be reset.
+        @return cfg: dict to be used by cloud-init set_passwd code.
+        """
+        logger.info('Starting password configuration')
+        if passwd:
+            passwd = self.decode_base64(passwd)
+        allRootUsers = []
+        for line in open('/etc/passwd', 'r'):
+            if line.split(':')[2] == '0':
+                allRootUsers.append(line.split(':')[0])
+        # read shadow file and check for each user, if its uid0 or root.
+        uidUsersList = []
+        rootUsersList = []
+        for line in open('/etc/shadow', 'r'):
+            user = line.split(':')[0]
+            if user in allRootUsers:
+                uidUsersList.append(user)
+                if user == 'root':
+                    rootUsersList.append('%s:%s' % (user, passwd))
+        rootUsers = '\n'.join(rootUsersList) + '\n'
+        cfg = {}
+        if passwd:
+            cfg = self.set_admin_password(rootUsers, resetPasswd)
+        if not passwd and resetPasswd:
+            self.reset_password(uidUsersList)
+        logger.info('Configure Password completed!')
+        return cfg
+
+    def decode_base64(self, passwd):
+        """
+        Method to decode password using base64.
+        Raise error if base64 doesnot exist to decode the password.
+        @param passwd: encoded password.
+        @return: decoded password.
+        """
+        logger.info('Checking BASE64 to decode password.')
+        if not os.path.exists(Apps.BASE64):
+            logger.debug('Base64 is not found! Exiting!')
+            raise ProgramNotFound('Base64 not found! Cant customize password')
+        decode_passwd = ''
+        if len(passwd) > 0:
+            try:
+                p1 = subprocess.Popen(['/bin/echo', passwd],
+                                      stdout=subprocess.PIPE)
+                p2 = subprocess.Popen([Apps.BASE64, '-di'], stdin=p1.stdout,
+                                      stdout=subprocess.PIPE)
+                decode_passwd, _ = p2.communicate()
+            except OSError as e:
+                raise util.ProcessExecutionError(description='Decoding passwd',
+                                                 reason=e, errno=e.errno)
+        logger.info('Decoding password completed')
+        return decode_passwd.decode().strip()
+
+    def set_admin_password(self, rootUsers, resetPasswd):
+        """
+        Method to setup a dict to be used by cloud-init code for configuring
+        password.
+        @param rootUsers: 'root:passwd' string to set.
+               resetPasswd: boolean to expire passwd
+        @return: dictionary of dictionary containing information for setting
+                 admin password. Format as per cloud-init's set passwd method.
+        """
+        logger.info('Setting admin password.')
+        cfg = {}
+        if os.path.exists(Apps.CHPASSWD):
+            cfg['chpasswd'] = {}
+            cfg['chpasswd']['list'] = rootUsers
+            cfg['chpasswd']['expire'] = resetPasswd
+        else:
+            logger.debug('Cannot set admin password. chpasswd not present')
+        return cfg
+
+    def reset_password(self, uidUserList):
+        """
+        Method to reset password. Use passwd --expire command. Use chage if
+        not succeeded using passwd command. Log failure message otherwise.
+        @param: list of users for which to expire password.
+        """
+        logger.info('Expiring password.')
+        for user in uidUserList:
+            try:
+                out, err = util.subp([Apps.PASSWD, '--expire', user])
+            except util.ProcessExecutionError as e:
+                if os.path.exists(Apps.CHAGE):
+                    out, e = util.subp([Apps.CHAGE, '-d', '0', user])
+                else:
+                    logger.debug('Failed to expire password for %s with error:\
+                                  %s' % (user, e))
diff --git a/tests/unittests/test_vmware/test_password_config.py b/tests/unittests/test_vmware/test_password_config.py
new file mode 100644
index 0000000..7418fc8
--- /dev/null
+++ b/tests/unittests/test_vmware/test_password_config.py
@@ -0,0 +1,53 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2015 Canonical Ltd.
+#    Copyright (C) 2016 VMware INC.
+#
+#    Author: Maitreyee Saikia <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import subprocess
+import unittest
+
+from cloudinit.sources.helpers.vmware.imc.config_passwd import (
+    Apps,
+    PasswordConfigurator,
+    ProgramNotFound
+)
+
+
+class TestVMwarePassword(unittest.TestCase):
+
+    def test_decode_passwd(self):
+        passwd = 'TestPass'
+        pwConf = PasswordConfigurator()
+        if os.path.exists(Apps.BASE64):
+            p1 = subprocess.Popen(['echo', passwd], stdout=subprocess.PIPE)
+            p2 = subprocess.Popen([Apps.BASE64], stdin=p1.stdout,
+                                  stdout=subprocess.PIPE)
+            encodedPw, _ = p2.communicate()
+            decodedPw = pwConf.decode_base64(encodedPw.decode().strip())
+            self.assertEqual(passwd, decodedPw)
+        else:
+            self.assertRaises(ProgramNotFound, pwConf.decode_base64, passwd)
+
+    def test_set_admin_pass(self):
+        pwConf = PasswordConfigurator()
+        cfg = pwConf.set_admin_password('root:passwd', False)
+        if os.path.exists(Apps.CHPASSWD):
+            self.assertEqual('root:passwd', cfg['chpasswd']['list'])
+            self.assertFalse(cfg['chpasswd']['expire'])
+        else:
+            self.assertEqual({}, cfg)