Merge ~msaikia/cloud-init:topic-msaikia-vmware into cloud-init:master

Proposed by Maitreyee Saikia
Status: Merged
Approved by: Chad Smith
Approved revision: d9e02b9b1d777445c2a8151f56fa3a1a3ba35e1a
Merge reported by: Chad Smith
Merged at revision: d9e02b9b1d777445c2a8151f56fa3a1a3ba35e1a
Proposed branch: ~msaikia/cloud-init:topic-msaikia-vmware
Merge into: cloud-init:master
Diff against target: 280 lines (+180/-6)
4 files modified
cloudinit/sources/DataSourceOVF.py (+62/-1)
cloudinit/sources/helpers/vmware/imc/config.py (+21/-3)
cloudinit/sources/helpers/vmware/imc/config_passwd.py (+67/-0)
tests/unittests/test_vmware_config_file.py (+30/-2)
Reviewer Review Type Date Requested Status
Chad Smith Approve
Server Team CI bot continuous-integration Approve
Scott Moser Pending
Review via email: mp+322991@code.launchpad.net

This proposal supersedes a proposal from 2016-09-12.

Commit message

vcloud directoy: Guest Customization support for passwords

This feature enables the following VMware VCloud Director functionalities:
1. Setting admin password
2. Expire password.
3. Set admin password and expire.
Password configuration is triggered only as part of a full recustomization, that happens either on first power on or when "poweron and full recustomization" is selected. Full customization flow is determined by marker files. Unique marker ids are generated when full recustomization is requested. And marker file based on these marker ids help to determine if we need to execute the above configuration.

Description of the change

1. Retrieve values from configuration file.
2. Parse data and call password customization.
3. Unit tests for retrieved password data

To post a comment you must log in.
Revision history for this message
Joshua Harlow (harlowja) : Posted in a previous version of this proposal
Revision history for this message
Maitreyee Saikia (msaikia) wrote : Posted in a previous version of this proposal

Thanks for the comments Joshua.
Updated the diff.

Revision history for this message
Maitreyee Saikia (msaikia) wrote : Posted in a previous version of this proposal

Updated some formatting.

Revision history for this message
Scott Moser (smoser) wrote : Posted in a previous version of this proposal

Please set a combined commit message ('Set commit message' above.

Can you explain where you're going also?
I'm concerned about adding more "vmware" paths for configuring things that are done elsewhere (or not elsewhere) in cloud-init. We'd like have consistent paths for doing things as much as possible.

I understand that you're trying to have cloud-init take over another more solution, and I'm not entirely opposed to that, but I want to integrate as much as possible rather than having specific paths and function on vmware.

Revision history for this message
Maitreyee Saikia (msaikia) wrote : Posted in a previous version of this proposal

> Please set a combined commit message ('Set commit message' above.
>
> Can you explain where you're going also?
> I'm concerned about adding more "vmware" paths for configuring things that are
> done elsewhere (or not elsewhere) in cloud-init. We'd like have consistent
> paths for doing things as much as possible.
>
> I understand that you're trying to have cloud-init take over another more
> solution, and I'm not entirely opposed to that, but I want to integrate as
> much as possible rather than having specific paths and function on vmware.

Thanks for your input Scott. I have updated the commit message.
This changeset is just to retrieve data from customization spec, for implementing some other customization functionalities like setting password, running post customization scripts etc. I shall post another changeset with the password configurator, which manipulates the values retrieved from these methods to be used by cloud-init's cc_set_password. We will be using as much cloud-init code as possible. For other functionalities like running pre and post customization specs that will be uploaded by the user, I dont think there is existing support. Please let me know if that sounds ok. Also for functions that are "vmware" specific, shall we still keep it in the helper path that we have created for vmware, or do we want to move them to some common code area?
Thanks
Maitreyee

Revision history for this message
Maitreyee Saikia (msaikia) : Posted in a previous version of this proposal
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal
review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote : Posted in a previous version of this proposal

There is one small nitpick in line.
There isn't anything wrong with this merge proposal, but it
a.) doesn't really do anything
  I realize you're planning on using these things going forward, and thats fine, but there is no reason to accept this MP until we also see that further work. So just add it here. You can rebase/squahs your commits into single function items in a "series" if that helps you.

b.) should add some unit tests.
unit tests will help stop others from breaking your code path inadvertently.

review: Needs Fixing
Revision history for this message
Scott Moser (smoser) wrote : Posted in a previous version of this proposal

Hi. I'm going to put this in "work in progress".
See the comments above. When youv'e addressed them, move it back to Needs Review.
Thanks
Scott

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Maitreyee Saikia (msaikia) wrote : Posted in a previous version of this proposal

Hi Scott
I have resubmitted my review request after addressing your comments.
https://code.launchpad.net/~msaikia/cloud-init/+git/cloud-init/+merge/322991
Please let me know if you get a chance to review the new changeset.

Thanks
Maitreyee

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Tuesday, March 14, 2017 7:25 AM
To: Maitreyee Saikia <email address hidden>
Subject: Re: [Merge] ~msaikia/cloud-init:topic-msaikia-vmware into cloud-init:master

Hi. I'm going to put this in "work in progress".
See the comments above. When youv'e addressed them, move it back to Needs Review.
Thanks
Scott
--
https://code.launchpad.net/~msaikia/cloud-init/+git/cloud-init/+merge/305427
You are the owner of ~msaikia/cloud-init:topic-msaikia-vmware.

Revision history for this message
Scott Moser (smoser) wrote :

Your commit message is probably out of date. Update that please.
Missing from it is at very least a mention of 'markerfile'. I have a feeling you're doing something that might be already done in util.FileSemaphores.

Thats just a quick review.
There are probably some other things.

Thanks!
Scott

a884268... by Maitreyee Saikia

Review comments

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Maitreyee Saikia (msaikia) wrote :

> Your commit message is probably out of date. Update that please.
> Missing from it is at very least a mention of 'markerfile'. I have a feeling
> you're doing something that might be already done in util.FileSemaphores.
>
> Thats just a quick review.
> There are probably some other things.
>
> Thanks!
> Scott

Thanks Scott for the reviews. I have updated the changeset and commit message with your comments. As for the Markerfile, it does not have anything to do with semaphores. Here is an explanation of what our marker file is used for. vCloud Director creates a new marker id every time a full customization is requested. And since a new marker id is generated and we do not have the corresponding marker file as yet, we go forward and execute password configurator and then create the marker file in order that this code execution doesnot happen in future unless a new marker id is generated (new marker id is generated only on first power on or when "power on and force recustomization" is selected). So, I do not feel we can use FileSemaphores here.

Revision history for this message
Scott Moser (smoser) wrote :

i think i'm fine with this. one tiny comment in line.

a080ca6... by Maitreyee Saikia

Addressed review comment to use warning when unable to reset password

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

> i think i'm fine with this. one tiny comment in line.
Addressed. Thanks.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:db962f2e31d9e6acb44828ee27b51a1a9b85a3e3
https://jenkins.ubuntu.com/server/job/cloud-init-ci/115/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/115/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:f9a1fa4f56cde70faf7755c6285ccf61a9a176df
https://jenkins.ubuntu.com/server/job/cloud-init-ci/117/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/117/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Ryan Harper (raharper) wrote :

I believe you need to rebase to master to resolve this issue. You'll likely need to push --force your branch here to overwrite the history because of the rebase.

0531d33... by Maitreyee Saikia

Fix tox errors

2395779... by Maitreyee Saikia

Pull from remote

3da515e... by Maitreyee Saikia

merge conflicts

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:3da515eeaaf5bd591aeee7b950b37b7553af0fcc
https://jenkins.ubuntu.com/server/job/cloud-init-ci/128/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: CentOS 6 & 7: Build & Test

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/128/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

Thanks for the submit and pinging us. Here's a brief review of your code. Most and nits, feel free to object if you feel strongly one way or another. Again thanks for your time on this. I'll watch for updates to see where we get.

4cec8b6... by Maitreyee Saikia

Addressed Chad's review comments

Revision history for this message
Maitreyee Saikia (msaikia) :
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:4cec8b64ce655496e2df894bc3b742571f116a8d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/129/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: CentOS 6 & 7: Build & Test

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/129/rebuild

review: Needs Fixing (continuous-integration)
d9e02b9... by Maitreyee Saikia

Fixed CI test in cloud-init for Centos 6/7 tests

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:d9e02b9b1d777445c2a8151f56fa3a1a3ba35e1a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/130/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/130/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

I'm +1 with this changeset, thanks for the good work.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/cloudinit/sources/DataSourceOVF.py b/cloudinit/sources/DataSourceOVF.py
index f20c9a6..73d3877 100644
--- a/cloudinit/sources/DataSourceOVF.py
+++ b/cloudinit/sources/DataSourceOVF.py
@@ -25,6 +25,8 @@ from cloudinit.sources.helpers.vmware.imc.config_file \
25 import ConfigFile25 import ConfigFile
26from cloudinit.sources.helpers.vmware.imc.config_nic \26from cloudinit.sources.helpers.vmware.imc.config_nic \
27 import NicConfigurator27 import NicConfigurator
28from cloudinit.sources.helpers.vmware.imc.config_passwd \
29 import PasswordConfigurator
28from cloudinit.sources.helpers.vmware.imc.guestcust_error \30from cloudinit.sources.helpers.vmware.imc.guestcust_error \
29 import GuestCustErrorEnum31 import GuestCustErrorEnum
30from cloudinit.sources.helpers.vmware.imc.guestcust_event \32from cloudinit.sources.helpers.vmware.imc.guestcust_event \
@@ -117,6 +119,8 @@ class DataSourceOVF(sources.DataSource):
117 (md, ud, cfg) = read_vmware_imc(conf)119 (md, ud, cfg) = read_vmware_imc(conf)
118 dirpath = os.path.dirname(vmwareImcConfigFilePath)120 dirpath = os.path.dirname(vmwareImcConfigFilePath)
119 nics = get_nics_to_enable(dirpath)121 nics = get_nics_to_enable(dirpath)
122 markerid = conf.marker_id
123 markerexists = check_marker_exists(markerid)
120 except Exception as e:124 except Exception as e:
121 LOG.debug("Error parsing the customization Config File")125 LOG.debug("Error parsing the customization Config File")
122 LOG.exception(e)126 LOG.exception(e)
@@ -127,7 +131,6 @@ class DataSourceOVF(sources.DataSource):
127 return False131 return False
128 finally:132 finally:
129 util.del_dir(os.path.dirname(vmwareImcConfigFilePath))133 util.del_dir(os.path.dirname(vmwareImcConfigFilePath))
130
131 try:134 try:
132 LOG.debug("Applying the Network customization")135 LOG.debug("Applying the Network customization")
133 nicConfigurator = NicConfigurator(conf.nics)136 nicConfigurator = NicConfigurator(conf.nics)
@@ -140,6 +143,35 @@ class DataSourceOVF(sources.DataSource):
140 GuestCustEventEnum.GUESTCUST_EVENT_NETWORK_SETUP_FAILED)143 GuestCustEventEnum.GUESTCUST_EVENT_NETWORK_SETUP_FAILED)
141 enable_nics(nics)144 enable_nics(nics)
142 return False145 return False
146 if markerid and not markerexists:
147 LOG.debug("Applying password customization")
148 pwdConfigurator = PasswordConfigurator()
149 adminpwd = conf.admin_password
150 try:
151 resetpwd = conf.reset_password
152 if adminpwd or resetpwd:
153 pwdConfigurator.configure(adminpwd, resetpwd,
154 self.distro)
155 else:
156 LOG.debug("Changing password is not needed")
157 except Exception as e:
158 LOG.debug("Error applying Password Configuration: %s", e)
159 set_customization_status(
160 GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
161 GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)
162 enable_nics(nics)
163 return False
164 if markerid:
165 LOG.debug("Handle marker creation")
166 try:
167 setup_marker_files(markerid)
168 except Exception as e:
169 LOG.debug("Error creating marker files: %s", e)
170 set_customization_status(
171 GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
172 GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)
173 enable_nics(nics)
174 return False
143175
144 vmwarePlatformFound = True176 vmwarePlatformFound = True
145 set_customization_status(177 set_customization_status(
@@ -445,4 +477,33 @@ datasources = (
445def get_datasource_list(depends):477def get_datasource_list(depends):
446 return sources.list_from_depends(depends, datasources)478 return sources.list_from_depends(depends, datasources)
447479
480
481# To check if marker file exists
482def check_marker_exists(markerid):
483 """
484 Check the existence of a marker file.
485 Presence of marker file determines whether a certain code path is to be
486 executed. It is needed for partial guest customization in VMware.
487 """
488 if not markerid:
489 return False
490 markerfile = "/.markerfile-" + markerid
491 if os.path.exists(markerfile):
492 return True
493 return False
494
495
496# Create a marker file
497def setup_marker_files(markerid):
498 """
499 Create a new marker file.
500 Marker files are unique to a full customization workflow in VMware
501 environment.
502 """
503 if not markerid:
504 return
505 markerfile = "/.markerfile-" + markerid
506 util.del_file("/.markerfile-*.txt")
507 open(markerfile, 'w').close()
508
448# vi: ts=4 expandtab509# vi: ts=4 expandtab
diff --git a/cloudinit/sources/helpers/vmware/imc/config.py b/cloudinit/sources/helpers/vmware/imc/config.py
index 9a5e3a8..49d441d 100644
--- a/cloudinit/sources/helpers/vmware/imc/config.py
+++ b/cloudinit/sources/helpers/vmware/imc/config.py
@@ -5,6 +5,7 @@
5#5#
6# This file is part of cloud-init. See LICENSE file for license information.6# This file is part of cloud-init. See LICENSE file for license information.
77
8
8from .nic import Nic9from .nic import Nic
910
1011
@@ -14,13 +15,16 @@ class Config(object):
14 Specification file.15 Specification file.
15 """16 """
1617
18 CUSTOM_SCRIPT = 'CUSTOM-SCRIPT|SCRIPT-NAME'
17 DNS = 'DNS|NAMESERVER|'19 DNS = 'DNS|NAMESERVER|'
18 SUFFIX = 'DNS|SUFFIX|'20 DOMAINNAME = 'NETWORK|DOMAINNAME'
21 HOSTNAME = 'NETWORK|HOSTNAME'
22 MARKERID = 'MISC|MARKER-ID'
19 PASS = 'PASSWORD|-PASS'23 PASS = 'PASSWORD|-PASS'
24 RESETPASS = 'PASSWORD|RESET'
25 SUFFIX = 'DNS|SUFFIX|'
20 TIMEZONE = 'DATETIME|TIMEZONE'26 TIMEZONE = 'DATETIME|TIMEZONE'
21 UTC = 'DATETIME|UTC'27 UTC = 'DATETIME|UTC'
22 HOSTNAME = 'NETWORK|HOSTNAME'
23 DOMAINNAME = 'NETWORK|DOMAINNAME'
2428
25 def __init__(self, configFile):29 def __init__(self, configFile):
26 self._configFile = configFile30 self._configFile = configFile
@@ -82,4 +86,18 @@ class Config(object):
8286
83 return res87 return res
8488
89 @property
90 def reset_password(self):
91 """Retreives if the root password needs to be reset."""
92 resetPass = self._configFile.get(Config.RESETPASS, 'no')
93 resetPass = resetPass.lower()
94 if resetPass not in ('yes', 'no'):
95 raise ValueError('ResetPassword value should be yes/no')
96 return resetPass == 'yes'
97
98 @property
99 def marker_id(self):
100 """Returns marker id."""
101 return self._configFile.get(Config.MARKERID, None)
102
85# vi: ts=4 expandtab103# vi: ts=4 expandtab
diff --git a/cloudinit/sources/helpers/vmware/imc/config_passwd.py b/cloudinit/sources/helpers/vmware/imc/config_passwd.py
86new file mode 100644104new file mode 100644
index 0000000..75cfbaa
--- /dev/null
+++ b/cloudinit/sources/helpers/vmware/imc/config_passwd.py
@@ -0,0 +1,67 @@
1# Copyright (C) 2016 Canonical Ltd.
2# Copyright (C) 2016 VMware INC.
3#
4# Author: Maitreyee Saikia <msaikia@vmware.com>
5#
6# This file is part of cloud-init. See LICENSE file for license information.
7
8
9import logging
10import os
11
12from cloudinit import util
13
14LOG = logging.getLogger(__name__)
15
16
17class PasswordConfigurator(object):
18 """
19 Class for changing configurations related to passwords in a VM. Includes
20 setting and expiring passwords.
21 """
22 def configure(self, passwd, resetPasswd, distro):
23 """
24 Main method to perform all functionalities based on configuration file
25 inputs.
26 @param passwd: encoded admin password.
27 @param resetPasswd: boolean to determine if password needs to be reset.
28 @return cfg: dict to be used by cloud-init set_passwd code.
29 """
30 LOG.info('Starting password configuration')
31 if passwd:
32 passwd = util.b64d(passwd)
33 allRootUsers = []
34 for line in open('/etc/passwd', 'r'):
35 if line.split(':')[2] == '0':
36 allRootUsers.append(line.split(':')[0])
37 # read shadow file and check for each user, if its uid0 or root.
38 uidUsersList = []
39 for line in open('/etc/shadow', 'r'):
40 user = line.split(':')[0]
41 if user in allRootUsers:
42 uidUsersList.append(user)
43 if passwd:
44 LOG.info('Setting admin password')
45 distro.set_passwd('root', passwd)
46 if resetPasswd:
47 self.reset_password(uidUsersList)
48 LOG.info('Configure Password completed!')
49
50 def reset_password(self, uidUserList):
51 """
52 Method to reset password. Use passwd --expire command. Use chage if
53 not succeeded using passwd command. Log failure message otherwise.
54 @param: list of users for which to expire password.
55 """
56 LOG.info('Expiring password.')
57 for user in uidUserList:
58 try:
59 out, err = util.subp(['passwd', '--expire', user])
60 except util.ProcessExecutionError as e:
61 if os.path.exists('/usr/bin/chage'):
62 out, e = util.subp(['chage', '-d', '0', user])
63 else:
64 LOG.warning('Failed to expire password for %s with error: '
65 '%s', user, e)
66
67# vi: ts=4 expandtab
diff --git a/tests/unittests/test_vmware_config_file.py b/tests/unittests/test_vmware_config_file.py
index 18475f1..03b36d3 100644
--- a/tests/unittests/test_vmware_config_file.py
+++ b/tests/unittests/test_vmware_config_file.py
@@ -7,8 +7,8 @@
77
8import logging8import logging
9import sys9import sys
10import unittest
1110
11from .helpers import CiTestCase
12from cloudinit.sources.helpers.vmware.imc.boot_proto import BootProtoEnum12from cloudinit.sources.helpers.vmware.imc.boot_proto import BootProtoEnum
13from cloudinit.sources.helpers.vmware.imc.config import Config13from cloudinit.sources.helpers.vmware.imc.config import Config
14from cloudinit.sources.helpers.vmware.imc.config_file import ConfigFile14from cloudinit.sources.helpers.vmware.imc.config_file import ConfigFile
@@ -17,7 +17,7 @@ logging.basicConfig(level=logging.DEBUG, stream=sys.stdout)
17logger = logging.getLogger(__name__)17logger = logging.getLogger(__name__)
1818
1919
20class TestVmwareConfigFile(unittest.TestCase):20class TestVmwareConfigFile(CiTestCase):
2121
22 def test_utility_methods(self):22 def test_utility_methods(self):
23 cf = ConfigFile("tests/data/vmware/cust-dhcp-2nic.cfg")23 cf = ConfigFile("tests/data/vmware/cust-dhcp-2nic.cfg")
@@ -90,4 +90,32 @@ class TestVmwareConfigFile(unittest.TestCase):
90 self.assertEqual('00:50:56:a6:8c:08', nics[0].mac, "mac0")90 self.assertEqual('00:50:56:a6:8c:08', nics[0].mac, "mac0")
91 self.assertEqual(BootProtoEnum.DHCP, nics[0].bootProto, "bootproto0")91 self.assertEqual(BootProtoEnum.DHCP, nics[0].bootProto, "bootproto0")
9292
93 def test_config_password(self):
94 cf = ConfigFile("tests/data/vmware/cust-dhcp-2nic.cfg")
95
96 cf._insertKey("PASSWORD|-PASS", "test-password")
97 cf._insertKey("PASSWORD|RESET", "no")
98
99 conf = Config(cf)
100 self.assertEqual('test-password', conf.admin_password, "password")
101 self.assertFalse(conf.reset_password, "do not reset password")
102
103 def test_config_reset_passwd(self):
104 cf = ConfigFile("tests/data/vmware/cust-dhcp-2nic.cfg")
105
106 cf._insertKey("PASSWORD|-PASS", "test-password")
107 cf._insertKey("PASSWORD|RESET", "random")
108
109 conf = Config(cf)
110 with self.assertRaises(ValueError):
111 conf.reset_password()
112
113 cf.clear()
114 cf._insertKey("PASSWORD|RESET", "yes")
115 self.assertEqual(1, len(cf), "insert size")
116
117 conf = Config(cf)
118 self.assertTrue(conf.reset_password, "reset password")
119
120
93# vi: ts=4 expandtab121# vi: ts=4 expandtab

Subscribers

People subscribed via source and target branches