cloud-init

Merge ~msaikia/cloud-init:topic-msaikia-vmware-custom-script into cloud-init:master

  1. Git
  2. lp:~msaikia/cloud-init
  3. topic-msaikia-vmware-custom-script
  4. Merge into master
Proposed by Maitreyee Saikia
Status: Merged
Approved by: Chad Smith
Approved revision: bde436d2775fdc1882a6f08c69aa528c0f638be5
Merged at revision: ce33e423cde806a0590fec635778d62836e1bd37
Proposed branch: ~msaikia/cloud-init:topic-msaikia-vmware-custom-script
Merge into: cloud-init:master
Diff against target: 650 lines (+459/-42)
8 files modified
cloudinit/sources/DataSourceOVF.py (+88/-37)
cloudinit/sources/helpers/vmware/imc/config.py (+4/-0)
cloudinit/sources/helpers/vmware/imc/config_custom_script.py (+153/-0)
cloudinit/sources/helpers/vmware/imc/config_nic.py (+1/-1)
tests/unittests/test_datasource/test_ovf.py (+107/-4)
tests/unittests/test_vmware/__init__.py (+0/-0)
tests/unittests/test_vmware/test_custom_script.py (+99/-0)
tests/unittests/test_vmware_config_file.py (+7/-0)
Reviewer Review Type Date Requested Status
Chad Smith Approve
Server Team CI bot continuous-integration Approve
Review via email: mp+330105@code.launchpad.net

This proposal supersedes a proposal from 2017-08-23.

Commit message

VMware: Support for user provided pre and post-customization scripts

In the VMware customization workflow, we have some options for the user
to upload scripts for additional customization. Based on user request,
those custom scripts can be either run before regular customization or
after. For post customization scripts, we decide whether to run the scripts
just after customization or post system reboot.

Description of the change

Support for users to upload custom scripts in VMware customization workflow.

In the VMware workflow, we have some options for the user to upload custom scripts for additional customization. Based on user request those custom scripts can be either run before regular customization or after. For post customization scripts, we decide whether to run the scripts just after customization or post reboot.

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

PASSED: Continuous integration, rev:dc7653be740c682a325f31162a59dbaf344b8505
https://jenkins.ubuntu.com/server/job/cloud-init-ci/188/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/188/rebuild

review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

PASSED: Continuous integration, rev:a339c450b53246be956c42168bb312d47f7c5598
https://jenkins.ubuntu.com/server/job/cloud-init-ci/191/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/191/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote : Posted in a previous version of this proposal

Thanks for this contribution, I have a couple of notes to get us started here. Will test more on this and respond Monday.

Revision history for this message
Chad Smith (chad.smith) : Posted in a previous version of this proposal
Revision history for this message
Chad Smith (chad.smith) wrote : Posted in a previous version of this proposal

Leaving as needs fixing for the review comments mentioned. Looks good again, thanks for the work here. Please mark this back to "needs review" when comments have been addressed.

review: Needs Fixing
Revision history for this message
Maitreyee Saikia (msaikia) wrote : Posted in a previous version of this proposal

Addressed review comments.
Thanks!

Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

PASSED: Continuous integration, rev:3b626e160697bb83fd1ad58687ac78f62669d48c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/250/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/250/rebuild

review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:3b626e160697bb83fd1ad58687ac78f62669d48c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/251/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/251/rebuild

review: Approve (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
3316786... by Joshua Powers

tox: add nose timer output

This adds the output of the nose timer plugin to the py3 environment to
tox. This will print out the 10 longest running tests and automatically
turn tests longer than 1 second "red" after the coverage output.

b39eb0a... by Joshua Powers

tox: add nose timer output

This adds the output of the nose timer plugin to the py3 environment to
tox. This will print out the 10 longest running tests and automatically
turn tests longer than 1 second "red" after the coverage output.

f759ce5... by Chad Smith

schema and docs: Add jsonschema to resizefs and bootcmd modules

Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.

This branch also does the following:
  - drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
    option only created the specified directory and didn't make use of
that directory for any resize operations.
  - Drop yaml.dumps calls from schema documentation generation to avoid
    yaml import costs on module load
  - Add __doc__ = get_schema_doc(schema) definitions it each module to
    supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
    cc_resizefs
  - Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
    contain more than one example

75d806c... by Chad Smith

schema and docs: Add jsonschema to resizefs and bootcmd modules

Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.

This branch also does the following:
  - drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
    option only created the specified directory and didn't make use of
that directory for any resize operations.
  - Drop yaml.dumps calls from schema documentation generation to avoid
    yaml import costs on module load
  - Add __doc__ = get_schema_doc(schema) definitions it each module to
    supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
    cc_resizefs
  - Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
    contain more than one example

0ea19ba... by Chad Smith

schema and docs: Add jsonschema to resizefs and bootcmd modules

Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.

This branch also does the following:
  - drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
    option only created the specified directory and didn't make use of
that directory for any resize operations.
  - Drop yaml.dumps calls from schema documentation generation to avoid
    yaml import costs on module load
  - Add __doc__ = get_schema_doc(schema) definitions it each module to
    supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
    cc_resizefs
  - Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
    contain more than one example

33f9546... by Lars Kellogg-Stedman

relocate tests/unittests/helpers.py to cloudinit/tests

This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.

9f80a27... by Lars Kellogg-Stedman

relocate tests/unittests/helpers.py to cloudinit/tests

This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.

eeb754d... by Scott Moser

ds-identify: Make OpenStack return maybe on arch other than intel.

OpenStack Nova identifies itself only to Intel guests.
Make ds-identify return 'MAYBE' for OpenStack on non-intel arches.

An unnecessary change here is to rename the 'policy_nodmi' kwarg
to 'policy_no_dmi' in the related unit tests.

LP: #1715241

7f54e3a... by Scott Moser

ds-identify: Make OpenStack return maybe on arch other than intel.

OpenStack Nova identifies itself only to Intel guests.
Make ds-identify return 'MAYBE' for OpenStack on non-intel arches.

An unnecessary change here is to rename the 'policy_nodmi' kwarg
to 'policy_no_dmi' in the related unit tests.

LP: #1715241

78b02e3... by Chad Smith

tests: mock missed openstack metadata uri network_data.json

This missed mock in test_openstack resulted in a costly unit test timeout.

LP: #1714376

83777af... by Chad Smith

tests: mock missed openstack metadata uri network_data.json

This missed mock in test_openstack resulted in a costly unit test timeout.

LP: #1714376

f08b473... by Scott Moser

Ec2: only attempt to operate at local mode on known platforms.

This change makes the DataSourceEc2Local do nothing unless it is on
actual AWS platform. The motivation is twofold:

a.) It is generally safer to only make this function available to Ec2
clones that explicitly identify themselves to the guest. (It also
gives them a reason to supply identification code to cloud-init.)

b.) On non-intel OpenStack platforms ds-identify would enable both the Ec2
and OpenStack sources. That is because there is not good data (such as
dmi) to positively identify the platform. Previously that would be fine
as OpenStack would run first and be successful. The change to add Ec2Local
meant that an Ec2 now runs first.

The best case for 'b' would be a slow down as attempts at the Ec2 metadata
service time out. The discovered case was worse.

Additionally we add a simple check for datatype of 'network' in the
metadata before attempting to read it.

LP: #1715128

1a738fa... by Scott Moser

Ec2: only attempt to operate at local mode on known platforms.

This change makes the DataSourceEc2Local do nothing unless it is on
actual AWS platform. The motivation is twofold:

a.) It is generally safer to only make this function available to Ec2
clones that explicitly identify themselves to the guest. (It also
gives them a reason to supply identification code to cloud-init.)

b.) On non-intel OpenStack platforms ds-identify would enable both the Ec2
and OpenStack sources. That is because there is not good data (such as
dmi) to positively identify the platform. Previously that would be fine
as OpenStack would run first and be successful. The change to add Ec2Local
meant that an Ec2 now runs first.

The best case for 'b' would be a slow down as attempts at the Ec2 metadata
service time out. The discovered case was worse.

Additionally we add a simple check for datatype of 'network' in the
metadata before attempting to read it.

LP: #1715128

5708cb2... by Sankar Tanguturi

vmware customization: return network config format

For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.

This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.

Modify the code to read the customization configuration and return the
converted data.

Added few tests.

LP: #1675063

8336907... by Sankar Tanguturi

vmware customization: return network config format

For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.

This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.

Modify the code to read the customization configuration and return the
converted data.

Added few tests.

LP: #1675063

190dbc7... by Sankar Tanguturi

vmware customization: return network config format

For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.

This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.

Modify the code to read the customization configuration and return the
converted data.

Added few tests.

LP: #1675063

78de454... by Joshua Powers

tests: execute: support command as string

If a string is passed to execute, then invoke 'bash', '-c',
'string'. That allows the less verbose execution of simple
commands:
  image.execute("ls /run")
compared to the more explicit but longer winded:
  image.execute(["ls", "/run"])

If 'env' was ever modified in execute or a method that it called,
then the next invocation's default value would be changed. Instead
use None and then set to a new empty dict in the method.

55e8a71... by Joshua Powers

tests: execute: support command as string

If a string is passed to execute, then invoke 'bash', '-c',
'string'. That allows the less verbose execution of simple
commands:
  image.execute("ls /run")
compared to the more explicit but longer winded:
  image.execute(["ls", "/run"])

If 'env' was ever modified in execute or a method that it called,
then the next invocation's default value would be changed. Instead
use None and then set to a new empty dict in the method.

4ed7511... by Joshua Powers

tools: Add xkvm script, wrapper around qemu-system

The xkvm script will be utilized by pending NoCloud qemu testing.
If this turns out to not be the case, then we will drop it.

37a3d71... by Joshua Powers

tools: Add xkvm script, wrapper around qemu-system

The xkvm script will be utilized by pending NoCloud qemu testing.
If this turns out to not be the case, then we will drop it.

157a632... by Maitreyee Saikia

Rebasing master

45194dd... by Maitreyee Saikia

Rebasing master

cef363b... by Ethan Apodaca

chef: Add option to pin chef omnibus install version

Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.

This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.

LP: #1462693

ae1683b... by Ethan Apodaca

chef: Add option to pin chef omnibus install version

Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.

This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.

LP: #1462693

b57a5d7... by Ethan Apodaca

chef: Add option to pin chef omnibus install version

Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.

This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.

LP: #1462693

f4a2806... by Maitreyee Saikia

Merge branch 'topic-msaikia-vmware-custom-script' of ssh://git.launchpad.net/~msaikia/cloud-init into topic-msaikia-vmware-custom-script

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:f4a2806ff2f0e8cf733335a558ade485f44df67d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/302/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/302/rebuild

review: Approve (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
8179b05... by Maitreyee Saikia

Merge branch 'topic-msaikia-vmware-custom-script' of ssh://git.launchpad.net/~msaikia/cloud-init into topic-msaikia-vmware-custom-script

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:8179b05c5408f22f9a750f1956d05f520ea5a796
https://jenkins.ubuntu.com/server/job/cloud-init-ci/303/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/303/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

Thanks again for your patience here. A couple more review comments and then I *think* we are there.
I'm still seeing merge conflicts with this branch that will likely require a git rebase to sort.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
c664a44... by Maitreyee Saikia

Rebase master and pull from topic branch

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:c664a4482194fcd85627b8d78d0fe1df09d76203
https://jenkins.ubuntu.com/server/job/cloud-init-ci/349/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/349/rebuild

review: Needs Fixing (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
d58e6af... by David Mulford

rh_subscription: Perform null checks for enabled and disabled repos.

The rh_subscription module doesn't perform null checks when attempting to
iterate on the enabled and disable repos arrays. When only one is
specified, cloud-init fails to run.

92e9944... by Maitreyee Saikia

Addressed review comments

7711408... by Maitreyee Saikia

Merge latest from main

a5d82c6... by Maitreyee Saikia

pull from remote branch

8785b65... by Maitreyee Saikia

test fix

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:8785b65633b36535dcaf2bf99ca68a25f2b935bb
https://jenkins.ubuntu.com/server/job/cloud-init-ci/439/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/439/rebuild

review: Approve (continuous-integration)
Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Hi Chad, thanks for taking a look at the review request. I have published a new review request after addressing your concerns. Please also find response to some other comments.
Thanks

Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Maitreyee Saikia (msaikia) :
Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Thanks Chad. Please find the responses below. Will address the rest.

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Thanks Chad for the review. I think the diff comments did not get updated in my previous reply. Re-posting comment.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
3f27def... by Scott Moser

Replace the temporary i9n.brickies.net with i9n.cloud-init.io.

We had used some dns records in i9n.brickies.net (my personal domain)
as a temporary solution until we got names registered in the cloud-init.io
namespace.

We now have CNAME records for:
  ubuntu.i9n.cloud-init.io
  cloudinit1.cloud-init.io
  cloudinit2.cloud-init.io

f530c36... by Robert Schweikert

Improve warning message when a template is not found.

At present the location for the template file look up upon failure
includes the template file itself. However based on the wording of the
message it should only contain the template directory issue

LP: #1731035

4694ebc... by Robert Schweikert

hosts: Fix openSUSE and SLES setup for /etc/hosts and clarify docs.

The etc/hosts file is was not properly setup for openSUSE or SLES
when manage_etc_hosts is set in the config file.

Improve the doc to address the fact that the 'localhost' ip is
distribution dependent (not always 127.0.0.1).

LP: #1731022

Revision history for this message
Chad Smith (chad.smith) :
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
7189c59... by Maitreyee Saikia

Addressed review comments

c773da4... by Maitreyee Saikia

Merge branch 'master' into topic-msaikia-vmware-custom-script

adf16c7... by Maitreyee Saikia

tox error

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:adf16c77ee2b65e464e29baffaecb8b83b88b40d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/481/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/481/rebuild

review: Approve (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
de5d6e9... by Scott Moser

Azure: don't generate network configuration for SRIOV devices

Azure kernel now configures the SRIOV devices itself so cloud-init
does not need to provide any SRIOV device configuration or udev
naming rules.

LP: #1721579

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Addressed review comments.

Revision history for this message
Chad Smith (chad.smith) wrote :

I'm +1 on this now. I know we have cleanup we need to address in the future, but for vmware's pre/post script flow, this current iteration makes sense.

review: Approve
Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Chad Smith (chad.smith) wrote :

Maitreyee, one last pass on this from me. Could you please add a couple unit tests which exercise the OVF get_data execution path with customizationscript present (and erroring) to ensure that behavior is properly handled. (probably looking for tests in tests/unittests/test_datasource/test_ovf.py

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

> Maitreyee, one last pass on this from me. Could you please add a couple unit
> tests which exercise the OVF get_data execution path with customizationscript
> present (and erroring) to ensure that behavior is properly handled. (probably
> looking for tests in tests/unittests/test_datasource/test_ovf.py

Hi Chad,
We have some tests in tests/unittests/test_vmware/test_custom_script.py, where we are testing for cases when customscript is not present.
Do you want me to test the calling function? The calling function does not error out if the variable "customscript" is not set. The if condition here is only to determine whether to go to that workflow or not? If not, we just move to the next configuration. All the error conditions are tested in the actual function, as to whether the correct error is raised or not.
Do you still want me to add tests on the calling function?

Revision history for this message
Chad Smith (chad.smith) wrote :

Thanks Maritreyee for those tests specific to the custom script. And yes please to add a couple tests special_customization and custom_script are set in get_data that it properly execs and handles an error from your custom scripts. This way, we track valid behavior of get_data() so that if we rework VMWare and OVF datasource we can be sure we didn't regress that calling function behavior.

We are trying to get cloud-init up out of the 60% range for unit test coverage so that we can have more certainty that we aren't breaking consumers out there when we add new features or refactor some code.

This is hard I realize as it doesn't seem like there are any get_data tests for OVF datasource currently. Here's a patch reference with a couple things in it for your branch:
1. Some tests for new functions you've added
2. Sample test for OVF.get_data()
3. Shuffle your marker files under /var/lib/cloud as that's where other cloud-init specific semaphores also live. It's better not to leak files into root '/'.

Here's a paste of the start http://paste.ubuntu.com/25963044/

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Thanks for the patch Chad.
Working on the tests now.
But I have a little concern about putting the marker file in /var/lib/cloud.
What if someone cleans up the entire /var/lib/cloud?
In that case, we will lose the marker file for the next run, and as such will
go through the entire workflow, which is not what we expect. Is there any way to handle that?

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
679b393... by Maitreyee Saikia

Adding unit tests for Datasource_OVF

Revision history for this message
Chad Smith (chad.smith) wrote :

Maitreyee, I believe that is exactly the behavior we want to be common across all of cloud-init. Right now, if /var/lib/cloud is cleaned up and the machine is rebooted, we expect cloud-init to re-perform all configuration steps as if it is run on against clean system. In fact we are adding a cloud-init clean commandline function to do just that:

https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/333513

Also, per filesystem heirarchy standard, we also shouldn't be leaking files to other directories beyond what the cloud-init package 'owns'. It's okay if we haven't yet sorted the details of leaving around vmware marker files (or cleaning up former config changes due to those customizations. This lack of 'cleanup' handling is something that cloud-init will have to tackle for ssh keys or users created which may have been left on the system by the first cloud-init run, and may need removing before reboot.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
50a025c... by Maitreyee Saikia

Merge branch 'master' into topic-msaikia-vmware-custom-script

582c798... by Maitreyee Saikia

Fix spacing in unittest

0cb4226... by Maitreyee Saikia

Fix spacing in unittest

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:582c798a3afabe7bac739378897d877c815aacbe
https://jenkins.ubuntu.com/server/job/cloud-init-ci/540/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/540/rebuild

review: Needs Fixing (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
da14017... by Maitreyee Saikia

Merge branch 'topic-msaikia-vmware-custom-script' of ssh://git.launchpad.net/~msaikia/cloud-init into topic-msaikia-vmware-custom-script

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:da14017dec15a4b787f62bb4ca40342bcd2c3454
https://jenkins.ubuntu.com/server/job/cloud-init-ci/541/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/541/rebuild

review: Approve (continuous-integration)
Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Thanks Chad,
I have addressed your comments.
Please take a look at the review request.
Regards,

> Maitreyee, I believe that is exactly the behavior we want to be common across
> all of cloud-init. Right now, if /var/lib/cloud is cleaned up and the machine
> is rebooted, we expect cloud-init to re-perform all configuration steps as if
> it is run on against clean system. In fact we are adding a cloud-init clean
> commandline function to do just that:
>
> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-
> init/+merge/333513
>
> Also, per filesystem heirarchy standard, we also shouldn't be leaking files to
> other directories beyond what the cloud-init package 'owns'. It's okay if we
> haven't yet sorted the details of leaving around vmware marker files (or
> cleaning up former config changes due to those customizations. This lack of
> 'cleanup' handling is something that cloud-init will have to tackle for ssh
> keys or users created which may have been left on the system by the first
> cloud-init run, and may need removing before reboot.

Revision history for this message
Chad Smith (chad.smith) wrote :

just a couple of tiny comments, thanks for adding the unit tests for datasourceovf.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
2f2070f... by Maitreyee Saikia

test_ovf review comments

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:2f2070fdafe04f2fa12657354b8a0cfefa775d56
https://jenkins.ubuntu.com/server/job/cloud-init-ci/558/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/558/rebuild

review: Approve (continuous-integration)
Revision history for this message
Maitreyee Saikia (msaikia) wrote :

> just a couple of tiny comments, thanks for adding the unit tests for
> datasourceovf.

Thanks. Updated new diff.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
028d945... by Maitreyee Saikia

Return error in older config_nic file

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:028d945d24b080428acfdeb10116d0ea68de62e4
https://jenkins.ubuntu.com/server/job/cloud-init-ci/578/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/578/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Chad Smith (chad.smith) wrote :

Thank you for all the work here Maitreyee. One minor comment above about where to store the marker files after discussion with the team.

I've tested your content on non-vmware kvm instances and validated that it doesn't break other consumer paths as well. Thank you again for the good work. If you agree to the above final change suggestion, I'll merge this proposal.

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

> Thank you for all the work here Maitreyee. One minor comment above about where
> to store the marker files after discussion with the team.
>
> I've tested your content on non-vmware kvm instances and validated that it
> doesn't break other consumer paths as well. Thank you again for the good work.
> If you agree to the above final change suggestion, I'll merge this proposal.

Thank you Chad.
I am ok with the suggestion.
Should I push a new diff?

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
06624c9... by Maitreyee Saikia

Merge branch 'master' into topic-msaikia-vmware-custom-script

bde436d... by Maitreyee Saikia

Move marker file to /var/lib/cloud/data

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:bde436d2775fdc1882a6f08c69aa528c0f638be5
https://jenkins.ubuntu.com/server/job/cloud-init-ci/588/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/588/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

Validated this branch against a non-vmware OVF kvm instance and the logic didn't adversely affect non-vmware paths. We should land this.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/cloudinit/sources/DataSourceOVF.py b/cloudinit/sources/DataSourceOVF.py
index ccebf11..489c583 100644
--- a/cloudinit/sources/DataSourceOVF.py
+++ b/cloudinit/sources/DataSourceOVF.py
@@ -21,6 +21,8 @@ from cloudinit import util
2121
22from cloudinit.sources.helpers.vmware.imc.config \22from cloudinit.sources.helpers.vmware.imc.config \
23 import Config23 import Config
24from cloudinit.sources.helpers.vmware.imc.config_custom_script \
25 import PreCustomScript, PostCustomScript
24from cloudinit.sources.helpers.vmware.imc.config_file \26from cloudinit.sources.helpers.vmware.imc.config_file \
25 import ConfigFile27 import ConfigFile
26from cloudinit.sources.helpers.vmware.imc.config_nic \28from cloudinit.sources.helpers.vmware.imc.config_nic \
@@ -30,7 +32,7 @@ from cloudinit.sources.helpers.vmware.imc.config_passwd \
30from cloudinit.sources.helpers.vmware.imc.guestcust_error \32from cloudinit.sources.helpers.vmware.imc.guestcust_error \
31 import GuestCustErrorEnum33 import GuestCustErrorEnum
32from cloudinit.sources.helpers.vmware.imc.guestcust_event \34from cloudinit.sources.helpers.vmware.imc.guestcust_event \
33 import GuestCustEventEnum35 import GuestCustEventEnum as GuestCustEvent
34from cloudinit.sources.helpers.vmware.imc.guestcust_state \36from cloudinit.sources.helpers.vmware.imc.guestcust_state \
35 import GuestCustStateEnum37 import GuestCustStateEnum
36from cloudinit.sources.helpers.vmware.imc.guestcust_util import (38from cloudinit.sources.helpers.vmware.imc.guestcust_util import (
@@ -124,17 +126,31 @@ class DataSourceOVF(sources.DataSource):
124 self._vmware_cust_conf = Config(cf)126 self._vmware_cust_conf = Config(cf)
125 (md, ud, cfg) = read_vmware_imc(self._vmware_cust_conf)127 (md, ud, cfg) = read_vmware_imc(self._vmware_cust_conf)
126 self._vmware_nics_to_enable = get_nics_to_enable(nicspath)128 self._vmware_nics_to_enable = get_nics_to_enable(nicspath)
127 markerid = self._vmware_cust_conf.marker_id129 imcdirpath = os.path.dirname(vmwareImcConfigFilePath)
128 markerexists = check_marker_exists(markerid)130 product_marker = self._vmware_cust_conf.marker_id
131 hasmarkerfile = check_marker_exists(
132 product_marker, os.path.join(self.paths.cloud_dir, 'data'))
133 special_customization = product_marker and not hasmarkerfile
134 customscript = self._vmware_cust_conf.custom_script_name
129 except Exception as e:135 except Exception as e:
130 LOG.debug("Error parsing the customization Config File")136 _raise_error_status(
131 LOG.exception(e)137 "Error parsing the customization Config File",
132 set_customization_status(138 e,
133 GuestCustStateEnum.GUESTCUST_STATE_RUNNING,139 GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
134 GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)140 vmwareImcConfigFilePath)
135 raise e141
136 finally:142 if special_customization:
137 util.del_dir(os.path.dirname(vmwareImcConfigFilePath))143 if customscript:
144 try:
145 precust = PreCustomScript(customscript, imcdirpath)
146 precust.execute()
147 except Exception as e:
148 _raise_error_status(
149 "Error executing pre-customization script",
150 e,
151 GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
152 vmwareImcConfigFilePath)
153
138 try:154 try:
139 LOG.debug("Preparing the Network configuration")155 LOG.debug("Preparing the Network configuration")
140 self._network_config = get_network_config_from_conf(156 self._network_config = get_network_config_from_conf(
@@ -143,13 +159,13 @@ class DataSourceOVF(sources.DataSource):
143 True,159 True,
144 self.distro.osfamily)160 self.distro.osfamily)
145 except Exception as e:161 except Exception as e:
146 LOG.exception(e)162 _raise_error_status(
147 set_customization_status(163 "Error preparing Network Configuration",
148 GuestCustStateEnum.GUESTCUST_STATE_RUNNING,164 e,
149 GuestCustEventEnum.GUESTCUST_EVENT_NETWORK_SETUP_FAILED)165 GuestCustEvent.GUESTCUST_EVENT_NETWORK_SETUP_FAILED,
150 raise e166 vmwareImcConfigFilePath)
151167
152 if markerid and not markerexists:168 if special_customization:
153 LOG.debug("Applying password customization")169 LOG.debug("Applying password customization")
154 pwdConfigurator = PasswordConfigurator()170 pwdConfigurator = PasswordConfigurator()
155 adminpwd = self._vmware_cust_conf.admin_password171 adminpwd = self._vmware_cust_conf.admin_password
@@ -161,27 +177,41 @@ class DataSourceOVF(sources.DataSource):
161 else:177 else:
162 LOG.debug("Changing password is not needed")178 LOG.debug("Changing password is not needed")
163 except Exception as e:179 except Exception as e:
164 LOG.debug("Error applying Password Configuration: %s", e)180 _raise_error_status(
165 set_customization_status(181 "Error applying Password Configuration",
166 GuestCustStateEnum.GUESTCUST_STATE_RUNNING,182 e,
167 GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)183 GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
168 return False184 vmwareImcConfigFilePath)
169 if markerid:185
170 LOG.debug("Handle marker creation")186 if customscript:
187 try:
188 postcust = PostCustomScript(customscript, imcdirpath)
189 postcust.execute()
190 except Exception as e:
191 _raise_error_status(
192 "Error executing post-customization script",
193 e,
194 GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
195 vmwareImcConfigFilePath)
196
197 if product_marker:
171 try:198 try:
172 setup_marker_files(markerid)199 setup_marker_files(
200 product_marker,
201 os.path.join(self.paths.cloud_dir, 'data'))
173 except Exception as e:202 except Exception as e:
174 LOG.debug("Error creating marker files: %s", e)203 _raise_error_status(
175 set_customization_status(204 "Error creating marker files",
176 GuestCustStateEnum.GUESTCUST_STATE_RUNNING,205 e,
177 GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)206 GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
178 return False207 vmwareImcConfigFilePath)
179208
180 self._vmware_cust_found = True209 self._vmware_cust_found = True
181 found.append('vmware-tools')210 found.append('vmware-tools')
182211
183 # TODO: Need to set the status to DONE only when the212 # TODO: Need to set the status to DONE only when the
184 # customization is done successfully.213 # customization is done successfully.
214 util.del_dir(os.path.dirname(vmwareImcConfigFilePath))
185 enable_nics(self._vmware_nics_to_enable)215 enable_nics(self._vmware_nics_to_enable)
186 set_customization_status(216 set_customization_status(
187 GuestCustStateEnum.GUESTCUST_STATE_DONE,217 GuestCustStateEnum.GUESTCUST_STATE_DONE,
@@ -536,31 +566,52 @@ def get_datasource_list(depends):
536566
537567
538# To check if marker file exists568# To check if marker file exists
539def check_marker_exists(markerid):569def check_marker_exists(markerid, marker_dir):
540 """570 """
541 Check the existence of a marker file.571 Check the existence of a marker file.
542 Presence of marker file determines whether a certain code path is to be572 Presence of marker file determines whether a certain code path is to be
543 executed. It is needed for partial guest customization in VMware.573 executed. It is needed for partial guest customization in VMware.
574 @param markerid: is an unique string representing a particular product
575 marker.
576 @param: marker_dir: The directory in which markers exist.
544 """577 """
545 if not markerid:578 if not markerid:
546 return False579 return False
547 markerfile = "/.markerfile-" + markerid580 markerfile = os.path.join(marker_dir, ".markerfile-" + markerid + ".txt")
548 if os.path.exists(markerfile):581 if os.path.exists(markerfile):
549 return True582 return True
550 return False583 return False
551584
552585
553# Create a marker file586# Create a marker file
554def setup_marker_files(markerid):587def setup_marker_files(markerid, marker_dir):
555 """588 """
556 Create a new marker file.589 Create a new marker file.
557 Marker files are unique to a full customization workflow in VMware590 Marker files are unique to a full customization workflow in VMware
558 environment.591 environment.
592 @param markerid: is an unique string representing a particular product
593 marker.
594 @param: marker_dir: The directory in which markers exist.
595
559 """596 """
560 if not markerid:597 LOG.debug("Handle marker creation")
561 return598 markerfile = os.path.join(marker_dir, ".markerfile-" + markerid + ".txt")
562 markerfile = "/.markerfile-" + markerid599 for fname in os.listdir(marker_dir):
563 util.del_file("/.markerfile-*.txt")600 if fname.startswith(".markerfile"):
601 util.del_file(os.path.join(marker_dir, fname))
564 open(markerfile, 'w').close()602 open(markerfile, 'w').close()
565603
604
605def _raise_error_status(prefix, error, event, config_file):
606 """
607 Raise error and send customization status to the underlying VMware
608 Virtualization Platform. Also, cleanup the imc directory.
609 """
610 LOG.debug('%s: %s', prefix, error)
611 set_customization_status(
612 GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
613 event)
614 util.del_dir(os.path.dirname(config_file))
615 raise error
616
566# vi: ts=4 expandtab617# vi: ts=4 expandtab
diff --git a/cloudinit/sources/helpers/vmware/imc/config.py b/cloudinit/sources/helpers/vmware/imc/config.py
index 49d441d..2eaeff3 100644
--- a/cloudinit/sources/helpers/vmware/imc/config.py
+++ b/cloudinit/sources/helpers/vmware/imc/config.py
@@ -100,4 +100,8 @@ class Config(object):
100 """Returns marker id."""100 """Returns marker id."""
101 return self._configFile.get(Config.MARKERID, None)101 return self._configFile.get(Config.MARKERID, None)
102102
103 @property
104 def custom_script_name(self):
105 """Return the name of custom (pre/post) script."""
106 return self._configFile.get(Config.CUSTOM_SCRIPT, None)
103# vi: ts=4 expandtab107# vi: ts=4 expandtab
diff --git a/cloudinit/sources/helpers/vmware/imc/config_custom_script.py b/cloudinit/sources/helpers/vmware/imc/config_custom_script.py
104new file mode 100644108new file mode 100644
index 0000000..a7d4ad9
--- /dev/null
+++ b/cloudinit/sources/helpers/vmware/imc/config_custom_script.py
@@ -0,0 +1,153 @@
1# Copyright (C) 2017 Canonical Ltd.
2# Copyright (C) 2017 VMware Inc.
3#
4# Author: Maitreyee Saikia <msaikia@vmware.com>
5#
6# This file is part of cloud-init. See LICENSE file for license information.
7
8import logging
9import os
10import stat
11from textwrap import dedent
12
13from cloudinit import util
14
15LOG = logging.getLogger(__name__)
16
17
18class CustomScriptNotFound(Exception):
19 pass
20
21
22class CustomScriptConstant(object):
23 RC_LOCAL = "/etc/rc.local"
24 POST_CUST_TMP_DIR = "/root/.customization"
25 POST_CUST_RUN_SCRIPT_NAME = "post-customize-guest.sh"
26 POST_CUST_RUN_SCRIPT = os.path.join(POST_CUST_TMP_DIR,
27 POST_CUST_RUN_SCRIPT_NAME)
28 POST_REBOOT_PENDING_MARKER = "/.guest-customization-post-reboot-pending"
29
30
31class RunCustomScript(object):
32 def __init__(self, scriptname, directory):
33 self.scriptname = scriptname
34 self.directory = directory
35 self.scriptpath = os.path.join(directory, scriptname)
36
37 def prepare_script(self):
38 if not os.path.exists(self.scriptpath):
39 raise CustomScriptNotFound("Script %s not found!! "
40 "Cannot execute custom script!"
41 % self.scriptpath)
42 # Strip any CR characters from the decoded script
43 util.load_file(self.scriptpath).replace("\r", "")
44 st = os.stat(self.scriptpath)
45 os.chmod(self.scriptpath, st.st_mode | stat.S_IEXEC)
46
47
48class PreCustomScript(RunCustomScript):
49 def execute(self):
50 """Executing custom script with precustomization argument."""
51 LOG.debug("Executing pre-customization script")
52 self.prepare_script()
53 util.subp(["/bin/sh", self.scriptpath, "precustomization"])
54
55
56class PostCustomScript(RunCustomScript):
57 def __init__(self, scriptname, directory):
58 super(PostCustomScript, self).__init__(scriptname, directory)
59 # Determine when to run custom script. When postreboot is True,
60 # the user uploaded script will run as part of rc.local after
61 # the machine reboots. This is determined by presence of rclocal.
62 # When postreboot is False, script will run as part of cloud-init.
63 self.postreboot = False
64
65 def _install_post_reboot_agent(self, rclocal):
66 """
67 Install post-reboot agent for running custom script after reboot.
68 As part of this process, we are editing the rclocal file to run a
69 VMware script, which in turn is resposible for handling the user
70 script.
71 @param: path to rc local.
72 """
73 LOG.debug("Installing post-reboot customization from %s to %s",
74 self.directory, rclocal)
75 if not self.has_previous_agent(rclocal):
76 LOG.info("Adding post-reboot customization agent to rc.local")
77 new_content = dedent("""
78 # Run post-reboot guest customization
79 /bin/sh %s
80 exit 0
81 """) % CustomScriptConstant.POST_CUST_RUN_SCRIPT
82 existing_rclocal = util.load_file(rclocal).replace('exit 0\n', '')
83 st = os.stat(rclocal)
84 # "x" flag should be set
85 mode = st.st_mode | stat.S_IEXEC
86 util.write_file(rclocal, existing_rclocal + new_content, mode)
87
88 else:
89 # We don't need to update rclocal file everytime a customization
90 # is requested. It just needs to be done for the first time.
91 LOG.info("Post-reboot guest customization agent is already "
92 "registered in rc.local")
93 LOG.debug("Installing post-reboot customization agent finished: %s",
94 self.postreboot)
95
96 def has_previous_agent(self, rclocal):
97 searchstring = "# Run post-reboot guest customization"
98 if searchstring in open(rclocal).read():
99 return True
100 return False
101
102 def find_rc_local(self):
103 """
104 Determine if rc local is present.
105 """
106 rclocal = ""
107 if os.path.exists(CustomScriptConstant.RC_LOCAL):
108 LOG.debug("rc.local detected.")
109 # resolving in case of symlink
110 rclocal = os.path.realpath(CustomScriptConstant.RC_LOCAL)
111 LOG.debug("rc.local resolved to %s", rclocal)
112 else:
113 LOG.warning("Can't find rc.local, post-customization "
114 "will be run before reboot")
115 return rclocal
116
117 def install_agent(self):
118 rclocal = self.find_rc_local()
119 if rclocal:
120 self._install_post_reboot_agent(rclocal)
121 self.postreboot = True
122
123 def execute(self):
124 """
125 This method executes post-customization script before or after reboot
126 based on the presence of rc local.
127 """
128 self.prepare_script()
129 self.install_agent()
130 if not self.postreboot:
131 LOG.warning("Executing post-customization script inline")
132 util.subp(["/bin/sh", self.scriptpath, "postcustomization"])
133 else:
134 LOG.debug("Scheduling custom script to run post reboot")
135 if not os.path.isdir(CustomScriptConstant.POST_CUST_TMP_DIR):
136 os.mkdir(CustomScriptConstant.POST_CUST_TMP_DIR)
137 # Script "post-customize-guest.sh" and user uploaded script are
138 # are present in the same directory and needs to copied to a temp
139 # directory to be executed post reboot. User uploaded script is
140 # saved as customize.sh in the temp directory.
141 # post-customize-guest.sh excutes customize.sh after reboot.
142 LOG.debug("Copying post-customization script")
143 util.copy(self.scriptpath,
144 CustomScriptConstant.POST_CUST_TMP_DIR + "/customize.sh")
145 LOG.debug("Copying script to run post-customization script")
146 util.copy(
147 os.path.join(self.directory,
148 CustomScriptConstant.POST_CUST_RUN_SCRIPT_NAME),
149 CustomScriptConstant.POST_CUST_RUN_SCRIPT)
150 LOG.info("Creating post-reboot pending marker")
151 util.ensure_file(CustomScriptConstant.POST_REBOOT_PENDING_MARKER)
152
153# vi: ts=4 expandtab
diff --git a/cloudinit/sources/helpers/vmware/imc/config_nic.py b/cloudinit/sources/helpers/vmware/imc/config_nic.py
index 2fb07c5..2d8900e 100644
--- a/cloudinit/sources/helpers/vmware/imc/config_nic.py
+++ b/cloudinit/sources/helpers/vmware/imc/config_nic.py
@@ -161,7 +161,7 @@ class NicConfigurator(object):
161 if nic.primary and v4.gateways:161 if nic.primary and v4.gateways:
162 self.ipv4PrimaryGateway = v4.gateways[0]162 self.ipv4PrimaryGateway = v4.gateways[0]
163 subnet.update({'gateway': self.ipv4PrimaryGateway})163 subnet.update({'gateway': self.ipv4PrimaryGateway})
164 return [subnet]164 return ([subnet], route_list)
165165
166 # Add routes if there is no primary nic166 # Add routes if there is no primary nic
167 if not self._primaryNic:167 if not self._primaryNic:
diff --git a/tests/unittests/test_datasource/test_ovf.py b/tests/unittests/test_datasource/test_ovf.py
index 700da86..fc4eb36 100644
--- a/tests/unittests/test_datasource/test_ovf.py
+++ b/tests/unittests/test_datasource/test_ovf.py
@@ -5,11 +5,17 @@
5# This file is part of cloud-init. See LICENSE file for license information.5# This file is part of cloud-init. See LICENSE file for license information.
66
7import base647import base64
8from collections import OrderedDict8import os
99
10from cloudinit.tests import helpers as test_helpers10from collections import OrderedDict
11from textwrap import dedent
1112
13from cloudinit import util
14from cloudinit.tests.helpers import CiTestCase, wrap_and_call
15from cloudinit.helpers import Paths
12from cloudinit.sources import DataSourceOVF as dsovf16from cloudinit.sources import DataSourceOVF as dsovf
17from cloudinit.sources.helpers.vmware.imc.config_custom_script import (
18 CustomScriptNotFound)
1319
14OVF_ENV_CONTENT = """<?xml version="1.0" encoding="UTF-8"?>20OVF_ENV_CONTENT = """<?xml version="1.0" encoding="UTF-8"?>
15<Environment xmlns="http://schemas.dmtf.org/ovf/environment/1"21<Environment xmlns="http://schemas.dmtf.org/ovf/environment/1"
@@ -42,7 +48,7 @@ def fill_properties(props, template=OVF_ENV_CONTENT):
42 return template.format(properties=properties)48 return template.format(properties=properties)
4349
4450
45class TestReadOvfEnv(test_helpers.TestCase):51class TestReadOvfEnv(CiTestCase):
46 def test_with_b64_userdata(self):52 def test_with_b64_userdata(self):
47 user_data = "#!/bin/sh\necho hello world\n"53 user_data = "#!/bin/sh\necho hello world\n"
48 user_data_b64 = base64.b64encode(user_data.encode()).decode()54 user_data_b64 = base64.b64encode(user_data.encode()).decode()
@@ -72,7 +78,104 @@ class TestReadOvfEnv(test_helpers.TestCase):
72 self.assertIsNone(ud)78 self.assertIsNone(ud)
7379
7480
75class TestTransportIso9660(test_helpers.CiTestCase):81class TestMarkerFiles(CiTestCase):
82
83 def setUp(self):
84 super(TestMarkerFiles, self).setUp()
85 self.tdir = self.tmp_dir()
86
87 def test_false_when_markerid_none(self):
88 """Return False when markerid provided is None."""
89 self.assertFalse(
90 dsovf.check_marker_exists(markerid=None, marker_dir=self.tdir))
91
92 def test_markerid_file_exist(self):
93 """Return False when markerid file path does not exist,
94 True otherwise."""
95 self.assertFalse(
96 dsovf.check_marker_exists('123', self.tdir))
97
98 marker_file = self.tmp_path('.markerfile-123.txt', self.tdir)
99 util.write_file(marker_file, '')
100 self.assertTrue(
101 dsovf.check_marker_exists('123', self.tdir)
102 )
103
104 def test_marker_file_setup(self):
105 """Test creation of marker files."""
106 markerfilepath = self.tmp_path('.markerfile-hi.txt', self.tdir)
107 self.assertFalse(os.path.exists(markerfilepath))
108 dsovf.setup_marker_files(markerid='hi', marker_dir=self.tdir)
109 self.assertTrue(os.path.exists(markerfilepath))
110
111
112class TestDatasourceOVF(CiTestCase):
113
114 with_logs = True
115
116 def setUp(self):
117 super(TestDatasourceOVF, self).setUp()
118 self.datasource = dsovf.DataSourceOVF
119 self.tdir = self.tmp_dir()
120
121 def test_get_data_false_on_none_dmi_data(self):
122 """When dmi for system-product-name is None, get_data returns False."""
123 paths = Paths({'seed_dir': self.tdir})
124 ds = self.datasource(sys_cfg={}, distro={}, paths=paths)
125 retcode = wrap_and_call(
126 'cloudinit.sources.DataSourceOVF',
127 {'util.read_dmi_data': None},
128 ds.get_data)
129 self.assertFalse(retcode, 'Expected False return from ds.get_data')
130 self.assertIn(
131 'DEBUG: No system-product-name found', self.logs.getvalue())
132
133 def test_get_data_no_vmware_customization_disabled(self):
134 """When vmware customization is disabled via sys_cfg log a message."""
135 paths = Paths({'seed_dir': self.tdir})
136 ds = self.datasource(
137 sys_cfg={'disable_vmware_customization': True}, distro={},
138 paths=paths)
139 retcode = wrap_and_call(
140 'cloudinit.sources.DataSourceOVF',
141 {'util.read_dmi_data': 'vmware'},
142 ds.get_data)
143 self.assertFalse(retcode, 'Expected False return from ds.get_data')
144 self.assertIn(
145 'DEBUG: Customization for VMware platform is disabled.',
146 self.logs.getvalue())
147
148 def test_get_data_vmware_customization_disabled(self):
149 """When cloud-init workflow for vmware is enabled via sys_cfg log a
150 message.
151 """
152 paths = Paths({'seed_dir': self.tdir})
153 ds = self.datasource(
154 sys_cfg={'disable_vmware_customization': False}, distro={},
155 paths=paths)
156 conf_file = self.tmp_path('test-cust', self.tdir)
157 conf_content = dedent("""\
158 [CUSTOM-SCRIPT]
159 SCRIPT-NAME = test-script
160 [MISC]
161 MARKER-ID = 12345345
162 """)
163 util.write_file(conf_file, conf_content)
164 with self.assertRaises(CustomScriptNotFound) as context:
165 wrap_and_call(
166 'cloudinit.sources.DataSourceOVF',
167 {'util.read_dmi_data': 'vmware',
168 'util.del_dir': True,
169 'search_file': self.tdir,
170 'wait_for_imc_cfg_file': conf_file,
171 'get_nics_to_enable': ''},
172 ds.get_data)
173 customscript = self.tmp_path('test-script', self.tdir)
174 self.assertIn('Script %s not found!!' % customscript,
175 str(context.exception))
176
177
178class TestTransportIso9660(CiTestCase):
76179
77 def setUp(self):180 def setUp(self):
78 super(TestTransportIso9660, self).setUp()181 super(TestTransportIso9660, self).setUp()
diff --git a/tests/unittests/test_vmware/__init__.py b/tests/unittests/test_vmware/__init__.py
79new file mode 100644182new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/tests/unittests/test_vmware/__init__.py
diff --git a/tests/unittests/test_vmware/test_custom_script.py b/tests/unittests/test_vmware/test_custom_script.py
80new file mode 100644183new file mode 100644
index 0000000..2d9519b
--- /dev/null
+++ b/tests/unittests/test_vmware/test_custom_script.py
@@ -0,0 +1,99 @@
1# Copyright (C) 2015 Canonical Ltd.
2# Copyright (C) 2017 VMware INC.
3#
4# Author: Maitreyee Saikia <msaikia@vmware.com>
5#
6# This file is part of cloud-init. See LICENSE file for license information.
7
8from cloudinit import util
9from cloudinit.sources.helpers.vmware.imc.config_custom_script import (
10 CustomScriptConstant,
11 CustomScriptNotFound,
12 PreCustomScript,
13 PostCustomScript,
14)
15from cloudinit.tests.helpers import CiTestCase, mock
16
17
18class TestVmwareCustomScript(CiTestCase):
19 def setUp(self):
20 self.tmpDir = self.tmp_dir()
21
22 def test_prepare_custom_script(self):
23 """
24 This test is designed to verify the behavior based on the presence of
25 custom script. Mainly needed for scenario where a custom script is
26 expected, but was not properly copied. "CustomScriptNotFound" exception
27 is raised in such cases.
28 """
29 # Custom script does not exist.
30 preCust = PreCustomScript("random-vmw-test", self.tmpDir)
31 self.assertEqual("random-vmw-test", preCust.scriptname)
32 self.assertEqual(self.tmpDir, preCust.directory)
33 self.assertEqual(self.tmp_path("random-vmw-test", self.tmpDir),
34 preCust.scriptpath)
35 with self.assertRaises(CustomScriptNotFound):
36 preCust.prepare_script()
37
38 # Custom script exists.
39 custScript = self.tmp_path("test-cust", self.tmpDir)
40 util.write_file(custScript, "test-CR-strip/r/r")
41 postCust = PostCustomScript("test-cust", self.tmpDir)
42 self.assertEqual("test-cust", postCust.scriptname)
43 self.assertEqual(self.tmpDir, postCust.directory)
44 self.assertEqual(custScript, postCust.scriptpath)
45 self.assertFalse(postCust.postreboot)
46 postCust.prepare_script()
47 # Check if all carraige returns are stripped from script.
48 self.assertFalse("/r" in custScript)
49
50 def test_rc_local_exists(self):
51 """
52 This test is designed to verify the different scenarios associated
53 with the presence of rclocal.
54 """
55 # test when rc local does not exist
56 postCust = PostCustomScript("test-cust", self.tmpDir)
57 with mock.patch.object(CustomScriptConstant, "RC_LOCAL", "/no/path"):
58 rclocal = postCust.find_rc_local()
59 self.assertEqual("", rclocal)
60
61 # test when rc local exists
62 rclocalFile = self.tmp_path("vmware-rclocal", self.tmpDir)
63 util.write_file(rclocalFile, "# Run post-reboot guest customization",
64 omode="w")
65 with mock.patch.object(CustomScriptConstant, "RC_LOCAL", rclocalFile):
66 rclocal = postCust.find_rc_local()
67 self.assertEqual(rclocalFile, rclocal)
68 self.assertTrue(postCust.has_previous_agent, rclocal)
69
70 # test when rc local is a symlink
71 rclocalLink = self.tmp_path("dummy-rclocal-link", self.tmpDir)
72 util.sym_link(rclocalFile, rclocalLink, True)
73 with mock.patch.object(CustomScriptConstant, "RC_LOCAL", rclocalLink):
74 rclocal = postCust.find_rc_local()
75 self.assertEqual(rclocalFile, rclocal)
76
77 def test_execute_post_cust(self):
78 """
79 This test is to identify if rclocal was properly populated to be
80 run after reboot.
81 """
82 customscript = self.tmp_path("vmware-post-cust-script", self.tmpDir)
83 rclocal = self.tmp_path("vmware-rclocal", self.tmpDir)
84 # Create a temporary rclocal file
85 open(customscript, "w")
86 util.write_file(rclocal, "tests\nexit 0", omode="w")
87 postCust = PostCustomScript("vmware-post-cust-script", self.tmpDir)
88 with mock.patch.object(CustomScriptConstant, "RC_LOCAL", rclocal):
89 # Test that guest customization agent is not installed initially.
90 self.assertFalse(postCust.postreboot)
91 self.assertIs(postCust.has_previous_agent(rclocal), False)
92 postCust.install_agent()
93
94 # Assert rclocal has been modified to have guest customization
95 # agent.
96 self.assertTrue(postCust.postreboot)
97 self.assertTrue(postCust.has_previous_agent, rclocal)
98
99# vi: ts=4 expandtab
diff --git a/tests/unittests/test_vmware_config_file.py b/tests/unittests/test_vmware_config_file.py
index 808d303..455d6c2 100644
--- a/tests/unittests/test_vmware_config_file.py
+++ b/tests/unittests/test_vmware_config_file.py
@@ -334,5 +334,12 @@ class TestVmwareConfigFile(CiTestCase):
334 self.assertEqual('255.255.0.0', subnet.get('netmask'),334 self.assertEqual('255.255.0.0', subnet.get('netmask'),
335 'Subnet netmask')335 'Subnet netmask')
336336
337 def test_custom_script(self):
338 cf = ConfigFile("tests/data/vmware/cust-dhcp-2nic.cfg")
339 conf = Config(cf)
340 self.assertIsNone(conf.custom_script_name)
341 cf._insertKey("CUSTOM-SCRIPT|SCRIPT-NAME", "test-script")
342 conf = Config(cf)
343 self.assertEqual("test-script", conf.custom_script_name)
337344
338# vi: ts=4 expandtab345# vi: ts=4 expandtab

Subscribers

People subscribed via source and target branches