cloud-init

Merge ~msaikia/cloud-init:topic-msaikia-vmware-custom-script into cloud-init:master

  1. Git
  2. lp:~msaikia/cloud-init
  3. topic-msaikia-vmware-custom-script
  4. Merge into master
Proposed by Maitreyee Saikia
Status: Merged
Approved by: Chad Smith
Approved revision: bde436d2775fdc1882a6f08c69aa528c0f638be5
Merged at revision: ce33e423cde806a0590fec635778d62836e1bd37
Proposed branch: ~msaikia/cloud-init:topic-msaikia-vmware-custom-script
Merge into: cloud-init:master
Diff against target: 650 lines (+459/-42)
8 files modified
cloudinit/sources/DataSourceOVF.py (+88/-37)
cloudinit/sources/helpers/vmware/imc/config.py (+4/-0)
cloudinit/sources/helpers/vmware/imc/config_custom_script.py (+153/-0)
cloudinit/sources/helpers/vmware/imc/config_nic.py (+1/-1)
tests/unittests/test_datasource/test_ovf.py (+107/-4)
tests/unittests/test_vmware/__init__.py (+0/-0)
tests/unittests/test_vmware/test_custom_script.py (+99/-0)
tests/unittests/test_vmware_config_file.py (+7/-0)
Reviewer Review Type Date Requested Status
Chad Smith Approve
Server Team CI bot continuous-integration Approve
Review via email: mp+330105@code.launchpad.net

This proposal supersedes a proposal from 2017-08-23.

Commit message

VMware: Support for user provided pre and post-customization scripts

In the VMware customization workflow, we have some options for the user
to upload scripts for additional customization. Based on user request,
those custom scripts can be either run before regular customization or
after. For post customization scripts, we decide whether to run the scripts
just after customization or post system reboot.

Description of the change

Support for users to upload custom scripts in VMware customization workflow.

In the VMware workflow, we have some options for the user to upload custom scripts for additional customization. Based on user request those custom scripts can be either run before regular customization or after. For post customization scripts, we decide whether to run the scripts just after customization or post reboot.

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

PASSED: Continuous integration, rev:dc7653be740c682a325f31162a59dbaf344b8505
https://jenkins.ubuntu.com/server/job/cloud-init-ci/188/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/188/rebuild

review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

PASSED: Continuous integration, rev:a339c450b53246be956c42168bb312d47f7c5598
https://jenkins.ubuntu.com/server/job/cloud-init-ci/191/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/191/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote : Posted in a previous version of this proposal

Thanks for this contribution, I have a couple of notes to get us started here. Will test more on this and respond Monday.

Revision history for this message
Chad Smith (chad.smith) : Posted in a previous version of this proposal
Revision history for this message
Chad Smith (chad.smith) wrote : Posted in a previous version of this proposal

Leaving as needs fixing for the review comments mentioned. Looks good again, thanks for the work here. Please mark this back to "needs review" when comments have been addressed.

review: Needs Fixing
Revision history for this message
Maitreyee Saikia (msaikia) wrote : Posted in a previous version of this proposal

Addressed review comments.
Thanks!

Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

PASSED: Continuous integration, rev:3b626e160697bb83fd1ad58687ac78f62669d48c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/250/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/250/rebuild

review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:3b626e160697bb83fd1ad58687ac78f62669d48c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/251/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/251/rebuild

review: Approve (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
3316786... by Joshua Powers

tox: add nose timer output

This adds the output of the nose timer plugin to the py3 environment to
tox. This will print out the 10 longest running tests and automatically
turn tests longer than 1 second "red" after the coverage output.

b39eb0a... by Joshua Powers

tox: add nose timer output

This adds the output of the nose timer plugin to the py3 environment to
tox. This will print out the 10 longest running tests and automatically
turn tests longer than 1 second "red" after the coverage output.

f759ce5... by Chad Smith

schema and docs: Add jsonschema to resizefs and bootcmd modules

Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.

This branch also does the following:
  - drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
    option only created the specified directory and didn't make use of
that directory for any resize operations.
  - Drop yaml.dumps calls from schema documentation generation to avoid
    yaml import costs on module load
  - Add __doc__ = get_schema_doc(schema) definitions it each module to
    supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
    cc_resizefs
  - Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
    contain more than one example

75d806c... by Chad Smith

schema and docs: Add jsonschema to resizefs and bootcmd modules

Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.

This branch also does the following:
  - drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
    option only created the specified directory and didn't make use of
that directory for any resize operations.
  - Drop yaml.dumps calls from schema documentation generation to avoid
    yaml import costs on module load
  - Add __doc__ = get_schema_doc(schema) definitions it each module to
    supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
    cc_resizefs
  - Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
    contain more than one example

0ea19ba... by Chad Smith

schema and docs: Add jsonschema to resizefs and bootcmd modules

Add schema definitions to both cc_resizefs and cc_bootcmd modules. Extend
schema.py to parse and document enumerated json types. Schema definitions
are used to generate module documention and log warnings for schema
infractions.

This branch also does the following:
  - drops vestigial 'resize_rootfs_tmp' option from cc_resizefs. That
    option only created the specified directory and didn't make use of
that directory for any resize operations.
  - Drop yaml.dumps calls from schema documentation generation to avoid
    yaml import costs on module load
  - Add __doc__ = get_schema_doc(schema) definitions it each module to
    supplement python help() calls for cc_runcmd, cc_bootcmd, cc_ntp and
    cc_resizefs
  - Add a SCHEMA_EXAMPLES_SPACER_TEMPLATE string to docs for modules which
    contain more than one example

33f9546... by Lars Kellogg-Stedman

relocate tests/unittests/helpers.py to cloudinit/tests

This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.

9f80a27... by Lars Kellogg-Stedman

relocate tests/unittests/helpers.py to cloudinit/tests

This moves the base test case classes into into cloudinit/tests and
updates all the corresponding imports.

eeb754d... by Scott Moser

ds-identify: Make OpenStack return maybe on arch other than intel.

OpenStack Nova identifies itself only to Intel guests.
Make ds-identify return 'MAYBE' for OpenStack on non-intel arches.

An unnecessary change here is to rename the 'policy_nodmi' kwarg
to 'policy_no_dmi' in the related unit tests.

LP: #1715241

7f54e3a... by Scott Moser

ds-identify: Make OpenStack return maybe on arch other than intel.

OpenStack Nova identifies itself only to Intel guests.
Make ds-identify return 'MAYBE' for OpenStack on non-intel arches.

An unnecessary change here is to rename the 'policy_nodmi' kwarg
to 'policy_no_dmi' in the related unit tests.

LP: #1715241

78b02e3... by Chad Smith

tests: mock missed openstack metadata uri network_data.json

This missed mock in test_openstack resulted in a costly unit test timeout.

LP: #1714376

83777af... by Chad Smith

tests: mock missed openstack metadata uri network_data.json

This missed mock in test_openstack resulted in a costly unit test timeout.

LP: #1714376

f08b473... by Scott Moser

Ec2: only attempt to operate at local mode on known platforms.

This change makes the DataSourceEc2Local do nothing unless it is on
actual AWS platform. The motivation is twofold:

a.) It is generally safer to only make this function available to Ec2
clones that explicitly identify themselves to the guest. (It also
gives them a reason to supply identification code to cloud-init.)

b.) On non-intel OpenStack platforms ds-identify would enable both the Ec2
and OpenStack sources. That is because there is not good data (such as
dmi) to positively identify the platform. Previously that would be fine
as OpenStack would run first and be successful. The change to add Ec2Local
meant that an Ec2 now runs first.

The best case for 'b' would be a slow down as attempts at the Ec2 metadata
service time out. The discovered case was worse.

Additionally we add a simple check for datatype of 'network' in the
metadata before attempting to read it.

LP: #1715128

1a738fa... by Scott Moser

Ec2: only attempt to operate at local mode on known platforms.

This change makes the DataSourceEc2Local do nothing unless it is on
actual AWS platform. The motivation is twofold:

a.) It is generally safer to only make this function available to Ec2
clones that explicitly identify themselves to the guest. (It also
gives them a reason to supply identification code to cloud-init.)

b.) On non-intel OpenStack platforms ds-identify would enable both the Ec2
and OpenStack sources. That is because there is not good data (such as
dmi) to positively identify the platform. Previously that would be fine
as OpenStack would run first and be successful. The change to add Ec2Local
meant that an Ec2 now runs first.

The best case for 'b' would be a slow down as attempts at the Ec2 metadata
service time out. The discovered case was worse.

Additionally we add a simple check for datatype of 'network' in the
metadata before attempting to read it.

LP: #1715128

5708cb2... by Sankar Tanguturi

vmware customization: return network config format

For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.

This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.

Modify the code to read the customization configuration and return the
converted data.

Added few tests.

LP: #1675063

8336907... by Sankar Tanguturi

vmware customization: return network config format

For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.

This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.

Modify the code to read the customization configuration and return the
converted data.

Added few tests.

LP: #1675063

190dbc7... by Sankar Tanguturi

vmware customization: return network config format

For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.

This branch also fixes /etc/network/interfaces replacing the line
"source /etc/network/interfaces.d/*.cfg" which is incorrectly removed
when VMWare's Perl Customization Engine writes /etc/network/interfaces.

Modify the code to read the customization configuration and return the
converted data.

Added few tests.

LP: #1675063

78de454... by Joshua Powers

tests: execute: support command as string

If a string is passed to execute, then invoke 'bash', '-c',
'string'. That allows the less verbose execution of simple
commands:
  image.execute("ls /run")
compared to the more explicit but longer winded:
  image.execute(["ls", "/run"])

If 'env' was ever modified in execute or a method that it called,
then the next invocation's default value would be changed. Instead
use None and then set to a new empty dict in the method.

55e8a71... by Joshua Powers

tests: execute: support command as string

If a string is passed to execute, then invoke 'bash', '-c',
'string'. That allows the less verbose execution of simple
commands:
  image.execute("ls /run")
compared to the more explicit but longer winded:
  image.execute(["ls", "/run"])

If 'env' was ever modified in execute or a method that it called,
then the next invocation's default value would be changed. Instead
use None and then set to a new empty dict in the method.

4ed7511... by Joshua Powers

tools: Add xkvm script, wrapper around qemu-system

The xkvm script will be utilized by pending NoCloud qemu testing.
If this turns out to not be the case, then we will drop it.

37a3d71... by Joshua Powers

tools: Add xkvm script, wrapper around qemu-system

The xkvm script will be utilized by pending NoCloud qemu testing.
If this turns out to not be the case, then we will drop it.

157a632... by Maitreyee Saikia

Rebasing master

45194dd... by Maitreyee Saikia

Rebasing master

cef363b... by Ethan Apodaca

chef: Add option to pin chef omnibus install version

Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.

This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.

LP: #1462693

ae1683b... by Ethan Apodaca

chef: Add option to pin chef omnibus install version

Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.

This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.

LP: #1462693

b57a5d7... by Ethan Apodaca

chef: Add option to pin chef omnibus install version

Most users of chef will want to pin the version that is installed.
Typically new versions of chef have to be evaluated for breakage etc.

This change proposes a new optional `omnibus_version` field to the chef
configuration. The changeset also adds documentation referencing the new
field.

LP: #1462693

f4a2806... by Maitreyee Saikia

Merge branch 'topic-msaikia-vmware-custom-script' of ssh://git.launchpad.net/~msaikia/cloud-init into topic-msaikia-vmware-custom-script

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:f4a2806ff2f0e8cf733335a558ade485f44df67d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/302/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/302/rebuild

review: Approve (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
8179b05... by Maitreyee Saikia

Merge branch 'topic-msaikia-vmware-custom-script' of ssh://git.launchpad.net/~msaikia/cloud-init into topic-msaikia-vmware-custom-script

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:8179b05c5408f22f9a750f1956d05f520ea5a796
https://jenkins.ubuntu.com/server/job/cloud-init-ci/303/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/303/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

Thanks again for your patience here. A couple more review comments and then I *think* we are there.
I'm still seeing merge conflicts with this branch that will likely require a git rebase to sort.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
c664a44... by Maitreyee Saikia

Rebase master and pull from topic branch

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:c664a4482194fcd85627b8d78d0fe1df09d76203
https://jenkins.ubuntu.com/server/job/cloud-init-ci/349/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/349/rebuild

review: Needs Fixing (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
d58e6af... by David Mulford

rh_subscription: Perform null checks for enabled and disabled repos.

The rh_subscription module doesn't perform null checks when attempting to
iterate on the enabled and disable repos arrays. When only one is
specified, cloud-init fails to run.

92e9944... by Maitreyee Saikia

Addressed review comments

7711408... by Maitreyee Saikia

Merge latest from main

a5d82c6... by Maitreyee Saikia

pull from remote branch

8785b65... by Maitreyee Saikia

test fix

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:8785b65633b36535dcaf2bf99ca68a25f2b935bb
https://jenkins.ubuntu.com/server/job/cloud-init-ci/439/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/439/rebuild

review: Approve (continuous-integration)
Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Hi Chad, thanks for taking a look at the review request. I have published a new review request after addressing your concerns. Please also find response to some other comments.
Thanks

Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Maitreyee Saikia (msaikia) :
Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Thanks Chad. Please find the responses below. Will address the rest.

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Thanks Chad for the review. I think the diff comments did not get updated in my previous reply. Re-posting comment.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
3f27def... by Scott Moser

Replace the temporary i9n.brickies.net with i9n.cloud-init.io.

We had used some dns records in i9n.brickies.net (my personal domain)
as a temporary solution until we got names registered in the cloud-init.io
namespace.

We now have CNAME records for:
  ubuntu.i9n.cloud-init.io
  cloudinit1.cloud-init.io
  cloudinit2.cloud-init.io

f530c36... by Robert Schweikert

Improve warning message when a template is not found.

At present the location for the template file look up upon failure
includes the template file itself. However based on the wording of the
message it should only contain the template directory issue

LP: #1731035

4694ebc... by Robert Schweikert

hosts: Fix openSUSE and SLES setup for /etc/hosts and clarify docs.

The etc/hosts file is was not properly setup for openSUSE or SLES
when manage_etc_hosts is set in the config file.

Improve the doc to address the fact that the 'localhost' ip is
distribution dependent (not always 127.0.0.1).

LP: #1731022

Revision history for this message
Chad Smith (chad.smith) :
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
7189c59... by Maitreyee Saikia

Addressed review comments

c773da4... by Maitreyee Saikia

Merge branch 'master' into topic-msaikia-vmware-custom-script

adf16c7... by Maitreyee Saikia

tox error

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:adf16c77ee2b65e464e29baffaecb8b83b88b40d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/481/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/481/rebuild

review: Approve (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
de5d6e9... by Scott Moser

Azure: don't generate network configuration for SRIOV devices

Azure kernel now configures the SRIOV devices itself so cloud-init
does not need to provide any SRIOV device configuration or udev
naming rules.

LP: #1721579

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Addressed review comments.

Revision history for this message
Chad Smith (chad.smith) wrote :

I'm +1 on this now. I know we have cleanup we need to address in the future, but for vmware's pre/post script flow, this current iteration makes sense.

review: Approve
Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Chad Smith (chad.smith) wrote :

Maitreyee, one last pass on this from me. Could you please add a couple unit tests which exercise the OVF get_data execution path with customizationscript present (and erroring) to ensure that behavior is properly handled. (probably looking for tests in tests/unittests/test_datasource/test_ovf.py

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

> Maitreyee, one last pass on this from me. Could you please add a couple unit
> tests which exercise the OVF get_data execution path with customizationscript
> present (and erroring) to ensure that behavior is properly handled. (probably
> looking for tests in tests/unittests/test_datasource/test_ovf.py

Hi Chad,
We have some tests in tests/unittests/test_vmware/test_custom_script.py, where we are testing for cases when customscript is not present.
Do you want me to test the calling function? The calling function does not error out if the variable "customscript" is not set. The if condition here is only to determine whether to go to that workflow or not? If not, we just move to the next configuration. All the error conditions are tested in the actual function, as to whether the correct error is raised or not.
Do you still want me to add tests on the calling function?

Revision history for this message
Chad Smith (chad.smith) wrote :

Thanks Maritreyee for those tests specific to the custom script. And yes please to add a couple tests special_customization and custom_script are set in get_data that it properly execs and handles an error from your custom scripts. This way, we track valid behavior of get_data() so that if we rework VMWare and OVF datasource we can be sure we didn't regress that calling function behavior.

We are trying to get cloud-init up out of the 60% range for unit test coverage so that we can have more certainty that we aren't breaking consumers out there when we add new features or refactor some code.

This is hard I realize as it doesn't seem like there are any get_data tests for OVF datasource currently. Here's a patch reference with a couple things in it for your branch:
1. Some tests for new functions you've added
2. Sample test for OVF.get_data()
3. Shuffle your marker files under /var/lib/cloud as that's where other cloud-init specific semaphores also live. It's better not to leak files into root '/'.

Here's a paste of the start http://paste.ubuntu.com/25963044/

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Thanks for the patch Chad.
Working on the tests now.
But I have a little concern about putting the marker file in /var/lib/cloud.
What if someone cleans up the entire /var/lib/cloud?
In that case, we will lose the marker file for the next run, and as such will
go through the entire workflow, which is not what we expect. Is there any way to handle that?

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
679b393... by Maitreyee Saikia

Adding unit tests for Datasource_OVF

Revision history for this message
Chad Smith (chad.smith) wrote :

Maitreyee, I believe that is exactly the behavior we want to be common across all of cloud-init. Right now, if /var/lib/cloud is cleaned up and the machine is rebooted, we expect cloud-init to re-perform all configuration steps as if it is run on against clean system. In fact we are adding a cloud-init clean commandline function to do just that:

https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/333513

Also, per filesystem heirarchy standard, we also shouldn't be leaking files to other directories beyond what the cloud-init package 'owns'. It's okay if we haven't yet sorted the details of leaving around vmware marker files (or cleaning up former config changes due to those customizations. This lack of 'cleanup' handling is something that cloud-init will have to tackle for ssh keys or users created which may have been left on the system by the first cloud-init run, and may need removing before reboot.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
50a025c... by Maitreyee Saikia

Merge branch 'master' into topic-msaikia-vmware-custom-script

582c798... by Maitreyee Saikia

Fix spacing in unittest

0cb4226... by Maitreyee Saikia

Fix spacing in unittest

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:582c798a3afabe7bac739378897d877c815aacbe
https://jenkins.ubuntu.com/server/job/cloud-init-ci/540/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/540/rebuild

review: Needs Fixing (continuous-integration)
~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
da14017... by Maitreyee Saikia

Merge branch 'topic-msaikia-vmware-custom-script' of ssh://git.launchpad.net/~msaikia/cloud-init into topic-msaikia-vmware-custom-script

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:da14017dec15a4b787f62bb4ca40342bcd2c3454
https://jenkins.ubuntu.com/server/job/cloud-init-ci/541/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/541/rebuild

review: Approve (continuous-integration)
Revision history for this message
Maitreyee Saikia (msaikia) wrote :

Thanks Chad,
I have addressed your comments.
Please take a look at the review request.
Regards,

> Maitreyee, I believe that is exactly the behavior we want to be common across
> all of cloud-init. Right now, if /var/lib/cloud is cleaned up and the machine
> is rebooted, we expect cloud-init to re-perform all configuration steps as if
> it is run on against clean system. In fact we are adding a cloud-init clean
> commandline function to do just that:
>
> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-
> init/+merge/333513
>
> Also, per filesystem heirarchy standard, we also shouldn't be leaking files to
> other directories beyond what the cloud-init package 'owns'. It's okay if we
> haven't yet sorted the details of leaving around vmware marker files (or
> cleaning up former config changes due to those customizations. This lack of
> 'cleanup' handling is something that cloud-init will have to tackle for ssh
> keys or users created which may have been left on the system by the first
> cloud-init run, and may need removing before reboot.

Revision history for this message
Chad Smith (chad.smith) wrote :

just a couple of tiny comments, thanks for adding the unit tests for datasourceovf.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
2f2070f... by Maitreyee Saikia

test_ovf review comments

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:2f2070fdafe04f2fa12657354b8a0cfefa775d56
https://jenkins.ubuntu.com/server/job/cloud-init-ci/558/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/558/rebuild

review: Approve (continuous-integration)
Revision history for this message
Maitreyee Saikia (msaikia) wrote :

> just a couple of tiny comments, thanks for adding the unit tests for
> datasourceovf.

Thanks. Updated new diff.

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
028d945... by Maitreyee Saikia

Return error in older config_nic file

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:028d945d24b080428acfdeb10116d0ea68de62e4
https://jenkins.ubuntu.com/server/job/cloud-init-ci/578/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/578/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Chad Smith (chad.smith) wrote :

Thank you for all the work here Maitreyee. One minor comment above about where to store the marker files after discussion with the team.

I've tested your content on non-vmware kvm instances and validated that it doesn't break other consumer paths as well. Thank you again for the good work. If you agree to the above final change suggestion, I'll merge this proposal.

Revision history for this message
Maitreyee Saikia (msaikia) wrote :

> Thank you for all the work here Maitreyee. One minor comment above about where
> to store the marker files after discussion with the team.
>
> I've tested your content on non-vmware kvm instances and validated that it
> doesn't break other consumer paths as well. Thank you again for the good work.
> If you agree to the above final change suggestion, I'll merge this proposal.

Thank you Chad.
I am ok with the suggestion.
Should I push a new diff?

~msaikia/cloud-init:topic-msaikia-vmware-custom-script updated
06624c9... by Maitreyee Saikia

Merge branch 'master' into topic-msaikia-vmware-custom-script

bde436d... by Maitreyee Saikia

Move marker file to /var/lib/cloud/data

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:bde436d2775fdc1882a6f08c69aa528c0f638be5
https://jenkins.ubuntu.com/server/job/cloud-init-ci/588/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/588/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

Validated this branch against a non-vmware OVF kvm instance and the logic didn't adversely affect non-vmware paths. We should land this.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/cloudinit/sources/DataSourceOVF.py b/cloudinit/sources/DataSourceOVF.py
2index ccebf11..489c583 100644
3--- a/cloudinit/sources/DataSourceOVF.py
4+++ b/cloudinit/sources/DataSourceOVF.py
5@@ -21,6 +21,8 @@ from cloudinit import util
6
7 from cloudinit.sources.helpers.vmware.imc.config \
8 import Config
9+from cloudinit.sources.helpers.vmware.imc.config_custom_script \
10+ import PreCustomScript, PostCustomScript
11 from cloudinit.sources.helpers.vmware.imc.config_file \
12 import ConfigFile
13 from cloudinit.sources.helpers.vmware.imc.config_nic \
14@@ -30,7 +32,7 @@ from cloudinit.sources.helpers.vmware.imc.config_passwd \
15 from cloudinit.sources.helpers.vmware.imc.guestcust_error \
16 import GuestCustErrorEnum
17 from cloudinit.sources.helpers.vmware.imc.guestcust_event \
18- import GuestCustEventEnum
19+ import GuestCustEventEnum as GuestCustEvent
20 from cloudinit.sources.helpers.vmware.imc.guestcust_state \
21 import GuestCustStateEnum
22 from cloudinit.sources.helpers.vmware.imc.guestcust_util import (
23@@ -124,17 +126,31 @@ class DataSourceOVF(sources.DataSource):
24 self._vmware_cust_conf = Config(cf)
25 (md, ud, cfg) = read_vmware_imc(self._vmware_cust_conf)
26 self._vmware_nics_to_enable = get_nics_to_enable(nicspath)
27- markerid = self._vmware_cust_conf.marker_id
28- markerexists = check_marker_exists(markerid)
29+ imcdirpath = os.path.dirname(vmwareImcConfigFilePath)
30+ product_marker = self._vmware_cust_conf.marker_id
31+ hasmarkerfile = check_marker_exists(
32+ product_marker, os.path.join(self.paths.cloud_dir, 'data'))
33+ special_customization = product_marker and not hasmarkerfile
34+ customscript = self._vmware_cust_conf.custom_script_name
35 except Exception as e:
36- LOG.debug("Error parsing the customization Config File")
37- LOG.exception(e)
38- set_customization_status(
39- GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
40- GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)
41- raise e
42- finally:
43- util.del_dir(os.path.dirname(vmwareImcConfigFilePath))
44+ _raise_error_status(
45+ "Error parsing the customization Config File",
46+ e,
47+ GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
48+ vmwareImcConfigFilePath)
49+
50+ if special_customization:
51+ if customscript:
52+ try:
53+ precust = PreCustomScript(customscript, imcdirpath)
54+ precust.execute()
55+ except Exception as e:
56+ _raise_error_status(
57+ "Error executing pre-customization script",
58+ e,
59+ GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
60+ vmwareImcConfigFilePath)
61+
62 try:
63 LOG.debug("Preparing the Network configuration")
64 self._network_config = get_network_config_from_conf(
65@@ -143,13 +159,13 @@ class DataSourceOVF(sources.DataSource):
66 True,
67 self.distro.osfamily)
68 except Exception as e:
69- LOG.exception(e)
70- set_customization_status(
71- GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
72- GuestCustEventEnum.GUESTCUST_EVENT_NETWORK_SETUP_FAILED)
73- raise e
74+ _raise_error_status(
75+ "Error preparing Network Configuration",
76+ e,
77+ GuestCustEvent.GUESTCUST_EVENT_NETWORK_SETUP_FAILED,
78+ vmwareImcConfigFilePath)
79
80- if markerid and not markerexists:
81+ if special_customization:
82 LOG.debug("Applying password customization")
83 pwdConfigurator = PasswordConfigurator()
84 adminpwd = self._vmware_cust_conf.admin_password
85@@ -161,27 +177,41 @@ class DataSourceOVF(sources.DataSource):
86 else:
87 LOG.debug("Changing password is not needed")
88 except Exception as e:
89- LOG.debug("Error applying Password Configuration: %s", e)
90- set_customization_status(
91- GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
92- GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)
93- return False
94- if markerid:
95- LOG.debug("Handle marker creation")
96+ _raise_error_status(
97+ "Error applying Password Configuration",
98+ e,
99+ GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
100+ vmwareImcConfigFilePath)
101+
102+ if customscript:
103+ try:
104+ postcust = PostCustomScript(customscript, imcdirpath)
105+ postcust.execute()
106+ except Exception as e:
107+ _raise_error_status(
108+ "Error executing post-customization script",
109+ e,
110+ GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
111+ vmwareImcConfigFilePath)
112+
113+ if product_marker:
114 try:
115- setup_marker_files(markerid)
116+ setup_marker_files(
117+ product_marker,
118+ os.path.join(self.paths.cloud_dir, 'data'))
119 except Exception as e:
120- LOG.debug("Error creating marker files: %s", e)
121- set_customization_status(
122- GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
123- GuestCustEventEnum.GUESTCUST_EVENT_CUSTOMIZE_FAILED)
124- return False
125+ _raise_error_status(
126+ "Error creating marker files",
127+ e,
128+ GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED,
129+ vmwareImcConfigFilePath)
130
131 self._vmware_cust_found = True
132 found.append('vmware-tools')
133
134 # TODO: Need to set the status to DONE only when the
135 # customization is done successfully.
136+ util.del_dir(os.path.dirname(vmwareImcConfigFilePath))
137 enable_nics(self._vmware_nics_to_enable)
138 set_customization_status(
139 GuestCustStateEnum.GUESTCUST_STATE_DONE,
140@@ -536,31 +566,52 @@ def get_datasource_list(depends):
141
142
143 # To check if marker file exists
144-def check_marker_exists(markerid):
145+def check_marker_exists(markerid, marker_dir):
146 """
147 Check the existence of a marker file.
148 Presence of marker file determines whether a certain code path is to be
149 executed. It is needed for partial guest customization in VMware.
150+ @param markerid: is an unique string representing a particular product
151+ marker.
152+ @param: marker_dir: The directory in which markers exist.
153 """
154 if not markerid:
155 return False
156- markerfile = "/.markerfile-" + markerid
157+ markerfile = os.path.join(marker_dir, ".markerfile-" + markerid + ".txt")
158 if os.path.exists(markerfile):
159 return True
160 return False
161
162
163 # Create a marker file
164-def setup_marker_files(markerid):
165+def setup_marker_files(markerid, marker_dir):
166 """
167 Create a new marker file.
168 Marker files are unique to a full customization workflow in VMware
169 environment.
170+ @param markerid: is an unique string representing a particular product
171+ marker.
172+ @param: marker_dir: The directory in which markers exist.
173+
174 """
175- if not markerid:
176- return
177- markerfile = "/.markerfile-" + markerid
178- util.del_file("/.markerfile-*.txt")
179+ LOG.debug("Handle marker creation")
180+ markerfile = os.path.join(marker_dir, ".markerfile-" + markerid + ".txt")
181+ for fname in os.listdir(marker_dir):
182+ if fname.startswith(".markerfile"):
183+ util.del_file(os.path.join(marker_dir, fname))
184 open(markerfile, 'w').close()
185
186+
187+def _raise_error_status(prefix, error, event, config_file):
188+ """
189+ Raise error and send customization status to the underlying VMware
190+ Virtualization Platform. Also, cleanup the imc directory.
191+ """
192+ LOG.debug('%s: %s', prefix, error)
193+ set_customization_status(
194+ GuestCustStateEnum.GUESTCUST_STATE_RUNNING,
195+ event)
196+ util.del_dir(os.path.dirname(config_file))
197+ raise error
198+
199 # vi: ts=4 expandtab
200diff --git a/cloudinit/sources/helpers/vmware/imc/config.py b/cloudinit/sources/helpers/vmware/imc/config.py
201index 49d441d..2eaeff3 100644
202--- a/cloudinit/sources/helpers/vmware/imc/config.py
203+++ b/cloudinit/sources/helpers/vmware/imc/config.py
204@@ -100,4 +100,8 @@ class Config(object):
205 """Returns marker id."""
206 return self._configFile.get(Config.MARKERID, None)
207
208+ @property
209+ def custom_script_name(self):
210+ """Return the name of custom (pre/post) script."""
211+ return self._configFile.get(Config.CUSTOM_SCRIPT, None)
212 # vi: ts=4 expandtab
213diff --git a/cloudinit/sources/helpers/vmware/imc/config_custom_script.py b/cloudinit/sources/helpers/vmware/imc/config_custom_script.py
214new file mode 100644
215index 0000000..a7d4ad9
216--- /dev/null
217+++ b/cloudinit/sources/helpers/vmware/imc/config_custom_script.py
218@@ -0,0 +1,153 @@
219+# Copyright (C) 2017 Canonical Ltd.
220+# Copyright (C) 2017 VMware Inc.
221+#
222+# Author: Maitreyee Saikia <msaikia@vmware.com>
223+#
224+# This file is part of cloud-init. See LICENSE file for license information.
225+
226+import logging
227+import os
228+import stat
229+from textwrap import dedent
230+
231+from cloudinit import util
232+
233+LOG = logging.getLogger(__name__)
234+
235+
236+class CustomScriptNotFound(Exception):
237+ pass
238+
239+
240+class CustomScriptConstant(object):
241+ RC_LOCAL = "/etc/rc.local"
242+ POST_CUST_TMP_DIR = "/root/.customization"
243+ POST_CUST_RUN_SCRIPT_NAME = "post-customize-guest.sh"
244+ POST_CUST_RUN_SCRIPT = os.path.join(POST_CUST_TMP_DIR,
245+ POST_CUST_RUN_SCRIPT_NAME)
246+ POST_REBOOT_PENDING_MARKER = "/.guest-customization-post-reboot-pending"
247+
248+
249+class RunCustomScript(object):
250+ def __init__(self, scriptname, directory):
251+ self.scriptname = scriptname
252+ self.directory = directory
253+ self.scriptpath = os.path.join(directory, scriptname)
254+
255+ def prepare_script(self):
256+ if not os.path.exists(self.scriptpath):
257+ raise CustomScriptNotFound("Script %s not found!! "
258+ "Cannot execute custom script!"
259+ % self.scriptpath)
260+ # Strip any CR characters from the decoded script
261+ util.load_file(self.scriptpath).replace("\r", "")
262+ st = os.stat(self.scriptpath)
263+ os.chmod(self.scriptpath, st.st_mode | stat.S_IEXEC)
264+
265+
266+class PreCustomScript(RunCustomScript):
267+ def execute(self):
268+ """Executing custom script with precustomization argument."""
269+ LOG.debug("Executing pre-customization script")
270+ self.prepare_script()
271+ util.subp(["/bin/sh", self.scriptpath, "precustomization"])
272+
273+
274+class PostCustomScript(RunCustomScript):
275+ def __init__(self, scriptname, directory):
276+ super(PostCustomScript, self).__init__(scriptname, directory)
277+ # Determine when to run custom script. When postreboot is True,
278+ # the user uploaded script will run as part of rc.local after
279+ # the machine reboots. This is determined by presence of rclocal.
280+ # When postreboot is False, script will run as part of cloud-init.
281+ self.postreboot = False
282+
283+ def _install_post_reboot_agent(self, rclocal):
284+ """
285+ Install post-reboot agent for running custom script after reboot.
286+ As part of this process, we are editing the rclocal file to run a
287+ VMware script, which in turn is resposible for handling the user
288+ script.
289+ @param: path to rc local.
290+ """
291+ LOG.debug("Installing post-reboot customization from %s to %s",
292+ self.directory, rclocal)
293+ if not self.has_previous_agent(rclocal):
294+ LOG.info("Adding post-reboot customization agent to rc.local")
295+ new_content = dedent("""
296+ # Run post-reboot guest customization
297+ /bin/sh %s
298+ exit 0
299+ """) % CustomScriptConstant.POST_CUST_RUN_SCRIPT
300+ existing_rclocal = util.load_file(rclocal).replace('exit 0\n', '')
301+ st = os.stat(rclocal)
302+ # "x" flag should be set
303+ mode = st.st_mode | stat.S_IEXEC
304+ util.write_file(rclocal, existing_rclocal + new_content, mode)
305+
306+ else:
307+ # We don't need to update rclocal file everytime a customization
308+ # is requested. It just needs to be done for the first time.
309+ LOG.info("Post-reboot guest customization agent is already "
310+ "registered in rc.local")
311+ LOG.debug("Installing post-reboot customization agent finished: %s",
312+ self.postreboot)
313+
314+ def has_previous_agent(self, rclocal):
315+ searchstring = "# Run post-reboot guest customization"
316+ if searchstring in open(rclocal).read():
317+ return True
318+ return False
319+
320+ def find_rc_local(self):
321+ """
322+ Determine if rc local is present.
323+ """
324+ rclocal = ""
325+ if os.path.exists(CustomScriptConstant.RC_LOCAL):
326+ LOG.debug("rc.local detected.")
327+ # resolving in case of symlink
328+ rclocal = os.path.realpath(CustomScriptConstant.RC_LOCAL)
329+ LOG.debug("rc.local resolved to %s", rclocal)
330+ else:
331+ LOG.warning("Can't find rc.local, post-customization "
332+ "will be run before reboot")
333+ return rclocal
334+
335+ def install_agent(self):
336+ rclocal = self.find_rc_local()
337+ if rclocal:
338+ self._install_post_reboot_agent(rclocal)
339+ self.postreboot = True
340+
341+ def execute(self):
342+ """
343+ This method executes post-customization script before or after reboot
344+ based on the presence of rc local.
345+ """
346+ self.prepare_script()
347+ self.install_agent()
348+ if not self.postreboot:
349+ LOG.warning("Executing post-customization script inline")
350+ util.subp(["/bin/sh", self.scriptpath, "postcustomization"])
351+ else:
352+ LOG.debug("Scheduling custom script to run post reboot")
353+ if not os.path.isdir(CustomScriptConstant.POST_CUST_TMP_DIR):
354+ os.mkdir(CustomScriptConstant.POST_CUST_TMP_DIR)
355+ # Script "post-customize-guest.sh" and user uploaded script are
356+ # are present in the same directory and needs to copied to a temp
357+ # directory to be executed post reboot. User uploaded script is
358+ # saved as customize.sh in the temp directory.
359+ # post-customize-guest.sh excutes customize.sh after reboot.
360+ LOG.debug("Copying post-customization script")
361+ util.copy(self.scriptpath,
362+ CustomScriptConstant.POST_CUST_TMP_DIR + "/customize.sh")
363+ LOG.debug("Copying script to run post-customization script")
364+ util.copy(
365+ os.path.join(self.directory,
366+ CustomScriptConstant.POST_CUST_RUN_SCRIPT_NAME),
367+ CustomScriptConstant.POST_CUST_RUN_SCRIPT)
368+ LOG.info("Creating post-reboot pending marker")
369+ util.ensure_file(CustomScriptConstant.POST_REBOOT_PENDING_MARKER)
370+
371+# vi: ts=4 expandtab
372diff --git a/cloudinit/sources/helpers/vmware/imc/config_nic.py b/cloudinit/sources/helpers/vmware/imc/config_nic.py
373index 2fb07c5..2d8900e 100644
374--- a/cloudinit/sources/helpers/vmware/imc/config_nic.py
375+++ b/cloudinit/sources/helpers/vmware/imc/config_nic.py
376@@ -161,7 +161,7 @@ class NicConfigurator(object):
377 if nic.primary and v4.gateways:
378 self.ipv4PrimaryGateway = v4.gateways[0]
379 subnet.update({'gateway': self.ipv4PrimaryGateway})
380- return [subnet]
381+ return ([subnet], route_list)
382
383 # Add routes if there is no primary nic
384 if not self._primaryNic:
385diff --git a/tests/unittests/test_datasource/test_ovf.py b/tests/unittests/test_datasource/test_ovf.py
386index 700da86..fc4eb36 100644
387--- a/tests/unittests/test_datasource/test_ovf.py
388+++ b/tests/unittests/test_datasource/test_ovf.py
389@@ -5,11 +5,17 @@
390 # This file is part of cloud-init. See LICENSE file for license information.
391
392 import base64
393-from collections import OrderedDict
394+import os
395
396-from cloudinit.tests import helpers as test_helpers
397+from collections import OrderedDict
398+from textwrap import dedent
399
400+from cloudinit import util
401+from cloudinit.tests.helpers import CiTestCase, wrap_and_call
402+from cloudinit.helpers import Paths
403 from cloudinit.sources import DataSourceOVF as dsovf
404+from cloudinit.sources.helpers.vmware.imc.config_custom_script import (
405+ CustomScriptNotFound)
406
407 OVF_ENV_CONTENT = """<?xml version="1.0" encoding="UTF-8"?>
408 <Environment xmlns="http://schemas.dmtf.org/ovf/environment/1"
409@@ -42,7 +48,7 @@ def fill_properties(props, template=OVF_ENV_CONTENT):
410 return template.format(properties=properties)
411
412
413-class TestReadOvfEnv(test_helpers.TestCase):
414+class TestReadOvfEnv(CiTestCase):
415 def test_with_b64_userdata(self):
416 user_data = "#!/bin/sh\necho hello world\n"
417 user_data_b64 = base64.b64encode(user_data.encode()).decode()
418@@ -72,7 +78,104 @@ class TestReadOvfEnv(test_helpers.TestCase):
419 self.assertIsNone(ud)
420
421
422-class TestTransportIso9660(test_helpers.CiTestCase):
423+class TestMarkerFiles(CiTestCase):
424+
425+ def setUp(self):
426+ super(TestMarkerFiles, self).setUp()
427+ self.tdir = self.tmp_dir()
428+
429+ def test_false_when_markerid_none(self):
430+ """Return False when markerid provided is None."""
431+ self.assertFalse(
432+ dsovf.check_marker_exists(markerid=None, marker_dir=self.tdir))
433+
434+ def test_markerid_file_exist(self):
435+ """Return False when markerid file path does not exist,
436+ True otherwise."""
437+ self.assertFalse(
438+ dsovf.check_marker_exists('123', self.tdir))
439+
440+ marker_file = self.tmp_path('.markerfile-123.txt', self.tdir)
441+ util.write_file(marker_file, '')
442+ self.assertTrue(
443+ dsovf.check_marker_exists('123', self.tdir)
444+ )
445+
446+ def test_marker_file_setup(self):
447+ """Test creation of marker files."""
448+ markerfilepath = self.tmp_path('.markerfile-hi.txt', self.tdir)
449+ self.assertFalse(os.path.exists(markerfilepath))
450+ dsovf.setup_marker_files(markerid='hi', marker_dir=self.tdir)
451+ self.assertTrue(os.path.exists(markerfilepath))
452+
453+
454+class TestDatasourceOVF(CiTestCase):
455+
456+ with_logs = True
457+
458+ def setUp(self):
459+ super(TestDatasourceOVF, self).setUp()
460+ self.datasource = dsovf.DataSourceOVF
461+ self.tdir = self.tmp_dir()
462+
463+ def test_get_data_false_on_none_dmi_data(self):
464+ """When dmi for system-product-name is None, get_data returns False."""
465+ paths = Paths({'seed_dir': self.tdir})
466+ ds = self.datasource(sys_cfg={}, distro={}, paths=paths)
467+ retcode = wrap_and_call(
468+ 'cloudinit.sources.DataSourceOVF',
469+ {'util.read_dmi_data': None},
470+ ds.get_data)
471+ self.assertFalse(retcode, 'Expected False return from ds.get_data')
472+ self.assertIn(
473+ 'DEBUG: No system-product-name found', self.logs.getvalue())
474+
475+ def test_get_data_no_vmware_customization_disabled(self):
476+ """When vmware customization is disabled via sys_cfg log a message."""
477+ paths = Paths({'seed_dir': self.tdir})
478+ ds = self.datasource(
479+ sys_cfg={'disable_vmware_customization': True}, distro={},
480+ paths=paths)
481+ retcode = wrap_and_call(
482+ 'cloudinit.sources.DataSourceOVF',
483+ {'util.read_dmi_data': 'vmware'},
484+ ds.get_data)
485+ self.assertFalse(retcode, 'Expected False return from ds.get_data')
486+ self.assertIn(
487+ 'DEBUG: Customization for VMware platform is disabled.',
488+ self.logs.getvalue())
489+
490+ def test_get_data_vmware_customization_disabled(self):
491+ """When cloud-init workflow for vmware is enabled via sys_cfg log a
492+ message.
493+ """
494+ paths = Paths({'seed_dir': self.tdir})
495+ ds = self.datasource(
496+ sys_cfg={'disable_vmware_customization': False}, distro={},
497+ paths=paths)
498+ conf_file = self.tmp_path('test-cust', self.tdir)
499+ conf_content = dedent("""\
500+ [CUSTOM-SCRIPT]
501+ SCRIPT-NAME = test-script
502+ [MISC]
503+ MARKER-ID = 12345345
504+ """)
505+ util.write_file(conf_file, conf_content)
506+ with self.assertRaises(CustomScriptNotFound) as context:
507+ wrap_and_call(
508+ 'cloudinit.sources.DataSourceOVF',
509+ {'util.read_dmi_data': 'vmware',
510+ 'util.del_dir': True,
511+ 'search_file': self.tdir,
512+ 'wait_for_imc_cfg_file': conf_file,
513+ 'get_nics_to_enable': ''},
514+ ds.get_data)
515+ customscript = self.tmp_path('test-script', self.tdir)
516+ self.assertIn('Script %s not found!!' % customscript,
517+ str(context.exception))
518+
519+
520+class TestTransportIso9660(CiTestCase):
521
522 def setUp(self):
523 super(TestTransportIso9660, self).setUp()
524diff --git a/tests/unittests/test_vmware/__init__.py b/tests/unittests/test_vmware/__init__.py
525new file mode 100644
526index 0000000..e69de29
527--- /dev/null
528+++ b/tests/unittests/test_vmware/__init__.py
529diff --git a/tests/unittests/test_vmware/test_custom_script.py b/tests/unittests/test_vmware/test_custom_script.py
530new file mode 100644
531index 0000000..2d9519b
532--- /dev/null
533+++ b/tests/unittests/test_vmware/test_custom_script.py
534@@ -0,0 +1,99 @@
535+# Copyright (C) 2015 Canonical Ltd.
536+# Copyright (C) 2017 VMware INC.
537+#
538+# Author: Maitreyee Saikia <msaikia@vmware.com>
539+#
540+# This file is part of cloud-init. See LICENSE file for license information.
541+
542+from cloudinit import util
543+from cloudinit.sources.helpers.vmware.imc.config_custom_script import (
544+ CustomScriptConstant,
545+ CustomScriptNotFound,
546+ PreCustomScript,
547+ PostCustomScript,
548+)
549+from cloudinit.tests.helpers import CiTestCase, mock
550+
551+
552+class TestVmwareCustomScript(CiTestCase):
553+ def setUp(self):
554+ self.tmpDir = self.tmp_dir()
555+
556+ def test_prepare_custom_script(self):
557+ """
558+ This test is designed to verify the behavior based on the presence of
559+ custom script. Mainly needed for scenario where a custom script is
560+ expected, but was not properly copied. "CustomScriptNotFound" exception
561+ is raised in such cases.
562+ """
563+ # Custom script does not exist.
564+ preCust = PreCustomScript("random-vmw-test", self.tmpDir)
565+ self.assertEqual("random-vmw-test", preCust.scriptname)
566+ self.assertEqual(self.tmpDir, preCust.directory)
567+ self.assertEqual(self.tmp_path("random-vmw-test", self.tmpDir),
568+ preCust.scriptpath)
569+ with self.assertRaises(CustomScriptNotFound):
570+ preCust.prepare_script()
571+
572+ # Custom script exists.
573+ custScript = self.tmp_path("test-cust", self.tmpDir)
574+ util.write_file(custScript, "test-CR-strip/r/r")
575+ postCust = PostCustomScript("test-cust", self.tmpDir)
576+ self.assertEqual("test-cust", postCust.scriptname)
577+ self.assertEqual(self.tmpDir, postCust.directory)
578+ self.assertEqual(custScript, postCust.scriptpath)
579+ self.assertFalse(postCust.postreboot)
580+ postCust.prepare_script()
581+ # Check if all carraige returns are stripped from script.
582+ self.assertFalse("/r" in custScript)
583+
584+ def test_rc_local_exists(self):
585+ """
586+ This test is designed to verify the different scenarios associated
587+ with the presence of rclocal.
588+ """
589+ # test when rc local does not exist
590+ postCust = PostCustomScript("test-cust", self.tmpDir)
591+ with mock.patch.object(CustomScriptConstant, "RC_LOCAL", "/no/path"):
592+ rclocal = postCust.find_rc_local()
593+ self.assertEqual("", rclocal)
594+
595+ # test when rc local exists
596+ rclocalFile = self.tmp_path("vmware-rclocal", self.tmpDir)
597+ util.write_file(rclocalFile, "# Run post-reboot guest customization",
598+ omode="w")
599+ with mock.patch.object(CustomScriptConstant, "RC_LOCAL", rclocalFile):
600+ rclocal = postCust.find_rc_local()
601+ self.assertEqual(rclocalFile, rclocal)
602+ self.assertTrue(postCust.has_previous_agent, rclocal)
603+
604+ # test when rc local is a symlink
605+ rclocalLink = self.tmp_path("dummy-rclocal-link", self.tmpDir)
606+ util.sym_link(rclocalFile, rclocalLink, True)
607+ with mock.patch.object(CustomScriptConstant, "RC_LOCAL", rclocalLink):
608+ rclocal = postCust.find_rc_local()
609+ self.assertEqual(rclocalFile, rclocal)
610+
611+ def test_execute_post_cust(self):
612+ """
613+ This test is to identify if rclocal was properly populated to be
614+ run after reboot.
615+ """
616+ customscript = self.tmp_path("vmware-post-cust-script", self.tmpDir)
617+ rclocal = self.tmp_path("vmware-rclocal", self.tmpDir)
618+ # Create a temporary rclocal file
619+ open(customscript, "w")
620+ util.write_file(rclocal, "tests\nexit 0", omode="w")
621+ postCust = PostCustomScript("vmware-post-cust-script", self.tmpDir)
622+ with mock.patch.object(CustomScriptConstant, "RC_LOCAL", rclocal):
623+ # Test that guest customization agent is not installed initially.
624+ self.assertFalse(postCust.postreboot)
625+ self.assertIs(postCust.has_previous_agent(rclocal), False)
626+ postCust.install_agent()
627+
628+ # Assert rclocal has been modified to have guest customization
629+ # agent.
630+ self.assertTrue(postCust.postreboot)
631+ self.assertTrue(postCust.has_previous_agent, rclocal)
632+
633+# vi: ts=4 expandtab
634diff --git a/tests/unittests/test_vmware_config_file.py b/tests/unittests/test_vmware_config_file.py
635index 808d303..455d6c2 100644
636--- a/tests/unittests/test_vmware_config_file.py
637+++ b/tests/unittests/test_vmware_config_file.py
638@@ -334,5 +334,12 @@ class TestVmwareConfigFile(CiTestCase):
639 self.assertEqual('255.255.0.0', subnet.get('netmask'),
640 'Subnet netmask')
641
642+ def test_custom_script(self):
643+ cf = ConfigFile("tests/data/vmware/cust-dhcp-2nic.cfg")
644+ conf = Config(cf)
645+ self.assertIsNone(conf.custom_script_name)
646+ cf._insertKey("CUSTOM-SCRIPT|SCRIPT-NAME", "test-script")
647+ conf = Config(cf)
648+ self.assertEqual("test-script", conf.custom_script_name)
649
650 # vi: ts=4 expandtab

Subscribers

People subscribed via source and target branches