Merge ~sankaraditya/cloud-init:topic-stanguturi-vmware-curtin-changes into cloud-init:master
| Status: | Merged | ||||
|---|---|---|---|---|---|
| Approved by: | Scott Moser on 2017-09-07 | ||||
| Approved revision: | f0b2b7cba92cd940ad3931a487087abb4126b019 | ||||
| Merged at revision: | a1dfdda2a2ae20fe026881980ddf7d16110f06e2 | ||||
| Proposed branch: | ~sankaraditya/cloud-init:topic-stanguturi-vmware-curtin-changes | ||||
| Merge into: | cloud-init:master | ||||
| Diff against target: |
808 lines (+418/-103) 4 files modified
cloudinit/sources/DataSourceOVF.py (+64/-27) cloudinit/sources/helpers/vmware/imc/config_nic.py (+130/-71) cloudinit/sources/helpers/vmware/imc/guestcust_util.py (+7/-5) tests/unittests/test_vmware_config_file.py (+217/-0) |
||||
| Related bugs: |
|
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Scott Moser | 2017-03-14 | Approve on 2017-09-07 | |
| Server Team CI bot | continuous-integration | Approve on 2017-09-01 | |
| Chad Smith | Approve on 2017-08-31 | ||
|
Review via email:
|
|||
Description of the Change
vmware customization: return network config format
For customizing the machines hosted on 'VMWare' hypervisor, the datasource
should return the 'network config' data in 'curtin' format.
Modify the code to read the customization configuration and return the converted data.
Added few tests.
LP: #1675063
- cab90ca... by Sankar Tanguturi on 2017-03-14
- 6f4a2e0... by Sankar Tanguturi on 2017-03-15
- 66391c3... by Sankar Tanguturi on 2017-03-15
| Scott Moser (smoser) wrote : | # |
- 394a132... by Sankar Tanguturi on 2017-03-16
| Sankar Tanguturi (sankaraditya) wrote : | # |
> a.) I think I'd like to move the conversion of the network config out of the
> 'get'.
> The LOG.debug "Applying the network customization" is actually incorrect in
> that
> no changes are done there. Lets move that to happen in the
> 'network_config' function.
>
> if self._network_
> return self._network_
>
> ## convert the network config from the data in 'self' here to to the
> desired
> ## 'network_config' format.
> nc = NicConcifugrato
> cfg = get_vmware_
> self._network_
> return self.network_config
Thanks for the comments. I tried modifying the code as per your suggestion and found an issue. When cloud-init runs in local mode initially, the datasourceOVF returned the data and then cloud-init saved the instance some where. During this stage, _network_config is saved as False. And then network-config property is called. The local mode finished. When the cloud-init in 'network' mode started, it loaded the saved instance. Since the _network_config is not saved, whenever this property is retrieved, the same code is executed again. This is not quite right for our use case.
Any suggestion to fix this?
| Scott Moser (smoser) wrote : | # |
Why was the network information not available in 'local' mode?
You'll have to show the code that you tried with, because nothing in your current tree here will set _network_config to false.
You're right that we have to get the network config in local mode otherwise it wont be applied early enough.
| Scott Moser (smoser) wrote : | # |
I'd also recommend storing the contents of vmwareImcConfig
self._vmware_
and then when you make a 'ConfigFile' object pass those contents in (you'll have ot modify it to take a string contents instead of having ConfigFile take a filename.
| Sankar Tanguturi (sankaraditya) wrote : | # |
> Why was the network information not available in 'local' mode?
>
> You'll have to show the code that you tried with, because nothing in your
> current tree here will set _network_config to false.
>
> You're right that we have to get the network config in local mode otherwise it
> wont be applied early enough.
Sorry if my previous comment was not clear. (_network_config was set to None and not False). Let me explain:
1. When the cloud-init initially runs in 'local' mode, first getdata is called and the necessary settings are retrieved. Now that we modified the code to retrieve the network code to 'network_config', getdata will not return the network data. At that point, cloud-init saves the datasource instance and it will have self._network_
2. After few seconds, cloud-init runs in 'network' mode. This time, the saved instance is retrieved and since the self._network_
So, to avoid this, I think, self._network_
Thanks
Sankar.
- 4bb42b4... by Sankar Tanguturi on 2017-03-17
| Sankar Tanguturi (sankaraditya) wrote : | # |
Addressed few review comments. Updated the code and also the tests.
In the getData() function itself, we want to parse the network configuration and return an error if parsing fails. For that reason, I didn't move the 'network config' part from getData() to network_config @property method.
Thanks
| Scott Moser (smoser) wrote : | # |
Can you open a bug for this? Just so I have something to track.
https:/
I think my request for get_network_config does not make any sense any more.
The way you've done it now, there is basically no value in that function. No one calls it directly, so just make it part of 'get_network_
Some other comments in line, nothing huge.
Good work Sankar,
Scott
| Sankar Tanguturi (sankaraditya) wrote : | # |
> Can you open a bug for this? Just so I have something to track.
> https:/
Scott,
Thanks for the valuable comments. For what part, you are asking me to file a bug?
| Sankar Tanguturi (sankaraditya) wrote : | # |
Will fix as per the review comments and update the diff. I have only one question about the 'activate' method. I mentioned below.
Thanks
Sankar.
- 2f2d810... by Sankar Tanguturi on 2017-03-21
| Scott Moser (smoser) wrote : | # |
Sankar, I went ahead and opened a bug. I just wanted something to track this larger set of changes.
One other request in this Merge. We should now be able to drop NicConfigurator and any tests that exist for it as this will all be getting rendered with cloud-init's generic renderers.
| Sankar Tanguturi (sankaraditya) wrote : | # |
> One other request in this Merge. We should now be able to drop
> NicConfigurator and any tests that exist for it as this will all be getting
> rendered with cloud-init's generic renderers.
I don't think we can drop the NicConfigurator. It the the class that generates the necessary network config in curtin format. It doesn't not actually configure the nics but just reads the mac addresses of the system and returns proper network data.
| Sankar Tanguturi (sankaraditya) wrote : | # |
Working on some small nits. Will update the diff.
PASSED: Continuous integration, rev:4bb42b4f22c
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
- 8e3bb47... by Sankar Tanguturi on 2017-03-27
PASSED: Continuous integration, rev:8e3bb474186
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
| Scott Moser (smoser) wrote : | # |
one small comment. this was as far as I got just yet.
| Chad Smith (chad.smith) wrote : | # |
Looks really good. Long term I would like to see us hoist some of the ip-command parsing utilities up into cloudinit.
Thanks again for this work. let's wrap up these branches early next week if we can. Sorry for the delay.
| Sankar Tanguturi (sankaraditya) wrote : | # |
Thanks Chad and Scott for your valuable inputs.
Will address all review comments and will update the diff.
- 4301c29... by Sankar Tanguturi on 2017-08-16
| Sankar Tanguturi (sankaraditya) wrote : | # |
Updating the diff after addressing review comments from Chat and SCott. Thanks.
FAILED: Continuous integration, rev:4301c299aaf
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
- 13e4e08... by Sankar Tanguturi on 2017-08-18
- 39d7aae... by Sankar Tanguturi on 2017-08-18
| Chad Smith (chad.smith) wrote : | # |
Thanks for the comment updates and questions answered Sankar!
Looks like this branch only needs a git rebase to fixup the following conflicts:
Conflict in cloudinit/
Conflict in tests/unittests
# from your branch: you probably already know this, it was kindof new to me
git pull origin/master
git rebase origin/master
# iterate through any conflicts, edit conflicts git add <conflict_files>; git rebase --continue
| Sankar Tanguturi (sankaraditya) wrote : | # |
Resolved conflicts and updated the diff. Thanks.
PASSED: Continuous integration, rev:39d7aaed101
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
- a3d130d... by Sankar Tanguturi on 2017-08-25
| Sankar Tanguturi (sankaraditya) wrote : | # |
As discussed with Chad Smith today during 'cloud-init' meeting, updated the diff to add 'source /etc/network/
Thanks Chad.
PASSED: Continuous integration, rev:a3d130dc31e
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
- 891e2d3... by Sankar Tanguturi on 2017-08-25
| Sankar Tanguturi (sankaraditya) wrote : | # |
In VMware guest customization workflow, we should be able to do re-customization. After discussing with Scott and Chad, modified the code to generate a unique id when there is a customization. This will force the cloud-init to do the re-customization.
Thanks Scott and Chad.
PASSED: Continuous integration, rev:891e2d38d3e
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
| Chad Smith (chad.smith) wrote : | # |
Good update on random instance-id Sankar, if you can instrument or add to an existing unit test that exposes the instance-id update +1 on this branch.
- e0a6eb6... by Sankar Tanguturi on 2017-08-25
| Sankar Tanguturi (sankaraditya) wrote : | # |
Addressed all review comments from Chad and updated the diff.
PASSED: Continuous integration, rev:e0a6eb60c18
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
| Chad Smith (chad.smith) wrote : | # |
Thanks Sankar for the fixes.
Since, as you mentioned, the python configuration engine overwrites /etc/network/
# DO NOT EDIT THIS FILE BY HAND -- AUTOMATICALLY GENERATED BY cloud-init.
Thanks for the additional unit test on randomized instance-id. etc.
Just some minor nits on the unit tests.
Wherever possible please don't use assertTrue( some conditional) because unit test errors in those cases are not helpful, they only say "AssertionError: False is not true". But if you use assertEqual() or assertIn you get slightly more informative test errors:
AssertionError: 'iid-vmware-' not found in 'some-random-
I approve after your unittest fixups are there
- f0b2b7c... by Sankar Tanguturi on 2017-08-31
| Sankar Tanguturi (sankaraditya) wrote : | # |
Fixed the new unit test case as Chad suggested. Thanks.
FAILED: Continuous integration, rev:f0b2b7cba92
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
FAILED: MAAS Compatability Testing
Click here to trigger a rebuild:
https:/
| Sankar Tanguturi (sankaraditya) wrote : | # |
> LGTM; +1 and thanks.
Thanks Chad.
FAILED: Continuous integration, rev:f0b2b7cba92
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
FAILED: Ubuntu LTS: Build
Click here to trigger a rebuild:
https:/
PASSED: Continuous integration, rev:f0b2b7cba92
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
| Scott Moser (smoser) wrote : | # |
approving based on Chad's review and talking with Sankar.


Sankar,
Hey, this looks really good. I do have some comments, but thanks for the good work.
a.) I think I'd like to move the conversion of the network config out of the 'get'.
The LOG.debug "Applying the network customization" is actually incorrect in that
no changes are done there. Lets move that to happen in the 'network_config' function.
if self._network_ config is not None: config
return self._network_
## convert the network config from the data in 'self' here to to the desired r(self. conf.nics) network_ config( ...) _network_ config = cfg
## 'network_config' format.
nc = NicConcifugrato
cfg = get_vmware_
self.
return self.network_config
b.) lets change get_vmware_ network_ config to either take no 'Config' input or only Config input.
as it is right now it takes a Config and a 'nics_cfg_list'.
but the nics_cfg_list is completely derived from the Config, so why not just pass the Config?
its probably helpful to just have a:
get_network_ config_ from_conf( )
get_network_ config( nameservers= None, search=None, nics=None)
that calls:
c.) It'll be helpful if there is an easy way to go from a vmwareImcConfig FilePath contents to a network config. Then your tests will just do: config_ from_vmware_ cfg("""
nc = network_
[NETWORK]
NETWORKING = yes
BOOTPROTO = dhcp
HOSTNAME = myhost1
DOMAINNAME = eng.vmware.com
[NIC-CONFIG]
NICS = NIC1,NIC2
[NIC1] assertEqual( expected, found)
...
""")
self.