Merge ~dojordan/cloud-init:azure-preprovisioning into cloud-init:master
- Git
- lp:~dojordan/cloud-init
- azure-preprovisioning
- Merge into master
Status: | Merged |
---|---|
Approved by: | Chad Smith |
Approved revision: | 573332c54ac77114beb0cdf13e0e8865f241ea18 |
Merge reported by: | Chad Smith |
Merged at revision: | c03bdd3d8ed762cada813c5e95a40b14d2047b57 |
Proposed branch: | ~dojordan/cloud-init:azure-preprovisioning |
Merge into: | cloud-init:master |
Diff against target: |
757 lines (+397/-53) 11 files modified
.gitignore (+1/-0) cloudinit/net/dhcp.py (+42/-1) cloudinit/net/network_state.py (+12/-0) cloudinit/sources/DataSourceAzure.py (+129/-9) cloudinit/sources/DataSourceEc2.py (+7/-16) cloudinit/sources/helpers/azure.py (+14/-8) cloudinit/temp_utils.py (+8/-3) cloudinit/url_helper.py (+20/-9) tests/unittests/test_datasource/test_azure.py (+151/-6) tests/unittests/test_datasource/test_ec2.py (+1/-1) tests/unittests/test_net.py (+12/-0) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Chad Smith | Approve | ||
Server Team CI bot | continuous-integration | Approve | |
Scott Moser | Approve | ||
Paul Meyer (community) | Approve | ||
Review via email: mp+334341@code.launchpad.net |
Commit message
Azure VM Preprovisioning support.
This change will enable azure vms to report provisioning has completed
twice, first to tell the fabric it has completed then a second time to
enable customer settings. The datasource for the second provisioning is
the Instance Metadata Service (IMDS),and the VM will poll indefinitely for
the new ovf-env.xml from IMDS.
This branch introduces EphemeralDHCPv4 which encapsulates common logic
used by both DataSourceEc2 an DataSourceAzure for temporary DHCP
interactions without side-effects.
LP: #1734991
Description of the change
Azure VM Preprovisioning support.
This change will enable azure vms to report provisioning has completed twice, first to tell the fabric it has completed then a second time to enable customer settings. The datasource for the second provisioning is the Instance Metadata Service (IMDS), and the VM will poll indefinitely for the new ovf-env.xml from IMDS.
LP: 1734991
Paul Meyer (paul-meyer) wrote : | # |
Scott Moser (smoser) wrote : | # |
Fix your commit message (press 'Set commit message' above).
Summary
<blank>
More info
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:
https:/
Executed test runs:
FAILED: Checkout
Click here to trigger a rebuild:
https:/
Douglas Jordan (dojordan) wrote : | # |
> FAILED: Continuous integration, rev:
> https:/
> Executed test runs:
> FAILED: Checkout
>
> Click here to trigger a rebuild:
> https:/
Looks like the git checkout failed. Thoughts?
stderr: remote: Authorisation required.
fatal: Authentication failed for 'https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:72e423b4a08
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
FAILED: MAAS Compatability Testing
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:384b7731e99
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
FAILED: MAAS Compatability Testing
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:d00ec2ec822
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:6021da2b61a
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Paul Meyer (paul-meyer) : | # |
Chad Smith (chad.smith) : | # |
Douglas Jordan (dojordan) wrote : | # |
Thanks for the feedback. I've resolved your comments.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:ff0b7b0b5fd
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) : | # |
Chad Smith (chad.smith) : | # |
Chad Smith (chad.smith) wrote : | # |
Douglas, thanks for working this. I'm going to mark it work in progress as you address or respond to comments. When you'd like another review pass please just mark it back to "Needs review"
Douglas Jordan (dojordan) : | # |
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:10c6b219ba7
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:e06a7627419
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) wrote : | # |
Thanks for the rework and comments. One more pass on this. I'll test against stock azure xenial today and report here.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:c500a0184b8
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) wrote : | # |
Ok one last nit from me and it looks good. I need a bit of testing still on this but content looks good.
Chad Smith (chad.smith) wrote : | # |
one last try on the nit
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:3c9509671f0
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) : | # |
Scott Moser (smoser) wrote : | # |
There are some comments in line.
I'm not sure I fully understand all of this.
I could be wrong here, but I think you're using
bounce_
We have 2 ways now of brining up ephemeral networking for this purpose:
a.) Chad has recently added code in the EC2 metadata service using cloudinit/
b.) the digital ocean datasource has code to negotiate a ipv4 link local address.
If 'b' is sufficient, I'd prefer that, but either one I'd prefer to the bounce_network_ which i think may actually not work for you if you rebased to trunk.
As mentioned in IRC, I'm still concerned about systemd giving up and deciding that boot has failed after some amount of time polling on a metadata service. As Douglas pointed out, cloud-init has timeouts set to 0 and is a 'oneshot', so *its* timeout is not an issue, but I think that things that it runs 'Before' or (pre-networking or other) might end up timing out.
Scott Moser (smoser) wrote : | # |
some things there need fixing, definitely need rebase to trunk (I suspect you'll have conflicts), but if not, some thought is needed.
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:c207d443ca3
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:ba329ca5cde
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:93d88e25a1d
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Douglas Jordan (dojordan) wrote : | # |
Thanks for the comments Scott-
The reason for the bounce_
> There are some comments in line.
> I'm not sure I fully understand all of this.
>
> I could be wrong here, but I think you're using
> bounce_
>
> We have 2 ways now of brining up ephemeral networking for this purpose:
> a.) Chad has recently added code in the EC2 metadata service using
> cloudinit/
> b.) the digital ocean datasource has code to negotiate a ipv4 link local
> address.
>
> If 'b' is sufficient, I'd prefer that, but either one I'd prefer to the
> bounce_network_ which i think may actually not work for you if you rebased to
> trunk.
>
> As mentioned in IRC, I'm still concerned about systemd giving up and deciding
> that boot has failed after some amount of time polling on a metadata service.
> As Douglas pointed out, cloud-init has timeouts set to 0 and is a 'oneshot',
> so *its* timeout is not an issue, but I think that things that it runs
> 'Before' or (pre-networking or other) might end up timing out.
Sushant Sharma (sushantsharma) : | # |
Sushant Sharma (sushantsharma) wrote : | # |
<not sure why the inline comment does not show up>
Hi Douglas, Can you please add a comment before line 83 (where you catch exception and re-DHCP) saying that this is temporary. We plan to add a networking module specific to azure in cloud-init and will address re-DHCP need in that module. The plan is to submit that module for review in a week or so. Thanks!
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:fc0154011d4
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
As it is right now, 'bounce_
on Ubuntu 18.04 (bionic) or any system without 'ifup'.
So while you want this fixed in 16.04, the general Ubuntu development
path insists that we make things work on the development release first
and the SRU them back to 16.04.
In summary: I can't let this in as broken on bionic. We'll need to find
a way to make that work on 18.04.
Your commit message says:
| This change will enable azure vms to report provisioning has completed
| twice, first to tell the fabric it has completed then a second time to
| enable customer settings.
Is that true, will it *always* report reprovisioning has completed twice?
there are some small inlinel things also.
those are the big things though. thanks for your work.
Douglas Jordan (dojordan) wrote : | # |
Regarding the DHCP stuff: We are exploring an alternate solution to bounce the nic from hyper-v, but in the mean time we would like to get this checked in. So an alternate solution for bionic would be to simply change the hostname. This way, systemd-networkd will keep re triggering DHCP. Once we get the final ovf_env.xml from IMDS, we will actually apply the real, customer provided hostname.
Regarding "Is that true, will it *always* report reprovisioning has completed twice?"-
tldr; yes. Technically, it will report "provisioning" has completed twice, while we are calling the second incarnation "reprovisioning"
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:7f23e5c4808
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
Chad and I had experience that lead us to believe that systemd-networkd just re-requested lease on hostname change. If that is true, then this seems reasonable.
Michael Hudson-Doyle in bug 1739516 indicates that we were mistaken.
Need to sort that out, and then I think i'm happy with this.
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:41daec29381
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
FAILED: Ubuntu LTS: Integration
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:4b2fb677802
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
FAILED: Ubuntu LTS: Integration
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:1091cb48d61
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
FAILED: Ubuntu LTS: Integration
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:1c3bb38b3a1
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:6610952f032
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
FAILED: Ubuntu LTS: Integration
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:6610952f032
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
Overall, I like it.
Some comments / questions inline.
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:f7b90fe5740
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:e83238aae58
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:280c3c383fc
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
I have a couple comments.
a.) i don't like the use of 'unknown-245'. we really should fix that somewhere centrally, but i wouldnt put that on Douglas to do. We should have something that finds that name and names it more reasonably.
b.) i want to make sure we're not doing multiple dhcp in the *non* provisioning path now.
c.) at some point we can start using the metdata service for data always?
d.) as discussed in #cloud-init http://
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:cbc353567df
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:f030e24e6fe
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) wrote : | # |
Thanks for the additional passes on the Douglas. Looks really good. Some quick inline comments/nits.
Chad Smith (chad.smith) : | # |
Scott Moser (smoser) : | # |
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:eaec80d3647
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:a73a578fd96
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) wrote : | # |
looks good to me. will test on azure/ec2 before landing, minor inline comments on latest branch, but I'm good to land after ec2/azure testing.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:d114c6377c3
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) wrote : | # |
Validated this patchset on amazon with no errors on clean install or upgrade path.
Scott Moser (smoser) wrote : | # |
I'd like a response to a comment in my last review:
| without passing in a nic to EphemeralDHCPv4, it will end up calling
| 'find_fallback_nic' each time. I wonder if that behavior is better or
| worse than picking one device and just sticking with it throuh the use
| of this class.
I dont expect that this is a big deal, but a hotplug of a nic during
the wait for the IMDS could result in the interface that is used for
dhcp to be different on retries than on the first.
Douglas Jordan (dojordan) wrote : | # |
Sorry about that, totally missed this one. I agree, I think we should pin the context manager to the nic. When we get an exception, I think it is reasonable to look for the fallback nic again. I'll make that change now.
> I'd like a response to a comment in my last review:
> | without passing in a nic to EphemeralDHCPv4, it will end up calling
> | 'find_fallback_nic' each time. I wonder if that behavior is better or
> | worse than picking one device and just sticking with it throuh the use
> | of this class.
>
> I dont expect that this is a big deal, but a hotplug of a nic during
> the wait for the IMDS could result in the interface that is used for
> dhcp to be different on retries than on the first.
Douglas Jordan (dojordan) wrote : | # |
Actually, ignore my comment. I think that is the correct behavior. For some reason, if we do hot plug a nic and it gets a new interface name, we would like to use it during the polling loop.
Scott Moser (smoser) wrote : | # |
thanks Douglas.
I like it.
If c-i approves, then so do I.
Thanks for your patience.
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:f466bd87777
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
FAILED: Ubuntu LTS: Integration
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:f466bd87777
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:f466bd87777
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) wrote : | # |
On checking latest commit on this branch I can't get ahold of the azure instance after reboot on bionic. douglas just confirmed same behavior on artful too.
sudo cloud-init clean --logs --reboot with a deb from this branch doesn't come back up for some reason.
Douglas picked up my boot logs for me https:/
haven't dug into what is the matter here. I briefly saw cloud-init get through the datasource get_data and modules:config, but no ssh is running on the ip.
Chad Smith (chad.smith) wrote : | # |
marked needs review so I don't accidentally publish this. Until we get to the bottom of why bionic artful aren't coming back after reboot.
- 0a4a1a0... by Douglas Jordan
-
Fixing bool('false') == true bug.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:0a4a1a092bc
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
- 3aa23e9... by Douglas Jordan
-
Using warning log to hit console.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:3aa23e90ab3
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
- 573332c... by Douglas Jordan
-
Using proper util for string to bool.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:573332c54ac
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) wrote : | # |
Validated on xenial and bionic that latest branch no longer blocks clean boot scenario.
Chad Smith (chad.smith) wrote : | # |
Hello Douglas and Paul, or anyone else affected,
As part of a stable release update, this changeset is committed and accepted cloud-init into artful-proposed and xenial-proposed as cloud-init version 17.2-35-
sed -i 's/ xenial / xenial-proposed /' /etc/apt/sources
sudo apt-get update
sudo apt-get install cloud-init;
sudo cloud-init clean --logs --reboot; # For a fresh install run
If there are concerns please comment on LP: #1747059.
Preview Diff
1 | diff --git a/.gitignore b/.gitignore |
2 | index b0500a6..75565ed 100644 |
3 | --- a/.gitignore |
4 | +++ b/.gitignore |
5 | @@ -10,3 +10,4 @@ parts |
6 | prime |
7 | stage |
8 | *.snap |
9 | +*.cover |
10 | diff --git a/cloudinit/net/dhcp.py b/cloudinit/net/dhcp.py |
11 | index 875a460..087c0c0 100644 |
12 | --- a/cloudinit/net/dhcp.py |
13 | +++ b/cloudinit/net/dhcp.py |
14 | @@ -10,7 +10,9 @@ import os |
15 | import re |
16 | import signal |
17 | |
18 | -from cloudinit.net import find_fallback_nic, get_devicelist |
19 | +from cloudinit.net import ( |
20 | + EphemeralIPv4Network, find_fallback_nic, get_devicelist) |
21 | +from cloudinit.net.network_state import mask_and_ipv4_to_bcast_addr as bcip |
22 | from cloudinit import temp_utils |
23 | from cloudinit import util |
24 | from six import StringIO |
25 | @@ -29,6 +31,45 @@ class InvalidDHCPLeaseFileError(Exception): |
26 | pass |
27 | |
28 | |
29 | +class NoDHCPLeaseError(Exception): |
30 | + """Raised when unable to get a DHCP lease.""" |
31 | + pass |
32 | + |
33 | + |
34 | +class EphemeralDHCPv4(object): |
35 | + def __init__(self, iface=None): |
36 | + self.iface = iface |
37 | + self._ephipv4 = None |
38 | + |
39 | + def __enter__(self): |
40 | + try: |
41 | + leases = maybe_perform_dhcp_discovery(self.iface) |
42 | + except InvalidDHCPLeaseFileError: |
43 | + raise NoDHCPLeaseError() |
44 | + if not leases: |
45 | + raise NoDHCPLeaseError() |
46 | + lease = leases[-1] |
47 | + LOG.debug("Received dhcp lease on %s for %s/%s", |
48 | + lease['interface'], lease['fixed-address'], |
49 | + lease['subnet-mask']) |
50 | + nmap = {'interface': 'interface', 'ip': 'fixed-address', |
51 | + 'prefix_or_mask': 'subnet-mask', |
52 | + 'broadcast': 'broadcast-address', |
53 | + 'router': 'routers'} |
54 | + kwargs = dict([(k, lease.get(v)) for k, v in nmap.items()]) |
55 | + if not kwargs['broadcast']: |
56 | + kwargs['broadcast'] = bcip(kwargs['prefix_or_mask'], kwargs['ip']) |
57 | + ephipv4 = EphemeralIPv4Network(**kwargs) |
58 | + ephipv4.__enter__() |
59 | + self._ephipv4 = ephipv4 |
60 | + return lease |
61 | + |
62 | + def __exit__(self, excp_type, excp_value, excp_traceback): |
63 | + if not self._ephipv4: |
64 | + return |
65 | + self._ephipv4.__exit__(excp_type, excp_value, excp_traceback) |
66 | + |
67 | + |
68 | def maybe_perform_dhcp_discovery(nic=None): |
69 | """Perform dhcp discovery if nic valid and dhclient command exists. |
70 | |
71 | diff --git a/cloudinit/net/network_state.py b/cloudinit/net/network_state.py |
72 | index 31738c7..fe667d8 100644 |
73 | --- a/cloudinit/net/network_state.py |
74 | +++ b/cloudinit/net/network_state.py |
75 | @@ -961,4 +961,16 @@ def mask_to_net_prefix(mask): |
76 | return ipv4_mask_to_net_prefix(mask) |
77 | |
78 | |
79 | +def mask_and_ipv4_to_bcast_addr(mask, ip): |
80 | + """Calculate the broadcast address from the subnet mask and ip addr. |
81 | + |
82 | + Supports ipv4 only.""" |
83 | + ip_bin = int(''.join([bin(int(x) + 256)[3:] for x in ip.split('.')]), 2) |
84 | + mask_dec = ipv4_mask_to_net_prefix(mask) |
85 | + bcast_bin = ip_bin | (2**(32 - mask_dec) - 1) |
86 | + bcast_str = '.'.join([str(bcast_bin >> (i << 3) & 0xFF) |
87 | + for i in range(4)[::-1]]) |
88 | + return bcast_str |
89 | + |
90 | + |
91 | # vi: ts=4 expandtab |
92 | diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py |
93 | index d1d0975..4bcbf3a 100644 |
94 | --- a/cloudinit/sources/DataSourceAzure.py |
95 | +++ b/cloudinit/sources/DataSourceAzure.py |
96 | @@ -11,13 +11,16 @@ from functools import partial |
97 | import os |
98 | import os.path |
99 | import re |
100 | +from time import time |
101 | from xml.dom import minidom |
102 | import xml.etree.ElementTree as ET |
103 | |
104 | from cloudinit import log as logging |
105 | from cloudinit import net |
106 | +from cloudinit.net.dhcp import EphemeralDHCPv4 |
107 | from cloudinit import sources |
108 | from cloudinit.sources.helpers.azure import get_metadata_from_fabric |
109 | +from cloudinit.url_helper import readurl, wait_for_url, UrlError |
110 | from cloudinit import util |
111 | |
112 | LOG = logging.getLogger(__name__) |
113 | @@ -44,6 +47,9 @@ LEASE_FILE = '/var/lib/dhcp/dhclient.eth0.leases' |
114 | DEFAULT_FS = 'ext4' |
115 | # DMI chassis-asset-tag is set static for all azure instances |
116 | AZURE_CHASSIS_ASSET_TAG = '7783-7084-3265-9085-8269-3286-77' |
117 | +REPROVISION_MARKER_FILE = "/var/lib/cloud/data/poll_imds" |
118 | +IMDS_URL = "http://169.254.169.254/metadata/reprovisiondata" |
119 | +IMDS_RETRIES = 5 |
120 | |
121 | |
122 | def find_storvscid_from_sysctl_pnpinfo(sysctl_out, deviceid): |
123 | @@ -276,19 +282,20 @@ class DataSourceAzure(sources.DataSource): |
124 | |
125 | with temporary_hostname(azure_hostname, self.ds_cfg, |
126 | hostname_command=hostname_command) \ |
127 | - as previous_hostname: |
128 | - if (previous_hostname is not None and |
129 | + as previous_hn: |
130 | + if (previous_hn is not None and |
131 | util.is_true(self.ds_cfg.get('set_hostname'))): |
132 | cfg = self.ds_cfg['hostname_bounce'] |
133 | |
134 | # "Bouncing" the network |
135 | try: |
136 | - perform_hostname_bounce(hostname=azure_hostname, |
137 | - cfg=cfg, |
138 | - prev_hostname=previous_hostname) |
139 | + return perform_hostname_bounce(hostname=azure_hostname, |
140 | + cfg=cfg, |
141 | + prev_hostname=previous_hn) |
142 | except Exception as e: |
143 | LOG.warning("Failed publishing hostname: %s", e) |
144 | util.logexc(LOG, "handling set_hostname failed") |
145 | + return False |
146 | |
147 | def get_metadata_from_agent(self): |
148 | temp_hostname = self.metadata.get('local-hostname') |
149 | @@ -345,15 +352,20 @@ class DataSourceAzure(sources.DataSource): |
150 | ddir = self.ds_cfg['data_dir'] |
151 | |
152 | candidates = [self.seed_dir] |
153 | + if os.path.isfile(REPROVISION_MARKER_FILE): |
154 | + candidates.insert(0, "IMDS") |
155 | candidates.extend(list_possible_azure_ds_devs()) |
156 | if ddir: |
157 | candidates.append(ddir) |
158 | |
159 | found = None |
160 | - |
161 | + reprovision = False |
162 | for cdev in candidates: |
163 | try: |
164 | - if cdev.startswith("/dev/"): |
165 | + if cdev == "IMDS": |
166 | + ret = None |
167 | + reprovision = True |
168 | + elif cdev.startswith("/dev/"): |
169 | if util.is_FreeBSD(): |
170 | ret = util.mount_cb(cdev, load_azure_ds_dir, |
171 | mtype="udf", sync=False) |
172 | @@ -370,6 +382,8 @@ class DataSourceAzure(sources.DataSource): |
173 | LOG.warning("%s was not mountable", cdev) |
174 | continue |
175 | |
176 | + if reprovision or self._should_reprovision(ret): |
177 | + ret = self._reprovision() |
178 | (md, self.userdata_raw, cfg, files) = ret |
179 | self.seed = cdev |
180 | self.metadata = util.mergemanydict([md, DEFAULT_METADATA]) |
181 | @@ -428,6 +442,83 @@ class DataSourceAzure(sources.DataSource): |
182 | LOG.debug("negotiating already done for %s", |
183 | self.get_instance_id()) |
184 | |
185 | + def _poll_imds(self, report_ready=True): |
186 | + """Poll IMDS for the new provisioning data until we get a valid |
187 | + response. Then return the returned JSON object.""" |
188 | + url = IMDS_URL + "?api-version=2017-04-02" |
189 | + headers = {"Metadata": "true"} |
190 | + LOG.debug("Start polling IMDS") |
191 | + |
192 | + def sleep_cb(response, loop_n): |
193 | + return 1 |
194 | + |
195 | + def exception_cb(msg, exception): |
196 | + if isinstance(exception, UrlError) and exception.code == 404: |
197 | + return |
198 | + LOG.warning("Exception during polling. Will try DHCP.", |
199 | + exc_info=True) |
200 | + |
201 | + # If we get an exception while trying to call IMDS, we |
202 | + # call DHCP and setup the ephemeral network to acquire the new IP. |
203 | + raise exception |
204 | + |
205 | + need_report = report_ready |
206 | + for i in range(IMDS_RETRIES): |
207 | + try: |
208 | + with EphemeralDHCPv4() as lease: |
209 | + if need_report: |
210 | + self._report_ready(lease=lease) |
211 | + need_report = False |
212 | + wait_for_url([url], max_wait=None, timeout=60, |
213 | + status_cb=LOG.info, |
214 | + headers_cb=lambda url: headers, sleep_time=1, |
215 | + exception_cb=exception_cb, |
216 | + sleep_time_cb=sleep_cb) |
217 | + return str(readurl(url, headers=headers)) |
218 | + except Exception: |
219 | + LOG.debug("Exception during polling-retrying dhcp" + |
220 | + " %d more time(s).", (IMDS_RETRIES - i), |
221 | + exc_info=True) |
222 | + |
223 | + def _report_ready(self, lease): |
224 | + """Tells the fabric provisioning has completed |
225 | + before we go into our polling loop.""" |
226 | + try: |
227 | + get_metadata_from_fabric(None, lease['unknown-245']) |
228 | + except Exception as exc: |
229 | + LOG.warning( |
230 | + "Error communicating with Azure fabric; You may experience." |
231 | + "connectivity issues.", exc_info=True) |
232 | + |
233 | + def _should_reprovision(self, ret): |
234 | + """Whether or not we should poll IMDS for reprovisioning data. |
235 | + Also sets a marker file to poll IMDS. |
236 | + |
237 | + The marker file is used for the following scenario: the VM boots into |
238 | + this polling loop, which we expect to be proceeding infinitely until |
239 | + the VM is picked. If for whatever reason the platform moves us to a |
240 | + new host (for instance a hardware issue), we need to keep polling. |
241 | + However, since the VM reports ready to the Fabric, we will not attach |
242 | + the ISO, thus cloud-init needs to have a way of knowing that it should |
243 | + jump back into the polling loop in order to retrieve the ovf_env.""" |
244 | + if not ret: |
245 | + return False |
246 | + (md, self.userdata_raw, cfg, files) = ret |
247 | + path = REPROVISION_MARKER_FILE |
248 | + if (cfg.get('PreprovisionedVm') is True or |
249 | + os.path.isfile(path)): |
250 | + if not os.path.isfile(path): |
251 | + LOG.info("Creating a marker file to poll imds") |
252 | + util.write_file(path, "%s: %s\n" % (os.getpid(), time())) |
253 | + return True |
254 | + return False |
255 | + |
256 | + def _reprovision(self): |
257 | + """Initiate the reprovisioning workflow.""" |
258 | + contents = self._poll_imds() |
259 | + md, ud, cfg = read_azure_ovf(contents) |
260 | + return (md, ud, cfg, {'ovf-env.xml': contents}) |
261 | + |
262 | def _negotiate(self): |
263 | """Negotiate with fabric and return data from it. |
264 | |
265 | @@ -453,7 +544,7 @@ class DataSourceAzure(sources.DataSource): |
266 | "Error communicating with Azure fabric; You may experience." |
267 | "connectivity issues.", exc_info=True) |
268 | return False |
269 | - |
270 | + util.del_file(REPROVISION_MARKER_FILE) |
271 | return fabric_data |
272 | |
273 | def activate(self, cfg, is_new_instance): |
274 | @@ -595,6 +686,7 @@ def address_ephemeral_resize(devpath=RESOURCE_DISK_PATH, maxwait=120, |
275 | def perform_hostname_bounce(hostname, cfg, prev_hostname): |
276 | # set the hostname to 'hostname' if it is not already set to that. |
277 | # then, if policy is not off, bounce the interface using command |
278 | + # Returns True if the network was bounced, False otherwise. |
279 | command = cfg['command'] |
280 | interface = cfg['interface'] |
281 | policy = cfg['policy'] |
282 | @@ -614,7 +706,8 @@ def perform_hostname_bounce(hostname, cfg, prev_hostname): |
283 | else: |
284 | LOG.debug( |
285 | "Skipping network bounce: ifupdown utils aren't present.") |
286 | - return # Don't bounce as networkd handles hostname DDNS updates |
287 | + # Don't bounce as networkd handles hostname DDNS updates |
288 | + return False |
289 | LOG.debug("pubhname: publishing hostname [%s]", msg) |
290 | shell = not isinstance(command, (list, tuple)) |
291 | # capture=False, see comments in bug 1202758 and bug 1206164. |
292 | @@ -622,6 +715,7 @@ def perform_hostname_bounce(hostname, cfg, prev_hostname): |
293 | get_uptime=True, func=util.subp, |
294 | kwargs={'args': command, 'shell': shell, 'capture': False, |
295 | 'env': env}) |
296 | + return True |
297 | |
298 | |
299 | def crtfile_to_pubkey(fname, data=None): |
300 | @@ -838,9 +932,35 @@ def read_azure_ovf(contents): |
301 | if 'ssh_pwauth' not in cfg and password: |
302 | cfg['ssh_pwauth'] = True |
303 | |
304 | + cfg['PreprovisionedVm'] = _extract_preprovisioned_vm_setting(dom) |
305 | + |
306 | return (md, ud, cfg) |
307 | |
308 | |
309 | +def _extract_preprovisioned_vm_setting(dom): |
310 | + """Read the preprovision flag from the ovf. It should not |
311 | + exist unless true.""" |
312 | + platform_settings_section = find_child( |
313 | + dom.documentElement, |
314 | + lambda n: n.localName == "PlatformSettingsSection") |
315 | + if not platform_settings_section or len(platform_settings_section) == 0: |
316 | + LOG.debug("PlatformSettingsSection not found") |
317 | + return False |
318 | + platform_settings = find_child( |
319 | + platform_settings_section[0], |
320 | + lambda n: n.localName == "PlatformSettings") |
321 | + if not platform_settings or len(platform_settings) == 0: |
322 | + LOG.debug("PlatformSettings not found") |
323 | + return False |
324 | + preprovisionedVm = find_child( |
325 | + platform_settings[0], |
326 | + lambda n: n.localName == "PreprovisionedVm") |
327 | + if not preprovisionedVm or len(preprovisionedVm) == 0: |
328 | + LOG.debug("PreprovisionedVm not found") |
329 | + return False |
330 | + return util.translate_bool(preprovisionedVm[0].firstChild.nodeValue) |
331 | + |
332 | + |
333 | def encrypt_pass(password, salt_id="$6$"): |
334 | return crypt.crypt(password, salt_id + util.rand_str(strlen=16)) |
335 | |
336 | diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourceEc2.py |
337 | index 0f89f34..e14553b 100644 |
338 | --- a/cloudinit/sources/DataSourceEc2.py |
339 | +++ b/cloudinit/sources/DataSourceEc2.py |
340 | @@ -14,7 +14,7 @@ import time |
341 | from cloudinit import ec2_utils as ec2 |
342 | from cloudinit import log as logging |
343 | from cloudinit import net |
344 | -from cloudinit.net import dhcp |
345 | +from cloudinit.net.dhcp import EphemeralDHCPv4, NoDHCPLeaseError |
346 | from cloudinit import sources |
347 | from cloudinit import url_helper as uhelp |
348 | from cloudinit import util |
349 | @@ -102,22 +102,13 @@ class DataSourceEc2(sources.DataSource): |
350 | if util.is_FreeBSD(): |
351 | LOG.debug("FreeBSD doesn't support running dhclient with -sf") |
352 | return False |
353 | - dhcp_leases = dhcp.maybe_perform_dhcp_discovery( |
354 | - self.fallback_interface) |
355 | - if not dhcp_leases: |
356 | - # DataSourceEc2Local failed in init-local stage. DataSourceEc2 |
357 | - # will still run in init-network stage. |
358 | + try: |
359 | + with EphemeralDHCPv4(self.fallback_interface): |
360 | + return util.log_time( |
361 | + logfunc=LOG.debug, msg='Crawl of metadata service', |
362 | + func=self._crawl_metadata) |
363 | + except NoDHCPLeaseError: |
364 | return False |
365 | - dhcp_opts = dhcp_leases[-1] |
366 | - net_params = {'interface': dhcp_opts.get('interface'), |
367 | - 'ip': dhcp_opts.get('fixed-address'), |
368 | - 'prefix_or_mask': dhcp_opts.get('subnet-mask'), |
369 | - 'broadcast': dhcp_opts.get('broadcast-address'), |
370 | - 'router': dhcp_opts.get('routers')} |
371 | - with net.EphemeralIPv4Network(**net_params): |
372 | - return util.log_time( |
373 | - logfunc=LOG.debug, msg='Crawl of metadata service', |
374 | - func=self._crawl_metadata) |
375 | else: |
376 | return self._crawl_metadata() |
377 | |
378 | diff --git a/cloudinit/sources/helpers/azure.py b/cloudinit/sources/helpers/azure.py |
379 | index 6cda572..90c12df 100644 |
380 | --- a/cloudinit/sources/helpers/azure.py |
381 | +++ b/cloudinit/sources/helpers/azure.py |
382 | @@ -199,10 +199,10 @@ class WALinuxAgentShim(object): |
383 | ' </Container>', |
384 | '</Health>']) |
385 | |
386 | - def __init__(self, fallback_lease_file=None): |
387 | + def __init__(self, fallback_lease_file=None, dhcp_options=None): |
388 | LOG.debug('WALinuxAgentShim instantiated, fallback_lease_file=%s', |
389 | fallback_lease_file) |
390 | - self.dhcpoptions = None |
391 | + self.dhcpoptions = dhcp_options |
392 | self._endpoint = None |
393 | self.openssl_manager = None |
394 | self.values = {} |
395 | @@ -220,7 +220,8 @@ class WALinuxAgentShim(object): |
396 | @property |
397 | def endpoint(self): |
398 | if self._endpoint is None: |
399 | - self._endpoint = self.find_endpoint(self.lease_file) |
400 | + self._endpoint = self.find_endpoint(self.lease_file, |
401 | + self.dhcpoptions) |
402 | return self._endpoint |
403 | |
404 | @staticmethod |
405 | @@ -292,10 +293,14 @@ class WALinuxAgentShim(object): |
406 | return _value |
407 | |
408 | @staticmethod |
409 | - def find_endpoint(fallback_lease_file=None): |
410 | + def find_endpoint(fallback_lease_file=None, dhcp245=None): |
411 | value = None |
412 | - LOG.debug('Finding Azure endpoint from networkd...') |
413 | - value = WALinuxAgentShim._networkd_get_value_from_leases() |
414 | + if dhcp245 is not None: |
415 | + value = dhcp245 |
416 | + LOG.debug("Using Azure Endpoint from dhcp options") |
417 | + if value is None: |
418 | + LOG.debug('Finding Azure endpoint from networkd...') |
419 | + value = WALinuxAgentShim._networkd_get_value_from_leases() |
420 | if value is None: |
421 | # Option-245 stored in /run/cloud-init/dhclient.hooks/<ifc>.json |
422 | # a dhclient exit hook that calls cloud-init-dhclient-hook |
423 | @@ -367,8 +372,9 @@ class WALinuxAgentShim(object): |
424 | LOG.info('Reported ready to Azure fabric.') |
425 | |
426 | |
427 | -def get_metadata_from_fabric(fallback_lease_file=None): |
428 | - shim = WALinuxAgentShim(fallback_lease_file=fallback_lease_file) |
429 | +def get_metadata_from_fabric(fallback_lease_file=None, dhcp_opts=None): |
430 | + shim = WALinuxAgentShim(fallback_lease_file=fallback_lease_file, |
431 | + dhcp_options=dhcp_opts) |
432 | try: |
433 | return shim.register_with_azure_and_fetch_data() |
434 | finally: |
435 | diff --git a/cloudinit/temp_utils.py b/cloudinit/temp_utils.py |
436 | index 5d7adf7..c98a1b5 100644 |
437 | --- a/cloudinit/temp_utils.py |
438 | +++ b/cloudinit/temp_utils.py |
439 | @@ -28,13 +28,18 @@ def _tempfile_dir_arg(odir=None, needs_exe=False): |
440 | if odir is not None: |
441 | return odir |
442 | |
443 | + if needs_exe: |
444 | + tdir = _EXE_ROOT_TMPDIR |
445 | + if not os.path.isdir(tdir): |
446 | + os.makedirs(tdir) |
447 | + os.chmod(tdir, 0o1777) |
448 | + return tdir |
449 | + |
450 | global _TMPDIR |
451 | if _TMPDIR: |
452 | return _TMPDIR |
453 | |
454 | - if needs_exe: |
455 | - tdir = _EXE_ROOT_TMPDIR |
456 | - elif os.getuid() == 0: |
457 | + if os.getuid() == 0: |
458 | tdir = _ROOT_TMPDIR |
459 | else: |
460 | tdir = os.environ.get('TMPDIR', '/tmp') |
461 | diff --git a/cloudinit/url_helper.py b/cloudinit/url_helper.py |
462 | index 0e0f5b4..0a5be0b 100644 |
463 | --- a/cloudinit/url_helper.py |
464 | +++ b/cloudinit/url_helper.py |
465 | @@ -273,7 +273,7 @@ def readurl(url, data=None, timeout=None, retries=0, sec_between=1, |
466 | |
467 | def wait_for_url(urls, max_wait=None, timeout=None, |
468 | status_cb=None, headers_cb=None, sleep_time=1, |
469 | - exception_cb=None): |
470 | + exception_cb=None, sleep_time_cb=None): |
471 | """ |
472 | urls: a list of urls to try |
473 | max_wait: roughly the maximum time to wait before giving up |
474 | @@ -286,6 +286,8 @@ def wait_for_url(urls, max_wait=None, timeout=None, |
475 | for request. |
476 | exception_cb: call method with 2 arguments 'msg' (per status_cb) and |
477 | 'exception', the exception that occurred. |
478 | + sleep_time_cb: call method with 2 arguments (response, loop_n) that |
479 | + generates the next sleep time. |
480 | |
481 | the idea of this routine is to wait for the EC2 metdata service to |
482 | come up. On both Eucalyptus and EC2 we have seen the case where |
483 | @@ -301,6 +303,8 @@ def wait_for_url(urls, max_wait=None, timeout=None, |
484 | service but is not going to find one. It is possible that the instance |
485 | data host (169.254.169.254) may be firewalled off Entirely for a sytem, |
486 | meaning that the connection will block forever unless a timeout is set. |
487 | + |
488 | + A value of None for max_wait will retry indefinitely. |
489 | """ |
490 | start_time = time.time() |
491 | |
492 | @@ -311,18 +315,24 @@ def wait_for_url(urls, max_wait=None, timeout=None, |
493 | status_cb = log_status_cb |
494 | |
495 | def timeup(max_wait, start_time): |
496 | - return ((max_wait <= 0 or max_wait is None) or |
497 | - (time.time() - start_time > max_wait)) |
498 | + if (max_wait is None): |
499 | + return False |
500 | + return ((max_wait <= 0) or (time.time() - start_time > max_wait)) |
501 | |
502 | loop_n = 0 |
503 | + response = None |
504 | while True: |
505 | - sleep_time = int(loop_n / 5) + 1 |
506 | + if sleep_time_cb is not None: |
507 | + sleep_time = sleep_time_cb(response, loop_n) |
508 | + else: |
509 | + sleep_time = int(loop_n / 5) + 1 |
510 | for url in urls: |
511 | now = time.time() |
512 | if loop_n != 0: |
513 | if timeup(max_wait, start_time): |
514 | break |
515 | - if timeout and (now + timeout > (start_time + max_wait)): |
516 | + if (max_wait is not None and |
517 | + timeout and (now + timeout > (start_time + max_wait))): |
518 | # shorten timeout to not run way over max_time |
519 | timeout = int((start_time + max_wait) - now) |
520 | |
521 | @@ -354,10 +364,11 @@ def wait_for_url(urls, max_wait=None, timeout=None, |
522 | url_exc = e |
523 | |
524 | time_taken = int(time.time() - start_time) |
525 | - status_msg = "Calling '%s' failed [%s/%ss]: %s" % (url, |
526 | - time_taken, |
527 | - max_wait, |
528 | - reason) |
529 | + max_wait_str = "%ss" % max_wait if max_wait else "unlimited" |
530 | + status_msg = "Calling '%s' failed [%s/%s]: %s" % (url, |
531 | + time_taken, |
532 | + max_wait_str, |
533 | + reason) |
534 | status_cb(status_msg) |
535 | if exception_cb: |
536 | # This can be used to alter the headers that will be sent |
537 | diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py |
538 | index 6341e1e..254e987 100644 |
539 | --- a/tests/unittests/test_datasource/test_azure.py |
540 | +++ b/tests/unittests/test_datasource/test_azure.py |
541 | @@ -5,7 +5,7 @@ from cloudinit.util import b64e, decode_binary, load_file, write_file |
542 | from cloudinit.sources import DataSourceAzure as dsaz |
543 | from cloudinit.util import find_freebsd_part |
544 | from cloudinit.util import get_path_dev_freebsd |
545 | - |
546 | +from cloudinit.version import version_string as vs |
547 | from cloudinit.tests.helpers import (CiTestCase, TestCase, populate_dir, mock, |
548 | ExitStack, PY26, SkipTest) |
549 | |
550 | @@ -16,7 +16,8 @@ import xml.etree.ElementTree as ET |
551 | import yaml |
552 | |
553 | |
554 | -def construct_valid_ovf_env(data=None, pubkeys=None, userdata=None): |
555 | +def construct_valid_ovf_env(data=None, pubkeys=None, |
556 | + userdata=None, platform_settings=None): |
557 | if data is None: |
558 | data = {'HostName': 'FOOHOST'} |
559 | if pubkeys is None: |
560 | @@ -66,10 +67,12 @@ def construct_valid_ovf_env(data=None, pubkeys=None, userdata=None): |
561 | xmlns:i="http://www.w3.org/2001/XMLSchema-instance"> |
562 | <KmsServerHostname>kms.core.windows.net</KmsServerHostname> |
563 | <ProvisionGuestAgent>false</ProvisionGuestAgent> |
564 | - <GuestAgentPackageName i:nil="true" /> |
565 | - </PlatformSettings></wa:PlatformSettingsSection> |
566 | -</Environment> |
567 | - """ |
568 | + <GuestAgentPackageName i:nil="true" />""" |
569 | + if platform_settings: |
570 | + for k, v in platform_settings.items(): |
571 | + content += "<%s>%s</%s>\n" % (k, v, k) |
572 | + content += """</PlatformSettings></wa:PlatformSettingsSection> |
573 | +</Environment>""" |
574 | |
575 | return content |
576 | |
577 | @@ -1107,4 +1110,146 @@ class TestAzureNetExists(CiTestCase): |
578 | self.assertTrue(hasattr(dsaz, "DataSourceAzureNet")) |
579 | |
580 | |
581 | +@mock.patch('cloudinit.sources.DataSourceAzure.util.subp') |
582 | +@mock.patch.object(dsaz, 'get_hostname') |
583 | +@mock.patch.object(dsaz, 'set_hostname') |
584 | +class TestAzureDataSourcePreprovisioning(CiTestCase): |
585 | + |
586 | + def setUp(self): |
587 | + super(TestAzureDataSourcePreprovisioning, self).setUp() |
588 | + tmp = self.tmp_dir() |
589 | + self.waagent_d = self.tmp_path('/var/lib/waagent', tmp) |
590 | + self.paths = helpers.Paths({'cloud_dir': tmp}) |
591 | + dsaz.BUILTIN_DS_CONFIG['data_dir'] = self.waagent_d |
592 | + |
593 | + def test_read_azure_ovf_with_true_flag(self, *args): |
594 | + """The read_azure_ovf method should set the PreprovisionedVM |
595 | + cfg flag if the proper setting is present.""" |
596 | + content = construct_valid_ovf_env( |
597 | + platform_settings={"PreprovisionedVm": "True"}) |
598 | + ret = dsaz.read_azure_ovf(content) |
599 | + cfg = ret[2] |
600 | + self.assertTrue(cfg['PreprovisionedVm']) |
601 | + |
602 | + def test_read_azure_ovf_with_false_flag(self, *args): |
603 | + """The read_azure_ovf method should set the PreprovisionedVM |
604 | + cfg flag to false if the proper setting is false.""" |
605 | + content = construct_valid_ovf_env( |
606 | + platform_settings={"PreprovisionedVm": "False"}) |
607 | + ret = dsaz.read_azure_ovf(content) |
608 | + cfg = ret[2] |
609 | + self.assertFalse(cfg['PreprovisionedVm']) |
610 | + |
611 | + def test_read_azure_ovf_without_flag(self, *args): |
612 | + """The read_azure_ovf method should not set the |
613 | + PreprovisionedVM cfg flag.""" |
614 | + content = construct_valid_ovf_env() |
615 | + ret = dsaz.read_azure_ovf(content) |
616 | + cfg = ret[2] |
617 | + self.assertFalse(cfg['PreprovisionedVm']) |
618 | + |
619 | + @mock.patch('cloudinit.sources.DataSourceAzure.util.is_FreeBSD') |
620 | + @mock.patch('cloudinit.net.dhcp.EphemeralIPv4Network') |
621 | + @mock.patch('cloudinit.net.dhcp.maybe_perform_dhcp_discovery') |
622 | + @mock.patch('requests.Session.request') |
623 | + def test_poll_imds_returns_ovf_env(self, fake_resp, m_dhcp, m_net, |
624 | + m_is_bsd, *args): |
625 | + """The _poll_imds method should return the ovf_env.xml.""" |
626 | + m_is_bsd.return_value = False |
627 | + m_dhcp.return_value = [{ |
628 | + 'interface': 'eth9', 'fixed-address': '192.168.2.9', |
629 | + 'routers': '192.168.2.1', 'subnet-mask': '255.255.255.0'}] |
630 | + url = 'http://{0}/metadata/reprovisiondata?api-version=2017-04-02' |
631 | + host = "169.254.169.254" |
632 | + full_url = url.format(host) |
633 | + fake_resp.return_value = mock.MagicMock(status_code=200, text="ovf") |
634 | + dsa = dsaz.DataSourceAzure({}, distro=None, paths=self.paths) |
635 | + self.assertTrue(len(dsa._poll_imds()) > 0) |
636 | + self.assertEqual(fake_resp.call_args_list, |
637 | + [mock.call(allow_redirects=True, |
638 | + headers={'Metadata': 'true', |
639 | + 'User-Agent': |
640 | + 'Cloud-Init/%s' % vs() |
641 | + }, method='GET', timeout=60.0, |
642 | + url=full_url), |
643 | + mock.call(allow_redirects=True, |
644 | + headers={'Metadata': 'true', |
645 | + 'User-Agent': |
646 | + 'Cloud-Init/%s' % vs() |
647 | + }, method='GET', url=full_url)]) |
648 | + self.assertEqual(m_dhcp.call_count, 1) |
649 | + m_net.assert_any_call( |
650 | + broadcast='192.168.2.255', interface='eth9', ip='192.168.2.9', |
651 | + prefix_or_mask='255.255.255.0', router='192.168.2.1') |
652 | + self.assertEqual(m_net.call_count, 1) |
653 | + |
654 | + @mock.patch('cloudinit.sources.DataSourceAzure.util.is_FreeBSD') |
655 | + @mock.patch('cloudinit.net.dhcp.EphemeralIPv4Network') |
656 | + @mock.patch('cloudinit.net.dhcp.maybe_perform_dhcp_discovery') |
657 | + @mock.patch('requests.Session.request') |
658 | + def test__reprovision_calls__poll_imds(self, fake_resp, m_dhcp, m_net, |
659 | + m_is_bsd, *args): |
660 | + """The _reprovision method should call poll IMDS.""" |
661 | + m_is_bsd.return_value = False |
662 | + m_dhcp.return_value = [{ |
663 | + 'interface': 'eth9', 'fixed-address': '192.168.2.9', |
664 | + 'routers': '192.168.2.1', 'subnet-mask': '255.255.255.0', |
665 | + 'unknown-245': '624c3620'}] |
666 | + url = 'http://{0}/metadata/reprovisiondata?api-version=2017-04-02' |
667 | + host = "169.254.169.254" |
668 | + full_url = url.format(host) |
669 | + hostname = "myhost" |
670 | + username = "myuser" |
671 | + odata = {'HostName': hostname, 'UserName': username} |
672 | + content = construct_valid_ovf_env(data=odata) |
673 | + fake_resp.return_value = mock.MagicMock(status_code=200, text=content) |
674 | + dsa = dsaz.DataSourceAzure({}, distro=None, paths=self.paths) |
675 | + md, ud, cfg, d = dsa._reprovision() |
676 | + self.assertEqual(md['local-hostname'], hostname) |
677 | + self.assertEqual(cfg['system_info']['default_user']['name'], username) |
678 | + self.assertEqual(fake_resp.call_args_list, |
679 | + [mock.call(allow_redirects=True, |
680 | + headers={'Metadata': 'true', |
681 | + 'User-Agent': |
682 | + 'Cloud-Init/%s' % vs()}, |
683 | + method='GET', timeout=60.0, url=full_url), |
684 | + mock.call(allow_redirects=True, |
685 | + headers={'Metadata': 'true', |
686 | + 'User-Agent': |
687 | + 'Cloud-Init/%s' % vs()}, |
688 | + method='GET', url=full_url)]) |
689 | + self.assertEqual(m_dhcp.call_count, 1) |
690 | + m_net.assert_any_call( |
691 | + broadcast='192.168.2.255', interface='eth9', ip='192.168.2.9', |
692 | + prefix_or_mask='255.255.255.0', router='192.168.2.1') |
693 | + self.assertEqual(m_net.call_count, 1) |
694 | + |
695 | + @mock.patch('cloudinit.sources.DataSourceAzure.util.write_file') |
696 | + @mock.patch('os.path.isfile') |
697 | + def test__should_reprovision_with_true_cfg(self, isfile, write_f, *args): |
698 | + """The _should_reprovision method should return true with config |
699 | + flag present.""" |
700 | + isfile.return_value = False |
701 | + dsa = dsaz.DataSourceAzure({}, distro=None, paths=self.paths) |
702 | + self.assertTrue(dsa._should_reprovision( |
703 | + (None, None, {'PreprovisionedVm': True}, None))) |
704 | + |
705 | + @mock.patch('os.path.isfile') |
706 | + def test__should_reprovision_with_file_existing(self, isfile, *args): |
707 | + """The _should_reprovision method should return True if the sentinal |
708 | + exists.""" |
709 | + isfile.return_value = True |
710 | + dsa = dsaz.DataSourceAzure({}, distro=None, paths=self.paths) |
711 | + self.assertTrue(dsa._should_reprovision( |
712 | + (None, None, {'preprovisionedvm': False}, None))) |
713 | + |
714 | + @mock.patch('os.path.isfile') |
715 | + def test__should_reprovision_returns_false(self, isfile, *args): |
716 | + """The _should_reprovision method should return False |
717 | + if config and sentinal are not present.""" |
718 | + isfile.return_value = False |
719 | + dsa = dsaz.DataSourceAzure({}, distro=None, paths=self.paths) |
720 | + self.assertFalse(dsa._should_reprovision((None, None, {}, None))) |
721 | + |
722 | + |
723 | # vi: ts=4 expandtab |
724 | diff --git a/tests/unittests/test_datasource/test_ec2.py b/tests/unittests/test_datasource/test_ec2.py |
725 | index f0dc833..0f7267b 100644 |
726 | --- a/tests/unittests/test_datasource/test_ec2.py |
727 | +++ b/tests/unittests/test_datasource/test_ec2.py |
728 | @@ -425,7 +425,7 @@ class TestEc2(test_helpers.HttprettyTestCase): |
729 | self.logs.getvalue()) |
730 | |
731 | @httpretty.activate |
732 | - @mock.patch('cloudinit.net.EphemeralIPv4Network') |
733 | + @mock.patch('cloudinit.net.dhcp.EphemeralIPv4Network') |
734 | @mock.patch('cloudinit.net.find_fallback_nic') |
735 | @mock.patch('cloudinit.net.dhcp.maybe_perform_dhcp_discovery') |
736 | @mock.patch('cloudinit.sources.DataSourceEc2.util.is_FreeBSD') |
737 | diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py |
738 | index ddea13d..ac33e8e 100644 |
739 | --- a/tests/unittests/test_net.py |
740 | +++ b/tests/unittests/test_net.py |
741 | @@ -2948,4 +2948,16 @@ class TestRenameInterfaces(CiTestCase): |
742 | mock_subp.assert_has_calls(expected) |
743 | |
744 | |
745 | +class TestNetworkState(CiTestCase): |
746 | + |
747 | + def test_bcast_addr(self): |
748 | + """Test mask_and_ipv4_to_bcast_addr proper execution.""" |
749 | + bcast_addr = network_state.mask_and_ipv4_to_bcast_addr |
750 | + self.assertEqual("192.168.1.255", |
751 | + bcast_addr("255.255.255.0", "192.168.1.1")) |
752 | + self.assertEqual("128.42.7.255", |
753 | + bcast_addr("255.255.248.0", "128.42.5.4")) |
754 | + self.assertEqual("10.1.21.255", |
755 | + bcast_addr("255.255.255.0", "10.1.21.4")) |
756 | + |
757 | # vi: ts=4 expandtab |
Looks good to me (after you fix the tests and flake8's)