cloud-init

Merge ~chad.smith/cloud-init:feature/os-local into cloud-init:master

  1. Git
  2. lp:~chad.smith/cloud-init
  3. feature/os-local
  4. Merge into master
Proposed by Chad Smith
Status: Merged
Approved by: Chad Smith
Approved revision: 99dbd5ee17cbda93f6dd30772baedfe6c08a1cf6
Merge reported by: Chad Smith
Merged at revision: cd1de5f47ab6b82f2c6fd61a5f6681f33b3e5705
Proposed branch: ~chad.smith/cloud-init:feature/os-local
Merge into: cloud-init:master
Diff against target: 900 lines (+416/-124)
9 files modified
cloudinit/sources/DataSourceCloudStack.py (+10/-21)
cloudinit/sources/DataSourceConfigDrive.py (+2/-2)
cloudinit/sources/DataSourceEc2.py (+15/-33)
cloudinit/sources/DataSourceOpenStack.py (+105/-56)
cloudinit/sources/DataSourceSmartOS.py (+4/-5)
cloudinit/sources/__init__.py (+76/-0)
cloudinit/sources/tests/test_init.py (+87/-2)
tests/unittests/test_datasource/test_common.py (+1/-0)
tests/unittests/test_datasource/test_openstack.py (+116/-5)
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve
Scott Moser Approve
Review via email: mp+345806@code.launchpad.net

Commit message

openstack: Allow discovery in init-local using dhclient in a sandbox.

Network has not yet been configured in the init-local stage so the
openstack datasource will use dhcp-client to temporarily obtain an ipv4
address and query the metadata service at http://169.254.169.254 to get
network_data.json configuration. If present, the datasource will return
network_config version 1 config based on that network_data.json content.
Previously OpenStack datasource only setup dhcp on the fallback interface
so this represents a change in behavior to react to the full config
provided by openstack.

Also significant to OpenStack is the separation of a _crawl_data operation
from get_data(). crawl_data walks the available metadata services and
returns a dict of discovered content. get_data consumes the crawled_data,
 caches it in the datasource and reacts to that data.
/run/cloud-init/instance-data.json now published network_data.json or
ec2_metadata key if that data is present on any datasource.

The main reasons for the separation of crawl from get_data:
 * Enable performance metrics of cloud-init's metadata crawls on each
 * Enable cloud-init modules and scripts to query and consume metadata
   content which may have updated/changed after cloud-init's initial cache
   during instance boot. (Think hotplug)

Also generalize common logic to base DataSource class/module:
 * Move to a common UNSET variable up into base datasource module fix EC2,
   ConfigDrive, OpenStack, SmartOS to use the global.
 * Drop get_url_settings from Ec2, CloudStack and OpenStack and generalize
   DataSource.get_url_params(). Allow subclasses to override url_max_wait,
   url_timeout and url_retries params.
 * Rename get_network_metadata bool to perform_dhcp_setup as it designates
   whether EphemeralDHCPv4 setup is required before crawling metadata.

LP: #1749717

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:ea957b9ddee350ff32c1112b1eabe984386fc5fc
https://jenkins.ubuntu.com/server/job/cloud-init-ci/9/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/9/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:69c82436e4074ca30c55cfb553bf2145a37f50e1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/10/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/10/rebuild

review: Approve (continuous-integration)
Revision history for this message
Ryan Harper (raharper) wrote :

The commit summary sound funny, dhclient returned network_data.json ?

Openstack: Allow Openstaack to run at init-local using dhclient in a sandbox

Which mirrors what we did for Ec2 and Azure, etc; the network_data.json is sort of an implementation detail.

Some questions inline as well.

Revision history for this message
Scott Moser (smoser) wrote :

as you suggested, the 'get_network_metadata' name is weird.

And we do have to think about what to do for SRU on this.
I guess we'd want to enable it in a datasource config so that then when
we SRU we could turn that setting to false to maintain the old path.

the rest of this i think looks pretty striaght forward.

comments inline

~chad.smith/cloud-init:feature/os-local updated
95f8c3c... by Chad Smith

Ec2/OS: rename get_network_metadata -> perform_dhcp_setup. Add URLParams namedtuple

4b538d7... by Chad Smith

use URLParams namedtuple in return values

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:b7ea947e56f1e25615a205c4b807e8153cf600c2
https://jenkins.ubuntu.com/server/job/cloud-init-ci/19/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/19/rebuild

review: Approve (continuous-integration)
~chad.smith/cloud-init:feature/os-local updated
422ff52... by Chad Smith

update Ec2 rtd on configuration options

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:8a2d4790773c5e5aa79030c546d4045390121837
https://jenkins.ubuntu.com/server/job/cloud-init-ci/20/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/20/rebuild

review: Approve (continuous-integration)
~chad.smith/cloud-init:feature/os-local updated
c843b38... by Chad Smith

doc: scrub ec2, openstack and cloudstack datasource configuration

cdc80ca... by Chad Smith

openstack: surface use_network_json datasource config option

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:b8a146e258c8991268be1ff765df8daefcd4ab91
https://jenkins.ubuntu.com/server/job/cloud-init-ci/23/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/23/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

I would rather you push 'doc: scrub ec2, ...'' as a separate merge proposal.

I dont want to make busywork, but contained commits make things easier.

Other than that I have only a nitpick on the name 'use_network_json'.
It would seem more "datasource generic" if we named it "apply_network_config" or something.
then this same variable could at least conceptually be used on other datasources.

~chad.smith/cloud-init:feature/os-local updated
9cddc4a... by Chad Smith

pull Ec2 fallback_interface upgrade logic into Ec2 only

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:36acdccc250dadd0be541c768493232335cd8fcd
https://jenkins.ubuntu.com/server/job/cloud-init-ci/26/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/26/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/os-local updated
ee83a76... by Chad Smith

doc: revert doc changes for separate branch

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:b0aed8d5fe04fd62269b8c112b17b001e713c1ac
https://jenkins.ubuntu.com/server/job/cloud-init-ci/28/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/28/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/os-local updated
a5dd971... by Chad Smith

fix DataSourceEc2.network_config. Add unit tests for DataSource.fallback_interface

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:fe328942e738b0b2e0759c3f3119ed18102bf7f5
https://jenkins.ubuntu.com/server/job/cloud-init-ci/29/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/29/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

Well, i approve, but that *$&%#*$ bot doesn't. Get him/her to approve and I'm happy.

cloudinit/sources/DataSourceEc2.py:340: [W0106(expression-not-assigned), DataSourceEc2.fallback_interface] Expression "super(DataSourceEc2, self).fallback_interface" is assigned to nothing

review: Approve
~chad.smith/cloud-init:feature/os-local updated
83cb40b... by Chad Smith

lint: return the fallback_interface value returned by DataSource super

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:99dbd5ee17cbda93f6dd30772baedfe6c08a1cf6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/35/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/35/rebuild

review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:83cb40b45e57a2313bf20435609c28afb3302d9a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/36/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/36/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=cd1de5f4

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/cloudinit/sources/DataSourceCloudStack.py b/cloudinit/sources/DataSourceCloudStack.py
2index 0df545f..d4b758f 100644
3--- a/cloudinit/sources/DataSourceCloudStack.py
4+++ b/cloudinit/sources/DataSourceCloudStack.py
5@@ -68,6 +68,10 @@ class DataSourceCloudStack(sources.DataSource):
6
7 dsname = 'CloudStack'
8
9+ # Setup read_url parameters per get_url_params.
10+ url_max_wait = 120
11+ url_timeout = 50
12+
13 def __init__(self, sys_cfg, distro, paths):
14 sources.DataSource.__init__(self, sys_cfg, distro, paths)
15 self.seed_dir = os.path.join(paths.seed_dir, 'cs')
16@@ -80,33 +84,18 @@ class DataSourceCloudStack(sources.DataSource):
17 self.metadata_address = "http://%s/" % (self.vr_addr,)
18 self.cfg = {}
19
20- def _get_url_settings(self):
21- mcfg = self.ds_cfg
22- max_wait = 120
23- try:
24- max_wait = int(mcfg.get("max_wait", max_wait))
25- except Exception:
26- util.logexc(LOG, "Failed to get max wait. using %s", max_wait)
27+ def wait_for_metadata_service(self):
28+ url_params = self.get_url_params()
29
30- if max_wait == 0:
31+ if url_params.max_wait_seconds <= 0:
32 return False
33
34- timeout = 50
35- try:
36- timeout = int(mcfg.get("timeout", timeout))
37- except Exception:
38- util.logexc(LOG, "Failed to get timeout, using %s", timeout)
39-
40- return (max_wait, timeout)
41-
42- def wait_for_metadata_service(self):
43- (max_wait, timeout) = self._get_url_settings()
44-
45 urls = [uhelp.combine_url(self.metadata_address,
46 'latest/meta-data/instance-id')]
47 start_time = time.time()
48- url = uhelp.wait_for_url(urls=urls, max_wait=max_wait,
49- timeout=timeout, status_cb=LOG.warn)
50+ url = uhelp.wait_for_url(
51+ urls=urls, max_wait=url_params.max_wait_seconds,
52+ timeout=url_params.timeout_seconds, status_cb=LOG.warn)
53
54 if url:
55 LOG.debug("Using metadata source: '%s'", url)
56diff --git a/cloudinit/sources/DataSourceConfigDrive.py b/cloudinit/sources/DataSourceConfigDrive.py
57index 121cf21..4cb2897 100644
58--- a/cloudinit/sources/DataSourceConfigDrive.py
59+++ b/cloudinit/sources/DataSourceConfigDrive.py
60@@ -43,7 +43,7 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource):
61 self.version = None
62 self.ec2_metadata = None
63 self._network_config = None
64- self.network_json = None
65+ self.network_json = sources.UNSET
66 self.network_eni = None
67 self.known_macs = None
68 self.files = {}
69@@ -149,7 +149,7 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource):
70 @property
71 def network_config(self):
72 if self._network_config is None:
73- if self.network_json is not None:
74+ if self.network_json not in (None, sources.UNSET):
75 LOG.debug("network config provided via network_json")
76 self._network_config = openstack.convert_net_json(
77 self.network_json, known_macs=self.known_macs)
78diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourceEc2.py
79index 21e9ef8..968ab3f 100644
80--- a/cloudinit/sources/DataSourceEc2.py
81+++ b/cloudinit/sources/DataSourceEc2.py
82@@ -27,8 +27,6 @@ SKIP_METADATA_URL_CODES = frozenset([uhelp.NOT_FOUND])
83 STRICT_ID_PATH = ("datasource", "Ec2", "strict_id")
84 STRICT_ID_DEFAULT = "warn"
85
86-_unset = "_unset"
87-
88
89 class Platforms(object):
90 # TODO Rename and move to cloudinit.cloud.CloudNames
91@@ -59,15 +57,16 @@ class DataSourceEc2(sources.DataSource):
92 # for extended metadata content. IPv6 support comes in 2016-09-02
93 extended_metadata_versions = ['2016-09-02']
94
95+ # Setup read_url parameters per get_url_params.
96+ url_max_wait = 120
97+ url_timeout = 50
98+
99 _cloud_platform = None
100
101- _network_config = _unset # Used for caching calculated network config v1
102+ _network_config = sources.UNSET # Used to cache calculated network cfg v1
103
104 # Whether we want to get network configuration from the metadata service.
105- get_network_metadata = False
106-
107- # Track the discovered fallback nic for use in configuration generation.
108- _fallback_interface = None
109+ perform_dhcp_setup = False
110
111 def __init__(self, sys_cfg, distro, paths):
112 super(DataSourceEc2, self).__init__(sys_cfg, distro, paths)
113@@ -98,7 +97,7 @@ class DataSourceEc2(sources.DataSource):
114 elif self.cloud_platform == Platforms.NO_EC2_METADATA:
115 return False
116
117- if self.get_network_metadata: # Setup networking in init-local stage.
118+ if self.perform_dhcp_setup: # Setup networking in init-local stage.
119 if util.is_FreeBSD():
120 LOG.debug("FreeBSD doesn't support running dhclient with -sf")
121 return False
122@@ -158,27 +157,11 @@ class DataSourceEc2(sources.DataSource):
123 else:
124 return self.metadata['instance-id']
125
126- def _get_url_settings(self):
127- mcfg = self.ds_cfg
128- max_wait = 120
129- try:
130- max_wait = int(mcfg.get("max_wait", max_wait))
131- except Exception:
132- util.logexc(LOG, "Failed to get max wait. using %s", max_wait)
133-
134- timeout = 50
135- try:
136- timeout = max(0, int(mcfg.get("timeout", timeout)))
137- except Exception:
138- util.logexc(LOG, "Failed to get timeout, using %s", timeout)
139-
140- return (max_wait, timeout)
141-
142 def wait_for_metadata_service(self):
143 mcfg = self.ds_cfg
144
145- (max_wait, timeout) = self._get_url_settings()
146- if max_wait <= 0:
147+ url_params = self.get_url_params()
148+ if url_params.max_wait_seconds <= 0:
149 return False
150
151 # Remove addresses from the list that wont resolve.
152@@ -205,7 +188,8 @@ class DataSourceEc2(sources.DataSource):
153
154 start_time = time.time()
155 url = uhelp.wait_for_url(
156- urls=urls, max_wait=max_wait, timeout=timeout, status_cb=LOG.warn)
157+ urls=urls, max_wait=url_params.max_wait_seconds,
158+ timeout=url_params.timeout_seconds, status_cb=LOG.warn)
159
160 if url:
161 self.metadata_address = url2base[url]
162@@ -310,11 +294,11 @@ class DataSourceEc2(sources.DataSource):
163 @property
164 def network_config(self):
165 """Return a network config dict for rendering ENI or netplan files."""
166- if self._network_config != _unset:
167+ if self._network_config != sources.UNSET:
168 return self._network_config
169
170 if self.metadata is None:
171- # this would happen if get_data hadn't been called. leave as _unset
172+ # this would happen if get_data hadn't been called. leave as UNSET
173 LOG.warning(
174 "Unexpected call to network_config when metadata is None.")
175 return None
176@@ -353,9 +337,7 @@ class DataSourceEc2(sources.DataSource):
177 self._fallback_interface = _legacy_fbnic
178 self.fallback_nic = None
179 else:
180- self._fallback_interface = net.find_fallback_nic()
181- if self._fallback_interface is None:
182- LOG.warning("Did not find a fallback interface on EC2.")
183+ return super(DataSourceEc2, self).fallback_interface
184 return self._fallback_interface
185
186 def _crawl_metadata(self):
187@@ -390,7 +372,7 @@ class DataSourceEc2Local(DataSourceEc2):
188 metadata service. If the metadata service provides network configuration
189 then render the network configuration for that instance based on metadata.
190 """
191- get_network_metadata = True # Get metadata network config if present
192+ perform_dhcp_setup = True # Use dhcp before querying metadata
193
194 def get_data(self):
195 supported_platforms = (Platforms.AWS,)
196diff --git a/cloudinit/sources/DataSourceOpenStack.py b/cloudinit/sources/DataSourceOpenStack.py
197index fb166ae..1a12a3f 100644
198--- a/cloudinit/sources/DataSourceOpenStack.py
199+++ b/cloudinit/sources/DataSourceOpenStack.py
200@@ -7,6 +7,7 @@
201 import time
202
203 from cloudinit import log as logging
204+from cloudinit.net.dhcp import EphemeralDHCPv4, NoDHCPLeaseError
205 from cloudinit import sources
206 from cloudinit import url_helper
207 from cloudinit import util
208@@ -27,46 +28,25 @@ class DataSourceOpenStack(openstack.SourceMixin, sources.DataSource):
209
210 dsname = "OpenStack"
211
212+ _network_config = sources.UNSET # Used to cache calculated network cfg v1
213+
214+ # Whether we want to get network configuration from the metadata service.
215+ perform_dhcp_setup = False
216+
217 def __init__(self, sys_cfg, distro, paths):
218 super(DataSourceOpenStack, self).__init__(sys_cfg, distro, paths)
219 self.metadata_address = None
220 self.ssl_details = util.fetch_ssl_details(self.paths)
221 self.version = None
222 self.files = {}
223- self.ec2_metadata = None
224+ self.ec2_metadata = sources.UNSET
225+ self.network_json = sources.UNSET
226
227 def __str__(self):
228 root = sources.DataSource.__str__(self)
229 mstr = "%s [%s,ver=%s]" % (root, self.dsmode, self.version)
230 return mstr
231
232- def _get_url_settings(self):
233- # TODO(harlowja): this is shared with ec2 datasource, we should just
234- # move it to a shared location instead...
235- # Note: the defaults here are different though.
236-
237- # max_wait < 0 indicates do not wait
238- max_wait = -1
239- timeout = 10
240- retries = 5
241-
242- try:
243- max_wait = int(self.ds_cfg.get("max_wait", max_wait))
244- except Exception:
245- util.logexc(LOG, "Failed to get max wait. using %s", max_wait)
246-
247- try:
248- timeout = max(0, int(self.ds_cfg.get("timeout", timeout)))
249- except Exception:
250- util.logexc(LOG, "Failed to get timeout, using %s", timeout)
251-
252- try:
253- retries = int(self.ds_cfg.get("retries", retries))
254- except Exception:
255- util.logexc(LOG, "Failed to get retries. using %s", retries)
256-
257- return (max_wait, timeout, retries)
258-
259 def wait_for_metadata_service(self):
260 urls = self.ds_cfg.get("metadata_urls", [DEF_MD_URL])
261 filtered = [x for x in urls if util.is_resolvable_url(x)]
262@@ -86,10 +66,11 @@ class DataSourceOpenStack(openstack.SourceMixin, sources.DataSource):
263 md_urls.append(md_url)
264 url2base[md_url] = url
265
266- (max_wait, timeout, _retries) = self._get_url_settings()
267+ url_params = self.get_url_params()
268 start_time = time.time()
269- avail_url = url_helper.wait_for_url(urls=md_urls, max_wait=max_wait,
270- timeout=timeout)
271+ avail_url = url_helper.wait_for_url(
272+ urls=md_urls, max_wait=url_params.max_wait_seconds,
273+ timeout=url_params.timeout_seconds)
274 if avail_url:
275 LOG.debug("Using metadata source: '%s'", url2base[avail_url])
276 else:
277@@ -99,38 +80,64 @@ class DataSourceOpenStack(openstack.SourceMixin, sources.DataSource):
278 self.metadata_address = url2base.get(avail_url)
279 return bool(avail_url)
280
281- def _get_data(self):
282- try:
283- if not self.wait_for_metadata_service():
284- return False
285- except IOError:
286- return False
287+ def check_instance_id(self, sys_cfg):
288+ # quickly (local check only) if self.instance_id is still valid
289+ return sources.instance_id_matches_system_uuid(self.get_instance_id())
290
291- (_max_wait, timeout, retries) = self._get_url_settings()
292+ @property
293+ def network_config(self):
294+ """Return a network config dict for rendering ENI or netplan files."""
295+ if self._network_config != sources.UNSET:
296+ return self._network_config
297+
298+ # RELEASE_BLOCKER: SRU to Xenial and Artful SRU should not provide
299+ # network_config by default unless configured in /etc/cloud/cloud.cfg*.
300+ # Patch Xenial and Artful before release to default to False.
301+ if util.is_false(self.ds_cfg.get('apply_network_config', True)):
302+ self._network_config = None
303+ return self._network_config
304+ if self.network_json == sources.UNSET:
305+ # this would happen if get_data hadn't been called. leave as UNSET
306+ LOG.warning(
307+ 'Unexpected call to network_config when network_json is None.')
308+ return None
309+
310+ LOG.debug('network config provided via network_json')
311+ self._network_config = openstack.convert_net_json(
312+ self.network_json, known_macs=None)
313+ return self._network_config
314
315- try:
316- results = util.log_time(LOG.debug,
317- 'Crawl of openstack metadata service',
318- read_metadata_service,
319- args=[self.metadata_address],
320- kwargs={'ssl_details': self.ssl_details,
321- 'retries': retries,
322- 'timeout': timeout})
323- except openstack.NonReadable:
324- return False
325- except (openstack.BrokenMetadata, IOError):
326- util.logexc(LOG, "Broken metadata address %s",
327- self.metadata_address)
328- return False
329+ def _get_data(self):
330+ """Crawl metadata, parse and persist that data for this instance.
331+
332+ @return: True when metadata discovered indicates OpenStack datasource.
333+ False when unable to contact metadata service or when metadata
334+ format is invalid or disabled.
335+ """
336+ if self.perform_dhcp_setup: # Setup networking in init-local stage.
337+ try:
338+ with EphemeralDHCPv4(self.fallback_interface):
339+ results = util.log_time(
340+ logfunc=LOG.debug, msg='Crawl of metadata service',
341+ func=self._crawl_metadata)
342+ except (NoDHCPLeaseError, sources.InvalidMetaDataException) as e:
343+ util.logexc(LOG, str(e))
344+ return False
345+ else:
346+ try:
347+ results = self._crawl_metadata()
348+ except sources.InvalidMetaDataException as e:
349+ util.logexc(LOG, str(e))
350+ return False
351
352 self.dsmode = self._determine_dsmode([results.get('dsmode')])
353 if self.dsmode == sources.DSMODE_DISABLED:
354 return False
355-
356 md = results.get('metadata', {})
357 md = util.mergemanydict([md, DEFAULT_METADATA])
358 self.metadata = md
359 self.ec2_metadata = results.get('ec2-metadata')
360+ self.network_json = results.get('networkdata')
361 self.userdata_raw = results.get('userdata')
362 self.version = results['version']
363 self.files.update(results.get('files', {}))
364@@ -145,9 +152,50 @@ class DataSourceOpenStack(openstack.SourceMixin, sources.DataSource):
365
366 return True
367
368- def check_instance_id(self, sys_cfg):
369- # quickly (local check only) if self.instance_id is still valid
370- return sources.instance_id_matches_system_uuid(self.get_instance_id())
371+ def _crawl_metadata(self):
372+ """Crawl metadata service when available.
373+
374+ @returns: Dictionary with all metadata discovered for this datasource.
375+ @raise: InvalidMetaDataException on unreadable or broken
376+ metadata.
377+ """
378+ try:
379+ if not self.wait_for_metadata_service():
380+ raise sources.InvalidMetaDataException(
381+ 'No active metadata service found')
382+ except IOError as e:
383+ raise sources.InvalidMetaDataException(
384+ 'IOError contacting metadata service: {error}'.format(
385+ error=str(e)))
386+
387+ url_params = self.get_url_params()
388+
389+ try:
390+ result = util.log_time(
391+ LOG.debug, 'Crawl of openstack metadata service',
392+ read_metadata_service, args=[self.metadata_address],
393+ kwargs={'ssl_details': self.ssl_details,
394+ 'retries': url_params.num_retries,
395+ 'timeout': url_params.timeout_seconds})
396+ except openstack.NonReadable as e:
397+ raise sources.InvalidMetaDataException(str(e))
398+ except (openstack.BrokenMetadata, IOError):
399+ msg = 'Broken metadata address {addr}'.format(
400+ addr=self.metadata_address)
401+ raise sources.InvalidMetaDataException(msg)
402+ return result
403+
404+
405+class DataSourceOpenStackLocal(DataSourceOpenStack):
406+ """Run in init-local using a dhcp discovery prior to metadata crawl.
407+
408+ In init-local, no network is available. This subclass sets up minimal
409+ networking with dhclient on a viable nic so that it can talk to the
410+ metadata service. If the metadata service provides network configuration
411+ then render the network configuration for that instance based on metadata.
412+ """
413+
414+ perform_dhcp_setup = True # Get metadata network config if present
415
416
417 def read_metadata_service(base_url, ssl_details=None,
418@@ -159,6 +207,7 @@ def read_metadata_service(base_url, ssl_details=None,
419
420 # Used to match classes to dependencies
421 datasources = [
422+ (DataSourceOpenStackLocal, (sources.DEP_FILESYSTEM,)),
423 (DataSourceOpenStack, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
424 ]
425
426diff --git a/cloudinit/sources/DataSourceSmartOS.py b/cloudinit/sources/DataSourceSmartOS.py
427index fcb46b1..c91e4d5 100644
428--- a/cloudinit/sources/DataSourceSmartOS.py
429+++ b/cloudinit/sources/DataSourceSmartOS.py
430@@ -165,9 +165,8 @@ class DataSourceSmartOS(sources.DataSource):
431
432 dsname = "Joyent"
433
434- _unset = "_unset"
435- smartos_type = _unset
436- md_client = _unset
437+ smartos_type = sources.UNSET
438+ md_client = sources.UNSET
439
440 def __init__(self, sys_cfg, distro, paths):
441 sources.DataSource.__init__(self, sys_cfg, distro, paths)
442@@ -189,12 +188,12 @@ class DataSourceSmartOS(sources.DataSource):
443 return "%s [client=%s]" % (root, self.md_client)
444
445 def _init(self):
446- if self.smartos_type == self._unset:
447+ if self.smartos_type == sources.UNSET:
448 self.smartos_type = get_smartos_environ()
449 if self.smartos_type is None:
450 self.md_client = None
451
452- if self.md_client == self._unset:
453+ if self.md_client == sources.UNSET:
454 self.md_client = jmc_client_factory(
455 smartos_type=self.smartos_type,
456 metadata_sockfile=self.ds_cfg['metadata_sockfile'],
457diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
458index df0b374..90d7457 100644
459--- a/cloudinit/sources/__init__.py
460+++ b/cloudinit/sources/__init__.py
461@@ -9,6 +9,7 @@
462 # This file is part of cloud-init. See LICENSE file for license information.
463
464 import abc
465+from collections import namedtuple
466 import copy
467 import json
468 import os
469@@ -17,6 +18,7 @@ import six
470 from cloudinit.atomic_helper import write_json
471 from cloudinit import importer
472 from cloudinit import log as logging
473+from cloudinit import net
474 from cloudinit import type_utils
475 from cloudinit import user_data as ud
476 from cloudinit import util
477@@ -41,6 +43,8 @@ INSTANCE_JSON_FILE = 'instance-data.json'
478 # Key which can be provide a cloud's official product name to cloud-init
479 METADATA_CLOUD_NAME_KEY = 'cloud-name'
480
481+UNSET = "_unset"
482+
483 LOG = logging.getLogger(__name__)
484
485
486@@ -48,6 +52,11 @@ class DataSourceNotFoundException(Exception):
487 pass
488
489
490+class InvalidMetaDataException(Exception):
491+ """Raised when metadata is broken, unavailable or disabled."""
492+ pass
493+
494+
495 def process_base64_metadata(metadata, key_path=''):
496 """Strip ci-b64 prefix and return metadata with base64-encoded-keys set."""
497 md_copy = copy.deepcopy(metadata)
498@@ -68,6 +77,10 @@ def process_base64_metadata(metadata, key_path=''):
499 return md_copy
500
501
502+URLParams = namedtuple(
503+ 'URLParms', ['max_wait_seconds', 'timeout_seconds', 'num_retries'])
504+
505+
506 @six.add_metaclass(abc.ABCMeta)
507 class DataSource(object):
508
509@@ -81,6 +94,14 @@ class DataSource(object):
510 # Cached cloud_name as determined by _get_cloud_name
511 _cloud_name = None
512
513+ # Track the discovered fallback nic for use in configuration generation.
514+ _fallback_interface = None
515+
516+ # read_url_params
517+ url_max_wait = -1 # max_wait < 0 means do not wait
518+ url_timeout = 10 # timeout for each metadata url read attempt
519+ url_retries = 5 # number of times to retry url upon 404
520+
521 def __init__(self, sys_cfg, distro, paths, ud_proc=None):
522 self.sys_cfg = sys_cfg
523 self.distro = distro
524@@ -128,6 +149,14 @@ class DataSource(object):
525 'meta-data': self.metadata,
526 'user-data': self.get_userdata_raw(),
527 'vendor-data': self.get_vendordata_raw()}}
528+ if hasattr(self, 'network_json'):
529+ network_json = getattr(self, 'network_json')
530+ if network_json != UNSET:
531+ instance_data['ds']['network_json'] = network_json
532+ if hasattr(self, 'ec2_metadata'):
533+ ec2_metadata = getattr(self, 'ec2_metadata')
534+ if ec2_metadata != UNSET:
535+ instance_data['ds']['ec2_metadata'] = ec2_metadata
536 instance_data.update(
537 self._get_standardized_metadata())
538 try:
539@@ -149,6 +178,42 @@ class DataSource(object):
540 'Subclasses of DataSource must implement _get_data which'
541 ' sets self.metadata, vendordata_raw and userdata_raw.')
542
543+ def get_url_params(self):
544+ """Return the Datasource's prefered url_read parameters.
545+
546+ Subclasses may override url_max_wait, url_timeout, url_retries.
547+
548+ @return: A URLParams object with max_wait_seconds, timeout_seconds,
549+ num_retries.
550+ """
551+ max_wait = self.url_max_wait
552+ try:
553+ max_wait = int(self.ds_cfg.get("max_wait", self.url_max_wait))
554+ except ValueError:
555+ util.logexc(
556+ LOG, "Config max_wait '%s' is not an int, using default '%s'",
557+ self.ds_cfg.get("max_wait"), max_wait)
558+
559+ timeout = self.url_timeout
560+ try:
561+ timeout = max(
562+ 0, int(self.ds_cfg.get("timeout", self.url_timeout)))
563+ except ValueError:
564+ timeout = self.url_timeout
565+ util.logexc(
566+ LOG, "Config timeout '%s' is not an int, using default '%s'",
567+ self.ds_cfg.get('timeout'), timeout)
568+
569+ retries = self.url_retries
570+ try:
571+ retries = int(self.ds_cfg.get("retries", self.url_retries))
572+ except Exception:
573+ util.logexc(
574+ LOG, "Config retries '%s' is not an int, using default '%s'",
575+ self.ds_cfg.get('retries'), retries)
576+
577+ return URLParams(max_wait, timeout, retries)
578+
579 def get_userdata(self, apply_filter=False):
580 if self.userdata is None:
581 self.userdata = self.ud_proc.process(self.get_userdata_raw())
582@@ -162,6 +227,17 @@ class DataSource(object):
583 return self.vendordata
584
585 @property
586+ def fallback_interface(self):
587+ """Determine the network interface used during local network config."""
588+ if self._fallback_interface is None:
589+ self._fallback_interface = net.find_fallback_nic()
590+ if self._fallback_interface is None:
591+ LOG.warning(
592+ "Did not find a fallback interface on %s.",
593+ self.cloud_name)
594+ return self._fallback_interface
595+
596+ @property
597 def cloud_name(self):
598 """Return lowercase cloud name as determined by the datasource.
599
600diff --git a/cloudinit/sources/tests/test_init.py b/cloudinit/sources/tests/test_init.py
601index 452e921..d5bc98a 100644
602--- a/cloudinit/sources/tests/test_init.py
603+++ b/cloudinit/sources/tests/test_init.py
604@@ -17,6 +17,7 @@ from cloudinit import util
605 class DataSourceTestSubclassNet(DataSource):
606
607 dsname = 'MyTestSubclass'
608+ url_max_wait = 55
609
610 def __init__(self, sys_cfg, distro, paths, custom_userdata=None):
611 super(DataSourceTestSubclassNet, self).__init__(
612@@ -70,8 +71,7 @@ class TestDataSource(CiTestCase):
613 """Init uses DataSource.dsname for sourcing ds_cfg."""
614 sys_cfg = {'datasource': {'MyTestSubclass': {'key2': False}}}
615 distro = 'distrotest' # generally should be a Distro object
616- paths = Paths({})
617- datasource = DataSourceTestSubclassNet(sys_cfg, distro, paths)
618+ datasource = DataSourceTestSubclassNet(sys_cfg, distro, self.paths)
619 self.assertEqual({'key2': False}, datasource.ds_cfg)
620
621 def test_str_is_classname(self):
622@@ -81,6 +81,91 @@ class TestDataSource(CiTestCase):
623 'DataSourceTestSubclassNet',
624 str(DataSourceTestSubclassNet('', '', self.paths)))
625
626+ def test_datasource_get_url_params_defaults(self):
627+ """get_url_params default url config settings for the datasource."""
628+ params = self.datasource.get_url_params()
629+ self.assertEqual(params.max_wait_seconds, self.datasource.url_max_wait)
630+ self.assertEqual(params.timeout_seconds, self.datasource.url_timeout)
631+ self.assertEqual(params.num_retries, self.datasource.url_retries)
632+
633+ def test_datasource_get_url_params_subclassed(self):
634+ """Subclasses can override get_url_params defaults."""
635+ sys_cfg = {'datasource': {'MyTestSubclass': {'key2': False}}}
636+ distro = 'distrotest' # generally should be a Distro object
637+ datasource = DataSourceTestSubclassNet(sys_cfg, distro, self.paths)
638+ expected = (datasource.url_max_wait, datasource.url_timeout,
639+ datasource.url_retries)
640+ url_params = datasource.get_url_params()
641+ self.assertNotEqual(self.datasource.get_url_params(), url_params)
642+ self.assertEqual(expected, url_params)
643+
644+ def test_datasource_get_url_params_ds_config_override(self):
645+ """Datasource configuration options can override url param defaults."""
646+ sys_cfg = {
647+ 'datasource': {
648+ 'MyTestSubclass': {
649+ 'max_wait': '1', 'timeout': '2', 'retries': '3'}}}
650+ datasource = DataSourceTestSubclassNet(
651+ sys_cfg, self.distro, self.paths)
652+ expected = (1, 2, 3)
653+ url_params = datasource.get_url_params()
654+ self.assertNotEqual(
655+ (datasource.url_max_wait, datasource.url_timeout,
656+ datasource.url_retries),
657+ url_params)
658+ self.assertEqual(expected, url_params)
659+
660+ def test_datasource_get_url_params_is_zero_or_greater(self):
661+ """get_url_params ignores timeouts with a value below 0."""
662+ # Set an override that is below 0 which gets ignored.
663+ sys_cfg = {'datasource': {'_undef': {'timeout': '-1'}}}
664+ datasource = DataSource(sys_cfg, self.distro, self.paths)
665+ (_max_wait, timeout, _retries) = datasource.get_url_params()
666+ self.assertEqual(0, timeout)
667+
668+ def test_datasource_get_url_uses_defaults_on_errors(self):
669+ """On invalid system config values for url_params defaults are used."""
670+ # All invalid values should be logged
671+ sys_cfg = {'datasource': {
672+ '_undef': {
673+ 'max_wait': 'nope', 'timeout': 'bug', 'retries': 'nonint'}}}
674+ datasource = DataSource(sys_cfg, self.distro, self.paths)
675+ url_params = datasource.get_url_params()
676+ expected = (datasource.url_max_wait, datasource.url_timeout,
677+ datasource.url_retries)
678+ self.assertEqual(expected, url_params)
679+ logs = self.logs.getvalue()
680+ expected_logs = [
681+ "Config max_wait 'nope' is not an int, using default '-1'",
682+ "Config timeout 'bug' is not an int, using default '10'",
683+ "Config retries 'nonint' is not an int, using default '5'",
684+ ]
685+ for log in expected_logs:
686+ self.assertIn(log, logs)
687+
688+ @mock.patch('cloudinit.sources.net.find_fallback_nic')
689+ def test_fallback_interface_is_discovered(self, m_get_fallback_nic):
690+ """The fallback_interface is discovered via find_fallback_nic."""
691+ m_get_fallback_nic.return_value = 'nic9'
692+ self.assertEqual('nic9', self.datasource.fallback_interface)
693+
694+ @mock.patch('cloudinit.sources.net.find_fallback_nic')
695+ def test_fallback_interface_logs_undiscovered(self, m_get_fallback_nic):
696+ """Log a warning when fallback_interface can not discover the nic."""
697+ self.datasource._cloud_name = 'MySupahCloud'
698+ m_get_fallback_nic.return_value = None # Couldn't discover nic
699+ self.assertIsNone(self.datasource.fallback_interface)
700+ self.assertEqual(
701+ 'WARNING: Did not find a fallback interface on MySupahCloud.\n',
702+ self.logs.getvalue())
703+
704+ @mock.patch('cloudinit.sources.net.find_fallback_nic')
705+ def test_wb_fallback_interface_is_cached(self, m_get_fallback_nic):
706+ """The fallback_interface is cached and won't be rediscovered."""
707+ self.datasource._fallback_interface = 'nic10'
708+ self.assertEqual('nic10', self.datasource.fallback_interface)
709+ m_get_fallback_nic.assert_not_called()
710+
711 def test__get_data_unimplemented(self):
712 """Raise an error when _get_data is not implemented."""
713 with self.assertRaises(NotImplementedError) as context_manager:
714diff --git a/tests/unittests/test_datasource/test_common.py b/tests/unittests/test_datasource/test_common.py
715index ec33388..0d35dc2 100644
716--- a/tests/unittests/test_datasource/test_common.py
717+++ b/tests/unittests/test_datasource/test_common.py
718@@ -40,6 +40,7 @@ DEFAULT_LOCAL = [
719 OVF.DataSourceOVF,
720 SmartOS.DataSourceSmartOS,
721 Ec2.DataSourceEc2Local,
722+ OpenStack.DataSourceOpenStackLocal,
723 ]
724
725 DEFAULT_NETWORK = [
726diff --git a/tests/unittests/test_datasource/test_openstack.py b/tests/unittests/test_datasource/test_openstack.py
727index bb180c0..fad73b2 100644
728--- a/tests/unittests/test_datasource/test_openstack.py
729+++ b/tests/unittests/test_datasource/test_openstack.py
730@@ -16,7 +16,7 @@ from six import StringIO
731
732 from cloudinit import helpers
733 from cloudinit import settings
734-from cloudinit.sources import convert_vendordata
735+from cloudinit.sources import convert_vendordata, UNSET
736 from cloudinit.sources import DataSourceOpenStack as ds
737 from cloudinit.sources.helpers import openstack
738 from cloudinit import util
739@@ -129,6 +129,8 @@ def _read_metadata_service():
740
741
742 class TestOpenStackDataSource(test_helpers.HttprettyTestCase):
743+
744+ with_logs = True
745 VERSION = 'latest'
746
747 def setUp(self):
748@@ -223,11 +225,11 @@ class TestOpenStackDataSource(test_helpers.HttprettyTestCase):
749 _register_uris(self.VERSION, {}, {}, os_files)
750 self.assertRaises(openstack.BrokenMetadata, _read_metadata_service)
751
752- def test_datasource(self):
753+ @test_helpers.mock.patch('cloudinit.net.dhcp.maybe_perform_dhcp_discovery')
754+ def test_datasource(self, m_dhcp):
755 _register_uris(self.VERSION, EC2_FILES, EC2_META, OS_FILES)
756- ds_os = ds.DataSourceOpenStack(settings.CFG_BUILTIN,
757- None,
758- helpers.Paths({'run_dir': self.tmp}))
759+ ds_os = ds.DataSourceOpenStack(
760+ settings.CFG_BUILTIN, None, helpers.Paths({'run_dir': self.tmp}))
761 self.assertIsNone(ds_os.version)
762 found = ds_os.get_data()
763 self.assertTrue(found)
764@@ -241,6 +243,36 @@ class TestOpenStackDataSource(test_helpers.HttprettyTestCase):
765 self.assertEqual(2, len(ds_os.files))
766 self.assertEqual(VENDOR_DATA, ds_os.vendordata_pure)
767 self.assertIsNone(ds_os.vendordata_raw)
768+ m_dhcp.assert_not_called()
769+
770+ @hp.activate
771+ @test_helpers.mock.patch('cloudinit.net.dhcp.EphemeralIPv4Network')
772+ @test_helpers.mock.patch('cloudinit.net.dhcp.maybe_perform_dhcp_discovery')
773+ def test_local_datasource(self, m_dhcp, m_net):
774+ """OpenStackLocal calls EphemeralDHCPNetwork and gets instance data."""
775+ _register_uris(self.VERSION, EC2_FILES, EC2_META, OS_FILES)
776+ ds_os_local = ds.DataSourceOpenStackLocal(
777+ settings.CFG_BUILTIN, None, helpers.Paths({'run_dir': self.tmp}))
778+ ds_os_local._fallback_interface = 'eth9' # Monkey patch for dhcp
779+ m_dhcp.return_value = [{
780+ 'interface': 'eth9', 'fixed-address': '192.168.2.9',
781+ 'routers': '192.168.2.1', 'subnet-mask': '255.255.255.0',
782+ 'broadcast-address': '192.168.2.255'}]
783+
784+ self.assertIsNone(ds_os_local.version)
785+ found = ds_os_local.get_data()
786+ self.assertTrue(found)
787+ self.assertEqual(2, ds_os_local.version)
788+ md = dict(ds_os_local.metadata)
789+ md.pop('instance-id', None)
790+ md.pop('local-hostname', None)
791+ self.assertEqual(OSTACK_META, md)
792+ self.assertEqual(EC2_META, ds_os_local.ec2_metadata)
793+ self.assertEqual(USER_DATA, ds_os_local.userdata_raw)
794+ self.assertEqual(2, len(ds_os_local.files))
795+ self.assertEqual(VENDOR_DATA, ds_os_local.vendordata_pure)
796+ self.assertIsNone(ds_os_local.vendordata_raw)
797+ m_dhcp.assert_called_with('eth9')
798
799 def test_bad_datasource_meta(self):
800 os_files = copy.deepcopy(OS_FILES)
801@@ -255,6 +287,10 @@ class TestOpenStackDataSource(test_helpers.HttprettyTestCase):
802 found = ds_os.get_data()
803 self.assertFalse(found)
804 self.assertIsNone(ds_os.version)
805+ self.assertIn(
806+ 'InvalidMetaDataException: Broken metadata address'
807+ ' http://169.254.169.25',
808+ self.logs.getvalue())
809
810 def test_no_datasource(self):
811 os_files = copy.deepcopy(OS_FILES)
812@@ -274,6 +310,52 @@ class TestOpenStackDataSource(test_helpers.HttprettyTestCase):
813 self.assertFalse(found)
814 self.assertIsNone(ds_os.version)
815
816+ def test_network_config_disabled_by_datasource_config(self):
817+ """The network_config can be disabled from datasource config."""
818+ mock_path = (
819+ 'cloudinit.sources.DataSourceOpenStack.openstack.'
820+ 'convert_net_json')
821+ ds_os = ds.DataSourceOpenStack(
822+ settings.CFG_BUILTIN, None, helpers.Paths({'run_dir': self.tmp}))
823+ ds_os.ds_cfg = {'apply_network_config': False}
824+ sample_json = {'links': [{'ethernet_mac_address': 'mymac'}],
825+ 'networks': [], 'services': []}
826+ ds_os.network_json = sample_json # Ignore this content from metadata
827+ with test_helpers.mock.patch(mock_path) as m_convert_json:
828+ self.assertIsNone(ds_os.network_config)
829+ m_convert_json.assert_not_called()
830+
831+ def test_network_config_from_network_json(self):
832+ """The datasource gets network_config from network_data.json."""
833+ mock_path = (
834+ 'cloudinit.sources.DataSourceOpenStack.openstack.'
835+ 'convert_net_json')
836+ example_cfg = {'version': 1, 'config': []}
837+ ds_os = ds.DataSourceOpenStack(
838+ settings.CFG_BUILTIN, None, helpers.Paths({'run_dir': self.tmp}))
839+ sample_json = {'links': [{'ethernet_mac_address': 'mymac'}],
840+ 'networks': [], 'services': []}
841+ ds_os.network_json = sample_json
842+ with test_helpers.mock.patch(mock_path) as m_convert_json:
843+ m_convert_json.return_value = example_cfg
844+ self.assertEqual(example_cfg, ds_os.network_config)
845+ self.assertIn(
846+ 'network config provided via network_json', self.logs.getvalue())
847+ m_convert_json.assert_called_with(sample_json, known_macs=None)
848+
849+ def test_network_config_cached(self):
850+ """The datasource caches the network_config property."""
851+ mock_path = (
852+ 'cloudinit.sources.DataSourceOpenStack.openstack.'
853+ 'convert_net_json')
854+ example_cfg = {'version': 1, 'config': []}
855+ ds_os = ds.DataSourceOpenStack(
856+ settings.CFG_BUILTIN, None, helpers.Paths({'run_dir': self.tmp}))
857+ ds_os._network_config = example_cfg
858+ with test_helpers.mock.patch(mock_path) as m_convert_json:
859+ self.assertEqual(example_cfg, ds_os.network_config)
860+ m_convert_json.assert_not_called()
861+
862 def test_disabled_datasource(self):
863 os_files = copy.deepcopy(OS_FILES)
864 os_meta = copy.deepcopy(OSTACK_META)
865@@ -296,6 +378,35 @@ class TestOpenStackDataSource(test_helpers.HttprettyTestCase):
866 self.assertFalse(found)
867 self.assertIsNone(ds_os.version)
868
869+ @hp.activate
870+ def test_wb__crawl_metadata_does_not_persist(self):
871+ """_crawl_metadata returns current metadata and does not cache."""
872+ _register_uris(self.VERSION, EC2_FILES, EC2_META, OS_FILES)
873+ ds_os = ds.DataSourceOpenStack(
874+ settings.CFG_BUILTIN, None, helpers.Paths({'run_dir': self.tmp}))
875+ crawled_data = ds_os._crawl_metadata()
876+ self.assertEqual(UNSET, ds_os.ec2_metadata)
877+ self.assertIsNone(ds_os.userdata_raw)
878+ self.assertEqual(0, len(ds_os.files))
879+ self.assertIsNone(ds_os.vendordata_raw)
880+ self.assertEqual(
881+ ['dsmode', 'ec2-metadata', 'files', 'metadata', 'networkdata',
882+ 'userdata', 'vendordata', 'version'],
883+ sorted(crawled_data.keys()))
884+ self.assertEqual('local', crawled_data['dsmode'])
885+ self.assertEqual(EC2_META, crawled_data['ec2-metadata'])
886+ self.assertEqual(2, len(crawled_data['files']))
887+ md = copy.deepcopy(crawled_data['metadata'])
888+ md.pop('instance-id')
889+ md.pop('local-hostname')
890+ self.assertEqual(OSTACK_META, md)
891+ self.assertEqual(
892+ json.loads(OS_FILES['openstack/latest/network_data.json']),
893+ crawled_data['networkdata'])
894+ self.assertEqual(USER_DATA, crawled_data['userdata'])
895+ self.assertEqual(VENDOR_DATA, crawled_data['vendordata'])
896+ self.assertEqual(2, crawled_data['version'])
897+
898
899 class TestVendorDataLoading(test_helpers.TestCase):
900 def cvj(self, data):

Subscribers

People subscribed via source and target branches