Merge ~smoser/cloud-init:azure_run_local into cloud-init:master

Proposed by Scott Moser on 2017-06-27
Status: Merged
Merged at revision: ebc9ecbc8a76bdf511a456fb72339a7eb4c20568
Proposed branch: ~smoser/cloud-init:azure_run_local
Merge into: cloud-init:master
Diff against target: 1463 lines (+887/-98)
11 files modified
cloudinit/cmd/main.py (+3/-0)
cloudinit/net/__init__.py (+154/-27)
cloudinit/net/eni.py (+2/-0)
cloudinit/net/renderer.py (+3/-1)
cloudinit/net/udev.py (+5/-2)
cloudinit/sources/DataSourceAzure.py (+95/-19)
cloudinit/sources/__init__.py (+14/-1)
cloudinit/stages.py (+5/-0)
tests/unittests/test_datasource/test_azure.py (+142/-32)
tests/unittests/test_datasource/test_common.py (+1/-1)
tests/unittests/test_net.py (+463/-15)
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve on 2017-06-27
cloud-init commiters 2017-06-27 Pending
Review via email: mp+326373@code.launchpad.net

Commit message

Azure: Add network-config, Refactor net layer to handle duplicate macs.

On systems with network devices with duplicate mac addresses, cloud-init
will fail to rename the devices according to the specified network
configuration. Refactor net layer to search by device driver and device
id if available. Azure systems may have duplicate mac addresses by
design.

Update Azure datasource to run at init-local time and let Azure datasource
generate a fallback networking config to handle advanced networking
configurations.

Lastly, add a 'setup' method to the datasources that is called before
userdata/vendordata is processed but after networking is up. That is
used here on Azure to interact with the 'fabric'.

Description of the change

To post a comment you must log in.

FAILED: Continuous integration, rev:5110bdc6723d7d7d4001256513f65a496663eb17
https://jenkins.ubuntu.com/server/job/cloud-init-ci/15/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/15/rebuild

review: Needs Fixing (continuous-integration)
Ryan Harper (raharper) :
~smoser/cloud-init:azure_run_local updated on 2017-06-27
34f9a51... by Scott Moser on 2017-06-27

tests: fix TestAzureDataSource

5d3f6f2... by Scott Moser on 2017-06-27

show is_new_instance in debug message.

66ae29e... by Scott Moser on 2017-06-27

change a info to a warn.

we moved this to run later, after 'get_data' succeeded.
At this poing (during 'setup') its too late to turn back.
we turn this into a warning instead.

a4297db... by Scott Moser on 2017-06-27

tests: fix TestAzureBounce

51a1571... by Scott Moser on 2017-06-27

add a comment for setup_datasource usage

6e31f5e... by Scott Moser on 2017-06-27

drop obsolete comment

bcdb00d... by Scott Moser on 2017-06-27

better debug message

9417e77... by Scott Moser on 2017-06-27

add a docstring to _negotiate

Scott Moser (smoser) :

FAILED: Continuous integration, rev:bcdb00d9a3b910a434f436f74f582bcdc20ada71
https://jenkins.ubuntu.com/server/job/cloud-init-ci/16/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/16/rebuild

review: Needs Fixing (continuous-integration)
~smoser/cloud-init:azure_run_local updated on 2017-06-27
fed926e... by Scott Moser on 2017-06-27

fix flake8

Ryan Harper (raharper) wrote :

That looks good.

PASSED: Continuous integration, rev:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/17/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/17/rebuild

review: Approve (continuous-integration)

PASSED: Continuous integration, rev:fed926e838c0984e4b94ddaea60eb38197e4a5ed
https://jenkins.ubuntu.com/server/job/cloud-init-ci/18/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/18/rebuild

review: Approve (continuous-integration)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/cloudinit/cmd/main.py b/cloudinit/cmd/main.py
2index ce3c10d..139e03b 100644
3--- a/cloudinit/cmd/main.py
4+++ b/cloudinit/cmd/main.py
5@@ -372,6 +372,9 @@ def main_init(name, args):
6 LOG.debug("[%s] %s is in local mode, will apply init modules now.",
7 mode, init.datasource)
8
9+ # Give the datasource a chance to use network resources.
10+ # This is used on Azure to communicate with the fabric over network.
11+ init.setup_datasource()
12 # update fully realizes user-data (pulling in #include if necessary)
13 init.update()
14 # Stage 7
15diff --git a/cloudinit/net/__init__.py b/cloudinit/net/__init__.py
16index 65accbb..cba991a 100644
17--- a/cloudinit/net/__init__.py
18+++ b/cloudinit/net/__init__.py
19@@ -97,6 +97,10 @@ def is_bridge(devname):
20 return os.path.exists(sys_dev_path(devname, "bridge"))
21
22
23+def is_bond(devname):
24+ return os.path.exists(sys_dev_path(devname, "bonding"))
25+
26+
27 def is_vlan(devname):
28 uevent = str(read_sys_net_safe(devname, "uevent"))
29 return 'DEVTYPE=vlan' in uevent.splitlines()
30@@ -124,6 +128,26 @@ def is_present(devname):
31 return os.path.exists(sys_dev_path(devname))
32
33
34+def device_driver(devname):
35+ """Return the device driver for net device named 'devname'."""
36+ driver = None
37+ driver_path = sys_dev_path(devname, "device/driver")
38+ # driver is a symlink to the driver *dir*
39+ if os.path.islink(driver_path):
40+ driver = os.path.basename(os.readlink(driver_path))
41+
42+ return driver
43+
44+
45+def device_devid(devname):
46+ """Return the device id string for net device named 'devname'."""
47+ dev_id = read_sys_net_safe(devname, "device/device")
48+ if dev_id is False:
49+ return None
50+
51+ return dev_id
52+
53+
54 def get_devicelist():
55 return os.listdir(SYS_CLASS_NET)
56
57@@ -138,12 +162,21 @@ def is_disabled_cfg(cfg):
58 return cfg.get('config') == "disabled"
59
60
61-def generate_fallback_config():
62+def generate_fallback_config(blacklist_drivers=None, config_driver=None):
63 """Determine which attached net dev is most likely to have a connection and
64 generate network state to run dhcp on that interface"""
65+
66+ if not config_driver:
67+ config_driver = False
68+
69+ if not blacklist_drivers:
70+ blacklist_drivers = []
71+
72 # get list of interfaces that could have connections
73 invalid_interfaces = set(['lo'])
74- potential_interfaces = set(get_devicelist())
75+ potential_interfaces = set([device for device in get_devicelist()
76+ if device_driver(device) not in
77+ blacklist_drivers])
78 potential_interfaces = potential_interfaces.difference(invalid_interfaces)
79 # sort into interfaces with carrier, interfaces which could have carrier,
80 # and ignore interfaces that are definitely disconnected
81@@ -155,6 +188,9 @@ def generate_fallback_config():
82 if is_bridge(interface):
83 # skip any bridges
84 continue
85+ if is_bond(interface):
86+ # skip any bonds
87+ continue
88 carrier = read_sys_net_int(interface, 'carrier')
89 if carrier:
90 connected.append(interface)
91@@ -194,9 +230,18 @@ def generate_fallback_config():
92 break
93 if target_mac and target_name:
94 nconf = {'config': [], 'version': 1}
95- nconf['config'].append(
96- {'type': 'physical', 'name': target_name,
97- 'mac_address': target_mac, 'subnets': [{'type': 'dhcp'}]})
98+ cfg = {'type': 'physical', 'name': target_name,
99+ 'mac_address': target_mac, 'subnets': [{'type': 'dhcp'}]}
100+ # inject the device driver name, dev_id into config if enabled and
101+ # device has a valid device driver value
102+ if config_driver:
103+ driver = device_driver(target_name)
104+ if driver:
105+ cfg['params'] = {
106+ 'driver': driver,
107+ 'device_id': device_devid(target_name),
108+ }
109+ nconf['config'].append(cfg)
110 return nconf
111 else:
112 # can't read any interfaces addresses (or there are none); give up
113@@ -217,10 +262,16 @@ def apply_network_config_names(netcfg, strict_present=True, strict_busy=True):
114 if ent.get('type') != 'physical':
115 continue
116 mac = ent.get('mac_address')
117- name = ent.get('name')
118 if not mac:
119 continue
120- renames.append([mac, name])
121+ name = ent.get('name')
122+ driver = ent.get('params', {}).get('driver')
123+ device_id = ent.get('params', {}).get('device_id')
124+ if not driver:
125+ driver = device_driver(name)
126+ if not device_id:
127+ device_id = device_devid(name)
128+ renames.append([mac, name, driver, device_id])
129
130 return _rename_interfaces(renames)
131
132@@ -245,15 +296,27 @@ def _get_current_rename_info(check_downable=True):
133 """Collect information necessary for rename_interfaces.
134
135 returns a dictionary by mac address like:
136- {mac:
137- {'name': name
138- 'up': boolean: is_up(name),
139+ {name:
140+ {
141 'downable': None or boolean indicating that the
142- device has only automatically assigned ip addrs.}}
143+ device has only automatically assigned ip addrs.
144+ 'device_id': Device id value (if it has one)
145+ 'driver': Device driver (if it has one)
146+ 'mac': mac address
147+ 'name': name
148+ 'up': boolean: is_up(name)
149+ }}
150 """
151- bymac = {}
152- for mac, name in get_interfaces_by_mac().items():
153- bymac[mac] = {'name': name, 'up': is_up(name), 'downable': None}
154+ cur_info = {}
155+ for (name, mac, driver, device_id) in get_interfaces():
156+ cur_info[name] = {
157+ 'downable': None,
158+ 'device_id': device_id,
159+ 'driver': driver,
160+ 'mac': mac,
161+ 'name': name,
162+ 'up': is_up(name),
163+ }
164
165 if check_downable:
166 nmatch = re.compile(r"[0-9]+:\s+(\w+)[@:]")
167@@ -265,11 +328,11 @@ def _get_current_rename_info(check_downable=True):
168 for bytes_out in (ipv6, ipv4):
169 nics_with_addresses.update(nmatch.findall(bytes_out))
170
171- for d in bymac.values():
172+ for d in cur_info.values():
173 d['downable'] = (d['up'] is False or
174 d['name'] not in nics_with_addresses)
175
176- return bymac
177+ return cur_info
178
179
180 def _rename_interfaces(renames, strict_present=True, strict_busy=True,
181@@ -282,15 +345,15 @@ def _rename_interfaces(renames, strict_present=True, strict_busy=True,
182 if current_info is None:
183 current_info = _get_current_rename_info()
184
185- cur_bymac = {}
186- for mac, data in current_info.items():
187+ cur_info = {}
188+ for name, data in current_info.items():
189 cur = data.copy()
190- cur['mac'] = mac
191- cur_bymac[mac] = cur
192+ cur['name'] = name
193+ cur_info[name] = cur
194
195 def update_byname(bymac):
196 return dict((data['name'], data)
197- for data in bymac.values())
198+ for data in cur_info.values())
199
200 def rename(cur, new):
201 util.subp(["ip", "link", "set", cur, "name", new], capture=True)
202@@ -304,14 +367,48 @@ def _rename_interfaces(renames, strict_present=True, strict_busy=True,
203 ops = []
204 errors = []
205 ups = []
206- cur_byname = update_byname(cur_bymac)
207+ cur_byname = update_byname(cur_info)
208 tmpname_fmt = "cirename%d"
209 tmpi = -1
210
211- for mac, new_name in renames:
212- cur = cur_bymac.get(mac, {})
213- cur_name = cur.get('name')
214+ def entry_match(data, mac, driver, device_id):
215+ """match if set and in data"""
216+ if mac and driver and device_id:
217+ return (data['mac'] == mac and
218+ data['driver'] == driver and
219+ data['device_id'] == device_id)
220+ elif mac and driver:
221+ return (data['mac'] == mac and
222+ data['driver'] == driver)
223+ elif mac:
224+ return (data['mac'] == mac)
225+
226+ return False
227+
228+ def find_entry(mac, driver, device_id):
229+ match = [data for data in cur_info.values()
230+ if entry_match(data, mac, driver, device_id)]
231+ if len(match):
232+ if len(match) > 1:
233+ msg = ('Failed to match a single device. Matched devices "%s"'
234+ ' with search values "(mac:%s driver:%s device_id:%s)"'
235+ % (match, mac, driver, device_id))
236+ raise ValueError(msg)
237+ return match[0]
238+
239+ return None
240+
241+ for mac, new_name, driver, device_id in renames:
242 cur_ops = []
243+ cur = find_entry(mac, driver, device_id)
244+ if not cur:
245+ if strict_present:
246+ errors.append(
247+ "[nic not present] Cannot rename mac=%s to %s"
248+ ", not available." % (mac, new_name))
249+ continue
250+
251+ cur_name = cur.get('name')
252 if cur_name == new_name:
253 # nothing to do
254 continue
255@@ -351,13 +448,13 @@ def _rename_interfaces(renames, strict_present=True, strict_busy=True,
256
257 cur_ops.append(("rename", mac, new_name, (new_name, tmp_name)))
258 target['name'] = tmp_name
259- cur_byname = update_byname(cur_bymac)
260+ cur_byname = update_byname(cur_info)
261 if target['up']:
262 ups.append(("up", mac, new_name, (tmp_name,)))
263
264 cur_ops.append(("rename", mac, new_name, (cur['name'], new_name)))
265 cur['name'] = new_name
266- cur_byname = update_byname(cur_bymac)
267+ cur_byname = update_byname(cur_info)
268 ops += cur_ops
269
270 opmap = {'rename': rename, 'down': down, 'up': up}
271@@ -426,6 +523,36 @@ def get_interfaces_by_mac():
272 return ret
273
274
275+def get_interfaces():
276+ """Return list of interface tuples (name, mac, driver, device_id)
277+
278+ Bridges and any devices that have a 'stolen' mac are excluded."""
279+ try:
280+ devs = get_devicelist()
281+ except OSError as e:
282+ if e.errno == errno.ENOENT:
283+ devs = []
284+ else:
285+ raise
286+ ret = []
287+ empty_mac = '00:00:00:00:00:00'
288+ for name in devs:
289+ if not interface_has_own_mac(name):
290+ continue
291+ if is_bridge(name):
292+ continue
293+ if is_vlan(name):
294+ continue
295+ mac = get_interface_mac(name)
296+ # some devices may not have a mac (tun0)
297+ if not mac:
298+ continue
299+ if mac == empty_mac and name != 'lo':
300+ continue
301+ ret.append((name, mac, device_driver(name), device_devid(name)))
302+ return ret
303+
304+
305 class RendererNotFoundError(RuntimeError):
306 pass
307
308diff --git a/cloudinit/net/eni.py b/cloudinit/net/eni.py
309index 98ce01e..b707146 100644
310--- a/cloudinit/net/eni.py
311+++ b/cloudinit/net/eni.py
312@@ -72,6 +72,8 @@ def _iface_add_attrs(iface, index):
313 content = []
314 ignore_map = [
315 'control',
316+ 'device_id',
317+ 'driver',
318 'index',
319 'inet',
320 'mode',
321diff --git a/cloudinit/net/renderer.py b/cloudinit/net/renderer.py
322index c68658d..bba139e 100644
323--- a/cloudinit/net/renderer.py
324+++ b/cloudinit/net/renderer.py
325@@ -34,8 +34,10 @@ class Renderer(object):
326 for iface in network_state.iter_interfaces(filter_by_physical):
327 # for physical interfaces write out a persist net udev rule
328 if 'name' in iface and iface.get('mac_address'):
329+ driver = iface.get('driver', None)
330 content.write(generate_udev_rule(iface['name'],
331- iface['mac_address']))
332+ iface['mac_address'],
333+ driver=driver))
334 return content.getvalue()
335
336 @abc.abstractmethod
337diff --git a/cloudinit/net/udev.py b/cloudinit/net/udev.py
338index fd2fd8c..58c0a70 100644
339--- a/cloudinit/net/udev.py
340+++ b/cloudinit/net/udev.py
341@@ -23,7 +23,7 @@ def compose_udev_setting(key, value):
342 return '%s="%s"' % (key, value)
343
344
345-def generate_udev_rule(interface, mac):
346+def generate_udev_rule(interface, mac, driver=None):
347 """Return a udev rule to set the name of network interface with `mac`.
348
349 The rule ends up as a single line looking something like:
350@@ -31,10 +31,13 @@ def generate_udev_rule(interface, mac):
351 SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*",
352 ATTR{address}="ff:ee:dd:cc:bb:aa", NAME="eth0"
353 """
354+ if not driver:
355+ driver = '?*'
356+
357 rule = ', '.join([
358 compose_udev_equality('SUBSYSTEM', 'net'),
359 compose_udev_equality('ACTION', 'add'),
360- compose_udev_equality('DRIVERS', '?*'),
361+ compose_udev_equality('DRIVERS', driver),
362 compose_udev_attr_equality('address', mac),
363 compose_udev_setting('NAME', interface),
364 ])
365diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py
366index 4fe0d63..b5a95a1 100644
367--- a/cloudinit/sources/DataSourceAzure.py
368+++ b/cloudinit/sources/DataSourceAzure.py
369@@ -16,6 +16,7 @@ from xml.dom import minidom
370 import xml.etree.ElementTree as ET
371
372 from cloudinit import log as logging
373+from cloudinit import net
374 from cloudinit import sources
375 from cloudinit.sources.helpers.azure import get_metadata_from_fabric
376 from cloudinit import util
377@@ -245,7 +246,9 @@ def temporary_hostname(temp_hostname, cfg, hostname_command='hostname'):
378 set_hostname(previous_hostname, hostname_command)
379
380
381-class DataSourceAzureNet(sources.DataSource):
382+class DataSourceAzure(sources.DataSource):
383+ _negotiated = False
384+
385 def __init__(self, sys_cfg, distro, paths):
386 sources.DataSource.__init__(self, sys_cfg, distro, paths)
387 self.seed_dir = os.path.join(paths.seed_dir, 'azure')
388@@ -255,6 +258,7 @@ class DataSourceAzureNet(sources.DataSource):
389 util.get_cfg_by_path(sys_cfg, DS_CFG_PATH, {}),
390 BUILTIN_DS_CONFIG])
391 self.dhclient_lease_file = self.ds_cfg.get('dhclient_lease_file')
392+ self._network_config = None
393
394 def __str__(self):
395 root = sources.DataSource.__str__(self)
396@@ -331,6 +335,7 @@ class DataSourceAzureNet(sources.DataSource):
397 if asset_tag != AZURE_CHASSIS_ASSET_TAG:
398 LOG.debug("Non-Azure DMI asset tag '%s' discovered.", asset_tag)
399 return False
400+
401 ddir = self.ds_cfg['data_dir']
402
403 candidates = [self.seed_dir]
404@@ -375,13 +380,14 @@ class DataSourceAzureNet(sources.DataSource):
405 LOG.debug("using files cached in %s", ddir)
406
407 # azure / hyper-v provides random data here
408+ # TODO. find the seed on FreeBSD platform
409+ # now update ds_cfg to reflect contents pass in config
410 if not util.is_FreeBSD():
411 seed = util.load_file("/sys/firmware/acpi/tables/OEM0",
412 quiet=True, decode=False)
413 if seed:
414 self.metadata['random_seed'] = seed
415- # TODO. find the seed on FreeBSD platform
416- # now update ds_cfg to reflect contents pass in config
417+
418 user_ds_cfg = util.get_cfg_by_path(self.cfg, DS_CFG_PATH, {})
419 self.ds_cfg = util.mergemanydict([user_ds_cfg, self.ds_cfg])
420
421@@ -389,6 +395,40 @@ class DataSourceAzureNet(sources.DataSource):
422 # the directory to be protected.
423 write_files(ddir, files, dirmode=0o700)
424
425+ self.metadata['instance-id'] = util.read_dmi_data('system-uuid')
426+
427+ return True
428+
429+ def device_name_to_device(self, name):
430+ return self.ds_cfg['disk_aliases'].get(name)
431+
432+ def get_config_obj(self):
433+ return self.cfg
434+
435+ def check_instance_id(self, sys_cfg):
436+ # quickly (local check only) if self.instance_id is still valid
437+ return sources.instance_id_matches_system_uuid(self.get_instance_id())
438+
439+ def setup(self, is_new_instance):
440+ if self._negotiated is False:
441+ LOG.debug("negotiating for %s (new_instance=%s)",
442+ self.get_instance_id(), is_new_instance)
443+ fabric_data = self._negotiate()
444+ LOG.debug("negotiating returned %s", fabric_data)
445+ if fabric_data:
446+ self.metadata.update(fabric_data)
447+ self._negotiated = True
448+ else:
449+ LOG.debug("negotiating already done for %s",
450+ self.get_instance_id())
451+
452+ def _negotiate(self):
453+ """Negotiate with fabric and return data from it.
454+
455+ On success, returns a dictionary including 'public_keys'.
456+ On failure, returns False.
457+ """
458+
459 if self.ds_cfg['agent_command'] == AGENT_START_BUILTIN:
460 self.bounce_network_with_azure_hostname()
461
462@@ -398,31 +438,64 @@ class DataSourceAzureNet(sources.DataSource):
463 else:
464 metadata_func = self.get_metadata_from_agent
465
466+ LOG.debug("negotiating with fabric via agent command %s",
467+ self.ds_cfg['agent_command'])
468 try:
469 fabric_data = metadata_func()
470 except Exception as exc:
471- LOG.info("Error communicating with Azure fabric; assume we aren't"
472- " on Azure.", exc_info=True)
473+ LOG.warning(
474+ "Error communicating with Azure fabric; You may experience."
475+ "connectivity issues.", exc_info=True)
476 return False
477- self.metadata['instance-id'] = util.read_dmi_data('system-uuid')
478- self.metadata.update(fabric_data)
479-
480- return True
481
482- def device_name_to_device(self, name):
483- return self.ds_cfg['disk_aliases'].get(name)
484-
485- def get_config_obj(self):
486- return self.cfg
487-
488- def check_instance_id(self, sys_cfg):
489- # quickly (local check only) if self.instance_id is still valid
490- return sources.instance_id_matches_system_uuid(self.get_instance_id())
491+ return fabric_data
492
493 def activate(self, cfg, is_new_instance):
494 address_ephemeral_resize(is_new_instance=is_new_instance)
495 return
496
497+ @property
498+ def network_config(self):
499+ """Generate a network config like net.generate_fallback_network() with
500+ the following execptions.
501+
502+ 1. Probe the drivers of the net-devices present and inject them in
503+ the network configuration under params: driver: <driver> value
504+ 2. If the driver value is 'mlx4_core', the control mode should be
505+ set to manual. The device will be later used to build a bond,
506+ for now we want to ensure the device gets named but does not
507+ break any network configuration
508+ """
509+ blacklist = ['mlx4_core']
510+ if not self._network_config:
511+ LOG.debug('Azure: generating fallback configuration')
512+ # generate a network config, blacklist picking any mlx4_core devs
513+ netconfig = net.generate_fallback_config(
514+ blacklist_drivers=blacklist, config_driver=True)
515+
516+ # if we have any blacklisted devices, update the network_config to
517+ # include the device, mac, and driver values, but with no ip
518+ # config; this ensures udev rules are generated but won't affect
519+ # ip configuration
520+ bl_found = 0
521+ for bl_dev in [dev for dev in net.get_devicelist()
522+ if net.device_driver(dev) in blacklist]:
523+ bl_found += 1
524+ cfg = {
525+ 'type': 'physical',
526+ 'name': 'vf%d' % bl_found,
527+ 'mac_address': net.get_interface_mac(bl_dev),
528+ 'params': {
529+ 'driver': net.device_driver(bl_dev),
530+ 'device_id': net.device_devid(bl_dev),
531+ },
532+ }
533+ netconfig['config'].append(cfg)
534+
535+ self._network_config = netconfig
536+
537+ return self._network_config
538+
539
540 def _partitions_on_device(devpath, maxnum=16):
541 # return a list of tuples (ptnum, path) for each part on devpath
542@@ -849,9 +922,12 @@ class NonAzureDataSource(Exception):
543 pass
544
545
546+# Legacy: Must be present in case we load an old pkl object
547+DataSourceAzureNet = DataSourceAzure
548+
549 # Used to match classes to dependencies
550 datasources = [
551- (DataSourceAzureNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
552+ (DataSourceAzure, (sources.DEP_FILESYSTEM, )),
553 ]
554
555
556diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
557index c3ce36d..952caf3 100644
558--- a/cloudinit/sources/__init__.py
559+++ b/cloudinit/sources/__init__.py
560@@ -251,10 +251,23 @@ class DataSource(object):
561 def first_instance_boot(self):
562 return
563
564+ def setup(self, is_new_instance):
565+ """setup(is_new_instance)
566+
567+ This is called before user-data and vendor-data have been processed.
568+
569+ Unless the datasource has set mode to 'local', then networking
570+ per 'fallback' or per 'network_config' will have been written and
571+ brought up the OS at this point.
572+ """
573+ return
574+
575 def activate(self, cfg, is_new_instance):
576 """activate(cfg, is_new_instance)
577
578- This is called before the init_modules will be called.
579+ This is called before the init_modules will be called but after
580+ the user-data and vendor-data have been fully processed.
581+
582 The cfg is fully up to date config, it contains a merged view of
583 system config, datasource config, user config, vendor config.
584 It should be used rather than the sys_cfg passed to __init__.
585diff --git a/cloudinit/stages.py b/cloudinit/stages.py
586index ad55782..a1c4a51 100644
587--- a/cloudinit/stages.py
588+++ b/cloudinit/stages.py
589@@ -362,6 +362,11 @@ class Init(object):
590 self._store_userdata()
591 self._store_vendordata()
592
593+ def setup_datasource(self):
594+ if self.datasource is None:
595+ raise RuntimeError("Datasource is None, cannot setup.")
596+ self.datasource.setup(is_new_instance=self.is_new_instance())
597+
598 def activate_datasource(self):
599 if self.datasource is None:
600 raise RuntimeError("Datasource is None, cannot activate.")
601diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py
602index 7d33daf..20e70fb 100644
603--- a/tests/unittests/test_datasource/test_azure.py
604+++ b/tests/unittests/test_datasource/test_azure.py
605@@ -181,13 +181,19 @@ scbus-1 on xpt0 bus 0
606 side_effect=_dmi_mocks)),
607 ])
608
609- dsrc = dsaz.DataSourceAzureNet(
610+ dsrc = dsaz.DataSourceAzure(
611 data.get('sys_cfg', {}), distro=None, paths=self.paths)
612 if agent_command is not None:
613 dsrc.ds_cfg['agent_command'] = agent_command
614
615 return dsrc
616
617+ def _get_and_setup(self, dsrc):
618+ ret = dsrc.get_data()
619+ if ret:
620+ dsrc.setup(True)
621+ return ret
622+
623 def xml_equals(self, oxml, nxml):
624 """Compare two sets of XML to make sure they are equal"""
625
626@@ -259,7 +265,7 @@ fdescfs /dev/fd fdescfs rw 0 0
627 # Return a non-matching asset tag value
628 nonazure_tag = dsaz.AZURE_CHASSIS_ASSET_TAG + 'X'
629 m_read_dmi_data.return_value = nonazure_tag
630- dsrc = dsaz.DataSourceAzureNet(
631+ dsrc = dsaz.DataSourceAzure(
632 {}, distro=None, paths=self.paths)
633 self.assertFalse(dsrc.get_data())
634 self.assertEqual(
635@@ -299,7 +305,7 @@ fdescfs /dev/fd fdescfs rw 0 0
636 data = {'ovfcontent': construct_valid_ovf_env(data=odata)}
637
638 dsrc = self._get_ds(data)
639- ret = dsrc.get_data()
640+ ret = self._get_and_setup(dsrc)
641 self.assertTrue(ret)
642 self.assertEqual(data['agent_invoked'], cfg['agent_command'])
643
644@@ -312,7 +318,7 @@ fdescfs /dev/fd fdescfs rw 0 0
645 data = {'ovfcontent': construct_valid_ovf_env(data=odata)}
646
647 dsrc = self._get_ds(data)
648- ret = dsrc.get_data()
649+ ret = self._get_and_setup(dsrc)
650 self.assertTrue(ret)
651 self.assertEqual(data['agent_invoked'], cfg['agent_command'])
652
653@@ -322,7 +328,7 @@ fdescfs /dev/fd fdescfs rw 0 0
654 'sys_cfg': sys_cfg}
655
656 dsrc = self._get_ds(data)
657- ret = dsrc.get_data()
658+ ret = self._get_and_setup(dsrc)
659 self.assertTrue(ret)
660 self.assertEqual(data['agent_invoked'], '_COMMAND')
661
662@@ -394,7 +400,7 @@ fdescfs /dev/fd fdescfs rw 0 0
663 pubkeys=pubkeys)}
664
665 dsrc = self._get_ds(data, agent_command=['not', '__builtin__'])
666- ret = dsrc.get_data()
667+ ret = self._get_and_setup(dsrc)
668 self.assertTrue(ret)
669 for mypk in mypklist:
670 self.assertIn(mypk, dsrc.cfg['_pubkeys'])
671@@ -409,7 +415,7 @@ fdescfs /dev/fd fdescfs rw 0 0
672 pubkeys=pubkeys)}
673
674 dsrc = self._get_ds(data, agent_command=['not', '__builtin__'])
675- ret = dsrc.get_data()
676+ ret = self._get_and_setup(dsrc)
677 self.assertTrue(ret)
678
679 for mypk in mypklist:
680@@ -425,7 +431,7 @@ fdescfs /dev/fd fdescfs rw 0 0
681 pubkeys=pubkeys)}
682
683 dsrc = self._get_ds(data, agent_command=['not', '__builtin__'])
684- ret = dsrc.get_data()
685+ ret = self._get_and_setup(dsrc)
686 self.assertTrue(ret)
687
688 for mypk in mypklist:
689@@ -519,18 +525,20 @@ fdescfs /dev/fd fdescfs rw 0 0
690 dsrc.get_data()
691
692 def test_exception_fetching_fabric_data_doesnt_propagate(self):
693- ds = self._get_ds({'ovfcontent': construct_valid_ovf_env()})
694- ds.ds_cfg['agent_command'] = '__builtin__'
695+ """Errors communicating with fabric should warn, but return True."""
696+ dsrc = self._get_ds({'ovfcontent': construct_valid_ovf_env()})
697+ dsrc.ds_cfg['agent_command'] = '__builtin__'
698 self.get_metadata_from_fabric.side_effect = Exception
699- self.assertFalse(ds.get_data())
700+ ret = self._get_and_setup(dsrc)
701+ self.assertTrue(ret)
702
703 def test_fabric_data_included_in_metadata(self):
704- ds = self._get_ds({'ovfcontent': construct_valid_ovf_env()})
705- ds.ds_cfg['agent_command'] = '__builtin__'
706+ dsrc = self._get_ds({'ovfcontent': construct_valid_ovf_env()})
707+ dsrc.ds_cfg['agent_command'] = '__builtin__'
708 self.get_metadata_from_fabric.return_value = {'test': 'value'}
709- ret = ds.get_data()
710+ ret = self._get_and_setup(dsrc)
711 self.assertTrue(ret)
712- self.assertEqual('value', ds.metadata['test'])
713+ self.assertEqual('value', dsrc.metadata['test'])
714
715 def test_instance_id_from_dmidecode_used(self):
716 ds = self._get_ds({'ovfcontent': construct_valid_ovf_env()})
717@@ -554,6 +562,84 @@ fdescfs /dev/fd fdescfs rw 0 0
718 self.assertEqual(
719 [mock.call("/dev/cd0")], m_check_fbsd_cdrom.call_args_list)
720
721+ @mock.patch('cloudinit.net.get_interface_mac')
722+ @mock.patch('cloudinit.net.get_devicelist')
723+ @mock.patch('cloudinit.net.device_driver')
724+ @mock.patch('cloudinit.net.generate_fallback_config')
725+ def test_network_config(self, mock_fallback, mock_dd,
726+ mock_devlist, mock_get_mac):
727+ odata = {'HostName': "myhost", 'UserName': "myuser"}
728+ data = {'ovfcontent': construct_valid_ovf_env(data=odata),
729+ 'sys_cfg': {}}
730+
731+ fallback_config = {
732+ 'version': 1,
733+ 'config': [{
734+ 'type': 'physical', 'name': 'eth0',
735+ 'mac_address': '00:11:22:33:44:55',
736+ 'params': {'driver': 'hv_netsvc'},
737+ 'subnets': [{'type': 'dhcp'}],
738+ }]
739+ }
740+ mock_fallback.return_value = fallback_config
741+
742+ mock_devlist.return_value = ['eth0']
743+ mock_dd.return_value = ['hv_netsvc']
744+ mock_get_mac.return_value = '00:11:22:33:44:55'
745+
746+ dsrc = self._get_ds(data)
747+ ret = dsrc.get_data()
748+ self.assertTrue(ret)
749+
750+ netconfig = dsrc.network_config
751+ self.assertEqual(netconfig, fallback_config)
752+ mock_fallback.assert_called_with(blacklist_drivers=['mlx4_core'],
753+ config_driver=True)
754+
755+ @mock.patch('cloudinit.net.get_interface_mac')
756+ @mock.patch('cloudinit.net.get_devicelist')
757+ @mock.patch('cloudinit.net.device_driver')
758+ @mock.patch('cloudinit.net.generate_fallback_config')
759+ def test_network_config_blacklist(self, mock_fallback, mock_dd,
760+ mock_devlist, mock_get_mac):
761+ odata = {'HostName': "myhost", 'UserName': "myuser"}
762+ data = {'ovfcontent': construct_valid_ovf_env(data=odata),
763+ 'sys_cfg': {}}
764+
765+ fallback_config = {
766+ 'version': 1,
767+ 'config': [{
768+ 'type': 'physical', 'name': 'eth0',
769+ 'mac_address': '00:11:22:33:44:55',
770+ 'params': {'driver': 'hv_netsvc'},
771+ 'subnets': [{'type': 'dhcp'}],
772+ }]
773+ }
774+ blacklist_config = {
775+ 'type': 'physical',
776+ 'name': 'eth1',
777+ 'mac_address': '00:11:22:33:44:55',
778+ 'params': {'driver': 'mlx4_core'}
779+ }
780+ mock_fallback.return_value = fallback_config
781+
782+ mock_devlist.return_value = ['eth0', 'eth1']
783+ mock_dd.side_effect = [
784+ 'hv_netsvc', # list composition, skipped
785+ 'mlx4_core', # list composition, match
786+ 'mlx4_core', # config get driver name
787+ ]
788+ mock_get_mac.return_value = '00:11:22:33:44:55'
789+
790+ dsrc = self._get_ds(data)
791+ ret = dsrc.get_data()
792+ self.assertTrue(ret)
793+
794+ netconfig = dsrc.network_config
795+ expected_config = fallback_config
796+ expected_config['config'].append(blacklist_config)
797+ self.assertEqual(netconfig, expected_config)
798+
799
800 class TestAzureBounce(TestCase):
801
802@@ -603,12 +689,18 @@ class TestAzureBounce(TestCase):
803 if ovfcontent is not None:
804 populate_dir(os.path.join(self.paths.seed_dir, "azure"),
805 {'ovf-env.xml': ovfcontent})
806- dsrc = dsaz.DataSourceAzureNet(
807+ dsrc = dsaz.DataSourceAzure(
808 {}, distro=None, paths=self.paths)
809 if agent_command is not None:
810 dsrc.ds_cfg['agent_command'] = agent_command
811 return dsrc
812
813+ def _get_and_setup(self, dsrc):
814+ ret = dsrc.get_data()
815+ if ret:
816+ dsrc.setup(True)
817+ return ret
818+
819 def get_ovf_env_with_dscfg(self, hostname, cfg):
820 odata = {
821 'HostName': hostname,
822@@ -652,17 +744,20 @@ class TestAzureBounce(TestCase):
823 host_name = 'unchanged-host-name'
824 self.get_hostname.return_value = host_name
825 cfg = {'hostname_bounce': {'policy': 'force'}}
826- self._get_ds(self.get_ovf_env_with_dscfg(host_name, cfg),
827- agent_command=['not', '__builtin__']).get_data()
828+ dsrc = self._get_ds(self.get_ovf_env_with_dscfg(host_name, cfg),
829+ agent_command=['not', '__builtin__'])
830+ ret = self._get_and_setup(dsrc)
831+ self.assertTrue(ret)
832 self.assertEqual(1, perform_hostname_bounce.call_count)
833
834 def test_different_hostnames_sets_hostname(self):
835 expected_hostname = 'azure-expected-host-name'
836 self.get_hostname.return_value = 'default-host-name'
837- self._get_ds(
838+ dsrc = self._get_ds(
839 self.get_ovf_env_with_dscfg(expected_hostname, {}),
840- agent_command=['not', '__builtin__'],
841- ).get_data()
842+ agent_command=['not', '__builtin__'])
843+ ret = self._get_and_setup(dsrc)
844+ self.assertTrue(ret)
845 self.assertEqual(expected_hostname,
846 self.set_hostname.call_args_list[0][0][0])
847
848@@ -671,19 +766,21 @@ class TestAzureBounce(TestCase):
849 self, perform_hostname_bounce):
850 expected_hostname = 'azure-expected-host-name'
851 self.get_hostname.return_value = 'default-host-name'
852- self._get_ds(
853+ dsrc = self._get_ds(
854 self.get_ovf_env_with_dscfg(expected_hostname, {}),
855- agent_command=['not', '__builtin__'],
856- ).get_data()
857+ agent_command=['not', '__builtin__'])
858+ ret = self._get_and_setup(dsrc)
859+ self.assertTrue(ret)
860 self.assertEqual(1, perform_hostname_bounce.call_count)
861
862 def test_different_hostnames_sets_hostname_back(self):
863 initial_host_name = 'default-host-name'
864 self.get_hostname.return_value = initial_host_name
865- self._get_ds(
866+ dsrc = self._get_ds(
867 self.get_ovf_env_with_dscfg('some-host-name', {}),
868- agent_command=['not', '__builtin__'],
869- ).get_data()
870+ agent_command=['not', '__builtin__'])
871+ ret = self._get_and_setup(dsrc)
872+ self.assertTrue(ret)
873 self.assertEqual(initial_host_name,
874 self.set_hostname.call_args_list[-1][0][0])
875
876@@ -693,10 +790,11 @@ class TestAzureBounce(TestCase):
877 perform_hostname_bounce.side_effect = Exception
878 initial_host_name = 'default-host-name'
879 self.get_hostname.return_value = initial_host_name
880- self._get_ds(
881+ dsrc = self._get_ds(
882 self.get_ovf_env_with_dscfg('some-host-name', {}),
883- agent_command=['not', '__builtin__'],
884- ).get_data()
885+ agent_command=['not', '__builtin__'])
886+ ret = self._get_and_setup(dsrc)
887+ self.assertTrue(ret)
888 self.assertEqual(initial_host_name,
889 self.set_hostname.call_args_list[-1][0][0])
890
891@@ -707,7 +805,9 @@ class TestAzureBounce(TestCase):
892 self.get_hostname.return_value = old_hostname
893 cfg = {'hostname_bounce': {'interface': interface, 'policy': 'force'}}
894 data = self.get_ovf_env_with_dscfg(hostname, cfg)
895- self._get_ds(data, agent_command=['not', '__builtin__']).get_data()
896+ dsrc = self._get_ds(data, agent_command=['not', '__builtin__'])
897+ ret = self._get_and_setup(dsrc)
898+ self.assertTrue(ret)
899 self.assertEqual(1, self.subp.call_count)
900 bounce_env = self.subp.call_args[1]['env']
901 self.assertEqual(interface, bounce_env['interface'])
902@@ -719,7 +819,9 @@ class TestAzureBounce(TestCase):
903 dsaz.BUILTIN_DS_CONFIG['hostname_bounce']['command'] = cmd
904 cfg = {'hostname_bounce': {'policy': 'force'}}
905 data = self.get_ovf_env_with_dscfg('some-hostname', cfg)
906- self._get_ds(data, agent_command=['not', '__builtin__']).get_data()
907+ dsrc = self._get_ds(data, agent_command=['not', '__builtin__'])
908+ ret = self._get_and_setup(dsrc)
909+ self.assertTrue(ret)
910 self.assertEqual(1, self.subp.call_count)
911 bounce_args = self.subp.call_args[1]['args']
912 self.assertEqual(cmd, bounce_args)
913@@ -975,4 +1077,12 @@ class TestCanDevBeReformatted(CiTestCase):
914 self.assertEqual(False, value)
915 self.assertIn("3 or more", msg.lower())
916
917+
918+class TestAzureNetExists(CiTestCase):
919+ def test_azure_net_must_exist_for_legacy_objpkl(self):
920+ """DataSourceAzureNet must exist for old obj.pkl files
921+ that reference it."""
922+ self.assertTrue(hasattr(dsaz, "DataSourceAzureNet"))
923+
924+
925 # vi: ts=4 expandtab
926diff --git a/tests/unittests/test_datasource/test_common.py b/tests/unittests/test_datasource/test_common.py
927index 7649b9a..2ff1d9d 100644
928--- a/tests/unittests/test_datasource/test_common.py
929+++ b/tests/unittests/test_datasource/test_common.py
930@@ -26,6 +26,7 @@ from cloudinit.sources import DataSourceNone as DSNone
931 from .. import helpers as test_helpers
932
933 DEFAULT_LOCAL = [
934+ Azure.DataSourceAzure,
935 CloudSigma.DataSourceCloudSigma,
936 ConfigDrive.DataSourceConfigDrive,
937 DigitalOcean.DataSourceDigitalOcean,
938@@ -38,7 +39,6 @@ DEFAULT_LOCAL = [
939 DEFAULT_NETWORK = [
940 AliYun.DataSourceAliYun,
941 AltCloud.DataSourceAltCloud,
942- Azure.DataSourceAzureNet,
943 Bigstep.DataSourceBigstep,
944 CloudStack.DataSourceCloudStack,
945 DSNone.DataSourceNone,
946diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py
947index 8edc0b8..06e8f09 100644
948--- a/tests/unittests/test_net.py
949+++ b/tests/unittests/test_net.py
950@@ -836,38 +836,176 @@ CONFIG_V1_EXPLICIT_LOOPBACK = {
951 'subnets': [{'control': 'auto', 'type': 'loopback'}]},
952 ]}
953
954+DEFAULT_DEV_ATTRS = {
955+ 'eth1000': {
956+ "bridge": False,
957+ "carrier": False,
958+ "dormant": False,
959+ "operstate": "down",
960+ "address": "07-1C-C6-75-A4-BE",
961+ "device/driver": None,
962+ "device/device": None,
963+ }
964+}
965+
966
967 def _setup_test(tmp_dir, mock_get_devicelist, mock_read_sys_net,
968- mock_sys_dev_path):
969- mock_get_devicelist.return_value = ['eth1000']
970- dev_characteristics = {
971- 'eth1000': {
972- "bridge": False,
973- "carrier": False,
974- "dormant": False,
975- "operstate": "down",
976- "address": "07-1C-C6-75-A4-BE",
977- }
978- }
979+ mock_sys_dev_path, dev_attrs=None):
980+ if not dev_attrs:
981+ dev_attrs = DEFAULT_DEV_ATTRS
982+
983+ mock_get_devicelist.return_value = dev_attrs.keys()
984
985 def fake_read(devname, path, translate=None,
986 on_enoent=None, on_keyerror=None,
987 on_einval=None):
988- return dev_characteristics[devname][path]
989+ return dev_attrs[devname][path]
990
991 mock_read_sys_net.side_effect = fake_read
992
993 def sys_dev_path(devname, path=""):
994- return tmp_dir + devname + "/" + path
995+ return tmp_dir + "/" + devname + "/" + path
996
997- for dev in dev_characteristics:
998+ for dev in dev_attrs:
999 os.makedirs(os.path.join(tmp_dir, dev))
1000 with open(os.path.join(tmp_dir, dev, 'operstate'), 'w') as fh:
1001- fh.write("down")
1002+ fh.write(dev_attrs[dev]['operstate'])
1003+ os.makedirs(os.path.join(tmp_dir, dev, "device"))
1004+ for key in ['device/driver']:
1005+ if key in dev_attrs[dev] and dev_attrs[dev][key]:
1006+ target = dev_attrs[dev][key]
1007+ link = os.path.join(tmp_dir, dev, key)
1008+ print('symlink %s -> %s' % (link, target))
1009+ os.symlink(target, link)
1010
1011 mock_sys_dev_path.side_effect = sys_dev_path
1012
1013
1014+class TestGenerateFallbackConfig(CiTestCase):
1015+
1016+ @mock.patch("cloudinit.net.sys_dev_path")
1017+ @mock.patch("cloudinit.net.read_sys_net")
1018+ @mock.patch("cloudinit.net.get_devicelist")
1019+ def test_device_driver(self, mock_get_devicelist, mock_read_sys_net,
1020+ mock_sys_dev_path):
1021+ devices = {
1022+ 'eth0': {
1023+ 'bridge': False, 'carrier': False, 'dormant': False,
1024+ 'operstate': 'down', 'address': '00:11:22:33:44:55',
1025+ 'device/driver': 'hv_netsvc', 'device/device': '0x3'},
1026+ 'eth1': {
1027+ 'bridge': False, 'carrier': False, 'dormant': False,
1028+ 'operstate': 'down', 'address': '00:11:22:33:44:55',
1029+ 'device/driver': 'mlx4_core', 'device/device': '0x7'},
1030+ }
1031+
1032+ tmp_dir = self.tmp_dir()
1033+ _setup_test(tmp_dir, mock_get_devicelist,
1034+ mock_read_sys_net, mock_sys_dev_path,
1035+ dev_attrs=devices)
1036+
1037+ network_cfg = net.generate_fallback_config(config_driver=True)
1038+ ns = network_state.parse_net_config_data(network_cfg,
1039+ skip_broken=False)
1040+
1041+ render_dir = os.path.join(tmp_dir, "render")
1042+ os.makedirs(render_dir)
1043+
1044+ # don't set rulepath so eni writes them
1045+ renderer = eni.Renderer(
1046+ {'eni_path': 'interfaces', 'netrules_path': 'netrules'})
1047+ renderer.render_network_state(ns, render_dir)
1048+
1049+ self.assertTrue(os.path.exists(os.path.join(render_dir,
1050+ 'interfaces')))
1051+ with open(os.path.join(render_dir, 'interfaces')) as fh:
1052+ contents = fh.read()
1053+ print(contents)
1054+ expected = """
1055+auto lo
1056+iface lo inet loopback
1057+
1058+auto eth0
1059+iface eth0 inet dhcp
1060+"""
1061+ self.assertEqual(expected.lstrip(), contents.lstrip())
1062+
1063+ self.assertTrue(os.path.exists(os.path.join(render_dir, 'netrules')))
1064+ with open(os.path.join(render_dir, 'netrules')) as fh:
1065+ contents = fh.read()
1066+ print(contents)
1067+ expected_rule = [
1068+ 'SUBSYSTEM=="net"',
1069+ 'ACTION=="add"',
1070+ 'DRIVERS=="hv_netsvc"',
1071+ 'ATTR{address}=="00:11:22:33:44:55"',
1072+ 'NAME="eth0"',
1073+ ]
1074+ self.assertEqual(", ".join(expected_rule) + '\n', contents.lstrip())
1075+
1076+ @mock.patch("cloudinit.net.sys_dev_path")
1077+ @mock.patch("cloudinit.net.read_sys_net")
1078+ @mock.patch("cloudinit.net.get_devicelist")
1079+ def test_device_driver_blacklist(self, mock_get_devicelist,
1080+ mock_read_sys_net, mock_sys_dev_path):
1081+ devices = {
1082+ 'eth1': {
1083+ 'bridge': False, 'carrier': False, 'dormant': False,
1084+ 'operstate': 'down', 'address': '00:11:22:33:44:55',
1085+ 'device/driver': 'hv_netsvc', 'device/device': '0x3'},
1086+ 'eth0': {
1087+ 'bridge': False, 'carrier': False, 'dormant': False,
1088+ 'operstate': 'down', 'address': '00:11:22:33:44:55',
1089+ 'device/driver': 'mlx4_core', 'device/device': '0x7'},
1090+ }
1091+
1092+ tmp_dir = self.tmp_dir()
1093+ _setup_test(tmp_dir, mock_get_devicelist,
1094+ mock_read_sys_net, mock_sys_dev_path,
1095+ dev_attrs=devices)
1096+
1097+ blacklist = ['mlx4_core']
1098+ network_cfg = net.generate_fallback_config(blacklist_drivers=blacklist,
1099+ config_driver=True)
1100+ ns = network_state.parse_net_config_data(network_cfg,
1101+ skip_broken=False)
1102+
1103+ render_dir = os.path.join(tmp_dir, "render")
1104+ os.makedirs(render_dir)
1105+
1106+ # don't set rulepath so eni writes them
1107+ renderer = eni.Renderer(
1108+ {'eni_path': 'interfaces', 'netrules_path': 'netrules'})
1109+ renderer.render_network_state(ns, render_dir)
1110+
1111+ self.assertTrue(os.path.exists(os.path.join(render_dir,
1112+ 'interfaces')))
1113+ with open(os.path.join(render_dir, 'interfaces')) as fh:
1114+ contents = fh.read()
1115+ print(contents)
1116+ expected = """
1117+auto lo
1118+iface lo inet loopback
1119+
1120+auto eth1
1121+iface eth1 inet dhcp
1122+"""
1123+ self.assertEqual(expected.lstrip(), contents.lstrip())
1124+
1125+ self.assertTrue(os.path.exists(os.path.join(render_dir, 'netrules')))
1126+ with open(os.path.join(render_dir, 'netrules')) as fh:
1127+ contents = fh.read()
1128+ print(contents)
1129+ expected_rule = [
1130+ 'SUBSYSTEM=="net"',
1131+ 'ACTION=="add"',
1132+ 'DRIVERS=="hv_netsvc"',
1133+ 'ATTR{address}=="00:11:22:33:44:55"',
1134+ 'NAME="eth1"',
1135+ ]
1136+ self.assertEqual(", ".join(expected_rule) + '\n', contents.lstrip())
1137+
1138+
1139 class TestSysConfigRendering(CiTestCase):
1140
1141 @mock.patch("cloudinit.net.sys_dev_path")
1142@@ -1560,6 +1698,118 @@ class TestNetRenderers(CiTestCase):
1143 priority=['sysconfig', 'eni'])
1144
1145
1146+class TestGetInterfaces(CiTestCase):
1147+ _data = {'bonds': ['bond1'],
1148+ 'bridges': ['bridge1'],
1149+ 'vlans': ['bond1.101'],
1150+ 'own_macs': ['enp0s1', 'enp0s2', 'bridge1-nic', 'bridge1',
1151+ 'bond1.101', 'lo', 'eth1'],
1152+ 'macs': {'enp0s1': 'aa:aa:aa:aa:aa:01',
1153+ 'enp0s2': 'aa:aa:aa:aa:aa:02',
1154+ 'bond1': 'aa:aa:aa:aa:aa:01',
1155+ 'bond1.101': 'aa:aa:aa:aa:aa:01',
1156+ 'bridge1': 'aa:aa:aa:aa:aa:03',
1157+ 'bridge1-nic': 'aa:aa:aa:aa:aa:03',
1158+ 'lo': '00:00:00:00:00:00',
1159+ 'greptap0': '00:00:00:00:00:00',
1160+ 'eth1': 'aa:aa:aa:aa:aa:01',
1161+ 'tun0': None},
1162+ 'drivers': {'enp0s1': 'virtio_net',
1163+ 'enp0s2': 'e1000',
1164+ 'bond1': None,
1165+ 'bond1.101': None,
1166+ 'bridge1': None,
1167+ 'bridge1-nic': None,
1168+ 'lo': None,
1169+ 'greptap0': None,
1170+ 'eth1': 'mlx4_core',
1171+ 'tun0': None}}
1172+ data = {}
1173+
1174+ def _se_get_devicelist(self):
1175+ return list(self.data['devices'])
1176+
1177+ def _se_device_driver(self, name):
1178+ return self.data['drivers'][name]
1179+
1180+ def _se_device_devid(self, name):
1181+ return '0x%s' % sorted(list(self.data['drivers'].keys())).index(name)
1182+
1183+ def _se_get_interface_mac(self, name):
1184+ return self.data['macs'][name]
1185+
1186+ def _se_is_bridge(self, name):
1187+ return name in self.data['bridges']
1188+
1189+ def _se_is_vlan(self, name):
1190+ return name in self.data['vlans']
1191+
1192+ def _se_interface_has_own_mac(self, name):
1193+ return name in self.data['own_macs']
1194+
1195+ def _mock_setup(self):
1196+ self.data = copy.deepcopy(self._data)
1197+ self.data['devices'] = set(list(self.data['macs'].keys()))
1198+ mocks = ('get_devicelist', 'get_interface_mac', 'is_bridge',
1199+ 'interface_has_own_mac', 'is_vlan', 'device_driver',
1200+ 'device_devid')
1201+ self.mocks = {}
1202+ for n in mocks:
1203+ m = mock.patch('cloudinit.net.' + n,
1204+ side_effect=getattr(self, '_se_' + n))
1205+ self.addCleanup(m.stop)
1206+ self.mocks[n] = m.start()
1207+
1208+ def test_gi_includes_duplicate_macs(self):
1209+ self._mock_setup()
1210+ ret = net.get_interfaces()
1211+
1212+ self.assertIn('enp0s1', self._se_get_devicelist())
1213+ self.assertIn('eth1', self._se_get_devicelist())
1214+ found = [ent for ent in ret if 'aa:aa:aa:aa:aa:01' in ent]
1215+ self.assertEqual(len(found), 2)
1216+
1217+ def test_gi_excludes_any_without_mac_address(self):
1218+ self._mock_setup()
1219+ ret = net.get_interfaces()
1220+
1221+ self.assertIn('tun0', self._se_get_devicelist())
1222+ found = [ent for ent in ret if 'tun0' in ent]
1223+ self.assertEqual(len(found), 0)
1224+
1225+ def test_gi_excludes_stolen_macs(self):
1226+ self._mock_setup()
1227+ ret = net.get_interfaces()
1228+ self.mocks['interface_has_own_mac'].assert_has_calls(
1229+ [mock.call('enp0s1'), mock.call('bond1')], any_order=True)
1230+ expected = [
1231+ ('enp0s2', 'aa:aa:aa:aa:aa:02', 'e1000', '0x5'),
1232+ ('enp0s1', 'aa:aa:aa:aa:aa:01', 'virtio_net', '0x4'),
1233+ ('eth1', 'aa:aa:aa:aa:aa:01', 'mlx4_core', '0x6'),
1234+ ('lo', '00:00:00:00:00:00', None, '0x8'),
1235+ ('bridge1-nic', 'aa:aa:aa:aa:aa:03', None, '0x3'),
1236+ ]
1237+ self.assertEqual(sorted(expected), sorted(ret))
1238+
1239+ def test_gi_excludes_bridges(self):
1240+ self._mock_setup()
1241+ # add a device 'b1', make all return they have their "own mac",
1242+ # set everything other than 'b1' to be a bridge.
1243+ # then expect b1 is the only thing left.
1244+ self.data['macs']['b1'] = 'aa:aa:aa:aa:aa:b1'
1245+ self.data['drivers']['b1'] = None
1246+ self.data['devices'].add('b1')
1247+ self.data['bonds'] = []
1248+ self.data['own_macs'] = self.data['devices']
1249+ self.data['bridges'] = [f for f in self.data['devices'] if f != "b1"]
1250+ ret = net.get_interfaces()
1251+ self.assertEqual([('b1', 'aa:aa:aa:aa:aa:b1', None, '0x0')], ret)
1252+ self.mocks['is_bridge'].assert_has_calls(
1253+ [mock.call('bridge1'), mock.call('enp0s1'), mock.call('bond1'),
1254+ mock.call('b1')],
1255+ any_order=True)
1256+
1257+
1258 class TestGetInterfacesByMac(CiTestCase):
1259 _data = {'bonds': ['bond1'],
1260 'bridges': ['bridge1'],
1261@@ -1691,4 +1941,202 @@ def _gzip_data(data):
1262 gzfp.close()
1263 return iobuf.getvalue()
1264
1265+
1266+class TestRenameInterfaces(CiTestCase):
1267+
1268+ @mock.patch('cloudinit.util.subp')
1269+ def test_rename_all(self, mock_subp):
1270+ renames = [
1271+ ('00:11:22:33:44:55', 'interface0', 'virtio_net', '0x3'),
1272+ ('00:11:22:33:44:aa', 'interface2', 'virtio_net', '0x5'),
1273+ ]
1274+ current_info = {
1275+ 'ens3': {
1276+ 'downable': True,
1277+ 'device_id': '0x3',
1278+ 'driver': 'virtio_net',
1279+ 'mac': '00:11:22:33:44:55',
1280+ 'name': 'ens3',
1281+ 'up': False},
1282+ 'ens5': {
1283+ 'downable': True,
1284+ 'device_id': '0x5',
1285+ 'driver': 'virtio_net',
1286+ 'mac': '00:11:22:33:44:aa',
1287+ 'name': 'ens5',
1288+ 'up': False},
1289+ }
1290+ net._rename_interfaces(renames, current_info=current_info)
1291+ print(mock_subp.call_args_list)
1292+ mock_subp.assert_has_calls([
1293+ mock.call(['ip', 'link', 'set', 'ens3', 'name', 'interface0'],
1294+ capture=True),
1295+ mock.call(['ip', 'link', 'set', 'ens5', 'name', 'interface2'],
1296+ capture=True),
1297+ ])
1298+
1299+ @mock.patch('cloudinit.util.subp')
1300+ def test_rename_no_driver_no_device_id(self, mock_subp):
1301+ renames = [
1302+ ('00:11:22:33:44:55', 'interface0', None, None),
1303+ ('00:11:22:33:44:aa', 'interface1', None, None),
1304+ ]
1305+ current_info = {
1306+ 'eth0': {
1307+ 'downable': True,
1308+ 'device_id': None,
1309+ 'driver': None,
1310+ 'mac': '00:11:22:33:44:55',
1311+ 'name': 'eth0',
1312+ 'up': False},
1313+ 'eth1': {
1314+ 'downable': True,
1315+ 'device_id': None,
1316+ 'driver': None,
1317+ 'mac': '00:11:22:33:44:aa',
1318+ 'name': 'eth1',
1319+ 'up': False},
1320+ }
1321+ net._rename_interfaces(renames, current_info=current_info)
1322+ print(mock_subp.call_args_list)
1323+ mock_subp.assert_has_calls([
1324+ mock.call(['ip', 'link', 'set', 'eth0', 'name', 'interface0'],
1325+ capture=True),
1326+ mock.call(['ip', 'link', 'set', 'eth1', 'name', 'interface1'],
1327+ capture=True),
1328+ ])
1329+
1330+ @mock.patch('cloudinit.util.subp')
1331+ def test_rename_all_bounce(self, mock_subp):
1332+ renames = [
1333+ ('00:11:22:33:44:55', 'interface0', 'virtio_net', '0x3'),
1334+ ('00:11:22:33:44:aa', 'interface2', 'virtio_net', '0x5'),
1335+ ]
1336+ current_info = {
1337+ 'ens3': {
1338+ 'downable': True,
1339+ 'device_id': '0x3',
1340+ 'driver': 'virtio_net',
1341+ 'mac': '00:11:22:33:44:55',
1342+ 'name': 'ens3',
1343+ 'up': True},
1344+ 'ens5': {
1345+ 'downable': True,
1346+ 'device_id': '0x5',
1347+ 'driver': 'virtio_net',
1348+ 'mac': '00:11:22:33:44:aa',
1349+ 'name': 'ens5',
1350+ 'up': True},
1351+ }
1352+ net._rename_interfaces(renames, current_info=current_info)
1353+ print(mock_subp.call_args_list)
1354+ mock_subp.assert_has_calls([
1355+ mock.call(['ip', 'link', 'set', 'ens3', 'down'], capture=True),
1356+ mock.call(['ip', 'link', 'set', 'ens3', 'name', 'interface0'],
1357+ capture=True),
1358+ mock.call(['ip', 'link', 'set', 'ens5', 'down'], capture=True),
1359+ mock.call(['ip', 'link', 'set', 'ens5', 'name', 'interface2'],
1360+ capture=True),
1361+ mock.call(['ip', 'link', 'set', 'interface0', 'up'], capture=True),
1362+ mock.call(['ip', 'link', 'set', 'interface2', 'up'], capture=True)
1363+ ])
1364+
1365+ @mock.patch('cloudinit.util.subp')
1366+ def test_rename_duplicate_macs(self, mock_subp):
1367+ renames = [
1368+ ('00:11:22:33:44:55', 'eth0', 'hv_netsvc', '0x3'),
1369+ ('00:11:22:33:44:55', 'vf1', 'mlx4_core', '0x5'),
1370+ ]
1371+ current_info = {
1372+ 'eth0': {
1373+ 'downable': True,
1374+ 'device_id': '0x3',
1375+ 'driver': 'hv_netsvc',
1376+ 'mac': '00:11:22:33:44:55',
1377+ 'name': 'eth0',
1378+ 'up': False},
1379+ 'eth1': {
1380+ 'downable': True,
1381+ 'device_id': '0x5',
1382+ 'driver': 'mlx4_core',
1383+ 'mac': '00:11:22:33:44:55',
1384+ 'name': 'eth1',
1385+ 'up': False},
1386+ }
1387+ net._rename_interfaces(renames, current_info=current_info)
1388+ print(mock_subp.call_args_list)
1389+ mock_subp.assert_has_calls([
1390+ mock.call(['ip', 'link', 'set', 'eth1', 'name', 'vf1'],
1391+ capture=True),
1392+ ])
1393+
1394+ @mock.patch('cloudinit.util.subp')
1395+ def test_rename_duplicate_macs_driver_no_devid(self, mock_subp):
1396+ renames = [
1397+ ('00:11:22:33:44:55', 'eth0', 'hv_netsvc', None),
1398+ ('00:11:22:33:44:55', 'vf1', 'mlx4_core', None),
1399+ ]
1400+ current_info = {
1401+ 'eth0': {
1402+ 'downable': True,
1403+ 'device_id': '0x3',
1404+ 'driver': 'hv_netsvc',
1405+ 'mac': '00:11:22:33:44:55',
1406+ 'name': 'eth0',
1407+ 'up': False},
1408+ 'eth1': {
1409+ 'downable': True,
1410+ 'device_id': '0x5',
1411+ 'driver': 'mlx4_core',
1412+ 'mac': '00:11:22:33:44:55',
1413+ 'name': 'eth1',
1414+ 'up': False},
1415+ }
1416+ net._rename_interfaces(renames, current_info=current_info)
1417+ print(mock_subp.call_args_list)
1418+ mock_subp.assert_has_calls([
1419+ mock.call(['ip', 'link', 'set', 'eth1', 'name', 'vf1'],
1420+ capture=True),
1421+ ])
1422+
1423+ @mock.patch('cloudinit.util.subp')
1424+ def test_rename_multi_mac_dups(self, mock_subp):
1425+ renames = [
1426+ ('00:11:22:33:44:55', 'eth0', 'hv_netsvc', '0x3'),
1427+ ('00:11:22:33:44:55', 'vf1', 'mlx4_core', '0x5'),
1428+ ('00:11:22:33:44:55', 'vf2', 'mlx4_core', '0x7'),
1429+ ]
1430+ current_info = {
1431+ 'eth0': {
1432+ 'downable': True,
1433+ 'device_id': '0x3',
1434+ 'driver': 'hv_netsvc',
1435+ 'mac': '00:11:22:33:44:55',
1436+ 'name': 'eth0',
1437+ 'up': False},
1438+ 'eth1': {
1439+ 'downable': True,
1440+ 'device_id': '0x5',
1441+ 'driver': 'mlx4_core',
1442+ 'mac': '00:11:22:33:44:55',
1443+ 'name': 'eth1',
1444+ 'up': False},
1445+ 'eth2': {
1446+ 'downable': True,
1447+ 'device_id': '0x7',
1448+ 'driver': 'mlx4_core',
1449+ 'mac': '00:11:22:33:44:55',
1450+ 'name': 'eth2',
1451+ 'up': False},
1452+ }
1453+ net._rename_interfaces(renames, current_info=current_info)
1454+ print(mock_subp.call_args_list)
1455+ mock_subp.assert_has_calls([
1456+ mock.call(['ip', 'link', 'set', 'eth1', 'name', 'vf1'],
1457+ capture=True),
1458+ mock.call(['ip', 'link', 'set', 'eth2', 'name', 'vf2'],
1459+ capture=True),
1460+ ])
1461+
1462+
1463 # vi: ts=4 expandtab

Subscribers

People subscribed via source and target branches