cloud-init

Merge ~chad.smith/cloud-init:feature/azure-network-on-boot into cloud-init:master

  1. Git
  2. lp:~chad.smith/cloud-init
  3. feature/azure-network-on-boot
  4. Merge into master
Proposed by Chad Smith
Status: Superseded
Proposed branch: ~chad.smith/cloud-init:feature/azure-network-on-boot
Merge into: cloud-init:master
Prerequisite: ~chad.smith/cloud-init:feature/maintain-network-on-boot
Diff against target: 1073 lines (+651/-83)
5 files modified
cloudinit/cmd/devel/net_convert.py (+7/-2)
cloudinit/sources/DataSourceAzure.py (+239/-29)
cloudinit/sources/__init__.py (+0/-19)
cloudinit/sources/tests/test_init.py (+0/-5)
tests/unittests/test_datasource/test_azure.py (+405/-28)
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve
cloud-init Commiters Pending
Review via email: mp+348704@code.launchpad.net

Commit message

azure: allow azure to generate network configuration from IMDS on each
boot

Azure datasource now queries IMDS metadata service for network
configuration at link local address
http://169.254.169.254/metadata/instance?api-version=2017-12-01. The
azure metadata service presents a list of macs and allocated ip addresses
associated with this instance. Azure will now also regenerate network
configuration on every boot because it subscribes to EventType.BOOT
maintenance events as well as the 'first boot'
EventType.BOOT_NEW_INSTANCE.

Also refactor _get_data into 3 discrete methods:
  - is_platform_viable: check quickly whether the datasource is
    potentially compatible with the platform on which is is running
  - crawl_metadata: walk all potential metadata candidates, returning a
    structured dict of all metadata and userdata. Raise InvalidMetaData on
    error.
  - _get_data: call crawl_metadata and process results or error. Cache
    instance data on class attributes: metadata, userdata_raw etc.

Description of the change

As a note: current supported Ubuntu images contain some magic udev rules and/or netplan config which automatically setup dhcp on any nic eth1 or greater which shows up on an azure instance during boot. With this functionality in cloud-init proper, we can drop the static script definitions from supported ubuntu images because cloud-init will own the network config on any new device that shows up across boot.

to test:
create an Azure vm;
ssh azcurl -H Metadata:true "http://169.254.169.254/metadata/instance?api-version=2017-12-01" > azure.json

python3 -m cloudinit.cmd.main devel net_convert --kind azure-imds --network-data azure.json -d output -O netplan

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:155c7688e9d7efa125a68c14b1148b2f06b32416
https://jenkins.ubuntu.com/server/job/cloud-init-ci/129/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    FAILED: Ubuntu LTS: Build

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/129/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:4bcee7ca6f7b5d2e22197f48b12dcaa5eaf3867e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/138/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    FAILED: Ubuntu LTS: Build

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/138/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:68e932545678c8330c753ad1c10be7d7c65cf424
https://jenkins.ubuntu.com/server/job/cloud-init-ci/139/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    FAILED: Ubuntu LTS: Build

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/139/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:0f5857a45285c2eb0a78f25e85ae7c0a55778fb4
https://jenkins.ubuntu.com/server/job/cloud-init-ci/140/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/140/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Ryan Harper (raharper) wrote :

This is coming together.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:c7562a61e272daa91c6e01363f5cb3d2abd88c6f
https://jenkins.ubuntu.com/server/job/cloud-init-ci/154/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/154/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:a8fe7604191724d72d895d26fbb6e8299fac3180
https://jenkins.ubuntu.com/server/job/cloud-init-ci/159/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/159/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/azure-network-on-boot updated
0e7a589... by Chad Smith

azure: azure to generate network configuration from IMDS on each boot

Azure datasource now queries IMDS metadata service for network
configuration at link local address
http://169.254.169.254/metadata/instance?api-version=2017-12-01. The
azure metadata service presents a list of macs and allocated ip addresses
associated with this instance. Azure will now also regenerate network
configuration on every boot because it subscribes to EventType.BOOT
maintenance events as well as the 'first boot'
EventType.BOOT_NEW_INSTANCE.

Also refactor _get_data into 3 discrete methods:
  - is_platform_viable: check quickly whether the datasource is
    potentially compatible with the platform on which is is running
  - crawl_metadata: walk all potential metadata candidates, returning a
    structured dict of all metadata and userdata. Raise InvalidMetaData on
    error.
  - _get_data: call crawl_metadata and process results or error. Cache
    instance data on class attributes: metadata, userdata_raw etc.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:fd6b0b18f6e99e7997b4cd198ed05ff533569442
https://jenkins.ubuntu.com/server/job/cloud-init-ci/190/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/190/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/azure-network-on-boot updated
fb7381b... by Chad Smith

pyflakes

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:6bcbd81055f16cbc2dec096884ca8463f37985a3
https://jenkins.ubuntu.com/server/job/cloud-init-ci/193/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    FAILED: Ubuntu LTS: Build

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/193/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/azure-network-on-boot updated
01ffbf2... by Chad Smith

mock find_fallback_nic to avoid udevadm calls leaking in tests

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:b1fffef776e344ee3a1b0842129f8ad260e4680c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/194/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/194/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

some little things/food for thought.
and you have some conflicts.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:ffabafc1137f3aee1a1eb8d685742f7262333024
https://jenkins.ubuntu.com/server/job/cloud-init-ci/199/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/199/rebuild

review: Approve (continuous-integration)
~chad.smith/cloud-init:feature/azure-network-on-boot updated
8390979... by Chad Smith

avoid side-effects in ds.network_config. remove ubuntu scripts from get_data.

9235c59... by Chad Smith

create separate private _is_platform_viable function called by DataSourceAzure method

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:799d6b68a347ea21fd28530cc1d53b4d07fb128e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/201/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: MAAS Compatability Testing

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/201/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/azure-network-on-boot updated
c27cec6... by Chad Smith

do not leak calls to systemd-detect-virt from unit tests. Also check if systemd-detect-virt exists before trying to run it

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:0182941850252bdf7d3c10466b43aed98e140f3f
https://jenkins.ubuntu.com/server/job/cloud-init-ci/202/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/202/rebuild

review: Approve (continuous-integration)
~chad.smith/cloud-init:feature/azure-network-on-boot updated
097df03... by Chad Smith

move DataSourceAzure.network_config logic to _parse_network_config

31145bc... by Chad Smith

since maybe_remove_ubuntu_network_config_scripts is moved to get_data, limit unit tests to only get_data calls, not network_config

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:a6e28e077d179d558bb48f135675d8845ece990e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/204/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/204/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

there are some merge issues shown below ("<<<<<").

I have some comments inline.

How feasible would it be to add support for azure conversion in the net-convert sub-command?

~chad.smith/cloud-init:feature/azure-network-on-boot updated
3f4a384... by Chad Smith

Removing network scripts log is info instead of debug. Unit test fixes

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:bc725f13f319cb75defeb39928f280efc7626031
https://jenkins.ubuntu.com/server/job/cloud-init-ci/205/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/205/rebuild

review: Approve (continuous-integration)
~chad.smith/cloud-init:feature/azure-network-on-boot updated
1d429c3... by Chad Smith

make Azure's parse_network_config function public and call it from net-convert cli

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:1d429c3cb514b35b84efc40533f5935ec2abdf33
https://jenkins.ubuntu.com/server/job/cloud-init-ci/206/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/206/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

andy thanks for this update: couple style/flakes errors showed up
cloudinit/reporting/handlers.py:287:80: E501 line too long (83 > 79 characters)
tests/unittests/test_reporting_hyperv.py:8: 'tempfile' imported but unused

Unmerged commits

1d429c3... by Chad Smith

make Azure's parse_network_config function public and call it from net-convert cli

3f4a384... by Chad Smith

Removing network scripts log is info instead of debug. Unit test fixes

31145bc... by Chad Smith

since maybe_remove_ubuntu_network_config_scripts is moved to get_data, limit unit tests to only get_data calls, not network_config

097df03... by Chad Smith

move DataSourceAzure.network_config logic to _parse_network_config

c27cec6... by Chad Smith

do not leak calls to systemd-detect-virt from unit tests. Also check if systemd-detect-virt exists before trying to run it

9235c59... by Chad Smith

create separate private _is_platform_viable function called by DataSourceAzure method

8390979... by Chad Smith

avoid side-effects in ds.network_config. remove ubuntu scripts from get_data.

01ffbf2... by Chad Smith

mock find_fallback_nic to avoid udevadm calls leaking in tests

fb7381b... by Chad Smith

pyflakes

0e7a589... by Chad Smith

azure: azure to generate network configuration from IMDS on each boot

Azure datasource now queries IMDS metadata service for network
configuration at link local address
http://169.254.169.254/metadata/instance?api-version=2017-12-01. The
azure metadata service presents a list of macs and allocated ip addresses
associated with this instance. Azure will now also regenerate network
configuration on every boot because it subscribes to EventType.BOOT
maintenance events as well as the 'first boot'
EventType.BOOT_NEW_INSTANCE.

Also refactor _get_data into 3 discrete methods:
  - is_platform_viable: check quickly whether the datasource is
    potentially compatible with the platform on which is is running
  - crawl_metadata: walk all potential metadata candidates, returning a
    structured dict of all metadata and userdata. Raise InvalidMetaData on
    error.
  - _get_data: call crawl_metadata and process results or error. Cache
    instance data on class attributes: metadata, userdata_raw etc.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/cloudinit/cmd/devel/net_convert.py b/cloudinit/cmd/devel/net_convert.py
2index 1ec08a3..271dc5e 100755
3--- a/cloudinit/cmd/devel/net_convert.py
4+++ b/cloudinit/cmd/devel/net_convert.py
5@@ -8,6 +8,7 @@ import sys
6 import yaml
7
8 from cloudinit.sources.helpers import openstack
9+from cloudinit.sources import DataSourceAzure as azure
10
11 from cloudinit.net import eni, netplan, network_state, sysconfig
12 from cloudinit import log
13@@ -28,7 +29,8 @@ def get_parser(parser=None):
14 parser.add_argument("-p", "--network-data", type=open,
15 metavar="PATH", required=True)
16 parser.add_argument("-k", "--kind",
17- choices=['eni', 'network_data.json', 'yaml'],
18+ choices=['eni', 'network_data.json', 'yaml',
19+ 'azure-imds'],
20 required=True)
21 parser.add_argument("-d", "--directory",
22 metavar="PATH",
23@@ -78,10 +80,13 @@ def handle_args(name, args):
24 ["Input YAML",
25 yaml.dump(pre_ns, default_flow_style=False, indent=4), ""]))
26 ns = network_state.parse_net_config_data(pre_ns)
27- else:
28+ elif args.kind == 'network_data.json':
29 pre_ns = openstack.convert_net_json(
30 json.loads(net_data), known_macs=known_macs)
31 ns = network_state.parse_net_config_data(pre_ns)
32+ elif args.kind == 'azure-imds':
33+ pre_ns = azure.parse_network_config(json.loads(net_data))
34+ ns = network_state.parse_net_config_data(pre_ns)
35
36 if not ns:
37 raise RuntimeError("No valid network_state object created from"
38diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py
39index 7007d9e..d8170c3 100644
40--- a/cloudinit/sources/DataSourceAzure.py
41+++ b/cloudinit/sources/DataSourceAzure.py
42@@ -8,6 +8,7 @@ import base64
43 import contextlib
44 import crypt
45 from functools import partial
46+import json
47 import os
48 import os.path
49 import re
50@@ -17,6 +18,7 @@ import xml.etree.ElementTree as ET
51
52 from cloudinit import log as logging
53 from cloudinit import net
54+from cloudinit.event import EventType
55 from cloudinit.net.dhcp import EphemeralDHCPv4
56 from cloudinit import sources
57 from cloudinit.sources.helpers.azure import get_metadata_from_fabric
58@@ -49,7 +51,17 @@ DEFAULT_FS = 'ext4'
59 AZURE_CHASSIS_ASSET_TAG = '7783-7084-3265-9085-8269-3286-77'
60 REPROVISION_MARKER_FILE = "/var/lib/cloud/data/poll_imds"
61 REPORTED_READY_MARKER_FILE = "/var/lib/cloud/data/reported_ready"
62-IMDS_URL = "http://169.254.169.254/metadata/reprovisiondata"
63+AGENT_SEED_DIR = '/var/lib/waagent'
64+IMDS_URL = "http://169.254.169.254/metadata/"
65+
66+# List of static scripts and network config artifacts created by
67+# stock ubuntu suported images.
68+UBUNTU_EXTENDED_NETWORK_SCRIPTS = [
69+ '/etc/netplan/90-azure-hotplug.yaml',
70+ '/usr/local/sbin/ephemeral_eth.sh',
71+ '/etc/udev/rules.d/10-net-device-added.rules',
72+ '/run/network/interfaces.ephemeral.d',
73+]
74
75
76 def find_storvscid_from_sysctl_pnpinfo(sysctl_out, deviceid):
77@@ -185,7 +197,7 @@ if util.is_FreeBSD():
78
79 BUILTIN_DS_CONFIG = {
80 'agent_command': AGENT_START_BUILTIN,
81- 'data_dir': "/var/lib/waagent",
82+ 'data_dir': AGENT_SEED_DIR,
83 'set_hostname': True,
84 'hostname_bounce': {
85 'interface': DEFAULT_PRIMARY_NIC,
86@@ -252,6 +264,10 @@ class DataSourceAzure(sources.DataSource):
87
88 dsname = 'Azure'
89 _negotiated = False
90+ _metadata_imds = sources.UNSET
91+
92+ # Regenerate network config new_instance boot and every boot
93+ update_events = {'network': [EventType.BOOT_NEW_INSTANCE, EventType.BOOT]}
94
95 def __init__(self, sys_cfg, distro, paths):
96 sources.DataSource.__init__(self, sys_cfg, distro, paths)
97@@ -336,15 +352,17 @@ class DataSourceAzure(sources.DataSource):
98 metadata['public-keys'] = key_value or pubkeys_from_crt_files(fp_files)
99 return metadata
100
101- def _get_data(self):
102+ def crawl_metadata(self):
103+ """Walk all instance metadata sources returning a dict on success.
104+
105+ @return: A dictionary of any metadata content for this instance.
106+ @raise: InvalidMetaDataException when the expected metadata service is
107+ unavailable, broken or disabled.
108+ """
109+ crawled_data = {}
110 # azure removes/ejects the cdrom containing the ovf-env.xml
111 # file on reboot. So, in order to successfully reboot we
112 # need to look in the datadir and consider that valid
113- asset_tag = util.read_dmi_data('chassis-asset-tag')
114- if asset_tag != AZURE_CHASSIS_ASSET_TAG:
115- LOG.debug("Non-Azure DMI asset tag '%s' discovered.", asset_tag)
116- return False
117-
118 ddir = self.ds_cfg['data_dir']
119
120 candidates = [self.seed_dir]
121@@ -373,41 +391,84 @@ class DataSourceAzure(sources.DataSource):
122 except NonAzureDataSource:
123 continue
124 except BrokenAzureDataSource as exc:
125- raise exc
126+ msg = 'BrokenAzureDataSource: %s' % exc
127+ raise sources.InvalidMetaDataException(msg)
128 except util.MountFailedError:
129 LOG.warning("%s was not mountable", cdev)
130 continue
131
132 if reprovision or self._should_reprovision(ret):
133 ret = self._reprovision()
134- (md, self.userdata_raw, cfg, files) = ret
135+ imds_md = get_metadata_from_imds(
136+ self.fallback_interface, retries=3)
137+ (md, userdata_raw, cfg, files) = ret
138 self.seed = cdev
139- self.metadata = util.mergemanydict([md, DEFAULT_METADATA])
140- self.cfg = util.mergemanydict([cfg, BUILTIN_CLOUD_CONFIG])
141+ crawled_data.update({
142+ 'cfg': cfg,
143+ 'files': files,
144+ 'metadata': util.mergemanydict(
145+ [md, {'imds': imds_md}]),
146+ 'userdata_raw': userdata_raw})
147 found = cdev
148
149 LOG.debug("found datasource in %s", cdev)
150 break
151
152 if not found:
153- return False
154+ raise sources.InvalidMetaDataException('No Azure metadata found')
155
156 if found == ddir:
157 LOG.debug("using files cached in %s", ddir)
158
159 seed = _get_random_seed()
160 if seed:
161- self.metadata['random_seed'] = seed
162+ crawled_data['metadata']['random_seed'] = seed
163+ crawled_data['metadata']['instance-id'] = util.read_dmi_data(
164+ 'system-uuid')
165+ return crawled_data
166+
167+ def _is_platform_viable(self):
168+ """Check platform environment to report if this datasource may run."""
169+ return _is_platform_viable(self.seed_dir)
170+
171+ def clear_cached_attrs(self, attr_defaults=()):
172+ """Reset any cached class attributes to defaults."""
173+ super(DataSourceAzure, self).clear_cached_attrs(attr_defaults)
174+ self._metadata_imds = sources.UNSET
175+
176+ def _get_data(self):
177+ """Crawl and process datasource metadata caching metadata as attrs.
178+
179+ @return: True on success, False on error, invalid or disabled
180+ datasource.
181+ """
182+ if not self._is_platform_viable():
183+ return False
184+ if self.distro and self.distro.name == 'ubuntu':
185+ maybe_remove_ubuntu_network_config_scripts()
186+ try:
187+ crawled_data = util.log_time(
188+ logfunc=LOG.debug, msg='Crawl of metadata service',
189+ func=self.crawl_metadata)
190+ except sources.InvalidMetaDataException as e:
191+ LOG.warning('Could not crawl Azure metadata: %s', e)
192+ return False
193+
194+ # Process crawled data and augment with various config defaults
195+ self.cfg = util.mergemanydict(
196+ [crawled_data['cfg'], BUILTIN_CLOUD_CONFIG])
197+ self._metadata_imds = crawled_data['metadata']['imds']
198+ self.metadata = util.mergemanydict(
199+ [crawled_data['metadata'], DEFAULT_METADATA])
200+ self.userdata_raw = crawled_data['userdata_raw']
201
202 user_ds_cfg = util.get_cfg_by_path(self.cfg, DS_CFG_PATH, {})
203 self.ds_cfg = util.mergemanydict([user_ds_cfg, self.ds_cfg])
204
205 # walinux agent writes files world readable, but expects
206 # the directory to be protected.
207- write_files(ddir, files, dirmode=0o700)
208-
209- self.metadata['instance-id'] = util.read_dmi_data('system-uuid')
210-
211+ write_files(
212+ self.ds_cfg['data_dir'], crawled_data['files'], dirmode=0o700)
213 return True
214
215 def device_name_to_device(self, name):
216@@ -436,7 +497,7 @@ class DataSourceAzure(sources.DataSource):
217 def _poll_imds(self):
218 """Poll IMDS for the new provisioning data until we get a valid
219 response. Then return the returned JSON object."""
220- url = IMDS_URL + "?api-version=2017-04-02"
221+ url = IMDS_URL + "reprovisiondata?api-version=2017-04-02"
222 headers = {"Metadata": "true"}
223 report_ready = bool(not os.path.isfile(REPORTED_READY_MARKER_FILE))
224 LOG.debug("Start polling IMDS")
225@@ -487,7 +548,7 @@ class DataSourceAzure(sources.DataSource):
226 jump back into the polling loop in order to retrieve the ovf_env."""
227 if not ret:
228 return False
229- (_md, self.userdata_raw, cfg, _files) = ret
230+ (_md, _userdata_raw, cfg, _files) = ret
231 path = REPROVISION_MARKER_FILE
232 if (cfg.get('PreprovisionedVm') is True or
233 os.path.isfile(path)):
234@@ -543,22 +604,15 @@ class DataSourceAzure(sources.DataSource):
235 @property
236 def network_config(self):
237 """Generate a network config like net.generate_fallback_network() with
238- the following execptions.
239+ the following exceptions.
240
241 1. Probe the drivers of the net-devices present and inject them in
242 the network configuration under params: driver: <driver> value
243 2. Generate a fallback network config that does not include any of
244 the blacklisted devices.
245 """
246- blacklist = ['mlx4_core']
247 if not self._network_config:
248- LOG.debug('Azure: generating fallback configuration')
249- # generate a network config, blacklist picking any mlx4_core devs
250- netconfig = net.generate_fallback_config(
251- blacklist_drivers=blacklist, config_driver=True)
252-
253- self._network_config = netconfig
254-
255+ self._network_config = parse_network_config(self._metadata_imds)
256 return self._network_config
257
258
259@@ -1025,6 +1079,162 @@ def load_azure_ds_dir(source_dir):
260 return (md, ud, cfg, {'ovf-env.xml': contents})
261
262
263+def parse_network_config(imds_metadata):
264+ """Convert imds_metadata dictionary to network v2 configuration.
265+
266+ Parses network configuration from imds metadata if present or generate
267+ fallback network config excluding mlx4_core devices.
268+
269+ @param: imds_metadata: Dict of content read from IMDS network service.
270+ @return: Dictionary containing network version 2 standard configuration.
271+ """
272+ if imds_metadata != sources.UNSET and imds_metadata:
273+ netconfig = {'version': 2, 'ethernets': {}}
274+ LOG.debug('Azure: generating network configuration from IMDS')
275+ network_metadata = imds_metadata['network']
276+ for idx, intf in enumerate(network_metadata['interface']):
277+ nicname = 'eth{idx}'.format(idx=idx)
278+ dev_config = {}
279+ for addr4 in intf['ipv4']['ipAddress']:
280+ privateIpv4 = addr4['privateIpAddress']
281+ if privateIpv4:
282+ if dev_config.get('dhcp4', False):
283+ # Append static address config for nic > 1
284+ netPrefix = intf['ipv4']['subnet'][0].get(
285+ 'prefix', '24')
286+ if not dev_config.get('addresses'):
287+ dev_config['addresses'] = []
288+ dev_config['addresses'].append(
289+ '{ip}/{prefix}'.format(
290+ ip=privateIpv4, prefix=netPrefix))
291+ else:
292+ dev_config['dhcp4'] = True
293+ for addr6 in intf['ipv6']['ipAddress']:
294+ privateIpv6 = addr6['privateIpAddress']
295+ if privateIpv6:
296+ dev_config['dhcp6'] = True
297+ break
298+ if dev_config:
299+ mac = ':'.join(re.findall(r'..', intf['macAddress']))
300+ dev_config.update(
301+ {'match': {'macaddress': mac.lower()},
302+ 'set-name': nicname})
303+ netconfig['ethernets'][nicname] = dev_config
304+ else:
305+ blacklist = ['mlx4_core']
306+ LOG.debug('Azure: generating fallback configuration')
307+ # generate a network config, blacklist picking mlx4_core devs
308+ netconfig = net.generate_fallback_config(
309+ blacklist_drivers=blacklist, config_driver=True)
310+ return netconfig
311+
312+
313+def get_metadata_from_imds(fallback_nic, retries):
314+ """Query Azure's network metadata service, returning a dictionary.
315+
316+ If network is not up, setup ephemeral dhcp on fallback_nic to talk to the
317+ IMDS. For more info on IMDS:
318+ https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service
319+
320+ @param fallback_nic: String. The name of the nic which requires active
321+ networ in order to query IMDS.
322+ @param retries: The number of retries of the IMDS_URL.
323+
324+ @return: A dict of instance metadata containing compute and network
325+ info.
326+ """
327+ if net.is_up(fallback_nic):
328+ return util.log_time(
329+ logfunc=LOG.debug,
330+ msg='Crawl of Azure Instance Metadata Service (IMDS)',
331+ func=_get_metadata_from_imds, args=(retries,))
332+ else:
333+ with EphemeralDHCPv4(fallback_nic):
334+ return util.log_time(
335+ logfunc=LOG.debug,
336+ msg='Crawl of Azure Instance Metadata Service (IMDS)',
337+ func=_get_metadata_from_imds, args=(retries,))
338+
339+
340+def _get_metadata_from_imds(retries):
341+
342+ def retry_on_url_error(msg, exception):
343+ if isinstance(exception, UrlError) and exception.code == 404:
344+ return True # Continue retries
345+ return False # Stop retries on all other exceptions, including 404s
346+
347+ url = IMDS_URL + "instance?api-version=2017-12-01"
348+ headers = {"Metadata": "true"}
349+ try:
350+ response = readurl(
351+ url, timeout=1, headers=headers, retries=retries,
352+ exception_cb=retry_on_url_error)
353+ except Exception as e:
354+ LOG.debug('Ignoring IMDS instance metadata: %s', e)
355+ return {}
356+ try:
357+ return util.load_json(str(response))
358+ except json.decoder.JSONDecodeError:
359+ LOG.warning(
360+ 'Ignoring non-json IMDS instance metadata: %s', str(response))
361+ return {}
362+
363+
364+def maybe_remove_ubuntu_network_config_scripts(paths=None):
365+ """Remove Azure-specific ubuntu network config for non-primary nics.
366+
367+ @param paths: List of networking scripts or directories to remove when
368+ present.
369+
370+ In certain supported ubuntu images, static udev rules or netplan yaml
371+ config is delivered in the base ubuntu image to support dhcp on any
372+ additional interfaces which get attached by a customer at some point
373+ after initial boot. Since the Azure datasource can now regenerate
374+ network configuration as metadata reports these new devices, we no longer
375+ want the udev rules or netplan's 90-azure-hotplug.yaml to configure
376+ networking on eth1 or greater as it might collide with cloud-init's
377+ configuration.
378+
379+ Remove the any existing extended network scripts if the datasource is
380+ enabled to write network per-boot.
381+ """
382+ if not paths:
383+ paths = UBUNTU_EXTENDED_NETWORK_SCRIPTS
384+ logged = False
385+ for path in paths:
386+ if os.path.exists(path):
387+ if not logged:
388+ LOG.info(
389+ 'Removing Ubuntu extended network scripts because'
390+ ' cloud-init updates Azure network configuration on the'
391+ ' following event: %s.',
392+ EventType.BOOT)
393+ logged = True
394+ if os.path.isdir(path):
395+ util.del_dir(path)
396+ else:
397+ util.del_file(path)
398+
399+
400+def _is_platform_viable(seed_dir):
401+ """Check platform environment to report if this datasource may run."""
402+ asset_tag = util.read_dmi_data('chassis-asset-tag')
403+ if asset_tag == AZURE_CHASSIS_ASSET_TAG:
404+ return True
405+ LOG.debug("Non-Azure DMI asset tag '%s' discovered.", asset_tag)
406+ for path in (AGENT_SEED_DIR, seed_dir):
407+ if os.path.exists(os.path.join(path, 'ovf-env.xml')):
408+ return True
409+ if util.which('systemd-detect-virt'):
410+ (virt_type, _err) = util.subp(
411+ ['systemd-detect-virt'], rcs=[0, 1], capture=True)
412+ if virt_type.strip() == 'microsoft':
413+ return True
414+ if util.find_devs_with(criteria='LABEL=rd_rdfe_*'):
415+ return True
416+ return False
417+
418+
419 class BrokenAzureDataSource(Exception):
420 pass
421
422diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
423index c50750a..06e613f 100644
424--- a/cloudinit/sources/__init__.py
425+++ b/cloudinit/sources/__init__.py
426@@ -103,7 +103,6 @@ class DataSource(object):
427 url_timeout = 10 # timeout for each metadata url read attempt
428 url_retries = 5 # number of times to retry url upon 404
429
430-<<<<<<< cloudinit/sources/__init__.py
431 # The datasource defines a set of supported EventTypes during which
432 # the datasource can react to changes in metadata and regenerate
433 # network configuration on metadata changes.
434@@ -112,16 +111,6 @@ class DataSource(object):
435
436 # Default: generate network config on new instance id (first boot).
437 update_events = {'network': set([EventType.BOOT_NEW_INSTANCE])}
438-=======
439- # The datasource defines a list of supported EventTypes during which
440- # the datasource can react to changes in metadata and regenerate
441- # network configuration on metadata changes.
442- # A datasource which supports writing network config on each system boot
443- # would set update_events = {'network': [EventType.BOOT]}
444-
445- # Default: generate network config on new instance id (first boot).
446- update_events = {'network': [EventType.BOOT_NEW_INSTANCE]}
447->>>>>>> cloudinit/sources/__init__.py
448
449 # N-tuple listing default values for any metadata-related class
450 # attributes cached on an instance by a process_data runs. These attribute
451@@ -486,13 +475,8 @@ class DataSource(object):
452 for update_scope, update_events in self.update_events.items():
453 if event in update_events:
454 if not supported_events.get(update_scope):
455-<<<<<<< cloudinit/sources/__init__.py
456 supported_events[update_scope] = set()
457 supported_events[update_scope].add(event)
458-=======
459- supported_events[update_scope] = []
460- supported_events[update_scope].append(event)
461->>>>>>> cloudinit/sources/__init__.py
462 for scope, matched_events in supported_events.items():
463 LOG.debug(
464 "Update datasource metadata and %s config due to events: %s",
465@@ -506,11 +490,8 @@ class DataSource(object):
466 result = self.get_data()
467 if result:
468 return True
469-<<<<<<< cloudinit/sources/__init__.py
470 LOG.debug("Datasource %s not updated for events: %s", self,
471 ', '.join(source_event_types))
472-=======
473->>>>>>> cloudinit/sources/__init__.py
474 return False
475
476 def check_instance_id(self, sys_cfg):
477diff --git a/cloudinit/sources/tests/test_init.py b/cloudinit/sources/tests/test_init.py
478index 30573cc..9e939c1 100644
479--- a/cloudinit/sources/tests/test_init.py
480+++ b/cloudinit/sources/tests/test_init.py
481@@ -429,14 +429,9 @@ class TestDataSource(CiTestCase):
482
483 def test_update_metadata_only_acts_on_supported_update_events(self):
484 """update_metadata won't get_data on unsupported update events."""
485-<<<<<<< cloudinit/sources/tests/test_init.py
486 self.datasource.update_events['network'].discard(EventType.BOOT)
487 self.assertEqual(
488 {'network': set([EventType.BOOT_NEW_INSTANCE])},
489-=======
490- self.assertEqual(
491- {'network': [EventType.BOOT_NEW_INSTANCE]},
492->>>>>>> cloudinit/sources/tests/test_init.py
493 self.datasource.update_events)
494
495 def fake_get_data():
496diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py
497index e82716e..5eae247 100644
498--- a/tests/unittests/test_datasource/test_azure.py
499+++ b/tests/unittests/test_datasource/test_azure.py
500@@ -1,15 +1,21 @@
501 # This file is part of cloud-init. See LICENSE file for license information.
502
503+from cloudinit import distros
504 from cloudinit import helpers
505-from cloudinit.sources import DataSourceAzure as dsaz
506+from cloudinit import url_helper
507+from cloudinit.sources import (
508+ UNSET, DataSourceAzure as dsaz, InvalidMetaDataException)
509 from cloudinit.util import (b64e, decode_binary, load_file, write_file,
510 find_freebsd_part, get_path_dev_freebsd,
511 MountFailedError)
512 from cloudinit.version import version_string as vs
513-from cloudinit.tests.helpers import (CiTestCase, TestCase, populate_dir, mock,
514- ExitStack, PY26, SkipTest)
515+from cloudinit.tests.helpers import (
516+ HttprettyTestCase, CiTestCase, populate_dir, mock, wrap_and_call,
517+ ExitStack, PY26, SkipTest)
518
519 import crypt
520+import httpretty
521+import json
522 import os
523 import stat
524 import xml.etree.ElementTree as ET
525@@ -77,6 +83,106 @@ def construct_valid_ovf_env(data=None, pubkeys=None,
526 return content
527
528
529+NETWORK_METADATA = {
530+ "network": {
531+ "interface": [
532+ {
533+ "macAddress": "000D3A047598",
534+ "ipv6": {
535+ "ipAddress": []
536+ },
537+ "ipv4": {
538+ "subnet": [
539+ {
540+ "prefix": "24",
541+ "address": "10.0.0.0"
542+ }
543+ ],
544+ "ipAddress": [
545+ {
546+ "privateIpAddress": "10.0.0.4",
547+ "publicIpAddress": "104.46.124.81"
548+ }
549+ ]
550+ }
551+ }
552+ ]
553+ }
554+}
555+
556+
557+class TestGetMetadataFromIMDS(HttprettyTestCase):
558+
559+ with_logs = True
560+
561+ def setUp(self):
562+ super(TestGetMetadataFromIMDS, self).setUp()
563+ self.network_md_url = dsaz.IMDS_URL + "instance?api-version=2017-12-01"
564+
565+ @mock.patch('cloudinit.sources.DataSourceAzure.readurl')
566+ @mock.patch('cloudinit.sources.DataSourceAzure.EphemeralDHCPv4')
567+ @mock.patch('cloudinit.sources.DataSourceAzure.net.is_up')
568+ def test_get_metadata_does_not_dhcp_if_network_is_up(
569+ self, m_net_is_up, m_dhcp, m_readurl):
570+ """Do not perform DHCP setup when nic is already up."""
571+ m_net_is_up.return_value = True
572+ m_readurl.return_value = url_helper.StringResponse(
573+ json.dumps(NETWORK_METADATA).encode('utf-8'))
574+ self.assertEqual(
575+ NETWORK_METADATA,
576+ dsaz.get_metadata_from_imds('eth9', retries=3))
577+
578+ m_net_is_up.assert_called_with('eth9')
579+ m_dhcp.assert_not_called()
580+ self.assertIn(
581+ "Crawl of Azure Instance Metadata Service (IMDS) took", # log_time
582+ self.logs.getvalue())
583+
584+ @mock.patch('cloudinit.sources.DataSourceAzure.readurl')
585+ @mock.patch('cloudinit.sources.DataSourceAzure.EphemeralDHCPv4')
586+ @mock.patch('cloudinit.sources.DataSourceAzure.net.is_up')
587+ def test_get_metadata_performs_dhcp_when_network_is_down(
588+ self, m_net_is_up, m_dhcp, m_readurl):
589+ """Perform DHCP setup when nic is not up."""
590+ m_net_is_up.return_value = False
591+ m_readurl.return_value = url_helper.StringResponse(
592+ json.dumps(NETWORK_METADATA).encode('utf-8'))
593+
594+ self.assertEqual(
595+ NETWORK_METADATA,
596+ dsaz.get_metadata_from_imds('eth9', retries=2))
597+
598+ m_net_is_up.assert_called_with('eth9')
599+ m_dhcp.assert_called_with('eth9')
600+ self.assertIn(
601+ "Crawl of Azure Instance Metadata Service (IMDS) took", # log_time
602+ self.logs.getvalue())
603+
604+ m_readurl.assert_called_with(
605+ self.network_md_url, exception_cb=mock.ANY,
606+ headers={'Metadata': 'true'}, retries=2, timeout=1)
607+
608+ @mock.patch('cloudinit.url_helper.time.sleep')
609+ @mock.patch('cloudinit.sources.DataSourceAzure.net.is_up')
610+ def test_get_metadata_from_imds_empty_when_no_imds_present(
611+ self, m_net_is_up, m_sleep):
612+ """Return empty dict when IMDS network metadata is absent."""
613+ httpretty.register_uri(
614+ httpretty.GET,
615+ dsaz.IMDS_URL + 'instance?api-version=2017-12-01',
616+ body={}, status=404)
617+
618+ m_net_is_up.return_value = True # skips dhcp
619+
620+ self.assertEqual({}, dsaz.get_metadata_from_imds('eth9', retries=2))
621+
622+ m_net_is_up.assert_called_with('eth9')
623+ self.assertEqual([mock.call(1), mock.call(1)], m_sleep.call_args_list)
624+ self.assertIn(
625+ "Crawl of Azure Instance Metadata Service (IMDS) took", # log_time
626+ self.logs.getvalue())
627+
628+
629 class TestAzureDataSource(CiTestCase):
630
631 with_logs = True
632@@ -95,8 +201,19 @@ class TestAzureDataSource(CiTestCase):
633 self.patches = ExitStack()
634 self.addCleanup(self.patches.close)
635
636- self.patches.enter_context(mock.patch.object(dsaz, '_get_random_seed'))
637-
638+ self.patches.enter_context(mock.patch.object(
639+ dsaz, '_get_random_seed', return_value='wild'))
640+ self.m_get_metadata_from_imds = self.patches.enter_context(
641+ mock.patch.object(
642+ dsaz, 'get_metadata_from_imds',
643+ mock.MagicMock(return_value=NETWORK_METADATA)))
644+ self.m_fallback_nic = self.patches.enter_context(
645+ mock.patch('cloudinit.sources.net.find_fallback_nic',
646+ return_value='eth9'))
647+ self.m_remove_ubuntu_network_scripts = self.patches.enter_context(
648+ mock.patch.object(
649+ dsaz, 'maybe_remove_ubuntu_network_config_scripts',
650+ mock.MagicMock()))
651 super(TestAzureDataSource, self).setUp()
652
653 def apply_patches(self, patches):
654@@ -137,7 +254,7 @@ scbus-1 on xpt0 bus 0
655 ])
656 return dsaz
657
658- def _get_ds(self, data, agent_command=None):
659+ def _get_ds(self, data, agent_command=None, distro=None):
660
661 def dsdevs():
662 return data.get('dsdevs', [])
663@@ -186,8 +303,11 @@ scbus-1 on xpt0 bus 0
664 side_effect=_wait_for_files)),
665 ])
666
667+ if distro is not None:
668+ distro_cls = distros.fetch(distro)
669+ distro = distro_cls(distro, data.get('sys_cfg', {}), self.paths)
670 dsrc = dsaz.DataSourceAzure(
671- data.get('sys_cfg', {}), distro=None, paths=self.paths)
672+ data.get('sys_cfg', {}), distro=distro, paths=self.paths)
673 if agent_command is not None:
674 dsrc.ds_cfg['agent_command'] = agent_command
675
676@@ -260,29 +380,20 @@ fdescfs /dev/fd fdescfs rw 0 0
677 res = get_path_dev_freebsd('/etc', mnt_list)
678 self.assertIsNotNone(res)
679
680- @mock.patch('cloudinit.sources.DataSourceAzure.util.read_dmi_data')
681- def test_non_azure_dmi_chassis_asset_tag(self, m_read_dmi_data):
682- """Report non-azure when DMI's chassis asset tag doesn't match.
683-
684- Return False when the asset tag doesn't match Azure's static
685- AZURE_CHASSIS_ASSET_TAG.
686- """
687+ @mock.patch('cloudinit.sources.DataSourceAzure._is_platform_viable')
688+ def test_call_is_platform_viable_seed(self, m_is_platform_viable):
689+ """Check seed_dir using _is_platform_viable and return False."""
690 # Return a non-matching asset tag value
691- nonazure_tag = dsaz.AZURE_CHASSIS_ASSET_TAG + 'X'
692- m_read_dmi_data.return_value = nonazure_tag
693+ m_is_platform_viable.return_value = False
694 dsrc = dsaz.DataSourceAzure(
695 {}, distro=None, paths=self.paths)
696 self.assertFalse(dsrc.get_data())
697- self.assertEqual(
698- "DEBUG: Non-Azure DMI asset tag '{0}' discovered.\n".format(
699- nonazure_tag),
700- self.logs.getvalue())
701+ m_is_platform_viable.assert_called_with(dsrc.seed_dir)
702
703 def test_basic_seed_dir(self):
704 odata = {'HostName': "myhost", 'UserName': "myuser"}
705 data = {'ovfcontent': construct_valid_ovf_env(data=odata),
706 'sys_cfg': {}}
707-
708 dsrc = self._get_ds(data)
709 ret = dsrc.get_data()
710 self.assertTrue(ret)
711@@ -291,6 +402,82 @@ fdescfs /dev/fd fdescfs rw 0 0
712 self.assertTrue(os.path.isfile(
713 os.path.join(self.waagent_d, 'ovf-env.xml')))
714
715+ def test_get_data_non_ubuntu_will_not_remove_network_scripts(self):
716+ """get_data on non-Ubuntu will not remove ubuntu net scripts."""
717+ odata = {'HostName': "myhost", 'UserName': "myuser"}
718+ data = {'ovfcontent': construct_valid_ovf_env(data=odata),
719+ 'sys_cfg': {}}
720+
721+ dsrc = self._get_ds(data, distro='debian')
722+ dsrc.get_data()
723+ self.m_remove_ubuntu_network_scripts.assert_not_called()
724+
725+ def test_get_data_on_ubuntu_will_remove_network_scripts(self):
726+ """get_data will remove ubuntu net scripts on Ubuntu distro."""
727+ odata = {'HostName': "myhost", 'UserName': "myuser"}
728+ data = {'ovfcontent': construct_valid_ovf_env(data=odata),
729+ 'sys_cfg': {}}
730+
731+ dsrc = self._get_ds(data, distro='ubuntu')
732+ dsrc.get_data()
733+ self.m_remove_ubuntu_network_scripts.assert_called_once_with()
734+
735+ def test_crawl_metadata_returns_structured_data_and_caches_nothing(self):
736+ """Return all structured metadata and cache no class attributes."""
737+ yaml_cfg = "{agent_command: my_command}\n"
738+ odata = {'HostName': "myhost", 'UserName': "myuser",
739+ 'UserData': {'text': 'FOOBAR', 'encoding': 'plain'},
740+ 'dscfg': {'text': yaml_cfg, 'encoding': 'plain'}}
741+ data = {'ovfcontent': construct_valid_ovf_env(data=odata),
742+ 'sys_cfg': {}}
743+ dsrc = self._get_ds(data)
744+ expected_cfg = {
745+ 'PreprovisionedVm': False,
746+ 'datasource': {'Azure': {'agent_command': 'my_command'}},
747+ 'system_info': {'default_user': {'name': u'myuser'}}}
748+ expected_metadata = {
749+ 'azure_data': {
750+ 'configurationsettype': 'LinuxProvisioningConfiguration'},
751+ 'imds': {'network': {'interface': [{
752+ 'ipv4': {'ipAddress': [
753+ {'privateIpAddress': '10.0.0.4',
754+ 'publicIpAddress': '104.46.124.81'}],
755+ 'subnet': [{'address': '10.0.0.0', 'prefix': '24'}]},
756+ 'ipv6': {'ipAddress': []},
757+ 'macAddress': '000D3A047598'}]}},
758+ 'instance-id': 'test-instance-id',
759+ 'local-hostname': u'myhost',
760+ 'random_seed': 'wild'}
761+
762+ crawled_metadata = dsrc.crawl_metadata()
763+
764+ self.assertItemsEqual(
765+ crawled_metadata.keys(),
766+ ['cfg', 'files', 'metadata', 'userdata_raw'])
767+ self.assertEqual(crawled_metadata['cfg'], expected_cfg)
768+ self.assertEqual(
769+ list(crawled_metadata['files'].keys()), ['ovf-env.xml'])
770+ self.assertIn(
771+ b'<HostName>myhost</HostName>',
772+ crawled_metadata['files']['ovf-env.xml'])
773+ self.assertEqual(crawled_metadata['metadata'], expected_metadata)
774+ self.assertEqual(crawled_metadata['userdata_raw'], 'FOOBAR')
775+ self.assertEqual(dsrc.userdata_raw, None)
776+ self.assertEqual(dsrc.metadata, {})
777+ self.assertEqual(dsrc._metadata_imds, UNSET)
778+ self.assertFalse(os.path.isfile(
779+ os.path.join(self.waagent_d, 'ovf-env.xml')))
780+
781+ def test_crawl_metadata_raises_invalid_metadata_on_error(self):
782+ """crawl_metadata raises an exception on invalid ovf-env.xml."""
783+ data = {'ovfcontent': "BOGUS", 'sys_cfg': {}}
784+ dsrc = self._get_ds(data)
785+ error_msg = ('BrokenAzureDataSource: Invalid ovf-env.xml:'
786+ ' syntax error: line 1, column 0')
787+ with self.assertRaises(InvalidMetaDataException) as cm:
788+ dsrc.crawl_metadata()
789+ self.assertEqual(str(cm.exception), error_msg)
790+
791 def test_waagent_d_has_0700_perms(self):
792 # we expect /var/lib/waagent to be created 0700
793 dsrc = self._get_ds({'ovfcontent': construct_valid_ovf_env()})
794@@ -314,6 +501,20 @@ fdescfs /dev/fd fdescfs rw 0 0
795 self.assertTrue(ret)
796 self.assertEqual(data['agent_invoked'], cfg['agent_command'])
797
798+ def test_network_config_set_from_imds(self):
799+ """Datasource.network_config returns IMDS network data."""
800+ odata = {}
801+ data = {'ovfcontent': construct_valid_ovf_env(data=odata)}
802+ expected_network_config = {
803+ 'ethernets': {
804+ 'eth0': {'set-name': 'eth0',
805+ 'match': {'macaddress': '00:0d:3a:04:75:98'},
806+ 'dhcp4': True}},
807+ 'version': 2}
808+ dsrc = self._get_ds(data)
809+ dsrc.get_data()
810+ self.assertEqual(expected_network_config, dsrc.network_config)
811+
812 def test_user_cfg_set_agent_command(self):
813 # set dscfg in via base64 encoded yaml
814 cfg = {'agent_command': "my_command"}
815@@ -579,12 +780,34 @@ fdescfs /dev/fd fdescfs rw 0 0
816 self.assertEqual(
817 [mock.call("/dev/cd0")], m_check_fbsd_cdrom.call_args_list)
818
819+ @mock.patch('cloudinit.net.generate_fallback_config')
820+ def test_imds_network_config(self, mock_fallback):
821+ """Network config is generated from IMDS network data when present."""
822+ odata = {'HostName': "myhost", 'UserName': "myuser"}
823+ data = {'ovfcontent': construct_valid_ovf_env(data=odata),
824+ 'sys_cfg': {}}
825+
826+ dsrc = self._get_ds(data)
827+ ret = dsrc.get_data()
828+ self.assertTrue(ret)
829+
830+ expected_cfg = {
831+ 'ethernets': {
832+ 'eth0': {'dhcp4': True,
833+ 'match': {'macaddress': '00:0d:3a:04:75:98'},
834+ 'set-name': 'eth0'}},
835+ 'version': 2}
836+
837+ self.assertEqual(expected_cfg, dsrc.network_config)
838+ mock_fallback.assert_not_called()
839+
840 @mock.patch('cloudinit.net.get_interface_mac')
841 @mock.patch('cloudinit.net.get_devicelist')
842 @mock.patch('cloudinit.net.device_driver')
843 @mock.patch('cloudinit.net.generate_fallback_config')
844- def test_network_config(self, mock_fallback, mock_dd,
845- mock_devlist, mock_get_mac):
846+ def test_fallback_network_config(self, mock_fallback, mock_dd,
847+ mock_devlist, mock_get_mac):
848+ """On absent IMDS network data, generate network fallback config."""
849 odata = {'HostName': "myhost", 'UserName': "myuser"}
850 data = {'ovfcontent': construct_valid_ovf_env(data=odata),
851 'sys_cfg': {}}
852@@ -605,6 +828,8 @@ fdescfs /dev/fd fdescfs rw 0 0
853 mock_get_mac.return_value = '00:11:22:33:44:55'
854
855 dsrc = self._get_ds(data)
856+ # Represent empty response from network imds
857+ self.m_get_metadata_from_imds.return_value = {}
858 ret = dsrc.get_data()
859 self.assertTrue(ret)
860
861@@ -617,8 +842,9 @@ fdescfs /dev/fd fdescfs rw 0 0
862 @mock.patch('cloudinit.net.get_devicelist')
863 @mock.patch('cloudinit.net.device_driver')
864 @mock.patch('cloudinit.net.generate_fallback_config')
865- def test_network_config_blacklist(self, mock_fallback, mock_dd,
866- mock_devlist, mock_get_mac):
867+ def test_fallback_network_config_blacklist(self, mock_fallback, mock_dd,
868+ mock_devlist, mock_get_mac):
869+ """On absent network metadata, blacklist mlx from fallback config."""
870 odata = {'HostName': "myhost", 'UserName': "myuser"}
871 data = {'ovfcontent': construct_valid_ovf_env(data=odata),
872 'sys_cfg': {}}
873@@ -649,6 +875,8 @@ fdescfs /dev/fd fdescfs rw 0 0
874 mock_get_mac.return_value = '00:11:22:33:44:55'
875
876 dsrc = self._get_ds(data)
877+ # Represent empty response from network imds
878+ self.m_get_metadata_from_imds.return_value = {}
879 ret = dsrc.get_data()
880 self.assertTrue(ret)
881
882@@ -689,9 +917,12 @@ class TestAzureBounce(CiTestCase):
883 mock.patch.object(dsaz, 'get_metadata_from_fabric',
884 mock.MagicMock(return_value={})))
885 self.patches.enter_context(
886- mock.patch.object(dsaz.util, 'which', lambda x: True))
887+ mock.patch.object(dsaz, 'get_metadata_from_imds',
888+ mock.MagicMock(return_value={})))
889 self.patches.enter_context(
890- mock.patch.object(dsaz, '_get_random_seed'))
891+ mock.patch.object(dsaz.util, 'which', lambda x: True))
892+ self.patches.enter_context(mock.patch.object(
893+ dsaz, '_get_random_seed', return_value='wild'))
894
895 def _dmi_mocks(key):
896 if key == 'system-uuid':
897@@ -719,9 +950,12 @@ class TestAzureBounce(CiTestCase):
898 mock.patch.object(dsaz, 'set_hostname'))
899 self.subp = self.patches.enter_context(
900 mock.patch('cloudinit.sources.DataSourceAzure.util.subp'))
901+ self.find_fallback_nic = self.patches.enter_context(
902+ mock.patch('cloudinit.net.find_fallback_nic', return_value='eth9'))
903
904 def tearDown(self):
905 self.patches.close()
906+ super(TestAzureBounce, self).tearDown()
907
908 def _get_ds(self, ovfcontent=None, agent_command=None):
909 if ovfcontent is not None:
910@@ -927,7 +1161,7 @@ class TestLoadAzureDsDir(CiTestCase):
911 str(context_manager.exception))
912
913
914-class TestReadAzureOvf(TestCase):
915+class TestReadAzureOvf(CiTestCase):
916
917 def test_invalid_xml_raises_non_azure_ds(self):
918 invalid_xml = "<foo>" + construct_valid_ovf_env(data={})
919@@ -1188,6 +1422,25 @@ class TestCanDevBeReformatted(CiTestCase):
920 "(datasource.Azure.never_destroy_ntfs)", msg)
921
922
923+class TestClearCachedData(CiTestCase):
924+
925+ def test_clear_cached_attrs_clears_imds(self):
926+ """All class attributes are reset to defaults, including imds data."""
927+ tmp = self.tmp_dir()
928+ paths = helpers.Paths(
929+ {'cloud_dir': tmp, 'run_dir': tmp})
930+ dsrc = dsaz.DataSourceAzure({}, distro=None, paths=paths)
931+ clean_values = [dsrc.metadata, dsrc.userdata, dsrc._metadata_imds]
932+ dsrc.metadata = 'md'
933+ dsrc.userdata = 'ud'
934+ dsrc._metadata_imds = 'imds'
935+ dsrc._dirty_cache = True
936+ dsrc.clear_cached_attrs()
937+ self.assertEqual(
938+ [dsrc.metadata, dsrc.userdata, dsrc._metadata_imds],
939+ clean_values)
940+
941+
942 class TestAzureNetExists(CiTestCase):
943
944 def test_azure_net_must_exist_for_legacy_objpkl(self):
945@@ -1398,4 +1651,128 @@ class TestAzureDataSourcePreprovisioning(CiTestCase):
946 self.assertEqual(m_net.call_count, 1)
947
948
949+class TestRemoveUbuntuNetworkConfigScripts(CiTestCase):
950+
951+ with_logs = True
952+
953+ def setUp(self):
954+ super(TestRemoveUbuntuNetworkConfigScripts, self).setUp()
955+ self.tmp = self.tmp_dir()
956+
957+ def test_remove_network_scripts_removes_both_files_and_directories(self):
958+ """Any files or directories in paths are removed when present."""
959+ file1 = self.tmp_path('file1', dir=self.tmp)
960+ subdir = self.tmp_path('sub1', dir=self.tmp)
961+ subfile = self.tmp_path('leaf1', dir=subdir)
962+ write_file(file1, 'file1content')
963+ write_file(subfile, 'leafcontent')
964+ dsaz.maybe_remove_ubuntu_network_config_scripts(paths=[subdir, file1])
965+
966+ for path in (file1, subdir, subfile):
967+ self.assertFalse(os.path.exists(path),
968+ 'Found unremoved: %s' % path)
969+
970+ expected_logs = [
971+ 'INFO: Removing Ubuntu extended network scripts because cloud-init'
972+ ' updates Azure network configuration on the following event:'
973+ ' System boot.',
974+ 'Recursively deleting %s' % subdir,
975+ 'Attempting to remove %s' % file1]
976+ for log in expected_logs:
977+ self.assertIn(log, self.logs.getvalue())
978+
979+ def test_remove_network_scripts_only_attempts_removal_if_path_exists(self):
980+ """Any files or directories absent are skipped without error."""
981+ dsaz.maybe_remove_ubuntu_network_config_scripts(paths=[
982+ self.tmp_path('nodirhere/', dir=self.tmp),
983+ self.tmp_path('notfilehere', dir=self.tmp)])
984+ self.assertNotIn('/not/a', self.logs.getvalue()) # No delete logs
985+
986+ @mock.patch('cloudinit.sources.DataSourceAzure.os.path.exists')
987+ def test_remove_network_scripts_default_removes_stock_scripts(self,
988+ m_exists):
989+ """Azure's stock ubuntu image scripts and artifacts are removed."""
990+ # Report path absent on all to avoid delete operation
991+ m_exists.return_value = False
992+ dsaz.maybe_remove_ubuntu_network_config_scripts()
993+ calls = m_exists.call_args_list
994+ for path in dsaz.UBUNTU_EXTENDED_NETWORK_SCRIPTS:
995+ self.assertIn(mock.call(path), calls)
996+
997+
998+class TestWBIsPlatformViable(CiTestCase):
999+ """White box tests for _is_platform_viable."""
1000+ with_logs = True
1001+
1002+ @mock.patch('cloudinit.sources.DataSourceAzure.util.read_dmi_data')
1003+ def test_true_on_non_azure_chassis(self, m_read_dmi_data):
1004+ """Return True if DMI chassis-asset-tag is AZURE_CHASSIS_ASSET_TAG."""
1005+ m_read_dmi_data.return_value = dsaz.AZURE_CHASSIS_ASSET_TAG
1006+ self.assertTrue(dsaz._is_platform_viable('doesnotmatter'))
1007+
1008+ @mock.patch('cloudinit.sources.DataSourceAzure.os.path.exists')
1009+ @mock.patch('cloudinit.sources.DataSourceAzure.util.read_dmi_data')
1010+ def test_true_on_azure_ovf_env_in_seed_dir(self, m_read_dmi_data, m_exist):
1011+ """Return True if ovf-env.xml exists in known seed dirs."""
1012+ # Non-matching Azure chassis-asset-tag
1013+ m_read_dmi_data.return_value = dsaz.AZURE_CHASSIS_ASSET_TAG + 'X'
1014+
1015+ m_exist.side_effect = [False, True]
1016+ self.assertTrue(dsaz._is_platform_viable('/other/seed/dir'))
1017+ self.assertEqual(
1018+ [mock.call('/var/lib/waagent/ovf-env.xml'),
1019+ mock.call('/other/seed/dir/ovf-env.xml')],
1020+ m_exist.call_args_list)
1021+
1022+ @mock.patch('cloudinit.sources.DataSourceAzure.util.which')
1023+ @mock.patch('cloudinit.sources.DataSourceAzure.util.subp')
1024+ def test_true_on_detect_virt_microsoft(self, m_subp, m_which):
1025+ """Return True if a partition label prefix rd_rdfe is present."""
1026+ m_which.return_value = '/usr/bin/systemd-detect-virt'
1027+ m_subp.return_value = ('microsoft\n', '')
1028+ self.assertTrue(wrap_and_call(
1029+ 'cloudinit.sources.DataSourceAzure',
1030+ {'os.path.exists': False,
1031+ # Non-matching Azure chassis-asset-tag
1032+ 'util.read_dmi_data': dsaz.AZURE_CHASSIS_ASSET_TAG + 'X'},
1033+ dsaz._is_platform_viable, 'doesnotmatter'))
1034+ m_which.assert_called_once_with('systemd-detect-virt')
1035+ m_subp.assert_called_once_with(
1036+ ['systemd-detect-virt'], capture=True, rcs=[0, 1])
1037+
1038+ @mock.patch('cloudinit.sources.DataSourceAzure.util.find_devs_with')
1039+ def test_true_on_azure_when_fs_label_prefix_rd_rdfe(self, m_find_devs):
1040+ """Return True if a partition label prefix rd_rdfe is present."""
1041+
1042+ m_find_devs.return_value = ['/dev/Imatched/azure']
1043+ self.assertTrue(wrap_and_call(
1044+ 'cloudinit.sources.DataSourceAzure',
1045+ {'os.path.exists': False,
1046+ # Non-matching Azure chassis-asset-tag
1047+ 'util.read_dmi_data': dsaz.AZURE_CHASSIS_ASSET_TAG + 'X',
1048+ 'util.which': None},
1049+ dsaz._is_platform_viable, 'doesnotmatter'))
1050+ m_find_devs.assert_called_once_with(criteria='LABEL=rd_rdfe_*')
1051+
1052+ def test_false_on_no_matching_azure_criteria(self):
1053+ """Report non-azure on unmatched asset tag, ovf-env absent and no dev.
1054+
1055+ Return False when the asset tag doesn't match Azure's static
1056+ AZURE_CHASSIS_ASSET_TAG, no ovf-env.xml files exist in known seed dirs
1057+ and no devices have a label starting with prefix 'rd_rdfe_'.
1058+ """
1059+ self.assertFalse(wrap_and_call(
1060+ 'cloudinit.sources.DataSourceAzure',
1061+ {'os.path.exists': False,
1062+ # Non-matching Azure chassis-asset-tag
1063+ 'util.read_dmi_data': dsaz.AZURE_CHASSIS_ASSET_TAG + 'X',
1064+ 'util.which': None,
1065+ 'util.find_devs_with': []},
1066+ dsaz._is_platform_viable, 'doesnotmatter'))
1067+ self.assertIn(
1068+ "DEBUG: Non-Azure DMI asset tag '{0}' discovered.\n".format(
1069+ dsaz.AZURE_CHASSIS_ASSET_TAG + 'X'),
1070+ self.logs.getvalue())
1071+
1072+
1073 # vi: ts=4 expandtab

Subscribers

People subscribed via source and target branches