Merge ~daniel-thewatkins/cloud-init/+git/cloud-init:ubuntu/xenial into cloud-init:ubuntu/xenial

Proposed by Dan Watkins on 2019-03-07
Status: Merged
Merged at revision: 0d05e437ef00065ff7259f957d02ce3237b20056
Proposed branch: ~daniel-thewatkins/cloud-init/+git/cloud-init:ubuntu/xenial
Merge into: cloud-init:ubuntu/xenial
Diff against target: 2255 lines (+1176/-160)
53 files modified
.gitignore (+1/-0)
cloudinit/cmd/clean.py (+14/-13)
cloudinit/cmd/tests/test_clean.py (+4/-2)
cloudinit/config/cc_apt_pipelining.py (+2/-2)
cloudinit/config/cc_chef.py (+3/-0)
cloudinit/config/cc_rsyslog.py (+1/-1)
cloudinit/config/schema.py (+1/-1)
cloudinit/config/tests/test_apt_pipelining.py (+28/-0)
cloudinit/distros/__init__.py (+9/-4)
cloudinit/handlers/upstart_job.py (+1/-1)
cloudinit/net/netplan.py (+2/-1)
cloudinit/net/network_state.py (+2/-2)
cloudinit/net/tests/test_dhcp.py (+1/-0)
cloudinit/netinfo.py (+5/-2)
cloudinit/safeyaml.py (+7/-0)
cloudinit/sources/DataSourceAzure.py (+9/-4)
cloudinit/sources/DataSourceEc2.py (+22/-1)
cloudinit/sources/DataSourceOVF.py (+3/-1)
cloudinit/sources/helpers/azure.py (+78/-31)
cloudinit/sources/helpers/openstack.py (+6/-6)
cloudinit/stages.py (+2/-2)
cloudinit/tests/helpers.py (+2/-21)
cloudinit/tests/test_netinfo.py (+14/-0)
cloudinit/url_helper.py (+1/-1)
cloudinit/util.py (+11/-17)
debian/changelog (+40/-0)
debian/cloud-init.postinst (+0/-14)
debian/control (+1/-1)
debian/patches/ec2-classic-dont-reapply-networking.patch (+25/-0)
debian/patches/series (+1/-0)
doc/examples/cloud-config-chef.txt (+3/-0)
doc/examples/cloud-config-disk-setup.txt (+17/-2)
doc/rtd/topics/datasources/ec2.rst (+11/-0)
doc/rtd/topics/instancedata.rst (+1/-1)
doc/rtd/topics/merging.rst (+84/-6)
templates/chef_client.rb.tmpl (+4/-1)
tests/cloud_tests/verify.py (+7/-2)
tests/data/azure/parse_certificates_fingerprints (+4/-0)
tests/data/azure/parse_certificates_pem (+152/-0)
tests/data/azure/pubkey_extract_cert (+13/-0)
tests/data/azure/pubkey_extract_ssh_key (+1/-0)
tests/data/netinfo/freebsd-ifconfig-output (+17/-0)
tests/data/netinfo/freebsd-netdev-formatted-output (+11/-0)
tests/unittests/test_datasource/test_azure.py (+2/-3)
tests/unittests/test_datasource/test_azure_helper.py (+65/-6)
tests/unittests/test_datasource/test_configdrive.py (+27/-8)
tests/unittests/test_datasource/test_ec2.py (+24/-0)
tests/unittests/test_distros/test_create_users.py (+28/-0)
tests/unittests/test_distros/test_netconfig.py (+1/-1)
tests/unittests/test_ds_identify.py (+1/-1)
tests/unittests/test_handler/test_handler_chef.py (+3/-0)
tests/unittests/test_net.py (+397/-0)
tools/cloud-init-per (+7/-1)
Reviewer Review Type Date Requested Status
Ryan Harper 2019-03-07 Approve on 2019-03-11
Server Team CI bot continuous-integration Needs Fixing on 2019-03-11
Review via email: mp+364127@code.launchpad.net
To post a comment you must log in.

FAILED: Continuous integration, rev:1f7d994b67569eea61ea1e8e2f7e88b56eb038b4
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/364127/+edit-commit-message

https://jenkins.ubuntu.com/server/job/cloud-init-ci/624/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/624/rebuild

review: Needs Fixing (continuous-integration)
Ryan Harper (raharper) :
review: Approve

FAILED: Continuous integration, rev:c72ad48fb1b3591d84784398920e413f03dc9247
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/364127/+edit-commit-message

https://jenkins.ubuntu.com/server/job/cloud-init-ci/626/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/626/rebuild

review: Needs Fixing (continuous-integration)

FAILED: Continuous integration, rev:ae685e7cad12300c129b0cbea37e754e6105a919
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/364127/+edit-commit-message

https://jenkins.ubuntu.com/server/job/cloud-init-ci/629/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/629/rebuild

review: Needs Fixing (continuous-integration)

FAILED: Continuous integration, rev:0d05e437ef00065ff7259f957d02ce3237b20056
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~daniel-thewatkins/cloud-init/+git/cloud-init/+merge/364127/+edit-commit-message

https://jenkins.ubuntu.com/server/job/cloud-init-ci/633/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/633/rebuild

review: Needs Fixing (continuous-integration)
Ryan Harper (raharper) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/.gitignore b/.gitignore
2index 75565ed..80c509e 100644
3--- a/.gitignore
4+++ b/.gitignore
5@@ -11,3 +11,4 @@ prime
6 stage
7 *.snap
8 *.cover
9+.idea/
10diff --git a/cloudinit/cmd/clean.py b/cloudinit/cmd/clean.py
11index de22f7f..30e49de 100644
12--- a/cloudinit/cmd/clean.py
13+++ b/cloudinit/cmd/clean.py
14@@ -5,12 +5,13 @@
15 """Define 'clean' utility and handler as part of cloud-init commandline."""
16
17 import argparse
18+import glob
19 import os
20 import sys
21
22 from cloudinit.stages import Init
23 from cloudinit.util import (
24- ProcessExecutionError, chdir, del_dir, del_file, get_config_logfiles,
25+ ProcessExecutionError, del_dir, del_file, get_config_logfiles,
26 is_link, subp)
27
28
29@@ -61,18 +62,18 @@ def remove_artifacts(remove_logs, remove_seed=False):
30
31 if not os.path.isdir(init.paths.cloud_dir):
32 return 0 # Artifacts dir already cleaned
33- with chdir(init.paths.cloud_dir):
34- for path in os.listdir('.'):
35- if path == 'seed' and not remove_seed:
36- continue
37- try:
38- if os.path.isdir(path) and not is_link(path):
39- del_dir(path)
40- else:
41- del_file(path)
42- except OSError as e:
43- error('Could not remove {0}: {1}'.format(path, str(e)))
44- return 1
45+ seed_path = os.path.join(init.paths.cloud_dir, 'seed')
46+ for path in glob.glob('%s/*' % init.paths.cloud_dir):
47+ if path == seed_path and not remove_seed:
48+ continue
49+ try:
50+ if os.path.isdir(path) and not is_link(path):
51+ del_dir(path)
52+ else:
53+ del_file(path)
54+ except OSError as e:
55+ error('Could not remove {0}: {1}'.format(path, str(e)))
56+ return 1
57 return 0
58
59
60diff --git a/cloudinit/cmd/tests/test_clean.py b/cloudinit/cmd/tests/test_clean.py
61index 5a3ec3b..f092ab3 100644
62--- a/cloudinit/cmd/tests/test_clean.py
63+++ b/cloudinit/cmd/tests/test_clean.py
64@@ -22,7 +22,8 @@ class TestClean(CiTestCase):
65 class FakeInit(object):
66 cfg = {'def_log_file': self.log1,
67 'output': {'all': '|tee -a {0}'.format(self.log2)}}
68- paths = mypaths(cloud_dir=self.artifact_dir)
69+ # Ensure cloud_dir has a trailing slash, to match real behaviour
70+ paths = mypaths(cloud_dir='{}/'.format(self.artifact_dir))
71
72 def __init__(self, ds_deps):
73 pass
74@@ -136,7 +137,8 @@ class TestClean(CiTestCase):
75 clean.remove_artifacts, remove_logs=False)
76 self.assertEqual(1, retcode)
77 self.assertEqual(
78- 'ERROR: Could not remove dir1: oops\n', m_stderr.getvalue())
79+ 'ERROR: Could not remove %s/dir1: oops\n' % self.artifact_dir,
80+ m_stderr.getvalue())
81
82 def test_handle_clean_args_reboots(self):
83 """handle_clean_args_reboots when reboot arg is provided."""
84diff --git a/cloudinit/config/cc_apt_pipelining.py b/cloudinit/config/cc_apt_pipelining.py
85index cdf28cd..459332a 100644
86--- a/cloudinit/config/cc_apt_pipelining.py
87+++ b/cloudinit/config/cc_apt_pipelining.py
88@@ -49,7 +49,7 @@ APT_PIPE_TPL = ("//Written by cloud-init per 'apt_pipelining'\n"
89
90 def handle(_name, cfg, _cloud, log, _args):
91
92- apt_pipe_value = util.get_cfg_option_str(cfg, "apt_pipelining", False)
93+ apt_pipe_value = util.get_cfg_option_str(cfg, "apt_pipelining", 'os')
94 apt_pipe_value_s = str(apt_pipe_value).lower().strip()
95
96 if apt_pipe_value_s == "false":
97@@ -59,7 +59,7 @@ def handle(_name, cfg, _cloud, log, _args):
98 elif apt_pipe_value_s in [str(b) for b in range(0, 6)]:
99 write_apt_snippet(apt_pipe_value_s, log, DEFAULT_FILE)
100 else:
101- log.warn("Invalid option for apt_pipeling: %s", apt_pipe_value)
102+ log.warn("Invalid option for apt_pipelining: %s", apt_pipe_value)
103
104
105 def write_apt_snippet(setting, log, f_name):
106diff --git a/cloudinit/config/cc_chef.py b/cloudinit/config/cc_chef.py
107index 46abedd..a624030 100644
108--- a/cloudinit/config/cc_chef.py
109+++ b/cloudinit/config/cc_chef.py
110@@ -51,6 +51,7 @@ file).
111
112 chef:
113 client_key:
114+ encrypted_data_bag_secret:
115 environment:
116 file_backup_path:
117 file_cache_path:
118@@ -114,6 +115,7 @@ CHEF_RB_TPL_DEFAULTS = {
119 'file_backup_path': "/var/backups/chef",
120 'pid_file': "/var/run/chef/client.pid",
121 'show_time': True,
122+ 'encrypted_data_bag_secret': None,
123 }
124 CHEF_RB_TPL_BOOL_KEYS = frozenset(['show_time'])
125 CHEF_RB_TPL_PATH_KEYS = frozenset([
126@@ -124,6 +126,7 @@ CHEF_RB_TPL_PATH_KEYS = frozenset([
127 'json_attribs',
128 'file_cache_path',
129 'pid_file',
130+ 'encrypted_data_bag_secret',
131 ])
132 CHEF_RB_TPL_KEYS = list(CHEF_RB_TPL_DEFAULTS.keys())
133 CHEF_RB_TPL_KEYS.extend(CHEF_RB_TPL_BOOL_KEYS)
134diff --git a/cloudinit/config/cc_rsyslog.py b/cloudinit/config/cc_rsyslog.py
135index 27d2366..22b1753 100644
136--- a/cloudinit/config/cc_rsyslog.py
137+++ b/cloudinit/config/cc_rsyslog.py
138@@ -203,7 +203,7 @@ LOG = logging.getLogger(__name__)
139 COMMENT_RE = re.compile(r'[ ]*[#]+[ ]*')
140 HOST_PORT_RE = re.compile(
141 r'^(?P<proto>[@]{0,2})'
142- r'(([[](?P<bracket_addr>[^\]]*)[\]])|(?P<addr>[^:]*))'
143+ r'(([\[](?P<bracket_addr>[^\]]*)[\]])|(?P<addr>[^:]*))'
144 r'([:](?P<port>[0-9]+))?$')
145
146
147diff --git a/cloudinit/config/schema.py b/cloudinit/config/schema.py
148index 080a6d0..807c3ee 100644
149--- a/cloudinit/config/schema.py
150+++ b/cloudinit/config/schema.py
151@@ -367,7 +367,7 @@ def handle_schema_args(name, args):
152 if not args.annotate:
153 error(str(e))
154 except RuntimeError as e:
155- error(str(e))
156+ error(str(e))
157 else:
158 print("Valid cloud-config file {0}".format(args.config_file))
159 if args.doc:
160diff --git a/cloudinit/config/tests/test_apt_pipelining.py b/cloudinit/config/tests/test_apt_pipelining.py
161new file mode 100644
162index 0000000..2a6bb10
163--- /dev/null
164+++ b/cloudinit/config/tests/test_apt_pipelining.py
165@@ -0,0 +1,28 @@
166+# This file is part of cloud-init. See LICENSE file for license information.
167+
168+"""Tests cc_apt_pipelining handler"""
169+
170+import cloudinit.config.cc_apt_pipelining as cc_apt_pipelining
171+
172+from cloudinit.tests.helpers import CiTestCase, mock
173+
174+
175+class TestAptPipelining(CiTestCase):
176+
177+ @mock.patch('cloudinit.config.cc_apt_pipelining.util.write_file')
178+ def test_not_disabled_by_default(self, m_write_file):
179+ """ensure that default behaviour is to not disable pipelining"""
180+ cc_apt_pipelining.handle('foo', {}, None, mock.MagicMock(), None)
181+ self.assertEqual(0, m_write_file.call_count)
182+
183+ @mock.patch('cloudinit.config.cc_apt_pipelining.util.write_file')
184+ def test_false_disables_pipelining(self, m_write_file):
185+ """ensure that pipelining can be disabled with correct config"""
186+ cc_apt_pipelining.handle(
187+ 'foo', {'apt_pipelining': 'false'}, None, mock.MagicMock(), None)
188+ self.assertEqual(1, m_write_file.call_count)
189+ args, _ = m_write_file.call_args
190+ self.assertEqual(cc_apt_pipelining.DEFAULT_FILE, args[0])
191+ self.assertIn('Pipeline-Depth "0"', args[1])
192+
193+# vi: ts=4 expandtab
194diff --git a/cloudinit/distros/__init__.py b/cloudinit/distros/__init__.py
195index ef618c2..20c994d 100644
196--- a/cloudinit/distros/__init__.py
197+++ b/cloudinit/distros/__init__.py
198@@ -577,11 +577,16 @@ class Distro(object):
199 """
200 Lock the password of a user, i.e., disable password logins
201 """
202+ # passwd must use short '-l' due to SLES11 lacking long form '--lock'
203+ lock_tools = (['passwd', '-l', name], ['usermod', '--lock', name])
204 try:
205- # Need to use the short option name '-l' instead of '--lock'
206- # (which would be more descriptive) since SLES 11 doesn't know
207- # about long names.
208- util.subp(['passwd', '-l', name])
209+ cmd = next(l for l in lock_tools if util.which(l[0]))
210+ except StopIteration:
211+ raise RuntimeError((
212+ "Unable to lock user account '%s'. No tools available. "
213+ " Tried: %s.") % (name, [c[0] for c in lock_tools]))
214+ try:
215+ util.subp(cmd)
216 except Exception as e:
217 util.logexc(LOG, 'Failed to disable password for user %s', name)
218 raise e
219diff --git a/cloudinit/handlers/upstart_job.py b/cloudinit/handlers/upstart_job.py
220index 83fb072..003cad6 100644
221--- a/cloudinit/handlers/upstart_job.py
222+++ b/cloudinit/handlers/upstart_job.py
223@@ -89,7 +89,7 @@ def _has_suitable_upstart():
224 util.subp(["dpkg", "--compare-versions", dpkg_ver, "ge", good])
225 return True
226 except util.ProcessExecutionError as e:
227- if e.exit_code is 1:
228+ if e.exit_code == 1:
229 pass
230 else:
231 util.logexc(LOG, "dpkg --compare-versions failed [%s]",
232diff --git a/cloudinit/net/netplan.py b/cloudinit/net/netplan.py
233index 21517fd..e54a34e 100644
234--- a/cloudinit/net/netplan.py
235+++ b/cloudinit/net/netplan.py
236@@ -361,7 +361,8 @@ class Renderer(renderer.Renderer):
237 if section:
238 dump = util.yaml_dumps({name: section},
239 explicit_start=False,
240- explicit_end=False)
241+ explicit_end=False,
242+ noalias=True)
243 txt = util.indent(dump, ' ' * 4)
244 return [txt]
245 return []
246diff --git a/cloudinit/net/network_state.py b/cloudinit/net/network_state.py
247index f76e508..539b76d 100644
248--- a/cloudinit/net/network_state.py
249+++ b/cloudinit/net/network_state.py
250@@ -706,9 +706,9 @@ class NetworkStateInterpreter(object):
251 """Common ipconfig extraction from v2 to v1 subnets array."""
252
253 subnets = []
254- if 'dhcp4' in cfg:
255+ if cfg.get('dhcp4'):
256 subnets.append({'type': 'dhcp4'})
257- if 'dhcp6' in cfg:
258+ if cfg.get('dhcp6'):
259 self.use_ipv6 = True
260 subnets.append({'type': 'dhcp6'})
261
262diff --git a/cloudinit/net/tests/test_dhcp.py b/cloudinit/net/tests/test_dhcp.py
263index 79e8842..5139024 100644
264--- a/cloudinit/net/tests/test_dhcp.py
265+++ b/cloudinit/net/tests/test_dhcp.py
266@@ -117,6 +117,7 @@ class TestDHCPDiscoveryClean(CiTestCase):
267 self.assertEqual('eth9', call[0][1])
268 self.assertIn('/var/tmp/cloud-init/cloud-init-dhcp-', call[0][2])
269
270+ @mock.patch('time.sleep', mock.MagicMock())
271 @mock.patch('cloudinit.net.dhcp.os.kill')
272 @mock.patch('cloudinit.net.dhcp.util.subp')
273 def test_dhcp_discovery_run_in_sandbox_warns_invalid_pid(self, m_subp,
274diff --git a/cloudinit/netinfo.py b/cloudinit/netinfo.py
275index 9ff929c..e91cd26 100644
276--- a/cloudinit/netinfo.py
277+++ b/cloudinit/netinfo.py
278@@ -141,6 +141,9 @@ def _netdev_info_ifconfig(ifconfig_data):
279 res = re.match(r'.*<(\S+)>', toks[i + 1])
280 if res:
281 devs[curdev]['ipv6'][-1]['scope6'] = res.group(1)
282+ else:
283+ devs[curdev]['ipv6'][-1]['scope6'] = toks[i + 1]
284+
285 return devs
286
287
288@@ -389,8 +392,8 @@ def netdev_pformat():
289 addr.get('scope', empty), data["hwaddr"]))
290 for addr in data.get('ipv6'):
291 tbl.add_row(
292- (dev, data["up"], addr["ip"], empty, addr["scope6"],
293- data["hwaddr"]))
294+ (dev, data["up"], addr["ip"], empty,
295+ addr.get("scope6", empty), data["hwaddr"]))
296 if len(data.get('ipv6')) + len(data.get('ipv4')) == 0:
297 tbl.add_row((dev, data["up"], empty, empty, empty,
298 data["hwaddr"]))
299diff --git a/cloudinit/safeyaml.py b/cloudinit/safeyaml.py
300index 7bcf9dd..3bd5e03 100644
301--- a/cloudinit/safeyaml.py
302+++ b/cloudinit/safeyaml.py
303@@ -17,6 +17,13 @@ _CustomSafeLoader.add_constructor(
304 _CustomSafeLoader.construct_python_unicode)
305
306
307+class NoAliasSafeDumper(yaml.dumper.SafeDumper):
308+ """A class which avoids constructing anchors/aliases on yaml dump"""
309+
310+ def ignore_aliases(self, data):
311+ return True
312+
313+
314 def load(blob):
315 return(yaml.load(blob, Loader=_CustomSafeLoader))
316
317diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py
318index a4f998b..eccbee5 100644
319--- a/cloudinit/sources/DataSourceAzure.py
320+++ b/cloudinit/sources/DataSourceAzure.py
321@@ -627,9 +627,11 @@ class DataSourceAzure(sources.DataSource):
322 if self.ds_cfg['agent_command'] == AGENT_START_BUILTIN:
323 self.bounce_network_with_azure_hostname()
324
325+ pubkey_info = self.cfg.get('_pubkeys', None)
326 metadata_func = partial(get_metadata_from_fabric,
327 fallback_lease_file=self.
328- dhclient_lease_file)
329+ dhclient_lease_file,
330+ pubkey_info=pubkey_info)
331 else:
332 metadata_func = self.get_metadata_from_agent
333
334@@ -642,6 +644,7 @@ class DataSourceAzure(sources.DataSource):
335 "Error communicating with Azure fabric; You may experience."
336 "connectivity issues.", exc_info=True)
337 return False
338+
339 util.del_file(REPORTED_READY_MARKER_FILE)
340 util.del_file(REPROVISION_MARKER_FILE)
341 return fabric_data
342@@ -909,13 +912,15 @@ def find_child(node, filter_func):
343 def load_azure_ovf_pubkeys(sshnode):
344 # This parses a 'SSH' node formatted like below, and returns
345 # an array of dicts.
346- # [{'fp': '6BE7A7C3C8A8F4B123CCA5D0C2F1BE4CA7B63ED7',
347- # 'path': 'where/to/go'}]
348+ # [{'fingerprint': '6BE7A7C3C8A8F4B123CCA5D0C2F1BE4CA7B63ED7',
349+ # 'path': '/where/to/go'}]
350 #
351 # <SSH><PublicKeys>
352- # <PublicKey><Fingerprint>ABC</FingerPrint><Path>/ABC</Path>
353+ # <PublicKey><Fingerprint>ABC</FingerPrint><Path>/x/y/z</Path>
354 # ...
355 # </PublicKeys></SSH>
356+ # Under some circumstances, there may be a <Value> element along with the
357+ # Fingerprint and Path. Pass those along if they appear.
358 results = find_child(sshnode, lambda n: n.localName == "PublicKeys")
359 if len(results) == 0:
360 return []
361diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourceEc2.py
362index 9ccf2cd..4f2f6cc 100644
363--- a/cloudinit/sources/DataSourceEc2.py
364+++ b/cloudinit/sources/DataSourceEc2.py
365@@ -19,6 +19,7 @@ from cloudinit import sources
366 from cloudinit import url_helper as uhelp
367 from cloudinit import util
368 from cloudinit import warnings
369+from cloudinit.event import EventType
370
371 LOG = logging.getLogger(__name__)
372
373@@ -107,6 +108,19 @@ class DataSourceEc2(sources.DataSource):
374 'dynamic', {}).get('instance-identity', {}).get('document', {})
375 return True
376
377+ def is_classic_instance(self):
378+ """Report if this instance type is Ec2 Classic (non-vpc)."""
379+ if not self.metadata:
380+ # Can return False on inconclusive as we are also called in
381+ # network_config where metadata will be present.
382+ # Secondary call site is in packaging postinst script.
383+ return False
384+ ifaces_md = self.metadata.get('network', {}).get('interfaces', {})
385+ for _mac, mac_data in ifaces_md.get('macs', {}).items():
386+ if 'vpc-id' in mac_data:
387+ return False
388+ return True
389+
390 @property
391 def launch_index(self):
392 if not self.metadata:
393@@ -320,6 +334,13 @@ class DataSourceEc2(sources.DataSource):
394 if isinstance(net_md, dict):
395 result = convert_ec2_metadata_network_config(
396 net_md, macs_to_nics=macs_to_nics, fallback_nic=iface)
397+ # RELEASE_BLOCKER: Xenial debian/postinst needs to add
398+ # EventType.BOOT on upgrade path for classic.
399+
400+ # Non-VPC (aka Classic) Ec2 instances need to rewrite the
401+ # network config file every boot due to MAC address change.
402+ if self.is_classic_instance():
403+ self.update_events['network'].add(EventType.BOOT)
404 else:
405 LOG.warning("Metadata 'network' key not valid: %s.", net_md)
406 self._network_config = result
407@@ -442,7 +463,7 @@ def identify_aws(data):
408 if (data['uuid'].startswith('ec2') and
409 (data['uuid_source'] == 'hypervisor' or
410 data['uuid'] == data['serial'])):
411- return CloudNames.AWS
412+ return CloudNames.AWS
413
414 return None
415
416diff --git a/cloudinit/sources/DataSourceOVF.py b/cloudinit/sources/DataSourceOVF.py
417index 3a3fcdf..70e7a5c 100644
418--- a/cloudinit/sources/DataSourceOVF.py
419+++ b/cloudinit/sources/DataSourceOVF.py
420@@ -15,6 +15,8 @@ import os
421 import re
422 import time
423
424+import six
425+
426 from cloudinit import log as logging
427 from cloudinit import sources
428 from cloudinit import util
429@@ -434,7 +436,7 @@ def maybe_cdrom_device(devname):
430 """
431 if not devname:
432 return False
433- elif not isinstance(devname, util.string_types):
434+ elif not isinstance(devname, six.string_types):
435 raise ValueError("Unexpected input for devname: %s" % devname)
436
437 # resolve '..' and multi '/' elements
438diff --git a/cloudinit/sources/helpers/azure.py b/cloudinit/sources/helpers/azure.py
439index e5696b1..2829dd2 100644
440--- a/cloudinit/sources/helpers/azure.py
441+++ b/cloudinit/sources/helpers/azure.py
442@@ -138,9 +138,36 @@ class OpenSSLManager(object):
443 self.certificate = certificate
444 LOG.debug('New certificate generated.')
445
446- def parse_certificates(self, certificates_xml):
447- tag = ElementTree.fromstring(certificates_xml).find(
448- './/Data')
449+ @staticmethod
450+ def _run_x509_action(action, cert):
451+ cmd = ['openssl', 'x509', '-noout', action]
452+ result, _ = util.subp(cmd, data=cert)
453+ return result
454+
455+ def _get_ssh_key_from_cert(self, certificate):
456+ pub_key = self._run_x509_action('-pubkey', certificate)
457+ keygen_cmd = ['ssh-keygen', '-i', '-m', 'PKCS8', '-f', '/dev/stdin']
458+ ssh_key, _ = util.subp(keygen_cmd, data=pub_key)
459+ return ssh_key
460+
461+ def _get_fingerprint_from_cert(self, certificate):
462+ """openssl x509 formats fingerprints as so:
463+ 'SHA1 Fingerprint=07:3E:19:D1:4D:1C:79:92:24:C6:A0:FD:8D:DA:\
464+ B6:A8:BF:27:D4:73\n'
465+
466+ Azure control plane passes that fingerprint as so:
467+ '073E19D14D1C799224C6A0FD8DDAB6A8BF27D473'
468+ """
469+ raw_fp = self._run_x509_action('-fingerprint', certificate)
470+ eq = raw_fp.find('=')
471+ octets = raw_fp[eq+1:-1].split(':')
472+ return ''.join(octets)
473+
474+ def _decrypt_certs_from_xml(self, certificates_xml):
475+ """Decrypt the certificates XML document using the our private key;
476+ return the list of certs and private keys contained in the doc.
477+ """
478+ tag = ElementTree.fromstring(certificates_xml).find('.//Data')
479 certificates_content = tag.text
480 lines = [
481 b'MIME-Version: 1.0',
482@@ -151,32 +178,30 @@ class OpenSSLManager(object):
483 certificates_content.encode('utf-8'),
484 ]
485 with cd(self.tmpdir):
486- with open('Certificates.p7m', 'wb') as f:
487- f.write(b'\n'.join(lines))
488 out, _ = util.subp(
489- 'openssl cms -decrypt -in Certificates.p7m -inkey'
490+ 'openssl cms -decrypt -in /dev/stdin -inkey'
491 ' {private_key} -recip {certificate} | openssl pkcs12 -nodes'
492 ' -password pass:'.format(**self.certificate_names),
493- shell=True)
494- private_keys, certificates = [], []
495+ shell=True, data=b'\n'.join(lines))
496+ return out
497+
498+ def parse_certificates(self, certificates_xml):
499+ """Given the Certificates XML document, return a dictionary of
500+ fingerprints and associated SSH keys derived from the certs."""
501+ out = self._decrypt_certs_from_xml(certificates_xml)
502 current = []
503+ keys = {}
504 for line in out.splitlines():
505 current.append(line)
506 if re.match(r'[-]+END .*?KEY[-]+$', line):
507- private_keys.append('\n'.join(current))
508+ # ignore private_keys
509 current = []
510 elif re.match(r'[-]+END .*?CERTIFICATE[-]+$', line):
511- certificates.append('\n'.join(current))
512+ certificate = '\n'.join(current)
513+ ssh_key = self._get_ssh_key_from_cert(certificate)
514+ fingerprint = self._get_fingerprint_from_cert(certificate)
515+ keys[fingerprint] = ssh_key
516 current = []
517- keys = []
518- for certificate in certificates:
519- with cd(self.tmpdir):
520- public_key, _ = util.subp(
521- 'openssl x509 -noout -pubkey |'
522- 'ssh-keygen -i -m PKCS8 -f /dev/stdin',
523- data=certificate,
524- shell=True)
525- keys.append(public_key)
526 return keys
527
528
529@@ -206,7 +231,6 @@ class WALinuxAgentShim(object):
530 self.dhcpoptions = dhcp_options
531 self._endpoint = None
532 self.openssl_manager = None
533- self.values = {}
534 self.lease_file = fallback_lease_file
535
536 def clean_up(self):
537@@ -328,8 +352,9 @@ class WALinuxAgentShim(object):
538 LOG.debug('Azure endpoint found at %s', endpoint_ip_address)
539 return endpoint_ip_address
540
541- def register_with_azure_and_fetch_data(self):
542- self.openssl_manager = OpenSSLManager()
543+ def register_with_azure_and_fetch_data(self, pubkey_info=None):
544+ if self.openssl_manager is None:
545+ self.openssl_manager = OpenSSLManager()
546 http_client = AzureEndpointHttpClient(self.openssl_manager.certificate)
547 LOG.info('Registering with Azure...')
548 attempts = 0
549@@ -347,16 +372,37 @@ class WALinuxAgentShim(object):
550 attempts += 1
551 LOG.debug('Successfully fetched GoalState XML.')
552 goal_state = GoalState(response.contents, http_client)
553- public_keys = []
554- if goal_state.certificates_xml is not None:
555+ ssh_keys = []
556+ if goal_state.certificates_xml is not None and pubkey_info is not None:
557 LOG.debug('Certificate XML found; parsing out public keys.')
558- public_keys = self.openssl_manager.parse_certificates(
559+ keys_by_fingerprint = self.openssl_manager.parse_certificates(
560 goal_state.certificates_xml)
561- data = {
562- 'public-keys': public_keys,
563- }
564+ ssh_keys = self._filter_pubkeys(keys_by_fingerprint, pubkey_info)
565 self._report_ready(goal_state, http_client)
566- return data
567+ return {'public-keys': ssh_keys}
568+
569+ def _filter_pubkeys(self, keys_by_fingerprint, pubkey_info):
570+ """cloud-init expects a straightforward array of keys to be dropped
571+ into the user's authorized_keys file. Azure control plane exposes
572+ multiple public keys to the VM via wireserver. Select just the
573+ user's key(s) and return them, ignoring any other certs.
574+ """
575+ keys = []
576+ for pubkey in pubkey_info:
577+ if 'value' in pubkey and pubkey['value']:
578+ keys.append(pubkey['value'])
579+ elif 'fingerprint' in pubkey and pubkey['fingerprint']:
580+ fingerprint = pubkey['fingerprint']
581+ if fingerprint in keys_by_fingerprint:
582+ keys.append(keys_by_fingerprint[fingerprint])
583+ else:
584+ LOG.warning("ovf-env.xml specified PublicKey fingerprint "
585+ "%s not found in goalstate XML", fingerprint)
586+ else:
587+ LOG.warning("ovf-env.xml specified PublicKey with neither "
588+ "value nor fingerprint: %s", pubkey)
589+
590+ return keys
591
592 def _report_ready(self, goal_state, http_client):
593 LOG.debug('Reporting ready to Azure fabric.')
594@@ -373,11 +419,12 @@ class WALinuxAgentShim(object):
595 LOG.info('Reported ready to Azure fabric.')
596
597
598-def get_metadata_from_fabric(fallback_lease_file=None, dhcp_opts=None):
599+def get_metadata_from_fabric(fallback_lease_file=None, dhcp_opts=None,
600+ pubkey_info=None):
601 shim = WALinuxAgentShim(fallback_lease_file=fallback_lease_file,
602 dhcp_options=dhcp_opts)
603 try:
604- return shim.register_with_azure_and_fetch_data()
605+ return shim.register_with_azure_and_fetch_data(pubkey_info=pubkey_info)
606 finally:
607 shim.clean_up()
608
609diff --git a/cloudinit/sources/helpers/openstack.py b/cloudinit/sources/helpers/openstack.py
610index 9c29cea..8f06911 100644
611--- a/cloudinit/sources/helpers/openstack.py
612+++ b/cloudinit/sources/helpers/openstack.py
613@@ -67,7 +67,7 @@ OS_VERSIONS = (
614 OS_ROCKY,
615 )
616
617-PHYSICAL_TYPES = (
618+KNOWN_PHYSICAL_TYPES = (
619 None,
620 'bgpovs', # not present in OpenStack upstream but used on OVH cloud.
621 'bridge',
622@@ -600,9 +600,7 @@ def convert_net_json(network_json=None, known_macs=None):
623 subnet['ipv6'] = True
624 subnets.append(subnet)
625 cfg.update({'subnets': subnets})
626- if link['type'] in PHYSICAL_TYPES:
627- cfg.update({'type': 'physical', 'mac_address': link_mac_addr})
628- elif link['type'] in ['bond']:
629+ if link['type'] in ['bond']:
630 params = {}
631 if link_mac_addr:
632 params['mac_address'] = link_mac_addr
633@@ -641,8 +639,10 @@ def convert_net_json(network_json=None, known_macs=None):
634 curinfo.update({'mac': link['vlan_mac_address'],
635 'name': name})
636 else:
637- raise ValueError(
638- 'Unknown network_data link type: %s' % link['type'])
639+ if link['type'] not in KNOWN_PHYSICAL_TYPES:
640+ LOG.warning('Unknown network_data link type (%s); treating as'
641+ ' physical', link['type'])
642+ cfg.update({'type': 'physical', 'mac_address': link_mac_addr})
643
644 config.append(cfg)
645 link_id_info[curinfo['id']] = curinfo
646diff --git a/cloudinit/stages.py b/cloudinit/stages.py
647index 8a06412..da7d349 100644
648--- a/cloudinit/stages.py
649+++ b/cloudinit/stages.py
650@@ -548,11 +548,11 @@ class Init(object):
651 with events.ReportEventStack("consume-user-data",
652 "reading and applying user-data",
653 parent=self.reporter):
654- self._consume_userdata(frequency)
655+ self._consume_userdata(frequency)
656 with events.ReportEventStack("consume-vendor-data",
657 "reading and applying vendor-data",
658 parent=self.reporter):
659- self._consume_vendordata(frequency)
660+ self._consume_vendordata(frequency)
661
662 # Perform post-consumption adjustments so that
663 # modules that run during the init stage reflect
664diff --git a/cloudinit/tests/helpers.py b/cloudinit/tests/helpers.py
665index 2eb7b0c..f41180f 100644
666--- a/cloudinit/tests/helpers.py
667+++ b/cloudinit/tests/helpers.py
668@@ -41,26 +41,6 @@ _real_subp = util.subp
669 SkipTest = unittest2.SkipTest
670 skipIf = unittest2.skipIf
671
672-# Used for detecting different python versions
673-PY2 = False
674-PY26 = False
675-PY27 = False
676-PY3 = False
677-
678-_PY_VER = sys.version_info
679-_PY_MAJOR, _PY_MINOR, _PY_MICRO = _PY_VER[0:3]
680-if (_PY_MAJOR, _PY_MINOR) <= (2, 6):
681- if (_PY_MAJOR, _PY_MINOR) == (2, 6):
682- PY26 = True
683- if (_PY_MAJOR, _PY_MINOR) >= (2, 0):
684- PY2 = True
685-else:
686- if (_PY_MAJOR, _PY_MINOR) == (2, 7):
687- PY27 = True
688- PY2 = True
689- if (_PY_MAJOR, _PY_MINOR) >= (3, 0):
690- PY3 = True
691-
692
693 # Makes the old path start
694 # with new base instead of whatever
695@@ -207,6 +187,7 @@ class CiTestCase(TestCase):
696 if self.with_logs:
697 # Remove the handler we setup
698 logging.getLogger().handlers = self.old_handlers
699+ logging.getLogger().level = None
700 util.subp = _real_subp
701 super(CiTestCase, self).tearDown()
702
703@@ -356,7 +337,7 @@ class FilesystemMockingTestCase(ResourceUsingTestCase):
704
705 def patchOpen(self, new_root):
706 trap_func = retarget_many_wrapper(new_root, 1, open)
707- name = 'builtins.open' if PY3 else '__builtin__.open'
708+ name = 'builtins.open' if six.PY3 else '__builtin__.open'
709 self.patched_funcs.enter_context(mock.patch(name, trap_func))
710
711 def patchStdoutAndStderr(self, stdout=None, stderr=None):
712diff --git a/cloudinit/tests/test_netinfo.py b/cloudinit/tests/test_netinfo.py
713index d76e768..1c8a791 100644
714--- a/cloudinit/tests/test_netinfo.py
715+++ b/cloudinit/tests/test_netinfo.py
716@@ -11,6 +11,7 @@ from cloudinit.tests.helpers import CiTestCase, mock, readResource
717 # Example ifconfig and route output
718 SAMPLE_OLD_IFCONFIG_OUT = readResource("netinfo/old-ifconfig-output")
719 SAMPLE_NEW_IFCONFIG_OUT = readResource("netinfo/new-ifconfig-output")
720+SAMPLE_FREEBSD_IFCONFIG_OUT = readResource("netinfo/freebsd-ifconfig-output")
721 SAMPLE_IPADDRSHOW_OUT = readResource("netinfo/sample-ipaddrshow-output")
722 SAMPLE_ROUTE_OUT_V4 = readResource("netinfo/sample-route-output-v4")
723 SAMPLE_ROUTE_OUT_V6 = readResource("netinfo/sample-route-output-v6")
724@@ -18,6 +19,7 @@ SAMPLE_IPROUTE_OUT_V4 = readResource("netinfo/sample-iproute-output-v4")
725 SAMPLE_IPROUTE_OUT_V6 = readResource("netinfo/sample-iproute-output-v6")
726 NETDEV_FORMATTED_OUT = readResource("netinfo/netdev-formatted-output")
727 ROUTE_FORMATTED_OUT = readResource("netinfo/route-formatted-output")
728+FREEBSD_NETDEV_OUT = readResource("netinfo/freebsd-netdev-formatted-output")
729
730
731 class TestNetInfo(CiTestCase):
732@@ -45,6 +47,18 @@ class TestNetInfo(CiTestCase):
733
734 @mock.patch('cloudinit.netinfo.util.which')
735 @mock.patch('cloudinit.netinfo.util.subp')
736+ def test_netdev_freebsd_nettools_pformat(self, m_subp, m_which):
737+ """netdev_pformat properly rendering netdev new nettools info."""
738+ m_subp.return_value = (SAMPLE_FREEBSD_IFCONFIG_OUT, '')
739+ m_which.side_effect = lambda x: x if x == 'ifconfig' else None
740+ content = netdev_pformat()
741+ print()
742+ print(content)
743+ print()
744+ self.assertEqual(FREEBSD_NETDEV_OUT, content)
745+
746+ @mock.patch('cloudinit.netinfo.util.which')
747+ @mock.patch('cloudinit.netinfo.util.subp')
748 def test_netdev_iproute_pformat(self, m_subp, m_which):
749 """netdev_pformat properly rendering ip route info."""
750 m_subp.return_value = (SAMPLE_IPADDRSHOW_OUT, '')
751diff --git a/cloudinit/url_helper.py b/cloudinit/url_helper.py
752index 396d69a..0af0d9e 100644
753--- a/cloudinit/url_helper.py
754+++ b/cloudinit/url_helper.py
755@@ -521,7 +521,7 @@ class OauthUrlHelper(object):
756 if extra_exception_cb:
757 ret = extra_exception_cb(msg, exception)
758 finally:
759- self.exception_cb(msg, exception)
760+ self.exception_cb(msg, exception)
761 return ret
762
763 def _headers_cb(self, extra_headers_cb, url):
764diff --git a/cloudinit/util.py b/cloudinit/util.py
765index a8a232b..a192091 100644
766--- a/cloudinit/util.py
767+++ b/cloudinit/util.py
768@@ -51,11 +51,6 @@ from cloudinit import version
769
770 from cloudinit.settings import (CFG_BUILTIN)
771
772-try:
773- string_types = (basestring,)
774-except NameError:
775- string_types = (str,)
776-
777 _DNS_REDIRECT_IP = None
778 LOG = logging.getLogger(__name__)
779
780@@ -77,7 +72,6 @@ CONTAINER_TESTS = (['systemd-detect-virt', '--quiet', '--container'],
781 PROC_CMDLINE = None
782
783 _LSB_RELEASE = {}
784-PY26 = sys.version_info[0:2] == (2, 6)
785
786
787 def get_architecture(target=None):
788@@ -125,7 +119,7 @@ def target_path(target, path=None):
789 # return 'path' inside target, accepting target as None
790 if target in (None, ""):
791 target = "/"
792- elif not isinstance(target, string_types):
793+ elif not isinstance(target, six.string_types):
794 raise ValueError("Unexpected input for target: %s" % target)
795 else:
796 target = os.path.abspath(target)
797@@ -1596,14 +1590,17 @@ def json_dumps(data):
798 separators=(',', ': '), default=json_serialize_default)
799
800
801-def yaml_dumps(obj, explicit_start=True, explicit_end=True):
802+def yaml_dumps(obj, explicit_start=True, explicit_end=True, noalias=False):
803 """Return data in nicely formatted yaml."""
804- return yaml.safe_dump(obj,
805- line_break="\n",
806- indent=4,
807- explicit_start=explicit_start,
808- explicit_end=explicit_end,
809- default_flow_style=False)
810+
811+ return yaml.dump(obj,
812+ line_break="\n",
813+ indent=4,
814+ explicit_start=explicit_start,
815+ explicit_end=explicit_end,
816+ default_flow_style=False,
817+ Dumper=(safeyaml.NoAliasSafeDumper
818+ if noalias else yaml.dumper.Dumper))
819
820
821 def ensure_dir(path, mode=None):
822@@ -2817,9 +2814,6 @@ def load_shell_content(content, add_empty=False, empty_val=None):
823 variables. Set their value to empty_val."""
824
825 def _shlex_split(blob):
826- if PY26 and isinstance(blob, six.text_type):
827- # Older versions don't support unicode input
828- blob = blob.encode("utf8")
829 return shlex.split(blob, comments=True)
830
831 data = {}
832diff --git a/debian/changelog b/debian/changelog
833index a62dba2..37fffe3 100644
834--- a/debian/changelog
835+++ b/debian/changelog
836@@ -1,3 +1,43 @@
837+cloud-init (18.5-45-g3554ffe8-0ubuntu1~16.04.1) xenial; urgency=medium
838+
839+ * New upstream snapshot. (LP: #1819067)
840+ - cloud-init-per: POSIX sh does not support string subst, use sed
841+ - Support locking user with usermod if passwd is not available.
842+ [Scott Moser]
843+ - Example for Microsoft Azure data disk added. [Anton Olifir]
844+ - clean: correctly determine the path for excluding seed directory
845+ - helpers/openstack: Treat unknown link types as physical
846+ - drop Python 2.6 support and our NIH version detection
847+ - tip-pylint: Fix assignment-from-return-none errors
848+ - net: append type:dhcp[46] only if dhcp[46] is True in v2 netconfig
849+ [Kurt Stieger]
850+ - cc_apt_pipelining: stop disabling pipelining by default
851+ - tests: fix some slow tests and some leaking state
852+ - util: don't determine string_types ourselves
853+ - cc_rsyslog: Escape possible nested set
854+ - Enable encrypted_data_bag_secret support for Chef [Eric Williams]
855+ - azure: Filter list of ssh keys pulled from fabric [Jason Zions (MSFT)]
856+ - doc: update merging doc with fixes and some additional details/examples
857+ - tests: integration test failure summary to use traceback if empty error
858+ - This is to fix https://bugs.launchpad.net/cloud-init/+bug/1812676
859+ [Vitaly Kuznetsov]
860+ - EC2: Rewrite network config on AWS Classic instances every boot
861+ [Guilherme G. Piccoli]
862+ - netinfo: Adjust ifconfig output parsing for FreeBSD ipv6 entries
863+ - netplan: Don't render yaml aliases when dumping netplan
864+ - add PyCharm IDE .idea/ path to .gitignore [Dominic Schlegel]
865+ - correct grammar issue in instance metadata documentation
866+ [Dominic Schlegel]
867+ - clean: cloud-init clean should not trace when run from within cloud_dir
868+ - Resolve flake8 comparison and pycodestyle over-ident issues
869+ [Paride Legovini]
870+ * Change Maintainer to Ubuntu Developers
871+ * d/postinst: remove now-incorrect apt pipelining configuration
872+ * d/patches/ec2-classic-dont-reapply-networking.patch: don't needlessly
873+ reapply networking configuration on every boot for EC2 classic instances
874+
875+ -- Daniel Watkins <oddbloke@ubuntu.com> Mon, 11 Mar 2019 17:09:59 -0400
876+
877 cloud-init (18.5-21-g8ee294d5-0ubuntu1~16.04.1) xenial; urgency=medium
878
879 * New upstream snapshot. (LP: #1813346)
880diff --git a/debian/cloud-init.postinst b/debian/cloud-init.postinst
881index 420420b..f16341e 100644
882--- a/debian/cloud-init.postinst
883+++ b/debian/cloud-init.postinst
884@@ -294,20 +294,6 @@ datasource_list: [ $values ]
885 EOF
886 fi
887
888- # we want to affect apt_pipelining on install, not wait for
889- # cloud-init to run it on next boot.
890- pipeline_f="/etc/apt/apt.conf.d/90cloud-init-pipelining"
891- if [ -f /var/lib/cloud/instance/obj.pkl ]; then
892- cloud-init single --name apt-pipelining --frequency once >/dev/null 2>&1 ||
893- echo "Warning: failed to setup apt-pipelining" 1>&2
894- elif [ ! -f "$pipeline_f" ]; then
895- # there was no cloud available, so populate it ourselves.
896- cat > "$pipeline_f" <<EOF
897-//Written by cloud-init per 'apt_pipelining'
898-Acquire::http::Pipeline-Depth "0";
899-EOF
900- fi
901-
902 # if there are maas settings pre-seeded apply them
903 handle_preseed_maas
904
905diff --git a/debian/control b/debian/control
906index 1de4f2f..17536e0 100644
907--- a/debian/control
908+++ b/debian/control
909@@ -1,7 +1,7 @@
910 Source: cloud-init
911 Section: admin
912 Priority: extra
913-Maintainer: Scott Moser <smoser@ubuntu.com>
914+Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
915 Build-Depends: debhelper (>= 9),
916 dh-python,
917 dh-systemd,
918diff --git a/debian/patches/ec2-classic-dont-reapply-networking.patch b/debian/patches/ec2-classic-dont-reapply-networking.patch
919new file mode 100644
920index 0000000..f5342f3
921--- /dev/null
922+++ b/debian/patches/ec2-classic-dont-reapply-networking.patch
923@@ -0,0 +1,25 @@
924+Description: don't configure networking on every boot for EC2 Classic instances
925+ The (ifupdown) networking configuration generated in xenial doesn't hardcode a
926+ MAC address, so xenial EC2 Classic instances don't need to apply networking
927+ configuration every boot to fix LP: #1802073
928+Author: Daniel Watkins <oddbloke@ubuntu.com>
929+Forwarded: not-needed
930+Last-Update: 2019-03-08
931+---
932+This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
933+--- a/cloudinit/sources/DataSourceEc2.py
934++++ b/cloudinit/sources/DataSourceEc2.py
935+@@ -334,13 +334,6 @@
936+ if isinstance(net_md, dict):
937+ result = convert_ec2_metadata_network_config(
938+ net_md, macs_to_nics=macs_to_nics, fallback_nic=iface)
939+- # RELEASE_BLOCKER: Xenial debian/postinst needs to add
940+- # EventType.BOOT on upgrade path for classic.
941+-
942+- # Non-VPC (aka Classic) Ec2 instances need to rewrite the
943+- # network config file every boot due to MAC address change.
944+- if self.is_classic_instance():
945+- self.update_events['network'].add(EventType.BOOT)
946+ else:
947+ LOG.warning("Metadata 'network' key not valid: %s.", net_md)
948+ self._network_config = result
949diff --git a/debian/patches/series b/debian/patches/series
950index 166a2d8..d37ae8a 100644
951--- a/debian/patches/series
952+++ b/debian/patches/series
953@@ -3,3 +3,4 @@ ds-identify-behavior-xenial.patch
954 stable-release-no-jsonschema-dep.patch
955 openstack-no-network-config.patch
956 azure-apply-network-config-false.patch
957+ec2-classic-dont-reapply-networking.patch
958diff --git a/doc/examples/cloud-config-chef.txt b/doc/examples/cloud-config-chef.txt
959index defc5a5..2320e01 100644
960--- a/doc/examples/cloud-config-chef.txt
961+++ b/doc/examples/cloud-config-chef.txt
962@@ -98,6 +98,9 @@ chef:
963 # to the install script
964 omnibus_version: "12.3.0"
965
966+ # If encrypted data bags are used, the client needs to have a secrets file
967+ # configured to decrypt them
968+ encrypted_data_bag_secret: "/etc/chef/encrypted_data_bag_secret"
969
970 # Capture all subprocess output into a logfile
971 # Useful for troubleshooting cloud-init issues
972diff --git a/doc/examples/cloud-config-disk-setup.txt b/doc/examples/cloud-config-disk-setup.txt
973index 43a62a2..89d9ff5 100644
974--- a/doc/examples/cloud-config-disk-setup.txt
975+++ b/doc/examples/cloud-config-disk-setup.txt
976@@ -17,7 +17,7 @@ fs_setup:
977 device: ephemeral0
978 partition: auto
979
980-# Default disk definitions for Windows Azure
981+# Default disk definitions for Microsoft Azure
982 # ------------------------------------------
983
984 device_aliases: {'ephemeral0': '/dev/sdb'}
985@@ -34,6 +34,21 @@ fs_setup:
986 replace_fs: ntfs
987
988
989+# Data disks definitions for Microsoft Azure
990+# ------------------------------------------
991+
992+disk_setup:
993+ /dev/disk/azure/scsi1/lun0:
994+ table_type: gpt
995+ layout: True
996+ overwrite: True
997+
998+fs_setup:
999+ - device: /dev/disk/azure/scsi1/lun0
1000+ partition: 1
1001+ filesystem: ext4
1002+
1003+
1004 # Default disk definitions for SmartOS
1005 # ------------------------------------
1006
1007@@ -242,7 +257,7 @@ fs_setup:
1008 #
1009 # "false": If an existing file system exists, skip the creation.
1010 #
1011-# <REPLACE_FS>: This is a special directive, used for Windows Azure that
1012+# <REPLACE_FS>: This is a special directive, used for Microsoft Azure that
1013 # instructs cloud-init to replace a file system of <FS_TYPE>. NOTE:
1014 # unless you define a label, this requires the use of the 'any' partition
1015 # directive.
1016diff --git a/doc/rtd/topics/datasources/ec2.rst b/doc/rtd/topics/datasources/ec2.rst
1017index 64c325d..76beca9 100644
1018--- a/doc/rtd/topics/datasources/ec2.rst
1019+++ b/doc/rtd/topics/datasources/ec2.rst
1020@@ -90,4 +90,15 @@ An example configuration with the default values is provided below:
1021 max_wait: 120
1022 timeout: 50
1023
1024+Notes
1025+-----
1026+ * There are 2 types of EC2 instances network-wise: VPC ones (Virtual Private
1027+ Cloud) and Classic ones (also known as non-VPC). One major difference
1028+ between them is that Classic instances have their MAC address changed on
1029+ stop/restart operations, so cloud-init will recreate the network config
1030+ file for EC2 Classic instances every boot. On VPC instances this file is
1031+ generated only in the first boot of the instance.
1032+ The check for the instance type is performed by is_classic_instance()
1033+ method.
1034+
1035 .. vi: textwidth=78
1036diff --git a/doc/rtd/topics/instancedata.rst b/doc/rtd/topics/instancedata.rst
1037index 5d2dc94..231a008 100644
1038--- a/doc/rtd/topics/instancedata.rst
1039+++ b/doc/rtd/topics/instancedata.rst
1040@@ -4,7 +4,7 @@
1041 Instance Metadata
1042 *****************
1043
1044-What is a instance data?
1045+What is instance data?
1046 ========================
1047
1048 Instance data is the collection of all configuration data that cloud-init
1049diff --git a/doc/rtd/topics/merging.rst b/doc/rtd/topics/merging.rst
1050index c75ca59..5f7ca18 100644
1051--- a/doc/rtd/topics/merging.rst
1052+++ b/doc/rtd/topics/merging.rst
1053@@ -21,12 +21,12 @@ For example.
1054 .. code-block:: yaml
1055
1056 #cloud-config (1)
1057- run_cmd:
1058+ runcmd:
1059 - bash1
1060 - bash2
1061
1062 #cloud-config (2)
1063- run_cmd:
1064+ runcmd:
1065 - bash3
1066 - bash4
1067
1068@@ -36,7 +36,7 @@ cloud-config object that contains the following.
1069 .. code-block:: yaml
1070
1071 #cloud-config (merged)
1072- run_cmd:
1073+ runcmd:
1074 - bash3
1075 - bash4
1076
1077@@ -45,7 +45,7 @@ Typically this is not what users want; instead they would likely prefer:
1078 .. code-block:: yaml
1079
1080 #cloud-config (merged)
1081- run_cmd:
1082+ runcmd:
1083 - bash1
1084 - bash2
1085 - bash3
1086@@ -55,6 +55,45 @@ This way makes it easier to combine the various cloud-config objects you have
1087 into a more useful list, thus reducing duplication necessary to accomplish the
1088 same result with the previous method.
1089
1090+
1091+Built-in Mergers
1092+================
1093+
1094+Cloud-init provides merging for the following built-in types:
1095+
1096+- Dict
1097+- List
1098+- String
1099+
1100+The ``Dict`` merger has the following options which control what is done with
1101+values contained within the config.
1102+
1103+- ``allow_delete``: Existing values not present in the new value can be deleted, defaults to False
1104+- ``no_replace``: Do not replace an existing value if one is already present, enabled by default.
1105+- ``replace``: Overwrite existing values with new ones.
1106+
1107+The ``List`` merger has the following options which control what is done with
1108+the values contained within the config.
1109+
1110+- ``append``: Add new value to the end of the list, defaults to False.
1111+- ``prepend``: Add new values to the start of the list, defaults to False.
1112+- ``no_replace``: Do not replace an existing value if one is already present, enabled by default.
1113+- ``replace``: Overwrite existing values with new ones.
1114+
1115+The ``Str`` merger has the following options which control what is done with
1116+the values contained within the config.
1117+
1118+- ``append``: Add new value to the end of the string, defaults to False.
1119+
1120+Common options for all merge types which control how recursive merging is
1121+done on other types.
1122+
1123+- ``recurse_dict``: If True merge the new values of the dictionary, defaults to True.
1124+- ``recurse_list``: If True merge the new values of the list, defaults to False.
1125+- ``recurse_array``: Alias for ``recurse_list``.
1126+- ``recurse_str``: If True merge the new values of the string, defaults to False.
1127+
1128+
1129 Customizability
1130 ===============
1131
1132@@ -164,8 +203,8 @@ string format (i.e. the second option above), for example:
1133
1134 .. code-block:: python
1135
1136- {'merge_how': [{'name': 'list', 'settings': ['extend']},
1137- {'name': 'dict', 'settings': []},
1138+ {'merge_how': [{'name': 'list', 'settings': ['append']},
1139+ {'name': 'dict', 'settings': ['no_replace', 'recurse_list']},
1140 {'name': 'str', 'settings': ['append']}]}
1141
1142 This would be the equivalent format for default string format but in dictionary
1143@@ -201,4 +240,43 @@ Note, however, that merge algorithms are not used *across* types of
1144 configuration. As was the case before merging was implemented,
1145 user-data will overwrite conf.d configuration without merging.
1146
1147+Example cloud-config
1148+====================
1149+
1150+A common request is to include multiple ``runcmd`` directives in different
1151+files and merge all of the commands together. To achieve this, we must modify
1152+the default merging to allow for dictionaries to join list values.
1153+
1154+
1155+The first config
1156+
1157+.. code-block:: yaml
1158+
1159+ #cloud-config
1160+ merge_how:
1161+ - name: list
1162+ settings: [append]
1163+ - name: dict
1164+ settings: [no_replace, recurse_list]
1165+
1166+ runcmd:
1167+ - bash1
1168+ - bash2
1169+
1170+The second config
1171+
1172+.. code-block:: yaml
1173+
1174+ #cloud-config
1175+ merge_how:
1176+ - name: list
1177+ settings: [append]
1178+ - name: dict
1179+ settings: [no_replace, recurse_list]
1180+
1181+ runcmd:
1182+ - bash3
1183+ - bash4
1184+
1185+
1186 .. vi: textwidth=78
1187diff --git a/templates/chef_client.rb.tmpl b/templates/chef_client.rb.tmpl
1188index cbb6b15..99978d3 100644
1189--- a/templates/chef_client.rb.tmpl
1190+++ b/templates/chef_client.rb.tmpl
1191@@ -1,6 +1,6 @@
1192 ## template:jinja
1193 {#
1194-This file is only utilized if the module 'cc_chef' is enabled in
1195+This file is only utilized if the module 'cc_chef' is enabled in
1196 cloud-config. Specifically, in order to enable it
1197 you need to add the following to config:
1198 chef:
1199@@ -56,3 +56,6 @@ pid_file "{{pid_file}}"
1200 {% if show_time %}
1201 Chef::Log::Formatter.show_time = true
1202 {% endif %}
1203+{% if encrypted_data_bag_secret %}
1204+encrypted_data_bag_secret "{{encrypted_data_bag_secret}}"
1205+{% endif %}
1206diff --git a/tests/cloud_tests/verify.py b/tests/cloud_tests/verify.py
1207index 9911ecf..7018f4d 100644
1208--- a/tests/cloud_tests/verify.py
1209+++ b/tests/cloud_tests/verify.py
1210@@ -61,12 +61,17 @@ def format_test_failures(test_result):
1211 if not test_result['failures']:
1212 return ''
1213 failure_hdr = ' test failures:'
1214- failure_fmt = ' * {module}.{class}.{function}\n {error}'
1215+ failure_fmt = ' * {module}.{class}.{function}\n '
1216 output = []
1217 for failure in test_result['failures']:
1218 if not output:
1219 output = [failure_hdr]
1220- output.append(failure_fmt.format(**failure))
1221+ msg = failure_fmt.format(**failure)
1222+ if failure.get('error'):
1223+ msg += failure['error']
1224+ else:
1225+ msg += failure.get('traceback', '')
1226+ output.append(msg)
1227 return '\n'.join(output)
1228
1229
1230diff --git a/tests/data/azure/parse_certificates_fingerprints b/tests/data/azure/parse_certificates_fingerprints
1231new file mode 100644
1232index 0000000..f7293c5
1233--- /dev/null
1234+++ b/tests/data/azure/parse_certificates_fingerprints
1235@@ -0,0 +1,4 @@
1236+ECEDEB3B8488D31AF3BC4CCED493F64B7D27D7B1
1237+073E19D14D1C799224C6A0FD8DDAB6A8BF27D473
1238+4C16E7FAD6297D74A9B25EB8F0A12808CEBE293E
1239+929130695289B450FE45DCD5F6EF0CDE69865867
1240diff --git a/tests/data/azure/parse_certificates_pem b/tests/data/azure/parse_certificates_pem
1241new file mode 100644
1242index 0000000..3521ea3
1243--- /dev/null
1244+++ b/tests/data/azure/parse_certificates_pem
1245@@ -0,0 +1,152 @@
1246+Bag Attributes
1247+ localKeyID: 01 00 00 00
1248+ Microsoft CSP Name: Microsoft Enhanced Cryptographic Provider v1.0
1249+Key Attributes
1250+ X509v3 Key Usage: 10
1251+-----BEGIN PRIVATE KEY-----
1252+MIIEwAIBADANBgkqhkiG9w0BAQEFAASCBKowggSmAgEAAoIBAQDlEe5fUqwdrQTP
1253+W2oVlGK2f31q/8ULT8KmOTyUvL0RPdJQ69vvHOc5Q2CKg2eviHC2LWhF8WmpnZj6
1254+61RL0GeFGizwvU8Moebw5p3oqdcgoGpHVtxf+mr4QcWF58/Fwez0dA4hcsimVNBz
1255+eNpBBUIKNBMTBG+4d6hcQBUAGKUdGRcCGEyTqXLU0MgHjxC9JgVqWJl+X2LcAGj5
1256+7J+tGYGTLzKJmeCeGVNN5ZtJ0T85MYHCKQk1/FElK+Kq5akovXffQHjlnCPcx0NJ
1257+47NBjlPaFp2gjnAChn79bT4iCjOFZ9avWpqRpeU517UCnY7djOr3fuod/MSQyh3L
1258+Wuem1tWBAgMBAAECggEBAM4ZXQRs6Kjmo95BHGiAEnSqrlgX+dycjcBq3QPh8KZT
1259+nifqnf48XhnackENy7tWIjr3DctoUq4mOp8AHt77ijhqfaa4XSg7fwKeK9NLBGC5
1260+lAXNtAey0o2894/sKrd+LMkgphoYIUnuI4LRaGV56potkj/ZDP/GwTcG/R4SDnTn
1261+C1Nb05PNTAPQtPZrgPo7TdM6gGsTnFbVrYHQLyg2Sq/osHfF15YohB01esRLCAwb
1262+EF8JkRC4hWIZoV7BsyQ39232zAJQGGla7+wKFs3kObwh3VnFkQpT94KZnNiZuEfG
1263+x5pW4Pn3gXgNsftscXsaNe/M9mYZqo//Qw7NvUIvAvECgYEA9AVveyK0HOA06fhh
1264++3hUWdvw7Pbrl+e06jO9+bT1RjQMbHKyI60DZyVGuAySN86iChJRoJr5c6xj+iXU
1265+cR6BVJDjGH5t1tyiK2aYf6hEpK9/j8Z54UiVQ486zPP0PGfT2TO4lBLK+8AUmoaH
1266+gk21ul8QeVCeCJa/o+xEoRFvzcUCgYEA8FCbbvInrUtNY+9eKaUYoNodsgBVjm5X
1267+I0YPUL9D4d+1nvupHSV2NVmQl0w1RaJwrNTafrl5LkqjhQbmuWNta6QgfZzSA3LB
1268+lWXo1Mm0azKdcD3qMGbvn0Q3zU+yGNEgmB/Yju3/NtgYRG6tc+FCWRbPbiCnZWT8
1269+v3C2Y0XggI0CgYEA2/jCZBgGkTkzue5kNVJlh5OS/aog+pCvL6hxCtarfBuTT3ed
1270+Sje+p46cz3DVpmUpATc+Si8py7KNdYQAm/BJ2be6X+woi9Xcgo87zWgcaPCjZzId
1271+0I2jsIE/Gl6XvpRCDrxnGWRPgt3GNP4szbPLrDPiH9oie8+Y9eYYf7G+PZkCgYEA
1272+nRSzZOPYV4f/QDF4pVQLMykfe/iH9B/fyWjEHg3He19VQmRReIHCMMEoqBziPXAe
1273+onpHj8oAkeer1wpZyhhZr6CKtFDLXgGm09bXSC/IRMHC81klORovyzU2HHfZfCtG
1274+WOmIDnU2+0xpIGIP8sztJ3qnf97MTJSkOSadsWo9gwkCgYEAh5AQmJQmck88Dff2
1275+qIfJIX8d+BDw47BFJ89OmMFjGV8TNB+JO+AV4Vkodg4hxKpLqTFZTTUFgoYfy5u1
1276+1/BhAjpmCDCrzubCFhx+8VEoM2+2+MmnuQoMAm9+/mD/IidwRaARgXgvEmp7sfdt
1277+RyWd+p2lYvFkC/jORQtDMY4uW1o=
1278+-----END PRIVATE KEY-----
1279+Bag Attributes
1280+ localKeyID: 02 00 00 00
1281+ Microsoft CSP Name: Microsoft Strong Cryptographic Provider
1282+Key Attributes
1283+ X509v3 Key Usage: 10
1284+-----BEGIN PRIVATE KEY-----
1285+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDlQhPrZwVQYFV4
1286+FBc0H1iTXYaznMpwZvEITKtXWACzTdguUderEVOkXW3HTi5HvC2rMayt0nqo3zcd
1287+x1eGiqdjpZQ/wMrkz9wNEM/nNMsXntEwxk0jCVNKB/jz6vf+BOtrSI01SritAGZW
1288+dpKoTUyztT8C2mA3X6D8g3m4Dd07ltnzxaDqAQIU5jBHh3f/Q14tlPNZWUIiqVTC
1289+gDxgAe7MDmfs9h3CInTBX1XM5J4UsLTL23/padgeSvP5YF5qr1+0c7Tdftxr2lwA
1290+N3rLkisf5EiLAToVyJJlgP/exo2I8DaIKe7DZzD3Y1CrurOpkcMKYu5kM1Htlbua
1291+tDkAa2oDAgMBAAECggEAOvdueS9DyiMlCKAeQb1IQosdQOh0l0ma+FgEABC2CWhd
1292+0LgjQTBRM6cGO+urcq7/jhdWQ1UuUG4tVn71z7itCi/F/Enhxc2C22d2GhFVpWsn
1293+giSXJYpZ/mIjkdVfWNo6FRuRmmHwMys1p0qTOS+8qUJWhSzW75csqJZGgeUrAI61
1294+LBV5F0SGR7dR2xZfy7PeDs9xpD0QivDt5DpsZWPaPvw4QlhdLgw6/YU1h9vtm6ci
1295+xLjnPRLZ7JMpcQHO8dUDl6FiEI7yQ11BDm253VQAVMddYRPQABn7SpEF8kD/aZVh
1296+2Clvz61Rz80SKjPUthMPLWMCRp7zB0xDMzt3/1i+tQKBgQD6Ar1/oD3eFnRnpi4u
1297+n/hdHJtMuXWNfUA4dspNjP6WGOid9sgIeUUdif1XyVJ+afITzvgpWc7nUWIqG2bQ
1298+WxJ/4q2rjUdvjNXTy1voVungR2jD5WLQ9DKeaTR0yCliWlx4JgdPG7qGI5MMwsr+
1299+R/PUoUUhGeEX+o/sCSieO3iUrQKBgQDqwBEMvIdhAv/CK2sG3fsKYX8rFT55ZNX3
1300+Tix9DbUGY3wQColNuI8U1nDlxE9U6VOfT9RPqKelBLCgbzB23kdEJnjSlnqlTxrx
1301+E+Hkndyf2ckdJAR3XNxoQ6SRLJNBsgoBj/z5tlfZE9/Jc+uh0mYy3e6g6XCVPBcz
1302+MgoIc+ofbwKBgQCGQhZ1hR30N+bHCozeaPW9OvGDIE0qcEqeh9xYDRFilXnF6pK9
1303+SjJ9jG7KR8jPLiHb1VebDSl5O1EV/6UU2vNyTc6pw7LLCryBgkGW4aWy1WZDXNnW
1304+EG1meGS9GghvUss5kmJ2bxOZmV0Mi0brisQ8OWagQf+JGvtS7BAt+Q3l+QKBgAb9
1305+8YQPmXiqPjPqVyW9Ntz4SnFeEJ5NApJ7IZgX8GxgSjGwHqbR+HEGchZl4ncE/Bii
1306+qBA3Vcb0fM5KgYcI19aPzsl28fA6ivLjRLcqfIfGVNcpW3iyq13vpdctHLW4N9QU
1307+FdTaOYOds+ysJziKq8CYG6NvUIshXw+HTgUybqbBAoGBAIIOqcmmtgOClAwipA17
1308+dAHsI9Sjk+J0+d4JU6o+5TsmhUfUKIjXf5+xqJkJcQZMEe5GhxcCuYkgFicvh4Hz
1309+kv2H/EU35LcJTqC6KTKZOWIbGcn1cqsvwm3GQJffYDiO8fRZSwCaif2J3F2lfH4Y
1310+R/fA67HXFSTT+OncdRpY1NOn
1311+-----END PRIVATE KEY-----
1312+Bag Attributes: <Empty Attributes>
1313+subject=/CN=CRP/OU=AzureRT/O=Microsoft Corporation/L=Redmond/ST=WA/C=US
1314+issuer=/CN=Root Agency
1315+-----BEGIN CERTIFICATE-----
1316+MIIB+TCCAeOgAwIBAgIBATANBgkqhkiG9w0BAQUFADAWMRQwEgYDVQQDDAtSb290
1317+IEFnZW5jeTAeFw0xOTAyMTUxOTA0MDRaFw0yOTAyMTUxOTE0MDRaMGwxDDAKBgNV
1318+BAMMA0NSUDEQMA4GA1UECwwHQXp1cmVSVDEeMBwGA1UECgwVTWljcm9zb2Z0IENv
1319+cnBvcmF0aW9uMRAwDgYDVQQHDAdSZWRtb25kMQswCQYDVQQIDAJXQTELMAkGA1UE
1320+BhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDIlPjJXzrRih4C
1321+k/XsoI01oqo7IUxH3dA2F7vHGXQoIpKCp8Qe6Z6cFfdD8Uj+s+B1BX6hngwzIwjN
1322+jE/23X3SALVzJVWzX4Y/IEjbgsuao6sOyNyB18wIU9YzZkVGj68fmMlUw3LnhPbe
1323+eWkufZaJCaLyhQOwlRMbOcn48D6Ys8fccOyXNzpq3rH1OzeQpxS2M8zaJYP4/VZ/
1324+sf6KRpI7bP+QwyFvNKfhcaO9/gj4kMo9lVGjvDU20FW6g8UVNJCV9N4GO6mOcyqo
1325+OhuhVfjCNGgW7N1qi0TIVn0/MQM4l4dcT2R7Z/bV9fhMJLjGsy5A4TLAdRrhKUHT
1326+bzi9HyDvAgMBAAEwDQYJKoZIhvcNAQEFBQADAQA=
1327+-----END CERTIFICATE-----
1328+Bag Attributes
1329+ localKeyID: 01 00 00 00
1330+subject=/C=US/ST=WASHINGTON/L=Seattle/O=Microsoft/OU=Azure/CN=AnhVo/emailAddress=redacted@microsoft.com
1331+issuer=/C=US/ST=WASHINGTON/L=Seattle/O=Microsoft/OU=Azure/CN=AnhVo/emailAddress=redacted@microsoft.com
1332+-----BEGIN CERTIFICATE-----
1333+MIID7TCCAtWgAwIBAgIJALQS3yMg3R41MA0GCSqGSIb3DQEBCwUAMIGMMQswCQYD
1334+VQQGEwJVUzETMBEGA1UECAwKV0FTSElOR1RPTjEQMA4GA1UEBwwHU2VhdHRsZTES
1335+MBAGA1UECgwJTWljcm9zb2Z0MQ4wDAYDVQQLDAVBenVyZTEOMAwGA1UEAwwFQW5o
1336+Vm8xIjAgBgkqhkiG9w0BCQEWE2FuaHZvQG1pY3Jvc29mdC5jb20wHhcNMTkwMjE0
1337+MjMxMjQwWhcNMjExMTEwMjMxMjQwWjCBjDELMAkGA1UEBhMCVVMxEzARBgNVBAgM
1338+CldBU0hJTkdUT04xEDAOBgNVBAcMB1NlYXR0bGUxEjAQBgNVBAoMCU1pY3Jvc29m
1339+dDEOMAwGA1UECwwFQXp1cmUxDjAMBgNVBAMMBUFuaFZvMSIwIAYJKoZIhvcNAQkB
1340+FhNhbmh2b0BtaWNyb3NvZnQuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
1341+CgKCAQEA5RHuX1KsHa0Ez1tqFZRitn99av/FC0/Cpjk8lLy9ET3SUOvb7xznOUNg
1342+ioNnr4hwti1oRfFpqZ2Y+utUS9BnhRos8L1PDKHm8Oad6KnXIKBqR1bcX/pq+EHF
1343+hefPxcHs9HQOIXLIplTQc3jaQQVCCjQTEwRvuHeoXEAVABilHRkXAhhMk6ly1NDI
1344+B48QvSYFaliZfl9i3ABo+eyfrRmBky8yiZngnhlTTeWbSdE/OTGBwikJNfxRJSvi
1345+quWpKL1330B45Zwj3MdDSeOzQY5T2hadoI5wAoZ+/W0+IgozhWfWr1qakaXlOde1
1346+Ap2O3Yzq937qHfzEkMody1rnptbVgQIDAQABo1AwTjAdBgNVHQ4EFgQUPvdgLiv3
1347+pAk4r0QTPZU3PFOZJvgwHwYDVR0jBBgwFoAUPvdgLiv3pAk4r0QTPZU3PFOZJvgw
1348+DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAVUHZT+h9+uCPLTEl5IDg
1349+kqd9WpzXA7PJd/V+7DeDDTkEd06FIKTWZLfxLVVDjQJnQqubQb//e0zGu1qKbXnX
1350+R7xqWabGU4eyPeUFWddmt1OHhxKLU3HbJNJJdL6XKiQtpGGUQt/mqNQ/DEr6hhNF
1351+im5I79iA8H/dXA2gyZrj5Rxea4mtsaYO0mfp1NrFtJpAh2Djy4B1lBXBIv4DWG9e
1352+mMEwzcLCOZj2cOMA6+mdLMUjYCvIRtnn5MKUHyZX5EmX79wsqMTvVpddlVLB9Kgz
1353+Qnvft9+SBWh9+F3ip7BsL6Q4Q9v8eHRbnP0ya7ddlgh64uwf9VOfZZdKCnwqudJP
1354+3g==
1355+-----END CERTIFICATE-----
1356+Bag Attributes
1357+ localKeyID: 02 00 00 00
1358+subject=/CN=/subscriptions/redacted/resourcegroups/redacted/providers/Microsoft.Compute/virtualMachines/redacted
1359+issuer=/CN=Microsoft.ManagedIdentity
1360+-----BEGIN CERTIFICATE-----
1361+MIIDnTCCAoWgAwIBAgIUB2lauSRccvFkoJybUfIwOUqBN7MwDQYJKoZIhvcNAQEL
1362+BQAwJDEiMCAGA1UEAxMZTWljcm9zb2Z0Lk1hbmFnZWRJZGVudGl0eTAeFw0xOTAy
1363+MTUxOTA5MDBaFw0xOTA4MTQxOTA5MDBaMIGUMYGRMIGOBgNVBAMTgYYvc3Vic2Ny
1364+aXB0aW9ucy8yN2I3NTBjZC1lZDQzLTQyZmQtOTA0NC04ZDc1ZTEyNGFlNTUvcmVz
1365+b3VyY2Vncm91cHMvYW5oZXh0cmFzc2gvcHJvdmlkZXJzL01pY3Jvc29mdC5Db21w
1366+dXRlL3ZpcnR1YWxNYWNoaW5lcy9hbmh0ZXN0Y2VydDCCASIwDQYJKoZIhvcNAQEB
1367+BQADggEPADCCAQoCggEBAOVCE+tnBVBgVXgUFzQfWJNdhrOcynBm8QhMq1dYALNN
1368+2C5R16sRU6RdbcdOLke8LasxrK3SeqjfNx3HV4aKp2OllD/AyuTP3A0Qz+c0yxee
1369+0TDGTSMJU0oH+PPq9/4E62tIjTVKuK0AZlZ2kqhNTLO1PwLaYDdfoPyDebgN3TuW
1370+2fPFoOoBAhTmMEeHd/9DXi2U81lZQiKpVMKAPGAB7swOZ+z2HcIidMFfVczknhSw
1371+tMvbf+lp2B5K8/lgXmqvX7RztN1+3GvaXAA3esuSKx/kSIsBOhXIkmWA/97GjYjw
1372+Nogp7sNnMPdjUKu6s6mRwwpi7mQzUe2Vu5q0OQBragMCAwEAAaNWMFQwDgYDVR0P
1373+AQH/BAQDAgeAMAwGA1UdEwEB/wQCMAAwEwYDVR0lBAwwCgYIKwYBBQUHAwIwHwYD
1374+VR0jBBgwFoAUOJvzEsriQWdJBndPrK+Me1bCPjYwDQYJKoZIhvcNAQELBQADggEB
1375+AFGP/g8o7Hv/to11M0UqfzJuW/AyH9RZtSRcNQFLZUndwweQ6fap8lFsA4REUdqe
1376+7Quqp5JNNY1XzKLWXMPoheIDH1A8FFXdsAroArzlNs9tO3TlIHE8A7HxEVZEmR4b
1377+7ZiixmkQPS2RkjEoV/GM6fheBrzuFn7X5kVZyE6cC5sfcebn8xhk3ZcXI0VmpdT0
1378+jFBsf5IvFCIXXLLhJI4KXc8VMoKFU1jT9na/jyaoGmfwovKj4ib8s2aiXGAp7Y38
1379+UCmY+bJapWom6Piy5Jzi/p/kzMVdJcSa+GqpuFxBoQYEVs2XYVl7cGu/wPM+NToC
1380+pkSoWwF1QAnHn0eokR9E1rU=
1381+-----END CERTIFICATE-----
1382+Bag Attributes: <Empty Attributes>
1383+subject=/CN=CRP/OU=AzureRT/O=Microsoft Corporation/L=Redmond/ST=WA/C=US
1384+issuer=/CN=Root Agency
1385+-----BEGIN CERTIFICATE-----
1386+MIIB+TCCAeOgAwIBAgIBATANBgkqhkiG9w0BAQUFADAWMRQwEgYDVQQDDAtSb290
1387+IEFnZW5jeTAeFw0xOTAyMTUxOTA0MDRaFw0yOTAyMTUxOTE0MDRaMGwxDDAKBgNV
1388+BAMMA0NSUDEQMA4GA1UECwwHQXp1cmVSVDEeMBwGA1UECgwVTWljcm9zb2Z0IENv
1389+cnBvcmF0aW9uMRAwDgYDVQQHDAdSZWRtb25kMQswCQYDVQQIDAJXQTELMAkGA1UE
1390+BhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDHU9IDclbKVYVb
1391+Yuv0+zViX+wTwlKspslmy/uf3hkWLh7pyzyrq70S7qtSW2EGixUPxZS/R8pOLHoi
1392+nlKF9ILgj0gVTCJsSwnWpXRg3rhZwIVoYMHN50BHS1SqVD0lsWNMXmo76LoJcjmW
1393+vwIznvj5C/gnhU+K7+c3m7AlCyU2wjwpBAEYj7PQs6l/wTqpEiaqC5NytNBd7qp+
1394+lYYysVrpa1PFL0Nj4MMZARIfjkiJtL9qDhy9YZeJRQ6q/Fhz0kjvkZnfxixfKF4y
1395+WzOfhBrAtpF6oOnuYKk3hxjh9KjTTX4/U8zdLojalX09iyHyEjwJKGlGEpzh1aY7
1396+t5btUyvpAgMBAAEwDQYJKoZIhvcNAQEFBQADAQA=
1397+-----END CERTIFICATE-----
1398diff --git a/tests/data/azure/pubkey_extract_cert b/tests/data/azure/pubkey_extract_cert
1399new file mode 100644
1400index 0000000..ce9b852
1401--- /dev/null
1402+++ b/tests/data/azure/pubkey_extract_cert
1403@@ -0,0 +1,13 @@
1404+-----BEGIN CERTIFICATE-----
1405+MIIB+TCCAeOgAwIBAgIBATANBgkqhkiG9w0BAQUFADAWMRQwEgYDVQQDDAtSb290
1406+IEFnZW5jeTAeFw0xOTAyMTUxOTA0MDRaFw0yOTAyMTUxOTE0MDRaMGwxDDAKBgNV
1407+BAMMA0NSUDEQMA4GA1UECwwHQXp1cmVSVDEeMBwGA1UECgwVTWljcm9zb2Z0IENv
1408+cnBvcmF0aW9uMRAwDgYDVQQHDAdSZWRtb25kMQswCQYDVQQIDAJXQTELMAkGA1UE
1409+BhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDHU9IDclbKVYVb
1410+Yuv0+zViX+wTwlKspslmy/uf3hkWLh7pyzyrq70S7qtSW2EGixUPxZS/R8pOLHoi
1411+nlKF9ILgj0gVTCJsSwnWpXRg3rhZwIVoYMHN50BHS1SqVD0lsWNMXmo76LoJcjmW
1412+vwIznvj5C/gnhU+K7+c3m7AlCyU2wjwpBAEYj7PQs6l/wTqpEiaqC5NytNBd7qp+
1413+lYYysVrpa1PFL0Nj4MMZARIfjkiJtL9qDhy9YZeJRQ6q/Fhz0kjvkZnfxixfKF4y
1414+WzOfhBrAtpF6oOnuYKk3hxjh9KjTTX4/U8zdLojalX09iyHyEjwJKGlGEpzh1aY7
1415+t5btUyvpAgMBAAEwDQYJKoZIhvcNAQEFBQADAQA=
1416+-----END CERTIFICATE-----
1417diff --git a/tests/data/azure/pubkey_extract_ssh_key b/tests/data/azure/pubkey_extract_ssh_key
1418new file mode 100644
1419index 0000000..54d749e
1420--- /dev/null
1421+++ b/tests/data/azure/pubkey_extract_ssh_key
1422@@ -0,0 +1 @@
1423+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHU9IDclbKVYVbYuv0+zViX+wTwlKspslmy/uf3hkWLh7pyzyrq70S7qtSW2EGixUPxZS/R8pOLHoinlKF9ILgj0gVTCJsSwnWpXRg3rhZwIVoYMHN50BHS1SqVD0lsWNMXmo76LoJcjmWvwIznvj5C/gnhU+K7+c3m7AlCyU2wjwpBAEYj7PQs6l/wTqpEiaqC5NytNBd7qp+lYYysVrpa1PFL0Nj4MMZARIfjkiJtL9qDhy9YZeJRQ6q/Fhz0kjvkZnfxixfKF4yWzOfhBrAtpF6oOnuYKk3hxjh9KjTTX4/U8zdLojalX09iyHyEjwJKGlGEpzh1aY7t5btUyvp
1424diff --git a/tests/data/netinfo/freebsd-ifconfig-output b/tests/data/netinfo/freebsd-ifconfig-output
1425new file mode 100644
1426index 0000000..3de15a5
1427--- /dev/null
1428+++ b/tests/data/netinfo/freebsd-ifconfig-output
1429@@ -0,0 +1,17 @@
1430+vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
1431+ options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
1432+ ether fa:16:3e:14:1f:99
1433+ hwaddr fa:16:3e:14:1f:99
1434+ inet 10.1.80.61 netmask 0xfffff000 broadcast 10.1.95.255
1435+ nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
1436+ media: Ethernet 10Gbase-T <full-duplex>
1437+ status: active
1438+pflog0: flags=0<> metric 0 mtu 33160
1439+pfsync0: flags=0<> metric 0 mtu 1500
1440+ syncpeer: 0.0.0.0 maxupd: 128 defer: off
1441+lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
1442+ options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
1443+ inet6 ::1 prefixlen 128
1444+ inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
1445+ inet 127.0.0.1 netmask 0xff000000
1446+ nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
1447diff --git a/tests/data/netinfo/freebsd-netdev-formatted-output b/tests/data/netinfo/freebsd-netdev-formatted-output
1448new file mode 100644
1449index 0000000..a9d2ac1
1450--- /dev/null
1451+++ b/tests/data/netinfo/freebsd-netdev-formatted-output
1452@@ -0,0 +1,11 @@
1453++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++
1454++---------+-------+----------------+------------+-------+-------------------+
1455+| Device | Up | Address | Mask | Scope | Hw-Address |
1456++---------+-------+----------------+------------+-------+-------------------+
1457+| lo0 | True | 127.0.0.1 | 0xff000000 | . | . |
1458+| lo0 | True | ::1/128 | . | . | . |
1459+| lo0 | True | fe80::1%lo0/64 | . | 0x4 | . |
1460+| pflog0 | False | . | . | . | . |
1461+| pfsync0 | False | . | . | . | . |
1462+| vtnet0 | True | 10.1.80.61 | 0xfffff000 | . | fa:16:3e:14:1f:99 |
1463++---------+-------+----------------+------------+-------+-------------------+
1464diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py
1465index 417d86a..6b05b8f 100644
1466--- a/tests/unittests/test_datasource/test_azure.py
1467+++ b/tests/unittests/test_datasource/test_azure.py
1468@@ -11,7 +11,7 @@ from cloudinit.util import (b64e, decode_binary, load_file, write_file,
1469 from cloudinit.version import version_string as vs
1470 from cloudinit.tests.helpers import (
1471 HttprettyTestCase, CiTestCase, populate_dir, mock, wrap_and_call,
1472- ExitStack, PY26, SkipTest)
1473+ ExitStack)
1474
1475 import crypt
1476 import httpretty
1477@@ -221,8 +221,6 @@ class TestAzureDataSource(CiTestCase):
1478
1479 def setUp(self):
1480 super(TestAzureDataSource, self).setUp()
1481- if PY26:
1482- raise SkipTest("Does not work on python 2.6")
1483 self.tmp = self.tmp_dir()
1484
1485 # patch cloud_dir, so our 'seed_dir' is guaranteed empty
1486@@ -1692,6 +1690,7 @@ class TestPreprovisioningPollIMDS(CiTestCase):
1487 self.paths = helpers.Paths({'cloud_dir': self.tmp})
1488 dsaz.BUILTIN_DS_CONFIG['data_dir'] = self.waagent_d
1489
1490+ @mock.patch('time.sleep', mock.MagicMock())
1491 @mock.patch(MOCKPATH + 'EphemeralDHCPv4')
1492 def test_poll_imds_re_dhcp_on_timeout(self, m_dhcpv4, report_ready_func,
1493 fake_resp, m_media_switch, m_dhcp,
1494diff --git a/tests/unittests/test_datasource/test_azure_helper.py b/tests/unittests/test_datasource/test_azure_helper.py
1495index 26b2b93..0255616 100644
1496--- a/tests/unittests/test_datasource/test_azure_helper.py
1497+++ b/tests/unittests/test_datasource/test_azure_helper.py
1498@@ -1,11 +1,13 @@
1499 # This file is part of cloud-init. See LICENSE file for license information.
1500
1501 import os
1502+import unittest2
1503 from textwrap import dedent
1504
1505 from cloudinit.sources.helpers import azure as azure_helper
1506 from cloudinit.tests.helpers import CiTestCase, ExitStack, mock, populate_dir
1507
1508+from cloudinit.util import load_file
1509 from cloudinit.sources.helpers.azure import WALinuxAgentShim as wa_shim
1510
1511 GOAL_STATE_TEMPLATE = """\
1512@@ -289,6 +291,50 @@ class TestOpenSSLManager(CiTestCase):
1513 self.assertEqual([mock.call(manager.tmpdir)], del_dir.call_args_list)
1514
1515
1516+class TestOpenSSLManagerActions(CiTestCase):
1517+
1518+ def setUp(self):
1519+ super(TestOpenSSLManagerActions, self).setUp()
1520+
1521+ self.allowed_subp = True
1522+
1523+ def _data_file(self, name):
1524+ path = 'tests/data/azure'
1525+ return os.path.join(path, name)
1526+
1527+ @unittest2.skip("todo move to cloud_test")
1528+ def test_pubkey_extract(self):
1529+ cert = load_file(self._data_file('pubkey_extract_cert'))
1530+ good_key = load_file(self._data_file('pubkey_extract_ssh_key'))
1531+ sslmgr = azure_helper.OpenSSLManager()
1532+ key = sslmgr._get_ssh_key_from_cert(cert)
1533+ self.assertEqual(good_key, key)
1534+
1535+ good_fingerprint = '073E19D14D1C799224C6A0FD8DDAB6A8BF27D473'
1536+ fingerprint = sslmgr._get_fingerprint_from_cert(cert)
1537+ self.assertEqual(good_fingerprint, fingerprint)
1538+
1539+ @unittest2.skip("todo move to cloud_test")
1540+ @mock.patch.object(azure_helper.OpenSSLManager, '_decrypt_certs_from_xml')
1541+ def test_parse_certificates(self, mock_decrypt_certs):
1542+ """Azure control plane puts private keys as well as certificates
1543+ into the Certificates XML object. Make sure only the public keys
1544+ from certs are extracted and that fingerprints are converted to
1545+ the form specified in the ovf-env.xml file.
1546+ """
1547+ cert_contents = load_file(self._data_file('parse_certificates_pem'))
1548+ fingerprints = load_file(self._data_file(
1549+ 'parse_certificates_fingerprints')
1550+ ).splitlines()
1551+ mock_decrypt_certs.return_value = cert_contents
1552+ sslmgr = azure_helper.OpenSSLManager()
1553+ keys_by_fp = sslmgr.parse_certificates('')
1554+ for fp in keys_by_fp.keys():
1555+ self.assertIn(fp, fingerprints)
1556+ for fp in fingerprints:
1557+ self.assertIn(fp, keys_by_fp)
1558+
1559+
1560 class TestWALinuxAgentShim(CiTestCase):
1561
1562 def setUp(self):
1563@@ -329,18 +375,31 @@ class TestWALinuxAgentShim(CiTestCase):
1564
1565 def test_certificates_used_to_determine_public_keys(self):
1566 shim = wa_shim()
1567- data = shim.register_with_azure_and_fetch_data()
1568+ """if register_with_azure_and_fetch_data() isn't passed some info about
1569+ the user's public keys, there's no point in even trying to parse
1570+ the certificates
1571+ """
1572+ mypk = [{'fingerprint': 'fp1', 'path': 'path1'},
1573+ {'fingerprint': 'fp3', 'path': 'path3', 'value': ''}]
1574+ certs = {'fp1': 'expected-key',
1575+ 'fp2': 'should-not-be-found',
1576+ 'fp3': 'expected-no-value-key',
1577+ }
1578+ sslmgr = self.OpenSSLManager.return_value
1579+ sslmgr.parse_certificates.return_value = certs
1580+ data = shim.register_with_azure_and_fetch_data(pubkey_info=mypk)
1581 self.assertEqual(
1582 [mock.call(self.GoalState.return_value.certificates_xml)],
1583- self.OpenSSLManager.return_value.parse_certificates.call_args_list)
1584- self.assertEqual(
1585- self.OpenSSLManager.return_value.parse_certificates.return_value,
1586- data['public-keys'])
1587+ sslmgr.parse_certificates.call_args_list)
1588+ self.assertIn('expected-key', data['public-keys'])
1589+ self.assertIn('expected-no-value-key', data['public-keys'])
1590+ self.assertNotIn('should-not-be-found', data['public-keys'])
1591
1592 def test_absent_certificates_produces_empty_public_keys(self):
1593+ mypk = [{'fingerprint': 'fp1', 'path': 'path1'}]
1594 self.GoalState.return_value.certificates_xml = None
1595 shim = wa_shim()
1596- data = shim.register_with_azure_and_fetch_data()
1597+ data = shim.register_with_azure_and_fetch_data(pubkey_info=mypk)
1598 self.assertEqual([], data['public-keys'])
1599
1600 def test_correct_url_used_for_report_ready(self):
1601diff --git a/tests/unittests/test_datasource/test_configdrive.py b/tests/unittests/test_datasource/test_configdrive.py
1602index dcdabea..520c50f 100644
1603--- a/tests/unittests/test_datasource/test_configdrive.py
1604+++ b/tests/unittests/test_datasource/test_configdrive.py
1605@@ -268,8 +268,7 @@ class TestConfigDriveDataSource(CiTestCase):
1606 exists_mock = mocks.enter_context(
1607 mock.patch.object(os.path, 'exists',
1608 side_effect=exists_side_effect()))
1609- device = cfg_ds.device_name_to_device(name)
1610- self.assertEqual(dev_name, device)
1611+ self.assertEqual(dev_name, cfg_ds.device_name_to_device(name))
1612
1613 find_mock.assert_called_once_with(mock.ANY)
1614 self.assertEqual(exists_mock.call_count, 2)
1615@@ -296,8 +295,7 @@ class TestConfigDriveDataSource(CiTestCase):
1616 exists_mock = mocks.enter_context(
1617 mock.patch.object(os.path, 'exists',
1618 return_value=True))
1619- device = cfg_ds.device_name_to_device(name)
1620- self.assertEqual(dev_name, device)
1621+ self.assertEqual(dev_name, cfg_ds.device_name_to_device(name))
1622
1623 find_mock.assert_called_once_with(mock.ANY)
1624 exists_mock.assert_called_once_with(mock.ANY)
1625@@ -331,8 +329,7 @@ class TestConfigDriveDataSource(CiTestCase):
1626 yield True
1627 with mock.patch.object(os.path, 'exists',
1628 side_effect=exists_side_effect()):
1629- device = cfg_ds.device_name_to_device(name)
1630- self.assertEqual(dev_name, device)
1631+ self.assertEqual(dev_name, cfg_ds.device_name_to_device(name))
1632 # We don't assert the call count for os.path.exists() because
1633 # not all of the entries in name_tests results in two calls to
1634 # that function. Specifically, 'root2k' doesn't seem to call
1635@@ -359,8 +356,7 @@ class TestConfigDriveDataSource(CiTestCase):
1636 }
1637 for name, dev_name in name_tests.items():
1638 with mock.patch.object(os.path, 'exists', return_value=True):
1639- device = cfg_ds.device_name_to_device(name)
1640- self.assertEqual(dev_name, device)
1641+ self.assertEqual(dev_name, cfg_ds.device_name_to_device(name))
1642
1643 def test_dir_valid(self):
1644 """Verify a dir is read as such."""
1645@@ -604,6 +600,9 @@ class TestNetJson(CiTestCase):
1646
1647
1648 class TestConvertNetworkData(CiTestCase):
1649+
1650+ with_logs = True
1651+
1652 def setUp(self):
1653 super(TestConvertNetworkData, self).setUp()
1654 self.tmp = self.tmp_dir()
1655@@ -730,6 +729,26 @@ class TestConvertNetworkData(CiTestCase):
1656 'enp0s2': 'fa:16:3e:d4:57:ad'}
1657 self.assertEqual(expected, config_name2mac)
1658
1659+ def test_unknown_device_types_accepted(self):
1660+ # If we don't recognise a link, we should treat it as physical for a
1661+ # best-effort boot
1662+ my_netdata = deepcopy(NETWORK_DATA)
1663+ my_netdata['links'][0]['type'] = 'my-special-link-type'
1664+
1665+ ncfg = openstack.convert_net_json(my_netdata, known_macs=KNOWN_MACS)
1666+ config_name2mac = {}
1667+ for n in ncfg['config']:
1668+ if n['type'] == 'physical':
1669+ config_name2mac[n['name']] = n['mac_address']
1670+
1671+ expected = {'nic0': 'fa:16:3e:05:30:fe', 'enp0s1': 'fa:16:3e:69:b0:58',
1672+ 'enp0s2': 'fa:16:3e:d4:57:ad'}
1673+ self.assertEqual(expected, config_name2mac)
1674+
1675+ # We should, however, warn the user that we don't recognise the type
1676+ self.assertIn('Unknown network_data link type (my-special-link-type)',
1677+ self.logs.getvalue())
1678+
1679
1680 def cfg_ds_from_dir(base_d, files=None):
1681 run = os.path.join(base_d, "run")
1682diff --git a/tests/unittests/test_datasource/test_ec2.py b/tests/unittests/test_datasource/test_ec2.py
1683index 1a5956d..20d59bf 100644
1684--- a/tests/unittests/test_datasource/test_ec2.py
1685+++ b/tests/unittests/test_datasource/test_ec2.py
1686@@ -401,6 +401,30 @@ class TestEc2(test_helpers.HttprettyTestCase):
1687 ds.metadata = DEFAULT_METADATA
1688 self.assertEqual('my-identity-id', ds.get_instance_id())
1689
1690+ def test_classic_instance_true(self):
1691+ """If no vpc-id in metadata, is_classic_instance must return true."""
1692+ md_copy = copy.deepcopy(DEFAULT_METADATA)
1693+ ifaces_md = md_copy.get('network', {}).get('interfaces', {})
1694+ for _mac, mac_data in ifaces_md.get('macs', {}).items():
1695+ if 'vpc-id' in mac_data:
1696+ del mac_data['vpc-id']
1697+
1698+ ds = self._setup_ds(
1699+ platform_data=self.valid_platform_data,
1700+ sys_cfg={'datasource': {'Ec2': {'strict_id': False}}},
1701+ md={'md': md_copy})
1702+ self.assertTrue(ds.get_data())
1703+ self.assertTrue(ds.is_classic_instance())
1704+
1705+ def test_classic_instance_false(self):
1706+ """If vpc-id in metadata, is_classic_instance must return false."""
1707+ ds = self._setup_ds(
1708+ platform_data=self.valid_platform_data,
1709+ sys_cfg={'datasource': {'Ec2': {'strict_id': False}}},
1710+ md={'md': DEFAULT_METADATA})
1711+ self.assertTrue(ds.get_data())
1712+ self.assertFalse(ds.is_classic_instance())
1713+
1714 @mock.patch('cloudinit.net.dhcp.maybe_perform_dhcp_discovery')
1715 def test_valid_platform_with_strict_true(self, m_dhcp):
1716 """Valid platform data should return true with strict_id true."""
1717diff --git a/tests/unittests/test_distros/test_create_users.py b/tests/unittests/test_distros/test_create_users.py
1718index c3f258d..4062495 100644
1719--- a/tests/unittests/test_distros/test_create_users.py
1720+++ b/tests/unittests/test_distros/test_create_users.py
1721@@ -240,4 +240,32 @@ class TestCreateUser(CiTestCase):
1722 [mock.call(set(['auth1']), user), # not disabled
1723 mock.call(set(['key1']), 'foouser', options=disable_prefix)])
1724
1725+ @mock.patch("cloudinit.distros.util.which")
1726+ def test_lock_with_usermod_if_no_passwd(self, m_which, m_subp,
1727+ m_is_snappy):
1728+ """Lock uses usermod --lock if no 'passwd' cmd available."""
1729+ m_which.side_effect = lambda m: m in ('usermod',)
1730+ self.dist.lock_passwd("bob")
1731+ self.assertEqual(
1732+ [mock.call(['usermod', '--lock', 'bob'])],
1733+ m_subp.call_args_list)
1734+
1735+ @mock.patch("cloudinit.distros.util.which")
1736+ def test_lock_with_passwd_if_available(self, m_which, m_subp,
1737+ m_is_snappy):
1738+ """Lock with only passwd will use passwd."""
1739+ m_which.side_effect = lambda m: m in ('passwd',)
1740+ self.dist.lock_passwd("bob")
1741+ self.assertEqual(
1742+ [mock.call(['passwd', '-l', 'bob'])],
1743+ m_subp.call_args_list)
1744+
1745+ @mock.patch("cloudinit.distros.util.which")
1746+ def test_lock_raises_runtime_if_no_commands(self, m_which, m_subp,
1747+ m_is_snappy):
1748+ """Lock with no commands available raises RuntimeError."""
1749+ m_which.return_value = None
1750+ with self.assertRaises(RuntimeError):
1751+ self.dist.lock_passwd("bob")
1752+
1753 # vi: ts=4 expandtab
1754diff --git a/tests/unittests/test_distros/test_netconfig.py b/tests/unittests/test_distros/test_netconfig.py
1755index e986b59..e453040 100644
1756--- a/tests/unittests/test_distros/test_netconfig.py
1757+++ b/tests/unittests/test_distros/test_netconfig.py
1758@@ -407,7 +407,7 @@ class TestNetCfgDistroUbuntuNetplan(TestNetCfgDistroBase):
1759 self.assertEqual(0o644, get_mode(cfgpath, tmpd))
1760
1761 def netplan_path(self):
1762- return '/etc/netplan/50-cloud-init.yaml'
1763+ return '/etc/netplan/50-cloud-init.yaml'
1764
1765 def test_apply_network_config_v1_to_netplan_ub(self):
1766 expected_cfgs = {
1767diff --git a/tests/unittests/test_ds_identify.py b/tests/unittests/test_ds_identify.py
1768index 756b4fb..d00c1b4 100644
1769--- a/tests/unittests/test_ds_identify.py
1770+++ b/tests/unittests/test_ds_identify.py
1771@@ -441,7 +441,7 @@ class TestDsIdentify(DsIdentifyBase):
1772 nova does not identify itself on platforms other than intel.
1773 https://bugs.launchpad.net/cloud-init/+bugs?field.tag=dsid-nova"""
1774
1775- data = VALID_CFG['OpenStack'].copy()
1776+ data = copy.deepcopy(VALID_CFG['OpenStack'])
1777 del data['files'][P_PRODUCT_NAME]
1778 data.update({'policy_dmi': POLICY_FOUND_OR_MAYBE,
1779 'policy_no_dmi': POLICY_FOUND_OR_MAYBE})
1780diff --git a/tests/unittests/test_handler/test_handler_chef.py b/tests/unittests/test_handler/test_handler_chef.py
1781index b16532e..f431126 100644
1782--- a/tests/unittests/test_handler/test_handler_chef.py
1783+++ b/tests/unittests/test_handler/test_handler_chef.py
1784@@ -145,6 +145,7 @@ class TestChef(FilesystemMockingTestCase):
1785 file_backup_path "/var/backups/chef"
1786 pid_file "/var/run/chef/client.pid"
1787 Chef::Log::Formatter.show_time = true
1788+ encrypted_data_bag_secret "/etc/chef/encrypted_data_bag_secret"
1789 """
1790 tpl_file = util.load_file('templates/chef_client.rb.tmpl')
1791 self.patchUtils(self.tmp)
1792@@ -157,6 +158,8 @@ class TestChef(FilesystemMockingTestCase):
1793 'validation_name': 'bob',
1794 'validation_key': "/etc/chef/vkey.pem",
1795 'validation_cert': "this is my cert",
1796+ 'encrypted_data_bag_secret':
1797+ '/etc/chef/encrypted_data_bag_secret'
1798 },
1799 }
1800 cc_chef.handle('chef', cfg, self.fetch_cloud('ubuntu'), LOG, [])
1801diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py
1802index e041e97..e3b9e02 100644
1803--- a/tests/unittests/test_net.py
1804+++ b/tests/unittests/test_net.py
1805@@ -19,6 +19,7 @@ import gzip
1806 import io
1807 import json
1808 import os
1809+import re
1810 import textwrap
1811 import yaml
1812
1813@@ -103,6 +104,326 @@ STATIC_EXPECTED_1 = {
1814 'address': '10.0.0.2'}],
1815 }
1816
1817+V1_NAMESERVER_ALIAS = """
1818+config:
1819+- id: eno1
1820+ mac_address: 08:94:ef:51:ae:e0
1821+ mtu: 1500
1822+ name: eno1
1823+ subnets:
1824+ - type: manual
1825+ type: physical
1826+- id: eno2
1827+ mac_address: 08:94:ef:51:ae:e1
1828+ mtu: 1500
1829+ name: eno2
1830+ subnets:
1831+ - type: manual
1832+ type: physical
1833+- id: eno3
1834+ mac_address: 08:94:ef:51:ae:de
1835+ mtu: 1500
1836+ name: eno3
1837+ subnets:
1838+ - type: manual
1839+ type: physical
1840+- bond_interfaces:
1841+ - eno1
1842+ - eno3
1843+ id: bondM
1844+ mac_address: 08:94:ef:51:ae:e0
1845+ mtu: 1500
1846+ name: bondM
1847+ params:
1848+ bond-downdelay: 0
1849+ bond-lacp-rate: fast
1850+ bond-miimon: 100
1851+ bond-mode: 802.3ad
1852+ bond-updelay: 0
1853+ bond-xmit-hash-policy: layer3+4
1854+ subnets:
1855+ - address: 10.101.10.47/23
1856+ gateway: 10.101.11.254
1857+ type: static
1858+ type: bond
1859+- id: eno4
1860+ mac_address: 08:94:ef:51:ae:df
1861+ mtu: 1500
1862+ name: eno4
1863+ subnets:
1864+ - type: manual
1865+ type: physical
1866+- id: enp0s20f0u1u6
1867+ mac_address: 0a:94:ef:51:a4:b9
1868+ mtu: 1500
1869+ name: enp0s20f0u1u6
1870+ subnets:
1871+ - type: manual
1872+ type: physical
1873+- id: enp216s0f0
1874+ mac_address: 68:05:ca:81:7c:e8
1875+ mtu: 9000
1876+ name: enp216s0f0
1877+ subnets:
1878+ - type: manual
1879+ type: physical
1880+- id: enp216s0f1
1881+ mac_address: 68:05:ca:81:7c:e9
1882+ mtu: 9000
1883+ name: enp216s0f1
1884+ subnets:
1885+ - type: manual
1886+ type: physical
1887+- id: enp47s0f0
1888+ mac_address: 68:05:ca:64:d3:6c
1889+ mtu: 9000
1890+ name: enp47s0f0
1891+ subnets:
1892+ - type: manual
1893+ type: physical
1894+- bond_interfaces:
1895+ - enp216s0f0
1896+ - enp47s0f0
1897+ id: bond0
1898+ mac_address: 68:05:ca:64:d3:6c
1899+ mtu: 9000
1900+ name: bond0
1901+ params:
1902+ bond-downdelay: 0
1903+ bond-lacp-rate: fast
1904+ bond-miimon: 100
1905+ bond-mode: 802.3ad
1906+ bond-updelay: 0
1907+ bond-xmit-hash-policy: layer3+4
1908+ subnets:
1909+ - type: manual
1910+ type: bond
1911+- id: bond0.3502
1912+ mtu: 9000
1913+ name: bond0.3502
1914+ subnets:
1915+ - address: 172.20.80.4/25
1916+ type: static
1917+ type: vlan
1918+ vlan_id: 3502
1919+ vlan_link: bond0
1920+- id: bond0.3503
1921+ mtu: 9000
1922+ name: bond0.3503
1923+ subnets:
1924+ - address: 172.20.80.129/25
1925+ type: static
1926+ type: vlan
1927+ vlan_id: 3503
1928+ vlan_link: bond0
1929+- id: enp47s0f1
1930+ mac_address: 68:05:ca:64:d3:6d
1931+ mtu: 9000
1932+ name: enp47s0f1
1933+ subnets:
1934+ - type: manual
1935+ type: physical
1936+- bond_interfaces:
1937+ - enp216s0f1
1938+ - enp47s0f1
1939+ id: bond1
1940+ mac_address: 68:05:ca:64:d3:6d
1941+ mtu: 9000
1942+ name: bond1
1943+ params:
1944+ bond-downdelay: 0
1945+ bond-lacp-rate: fast
1946+ bond-miimon: 100
1947+ bond-mode: 802.3ad
1948+ bond-updelay: 0
1949+ bond-xmit-hash-policy: layer3+4
1950+ subnets:
1951+ - address: 10.101.8.65/26
1952+ routes:
1953+ - destination: 213.119.192.0/24
1954+ gateway: 10.101.8.126
1955+ metric: 0
1956+ type: static
1957+ type: bond
1958+- address:
1959+ - 10.101.10.1
1960+ - 10.101.10.2
1961+ - 10.101.10.3
1962+ - 10.101.10.5
1963+ search:
1964+ - foo.bar
1965+ - maas
1966+ type: nameserver
1967+version: 1
1968+"""
1969+
1970+NETPLAN_NO_ALIAS = """
1971+network:
1972+ version: 2
1973+ ethernets:
1974+ eno1:
1975+ match:
1976+ macaddress: 08:94:ef:51:ae:e0
1977+ mtu: 1500
1978+ set-name: eno1
1979+ eno2:
1980+ match:
1981+ macaddress: 08:94:ef:51:ae:e1
1982+ mtu: 1500
1983+ set-name: eno2
1984+ eno3:
1985+ match:
1986+ macaddress: 08:94:ef:51:ae:de
1987+ mtu: 1500
1988+ set-name: eno3
1989+ eno4:
1990+ match:
1991+ macaddress: 08:94:ef:51:ae:df
1992+ mtu: 1500
1993+ set-name: eno4
1994+ enp0s20f0u1u6:
1995+ match:
1996+ macaddress: 0a:94:ef:51:a4:b9
1997+ mtu: 1500
1998+ set-name: enp0s20f0u1u6
1999+ enp216s0f0:
2000+ match:
2001+ macaddress: 68:05:ca:81:7c:e8
2002+ mtu: 9000
2003+ set-name: enp216s0f0
2004+ enp216s0f1:
2005+ match:
2006+ macaddress: 68:05:ca:81:7c:e9
2007+ mtu: 9000
2008+ set-name: enp216s0f1
2009+ enp47s0f0:
2010+ match:
2011+ macaddress: 68:05:ca:64:d3:6c
2012+ mtu: 9000
2013+ set-name: enp47s0f0
2014+ enp47s0f1:
2015+ match:
2016+ macaddress: 68:05:ca:64:d3:6d
2017+ mtu: 9000
2018+ set-name: enp47s0f1
2019+ bonds:
2020+ bond0:
2021+ interfaces:
2022+ - enp216s0f0
2023+ - enp47s0f0
2024+ macaddress: 68:05:ca:64:d3:6c
2025+ mtu: 9000
2026+ parameters:
2027+ down-delay: 0
2028+ lacp-rate: fast
2029+ mii-monitor-interval: 100
2030+ mode: 802.3ad
2031+ transmit-hash-policy: layer3+4
2032+ up-delay: 0
2033+ bond1:
2034+ addresses:
2035+ - 10.101.8.65/26
2036+ interfaces:
2037+ - enp216s0f1
2038+ - enp47s0f1
2039+ macaddress: 68:05:ca:64:d3:6d
2040+ mtu: 9000
2041+ nameservers:
2042+ addresses:
2043+ - 10.101.10.1
2044+ - 10.101.10.2
2045+ - 10.101.10.3
2046+ - 10.101.10.5
2047+ search:
2048+ - foo.bar
2049+ - maas
2050+ parameters:
2051+ down-delay: 0
2052+ lacp-rate: fast
2053+ mii-monitor-interval: 100
2054+ mode: 802.3ad
2055+ transmit-hash-policy: layer3+4
2056+ up-delay: 0
2057+ routes:
2058+ - metric: 0
2059+ to: 213.119.192.0/24
2060+ via: 10.101.8.126
2061+ bondM:
2062+ addresses:
2063+ - 10.101.10.47/23
2064+ gateway4: 10.101.11.254
2065+ interfaces:
2066+ - eno1
2067+ - eno3
2068+ macaddress: 08:94:ef:51:ae:e0
2069+ mtu: 1500
2070+ nameservers:
2071+ addresses:
2072+ - 10.101.10.1
2073+ - 10.101.10.2
2074+ - 10.101.10.3
2075+ - 10.101.10.5
2076+ search:
2077+ - foo.bar
2078+ - maas
2079+ parameters:
2080+ down-delay: 0
2081+ lacp-rate: fast
2082+ mii-monitor-interval: 100
2083+ mode: 802.3ad
2084+ transmit-hash-policy: layer3+4
2085+ up-delay: 0
2086+ vlans:
2087+ bond0.3502:
2088+ addresses:
2089+ - 172.20.80.4/25
2090+ id: 3502
2091+ link: bond0
2092+ mtu: 9000
2093+ nameservers:
2094+ addresses:
2095+ - 10.101.10.1
2096+ - 10.101.10.2
2097+ - 10.101.10.3
2098+ - 10.101.10.5
2099+ search:
2100+ - foo.bar
2101+ - maas
2102+ bond0.3503:
2103+ addresses:
2104+ - 172.20.80.129/25
2105+ id: 3503
2106+ link: bond0
2107+ mtu: 9000
2108+ nameservers:
2109+ addresses:
2110+ - 10.101.10.1
2111+ - 10.101.10.2
2112+ - 10.101.10.3
2113+ - 10.101.10.5
2114+ search:
2115+ - foo.bar
2116+ - maas
2117+"""
2118+
2119+NETPLAN_DHCP_FALSE = """
2120+version: 2
2121+ethernets:
2122+ ens3:
2123+ match:
2124+ macaddress: 52:54:00:ab:cd:ef
2125+ dhcp4: false
2126+ dhcp6: false
2127+ addresses:
2128+ - 192.168.42.100/24
2129+ - 2001:db8::100/32
2130+ gateway4: 192.168.42.1
2131+ gateway6: 2001:db8::1
2132+ nameservers:
2133+ search: [example.com]
2134+ addresses: [192.168.42.53, 1.1.1.1]
2135+"""
2136+
2137 # Examples (and expected outputs for various renderers).
2138 OS_SAMPLES = [
2139 {
2140@@ -2286,6 +2607,50 @@ USERCTL=no
2141 config = sysconfig.ConfigObj(nm_cfg)
2142 self.assertIn('ifcfg-rh', config['main']['plugins'])
2143
2144+ def test_netplan_dhcp_false_disable_dhcp_in_state(self):
2145+ """netplan config with dhcp[46]: False should not add dhcp in state"""
2146+ net_config = yaml.load(NETPLAN_DHCP_FALSE)
2147+ ns = network_state.parse_net_config_data(net_config,
2148+ skip_broken=False)
2149+
2150+ dhcp_found = [snet for iface in ns.iter_interfaces()
2151+ for snet in iface['subnets'] if 'dhcp' in snet['type']]
2152+
2153+ self.assertEqual([], dhcp_found)
2154+
2155+ def test_netplan_dhcp_false_no_dhcp_in_sysconfig(self):
2156+ """netplan cfg with dhcp[46]: False should not have bootproto=dhcp"""
2157+
2158+ entry = {
2159+ 'yaml': NETPLAN_DHCP_FALSE,
2160+ 'expected_sysconfig': {
2161+ 'ifcfg-ens3': textwrap.dedent("""\
2162+ BOOTPROTO=none
2163+ DEFROUTE=yes
2164+ DEVICE=ens3
2165+ DNS1=192.168.42.53
2166+ DNS2=1.1.1.1
2167+ DOMAIN=example.com
2168+ GATEWAY=192.168.42.1
2169+ HWADDR=52:54:00:ab:cd:ef
2170+ IPADDR=192.168.42.100
2171+ IPV6ADDR=2001:db8::100/32
2172+ IPV6INIT=yes
2173+ IPV6_DEFAULTGW=2001:db8::1
2174+ NETMASK=255.255.255.0
2175+ NM_CONTROLLED=no
2176+ ONBOOT=yes
2177+ STARTMODE=auto
2178+ TYPE=Ethernet
2179+ USERCTL=no
2180+ """),
2181+ }
2182+ }
2183+
2184+ found = self._render_and_read(network_config=yaml.load(entry['yaml']))
2185+ self._compare_files_to_expected(entry['expected_sysconfig'], found)
2186+ self._assert_headers(found)
2187+
2188
2189 class TestOpenSuseSysConfigRendering(CiTestCase):
2190
2191@@ -3065,6 +3430,38 @@ class TestNetplanRoundTrip(CiTestCase):
2192 entry['expected_netplan'].splitlines(),
2193 files['/etc/netplan/50-cloud-init.yaml'].splitlines())
2194
2195+ def test_render_output_has_yaml_no_aliases(self):
2196+ entry = {
2197+ 'yaml': V1_NAMESERVER_ALIAS,
2198+ 'expected_netplan': NETPLAN_NO_ALIAS,
2199+ }
2200+ network_config = yaml.load(entry['yaml'])
2201+ ns = network_state.parse_net_config_data(network_config)
2202+ files = self._render_and_read(state=ns)
2203+ # check for alias
2204+ content = files['/etc/netplan/50-cloud-init.yaml']
2205+
2206+ # test load the yaml to ensure we don't render something not loadable
2207+ # this allows single aliases, but not duplicate ones
2208+ parsed = yaml.load(files['/etc/netplan/50-cloud-init.yaml'])
2209+ self.assertNotEqual(None, parsed)
2210+
2211+ # now look for any alias, avoid rendering them entirely
2212+ # generate the first anchor string using the template
2213+ # as of this writing, looks like "&id001"
2214+ anchor = r'&' + yaml.serializer.Serializer.ANCHOR_TEMPLATE % 1
2215+ found_alias = re.search(anchor, content, re.MULTILINE)
2216+ if found_alias:
2217+ msg = "Error at: %s\nContent:\n%s" % (found_alias, content)
2218+ raise ValueError('Found yaml alias in rendered netplan: ' + msg)
2219+
2220+ print(entry['expected_netplan'])
2221+ print('-- expected ^ | v rendered --')
2222+ print(files['/etc/netplan/50-cloud-init.yaml'])
2223+ self.assertEqual(
2224+ entry['expected_netplan'].splitlines(),
2225+ files['/etc/netplan/50-cloud-init.yaml'].splitlines())
2226+
2227
2228 class TestEniRoundTrip(CiTestCase):
2229
2230diff --git a/tools/cloud-init-per b/tools/cloud-init-per
2231index 7d6754b..fcd1ea7 100755
2232--- a/tools/cloud-init-per
2233+++ b/tools/cloud-init-per
2234@@ -38,7 +38,7 @@ fi
2235 [ "$1" = "-h" -o "$1" = "--help" ] && { Usage ; exit 0; }
2236 [ $# -ge 3 ] || { Usage 1>&2; exit 1; }
2237 freq=$1
2238-name=$2
2239+name=$(echo $2 | sed 's/-/_/g')
2240 shift 2;
2241
2242 [ "${name#*/}" = "${name}" ] || fail "name cannot contain a /"
2243@@ -53,6 +53,12 @@ esac
2244 [ -d "${sem%/*}" ] || mkdir -p "${sem%/*}" ||
2245 fail "failed to make directory for ${sem}"
2246
2247+# Rename legacy sem files with dashes in their names. Do not overwrite existing
2248+# sem files to prevent clobbering those which may have been created from calls
2249+# outside of cloud-init.
2250+sem_legacy=$(echo $sem | sed 's/_/-/g')
2251+[ "$sem" != "$sem_legacy" -a -e "$sem_legacy" ] && mv -n "$sem_legacy" "$sem"
2252+
2253 [ "$freq" != "always" -a -e "$sem" ] && exit 0
2254 "$@"
2255 ret=$?

Subscribers

People subscribed via source and target branches