Merge ~chad.smith/cloud-init:ubuntu/disco into cloud-init:ubuntu/disco
- Git
- lp:~chad.smith/cloud-init
- ubuntu/disco
- Merge into ubuntu/disco
Proposed by
Chad Smith
Status: | Merged | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Merged at revision: | 59835f9211e70101bde10c63c7e84cc6a74c930c | ||||||||||||
Proposed branch: | ~chad.smith/cloud-init:ubuntu/disco | ||||||||||||
Merge into: | cloud-init:ubuntu/disco | ||||||||||||
Diff against target: |
1142 lines (+417/-168) 25 files modified
ChangeLog (+117/-0) cloudinit/config/cc_apt_configure.py (+1/-1) cloudinit/config/cc_mounts.py (+11/-0) cloudinit/net/sysconfig.py (+4/-2) cloudinit/net/tests/test_init.py (+1/-1) cloudinit/reporting/handlers.py (+57/-60) cloudinit/sources/DataSourceAzure.py (+11/-6) cloudinit/sources/DataSourceCloudStack.py (+1/-1) cloudinit/sources/DataSourceConfigDrive.py (+2/-5) cloudinit/sources/DataSourceEc2.py (+1/-1) cloudinit/sources/helpers/azure.py (+11/-3) cloudinit/util.py (+2/-13) cloudinit/version.py (+1/-1) debian/changelog (+27/-0) packages/redhat/cloud-init.spec.in (+3/-1) packages/suse/cloud-init.spec.in (+3/-1) setup.py (+2/-1) tests/cloud_tests/releases.yaml (+16/-0) tests/unittests/test_datasource/test_azure.py (+10/-3) tests/unittests/test_datasource/test_azure_helper.py (+7/-2) tests/unittests/test_handler/test_handler_mounts.py (+29/-1) tests/unittests/test_net.py (+42/-3) tests/unittests/test_reporting_hyperv.py (+49/-55) tools/build-on-freebsd (+4/-5) tools/read-version (+5/-2) |
||||||||||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Server Team CI bot | continuous-integration | Approve | |
cloud-init Commiters | Pending | ||
Review via email: mp+367300@code.launchpad.net |
Commit message
new upstream snapshot for release into disco
nothing special here for ubuntu advantage config module as ubuntu-
Description of the change
To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote : | # |
review:
Approve
(continuous-integration)
There was an error fetching revisions from git servers. Please try again in a few minutes. If the problem persists, contact Launchpad support.
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1 | diff --git a/ChangeLog b/ChangeLog |
2 | index 8fa6fdd..bf48fd4 100644 |
3 | --- a/ChangeLog |
4 | +++ b/ChangeLog |
5 | @@ -1,3 +1,120 @@ |
6 | +19.1: |
7 | + - freebsd: add chpasswd pkg in the image [Gonéri Le Bouder] |
8 | + - tests: add Eoan release [Paride Legovini] |
9 | + - cc_mounts: check if mount -a on no-change fstab path |
10 | + [Jason Zions (MSFT)] (LP: #1825596) |
11 | + - replace remaining occurrences of LOG.warn [Daniel Watkins] |
12 | + - DataSourceAzure: Adjust timeout for polling IMDS [Anh Vo] |
13 | + - Azure: Changes to the Hyper-V KVP Reporter [Anh Vo] |
14 | + - git tests: no longer show warning about safe yaml. |
15 | + - tools/read-version: handle errors [Chad Miller] |
16 | + - net/sysconfig: only indicate available on known sysconfig distros |
17 | + (LP: #1819994) |
18 | + - packages: update rpm specs for new bash completion path |
19 | + [Daniel Watkins] (LP: #1825444) |
20 | + - test_azure: mock util.SeLinuxGuard where needed |
21 | + [Jason Zions (MSFT)] (LP: #1825253) |
22 | + - setup.py: install bash completion script in new location [Daniel Watkins] |
23 | + - mount_cb: do not pass sync and rw options to mount |
24 | + [Gonéri Le Bouder] (LP: #1645824) |
25 | + - cc_apt_configure: fix typo in apt documentation [Dominic Schlegel] |
26 | + - Revert "DataSource: move update_events from a class to an instance..." |
27 | + [Daniel Watkins] |
28 | + - Change DataSourceNoCloud to ignore file system label's case. |
29 | + [Risto Oikarinen] |
30 | + - cmd:main.py: Fix missing 'modules-init' key in modes dict |
31 | + [Antonio Romito] (LP: #1815109) |
32 | + - ubuntu_advantage: rewrite cloud-config module |
33 | + - Azure: Treat _unset network configuration as if it were absent |
34 | + [Jason Zions (MSFT)] (LP: #1823084) |
35 | + - DatasourceAzure: add additional logging for azure datasource [Anh Vo] |
36 | + - cloud_tests: fix apt_pipelining test-cases |
37 | + - Azure: Ensure platform random_seed is always serializable as JSON. |
38 | + [Jason Zions (MSFT)] |
39 | + - net/sysconfig: write out SUSE-compatible IPv6 config [Robert Schweikert] |
40 | + - tox: Update testenv for openSUSE Leap to 15.0 [Thomas Bechtold] |
41 | + - net: Fix ipv6 static routes when using eni renderer |
42 | + [Raphael Glon] (LP: #1818669) |
43 | + - Add ubuntu_drivers config module [Daniel Watkins] |
44 | + - doc: Refresh Azure walinuxagent docs [Daniel Watkins] |
45 | + - tox: bump pylint version to latest (2.3.1) [Daniel Watkins] |
46 | + - DataSource: move update_events from a class to an instance attribute |
47 | + [Daniel Watkins] (LP: #1819913) |
48 | + - net/sysconfig: Handle default route setup for dhcp configured NICs |
49 | + [Robert Schweikert] (LP: #1812117) |
50 | + - DataSourceEc2: update RELEASE_BLOCKER to be more accurate |
51 | + [Daniel Watkins] |
52 | + - cloud-init-per: POSIX sh does not support string subst, use sed |
53 | + (LP: #1819222) |
54 | + - Support locking user with usermod if passwd is not available. |
55 | + - Example for Microsoft Azure data disk added. [Anton Olifir] |
56 | + - clean: correctly determine the path for excluding seed directory |
57 | + [Daniel Watkins] (LP: #1818571) |
58 | + - helpers/openstack: Treat unknown link types as physical |
59 | + [Daniel Watkins] (LP: #1639263) |
60 | + - drop Python 2.6 support and our NIH version detection [Daniel Watkins] |
61 | + - tip-pylint: Fix assignment-from-return-none errors |
62 | + - net: append type:dhcp[46] only if dhcp[46] is True in v2 netconfig |
63 | + [Kurt Stieger] (LP: #1818032) |
64 | + - cc_apt_pipelining: stop disabling pipelining by default |
65 | + [Daniel Watkins] (LP: #1794982) |
66 | + - tests: fix some slow tests and some leaking state [Daniel Watkins] |
67 | + - util: don't determine string_types ourselves [Daniel Watkins] |
68 | + - cc_rsyslog: Escape possible nested set [Daniel Watkins] (LP: #1816967) |
69 | + - Enable encrypted_data_bag_secret support for Chef |
70 | + [Eric Williams] (LP: #1817082) |
71 | + - azure: Filter list of ssh keys pulled from fabric [Jason Zions (MSFT)] |
72 | + - doc: update merging doc with fixes and some additional details/examples |
73 | + - tests: integration test failure summary to use traceback if empty error |
74 | + - This is to fix https://bugs.launchpad.net/cloud-init/+bug/1812676 |
75 | + [Vitaly Kuznetsov] |
76 | + - EC2: Rewrite network config on AWS Classic instances every boot |
77 | + [Guilherme G. Piccoli] (LP: #1802073) |
78 | + - netinfo: Adjust ifconfig output parsing for FreeBSD ipv6 entries |
79 | + (LP: #1779672) |
80 | + - netplan: Don't render yaml aliases when dumping netplan (LP: #1815051) |
81 | + - add PyCharm IDE .idea/ path to .gitignore [Dominic Schlegel] |
82 | + - correct grammar issue in instance metadata documentation |
83 | + [Dominic Schlegel] (LP: #1802188) |
84 | + - clean: cloud-init clean should not trace when run from within cloud_dir |
85 | + (LP: #1795508) |
86 | + - Resolve flake8 comparison and pycodestyle over-ident issues |
87 | + [Paride Legovini] |
88 | + - opennebula: also exclude epochseconds from changed environment vars |
89 | + (LP: #1813641) |
90 | + - systemd: Render generator from template to account for system |
91 | + differences. [Robert Schweikert] |
92 | + - sysconfig: On SUSE, use STARTMODE instead of ONBOOT |
93 | + [Robert Schweikert] (LP: #1799540) |
94 | + - flake8: use ==/!= to compare str, bytes, and int literals |
95 | + [Paride Legovini] |
96 | + - opennebula: exclude EPOCHREALTIME as known bash env variable with a |
97 | + delta (LP: #1813383) |
98 | + - tox: fix disco httpretty dependencies for py37 (LP: #1813361) |
99 | + - run-container: uncomment baseurl in yum.repos.d/*.repo when using a |
100 | + proxy [Paride Legovini] |
101 | + - lxd: install zfs-linux instead of zfs meta package |
102 | + [Johnson Shi] (LP: #1799779) |
103 | + - net/sysconfig: do not write a resolv.conf file with only the header. |
104 | + [Robert Schweikert] |
105 | + - net: Make sysconfig renderer compatible with Network Manager. |
106 | + [Eduardo Otubo] |
107 | + - cc_set_passwords: Fix regex when parsing hashed passwords |
108 | + [Marlin Cremers] (LP: #1811446) |
109 | + - net: Wait for dhclient to daemonize before reading lease file |
110 | + [Jason Zions] (LP: #1794399) |
111 | + - [Azure] Increase retries when talking to Wireserver during metadata walk |
112 | + [Jason Zions] |
113 | + - Add documentation on adding a datasource. |
114 | + - doc: clean up some datasource documentation. |
115 | + - ds-identify: fix wrong variable name in ovf_vmware_transport_guestinfo. |
116 | + - Scaleway: Support ssh keys provided inside an instance tag. [PORTE Loïc] |
117 | + - OVF: simplify expected return values of transport functions. |
118 | + - Vmware: Add support for the com.vmware.guestInfo OVF transport. |
119 | + (LP: #1807466) |
120 | + - HACKING.rst: change contact info to Josh Powers |
121 | + - Update to pylint 2.2.2. |
122 | + |
123 | 18.5: |
124 | - tests: add Disco release [Joshua Powers] |
125 | - net: render 'metric' values in per-subnet routes (LP: #1805871) |
126 | diff --git a/cloudinit/config/cc_apt_configure.py b/cloudinit/config/cc_apt_configure.py |
127 | index e18944e..919d199 100644 |
128 | --- a/cloudinit/config/cc_apt_configure.py |
129 | +++ b/cloudinit/config/cc_apt_configure.py |
130 | @@ -127,7 +127,7 @@ to ``^[\\w-]+:\\w`` |
131 | |
132 | Source list entries can be specified as a dictionary under the ``sources`` |
133 | config key, with key in the dict representing a different source file. The key |
134 | -The key of each source entry will be used as an id that can be referenced in |
135 | +of each source entry will be used as an id that can be referenced in |
136 | other config entries, as well as the filename for the source's configuration |
137 | under ``/etc/apt/sources.list.d``. If the name does not end with ``.list``, |
138 | it will be appended. If there is no configuration for a key in ``sources``, no |
139 | diff --git a/cloudinit/config/cc_mounts.py b/cloudinit/config/cc_mounts.py |
140 | index 339baba..123ffb8 100644 |
141 | --- a/cloudinit/config/cc_mounts.py |
142 | +++ b/cloudinit/config/cc_mounts.py |
143 | @@ -439,6 +439,7 @@ def handle(_name, cfg, cloud, log, _args): |
144 | |
145 | cc_lines = [] |
146 | needswap = False |
147 | + need_mount_all = False |
148 | dirs = [] |
149 | for line in actlist: |
150 | # write 'comment' in the fs_mntops, entry, claiming this |
151 | @@ -449,11 +450,18 @@ def handle(_name, cfg, cloud, log, _args): |
152 | dirs.append(line[1]) |
153 | cc_lines.append('\t'.join(line)) |
154 | |
155 | + mount_points = [v['mountpoint'] for k, v in util.mounts().items() |
156 | + if 'mountpoint' in v] |
157 | for d in dirs: |
158 | try: |
159 | util.ensure_dir(d) |
160 | except Exception: |
161 | util.logexc(log, "Failed to make '%s' config-mount", d) |
162 | + # dirs is list of directories on which a volume should be mounted. |
163 | + # If any of them does not already show up in the list of current |
164 | + # mount points, we will definitely need to do mount -a. |
165 | + if not need_mount_all and d not in mount_points: |
166 | + need_mount_all = True |
167 | |
168 | sadds = [WS.sub(" ", n) for n in cc_lines] |
169 | sdrops = [WS.sub(" ", n) for n in fstab_removed] |
170 | @@ -473,6 +481,9 @@ def handle(_name, cfg, cloud, log, _args): |
171 | log.debug("No changes to /etc/fstab made.") |
172 | else: |
173 | log.debug("Changes to fstab: %s", sops) |
174 | + need_mount_all = True |
175 | + |
176 | + if need_mount_all: |
177 | activate_cmds.append(["mount", "-a"]) |
178 | if uses_systemd: |
179 | activate_cmds.append(["systemctl", "daemon-reload"]) |
180 | diff --git a/cloudinit/net/sysconfig.py b/cloudinit/net/sysconfig.py |
181 | index 0998392..a47da0a 100644 |
182 | --- a/cloudinit/net/sysconfig.py |
183 | +++ b/cloudinit/net/sysconfig.py |
184 | @@ -18,6 +18,8 @@ from .network_state import ( |
185 | |
186 | LOG = logging.getLogger(__name__) |
187 | NM_CFG_FILE = "/etc/NetworkManager/NetworkManager.conf" |
188 | +KNOWN_DISTROS = [ |
189 | + 'opensuse', 'sles', 'suse', 'redhat', 'fedora', 'centos'] |
190 | |
191 | |
192 | def _make_header(sep='#'): |
193 | @@ -717,8 +719,8 @@ class Renderer(renderer.Renderer): |
194 | def available(target=None): |
195 | sysconfig = available_sysconfig(target=target) |
196 | nm = available_nm(target=target) |
197 | - |
198 | - return any([nm, sysconfig]) |
199 | + return (util.get_linux_distro()[0] in KNOWN_DISTROS |
200 | + and any([nm, sysconfig])) |
201 | |
202 | |
203 | def available_sysconfig(target=None): |
204 | diff --git a/cloudinit/net/tests/test_init.py b/cloudinit/net/tests/test_init.py |
205 | index f55c31e..6d2affe 100644 |
206 | --- a/cloudinit/net/tests/test_init.py |
207 | +++ b/cloudinit/net/tests/test_init.py |
208 | @@ -7,11 +7,11 @@ import mock |
209 | import os |
210 | import requests |
211 | import textwrap |
212 | -import yaml |
213 | |
214 | import cloudinit.net as net |
215 | from cloudinit.util import ensure_file, write_file, ProcessExecutionError |
216 | from cloudinit.tests.helpers import CiTestCase, HttprettyTestCase |
217 | +from cloudinit import safeyaml as yaml |
218 | |
219 | |
220 | class TestSysDevPath(CiTestCase): |
221 | diff --git a/cloudinit/reporting/handlers.py b/cloudinit/reporting/handlers.py |
222 | old mode 100644 |
223 | new mode 100755 |
224 | index 6d23558..10165ae |
225 | --- a/cloudinit/reporting/handlers.py |
226 | +++ b/cloudinit/reporting/handlers.py |
227 | @@ -5,7 +5,6 @@ import fcntl |
228 | import json |
229 | import six |
230 | import os |
231 | -import re |
232 | import struct |
233 | import threading |
234 | import time |
235 | @@ -14,6 +13,7 @@ from cloudinit import log as logging |
236 | from cloudinit.registry import DictRegistry |
237 | from cloudinit import (url_helper, util) |
238 | from datetime import datetime |
239 | +from six.moves.queue import Empty as QueueEmptyError |
240 | |
241 | if six.PY2: |
242 | from multiprocessing.queues import JoinableQueue as JQueue |
243 | @@ -129,24 +129,50 @@ class HyperVKvpReportingHandler(ReportingHandler): |
244 | DESC_IDX_KEY = 'msg_i' |
245 | JSON_SEPARATORS = (',', ':') |
246 | KVP_POOL_FILE_GUEST = '/var/lib/hyperv/.kvp_pool_1' |
247 | + _already_truncated_pool_file = False |
248 | |
249 | def __init__(self, |
250 | kvp_file_path=KVP_POOL_FILE_GUEST, |
251 | event_types=None): |
252 | super(HyperVKvpReportingHandler, self).__init__() |
253 | self._kvp_file_path = kvp_file_path |
254 | + HyperVKvpReportingHandler._truncate_guest_pool_file( |
255 | + self._kvp_file_path) |
256 | + |
257 | self._event_types = event_types |
258 | self.q = JQueue() |
259 | - self.kvp_file = None |
260 | self.incarnation_no = self._get_incarnation_no() |
261 | self.event_key_prefix = u"{0}|{1}".format(self.EVENT_PREFIX, |
262 | self.incarnation_no) |
263 | - self._current_offset = 0 |
264 | self.publish_thread = threading.Thread( |
265 | target=self._publish_event_routine) |
266 | self.publish_thread.daemon = True |
267 | self.publish_thread.start() |
268 | |
269 | + @classmethod |
270 | + def _truncate_guest_pool_file(cls, kvp_file): |
271 | + """ |
272 | + Truncate the pool file if it has not been truncated since boot. |
273 | + This should be done exactly once for the file indicated by |
274 | + KVP_POOL_FILE_GUEST constant above. This method takes a filename |
275 | + so that we can use an arbitrary file during unit testing. |
276 | + Since KVP is a best-effort telemetry channel we only attempt to |
277 | + truncate the file once and only if the file has not been modified |
278 | + since boot. Additional truncation can lead to loss of existing |
279 | + KVPs. |
280 | + """ |
281 | + if cls._already_truncated_pool_file: |
282 | + return |
283 | + boot_time = time.time() - float(util.uptime()) |
284 | + try: |
285 | + if os.path.getmtime(kvp_file) < boot_time: |
286 | + with open(kvp_file, "w"): |
287 | + pass |
288 | + except (OSError, IOError) as e: |
289 | + LOG.warning("failed to truncate kvp pool file, %s", e) |
290 | + finally: |
291 | + cls._already_truncated_pool_file = True |
292 | + |
293 | def _get_incarnation_no(self): |
294 | """ |
295 | use the time passed as the incarnation number. |
296 | @@ -162,20 +188,15 @@ class HyperVKvpReportingHandler(ReportingHandler): |
297 | |
298 | def _iterate_kvps(self, offset): |
299 | """iterate the kvp file from the current offset.""" |
300 | - try: |
301 | - with open(self._kvp_file_path, 'rb+') as f: |
302 | - self.kvp_file = f |
303 | - fcntl.flock(f, fcntl.LOCK_EX) |
304 | - f.seek(offset) |
305 | + with open(self._kvp_file_path, 'rb') as f: |
306 | + fcntl.flock(f, fcntl.LOCK_EX) |
307 | + f.seek(offset) |
308 | + record_data = f.read(self.HV_KVP_RECORD_SIZE) |
309 | + while len(record_data) == self.HV_KVP_RECORD_SIZE: |
310 | + kvp_item = self._decode_kvp_item(record_data) |
311 | + yield kvp_item |
312 | record_data = f.read(self.HV_KVP_RECORD_SIZE) |
313 | - while len(record_data) == self.HV_KVP_RECORD_SIZE: |
314 | - self._current_offset += self.HV_KVP_RECORD_SIZE |
315 | - kvp_item = self._decode_kvp_item(record_data) |
316 | - yield kvp_item |
317 | - record_data = f.read(self.HV_KVP_RECORD_SIZE) |
318 | - fcntl.flock(f, fcntl.LOCK_UN) |
319 | - finally: |
320 | - self.kvp_file = None |
321 | + fcntl.flock(f, fcntl.LOCK_UN) |
322 | |
323 | def _event_key(self, event): |
324 | """ |
325 | @@ -207,23 +228,13 @@ class HyperVKvpReportingHandler(ReportingHandler): |
326 | |
327 | return {'key': k, 'value': v} |
328 | |
329 | - def _update_kvp_item(self, record_data): |
330 | - if self.kvp_file is None: |
331 | - raise ReportException( |
332 | - "kvp file '{0}' not opened." |
333 | - .format(self._kvp_file_path)) |
334 | - self.kvp_file.seek(-self.HV_KVP_RECORD_SIZE, 1) |
335 | - self.kvp_file.write(record_data) |
336 | - |
337 | def _append_kvp_item(self, record_data): |
338 | - with open(self._kvp_file_path, 'rb+') as f: |
339 | + with open(self._kvp_file_path, 'ab') as f: |
340 | fcntl.flock(f, fcntl.LOCK_EX) |
341 | - # seek to end of the file |
342 | - f.seek(0, 2) |
343 | - f.write(record_data) |
344 | + for data in record_data: |
345 | + f.write(data) |
346 | f.flush() |
347 | fcntl.flock(f, fcntl.LOCK_UN) |
348 | - self._current_offset = f.tell() |
349 | |
350 | def _break_down(self, key, meta_data, description): |
351 | del meta_data[self.MSG_KEY] |
352 | @@ -279,40 +290,26 @@ class HyperVKvpReportingHandler(ReportingHandler): |
353 | |
354 | def _publish_event_routine(self): |
355 | while True: |
356 | + items_from_queue = 0 |
357 | try: |
358 | event = self.q.get(block=True) |
359 | - need_append = True |
360 | + items_from_queue += 1 |
361 | + encoded_data = [] |
362 | + while event is not None: |
363 | + encoded_data += self._encode_event(event) |
364 | + try: |
365 | + # get all the rest of the events in the queue |
366 | + event = self.q.get(block=False) |
367 | + items_from_queue += 1 |
368 | + except QueueEmptyError: |
369 | + event = None |
370 | try: |
371 | - if not os.path.exists(self._kvp_file_path): |
372 | - LOG.warning( |
373 | - "skip writing events %s to %s. file not present.", |
374 | - event.as_string(), |
375 | - self._kvp_file_path) |
376 | - encoded_event = self._encode_event(event) |
377 | - # for each encoded_event |
378 | - for encoded_data in (encoded_event): |
379 | - for kvp in self._iterate_kvps(self._current_offset): |
380 | - match = ( |
381 | - re.match( |
382 | - r"^{0}\|(\d+)\|.+" |
383 | - .format(self.EVENT_PREFIX), |
384 | - kvp['key'] |
385 | - )) |
386 | - if match: |
387 | - match_groups = match.groups(0) |
388 | - if int(match_groups[0]) < self.incarnation_no: |
389 | - need_append = False |
390 | - self._update_kvp_item(encoded_data) |
391 | - continue |
392 | - if need_append: |
393 | - self._append_kvp_item(encoded_data) |
394 | - except IOError as e: |
395 | - LOG.warning( |
396 | - "failed posting event to kvp: %s e:%s", |
397 | - event.as_string(), e) |
398 | + self._append_kvp_item(encoded_data) |
399 | + except (OSError, IOError) as e: |
400 | + LOG.warning("failed posting events to kvp, %s", e) |
401 | finally: |
402 | - self.q.task_done() |
403 | - |
404 | + for _ in range(items_from_queue): |
405 | + self.q.task_done() |
406 | # when main process exits, q.get() will through EOFError |
407 | # indicating we should exit this thread. |
408 | except EOFError: |
409 | @@ -322,7 +319,7 @@ class HyperVKvpReportingHandler(ReportingHandler): |
410 | # if the kvp pool already contains a chunk of data, |
411 | # so defer it to another thread. |
412 | def publish_event(self, event): |
413 | - if (not self._event_types or event.event_type in self._event_types): |
414 | + if not self._event_types or event.event_type in self._event_types: |
415 | self.q.put(event) |
416 | |
417 | def flush(self): |
418 | diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py |
419 | index 76b1661..b7440c1 100755 |
420 | --- a/cloudinit/sources/DataSourceAzure.py |
421 | +++ b/cloudinit/sources/DataSourceAzure.py |
422 | @@ -57,7 +57,12 @@ AZURE_CHASSIS_ASSET_TAG = '7783-7084-3265-9085-8269-3286-77' |
423 | REPROVISION_MARKER_FILE = "/var/lib/cloud/data/poll_imds" |
424 | REPORTED_READY_MARKER_FILE = "/var/lib/cloud/data/reported_ready" |
425 | AGENT_SEED_DIR = '/var/lib/waagent' |
426 | + |
427 | +# In the event where the IMDS primary server is not |
428 | +# available, it takes 1s to fallback to the secondary one |
429 | +IMDS_TIMEOUT_IN_SECONDS = 2 |
430 | IMDS_URL = "http://169.254.169.254/metadata/" |
431 | + |
432 | PLATFORM_ENTROPY_SOURCE = "/sys/firmware/acpi/tables/OEM0" |
433 | |
434 | # List of static scripts and network config artifacts created by |
435 | @@ -407,7 +412,7 @@ class DataSourceAzure(sources.DataSource): |
436 | elif cdev.startswith("/dev/"): |
437 | if util.is_FreeBSD(): |
438 | ret = util.mount_cb(cdev, load_azure_ds_dir, |
439 | - mtype="udf", sync=False) |
440 | + mtype="udf") |
441 | else: |
442 | ret = util.mount_cb(cdev, load_azure_ds_dir) |
443 | else: |
444 | @@ -582,9 +587,9 @@ class DataSourceAzure(sources.DataSource): |
445 | return |
446 | self._ephemeral_dhcp_ctx.clean_network() |
447 | else: |
448 | - return readurl(url, timeout=1, headers=headers, |
449 | - exception_cb=exc_cb, infinite=True, |
450 | - log_req_resp=False).contents |
451 | + return readurl(url, timeout=IMDS_TIMEOUT_IN_SECONDS, |
452 | + headers=headers, exception_cb=exc_cb, |
453 | + infinite=True, log_req_resp=False).contents |
454 | except UrlError: |
455 | # Teardown our EphemeralDHCPv4 context on failure as we retry |
456 | self._ephemeral_dhcp_ctx.clean_network() |
457 | @@ -1291,8 +1296,8 @@ def _get_metadata_from_imds(retries): |
458 | headers = {"Metadata": "true"} |
459 | try: |
460 | response = readurl( |
461 | - url, timeout=1, headers=headers, retries=retries, |
462 | - exception_cb=retry_on_url_exc) |
463 | + url, timeout=IMDS_TIMEOUT_IN_SECONDS, headers=headers, |
464 | + retries=retries, exception_cb=retry_on_url_exc) |
465 | except Exception as e: |
466 | LOG.debug('Ignoring IMDS instance metadata: %s', e) |
467 | return {} |
468 | diff --git a/cloudinit/sources/DataSourceCloudStack.py b/cloudinit/sources/DataSourceCloudStack.py |
469 | index d4b758f..f185dc7 100644 |
470 | --- a/cloudinit/sources/DataSourceCloudStack.py |
471 | +++ b/cloudinit/sources/DataSourceCloudStack.py |
472 | @@ -95,7 +95,7 @@ class DataSourceCloudStack(sources.DataSource): |
473 | start_time = time.time() |
474 | url = uhelp.wait_for_url( |
475 | urls=urls, max_wait=url_params.max_wait_seconds, |
476 | - timeout=url_params.timeout_seconds, status_cb=LOG.warn) |
477 | + timeout=url_params.timeout_seconds, status_cb=LOG.warning) |
478 | |
479 | if url: |
480 | LOG.debug("Using metadata source: '%s'", url) |
481 | diff --git a/cloudinit/sources/DataSourceConfigDrive.py b/cloudinit/sources/DataSourceConfigDrive.py |
482 | index 564e3eb..571d30d 100644 |
483 | --- a/cloudinit/sources/DataSourceConfigDrive.py |
484 | +++ b/cloudinit/sources/DataSourceConfigDrive.py |
485 | @@ -72,15 +72,12 @@ class DataSourceConfigDrive(openstack.SourceMixin, sources.DataSource): |
486 | dslist = self.sys_cfg.get('datasource_list') |
487 | for dev in find_candidate_devs(dslist=dslist): |
488 | try: |
489 | - # Set mtype if freebsd and turn off sync |
490 | - if dev.startswith("/dev/cd"): |
491 | + if util.is_FreeBSD() and dev.startswith("/dev/cd"): |
492 | mtype = "cd9660" |
493 | - sync = False |
494 | else: |
495 | mtype = None |
496 | - sync = True |
497 | results = util.mount_cb(dev, read_config_drive, |
498 | - mtype=mtype, sync=sync) |
499 | + mtype=mtype) |
500 | found = dev |
501 | except openstack.NonReadable: |
502 | pass |
503 | diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourceEc2.py |
504 | index ac28f1d..5c017bf 100644 |
505 | --- a/cloudinit/sources/DataSourceEc2.py |
506 | +++ b/cloudinit/sources/DataSourceEc2.py |
507 | @@ -208,7 +208,7 @@ class DataSourceEc2(sources.DataSource): |
508 | start_time = time.time() |
509 | url = uhelp.wait_for_url( |
510 | urls=urls, max_wait=url_params.max_wait_seconds, |
511 | - timeout=url_params.timeout_seconds, status_cb=LOG.warn) |
512 | + timeout=url_params.timeout_seconds, status_cb=LOG.warning) |
513 | |
514 | if url: |
515 | self.metadata_address = url2base[url] |
516 | diff --git a/cloudinit/sources/helpers/azure.py b/cloudinit/sources/helpers/azure.py |
517 | index d3af05e..82c4c8c 100755 |
518 | --- a/cloudinit/sources/helpers/azure.py |
519 | +++ b/cloudinit/sources/helpers/azure.py |
520 | @@ -20,6 +20,9 @@ from cloudinit.reporting import events |
521 | |
522 | LOG = logging.getLogger(__name__) |
523 | |
524 | +# This endpoint matches the format as found in dhcp lease files, since this |
525 | +# value is applied if the endpoint can't be found within a lease file |
526 | +DEFAULT_WIRESERVER_ENDPOINT = "a8:3f:81:10" |
527 | |
528 | azure_ds_reporter = events.ReportEventStack( |
529 | name="azure-ds", |
530 | @@ -297,7 +300,12 @@ class WALinuxAgentShim(object): |
531 | @azure_ds_telemetry_reporter |
532 | def _get_value_from_leases_file(fallback_lease_file): |
533 | leases = [] |
534 | - content = util.load_file(fallback_lease_file) |
535 | + try: |
536 | + content = util.load_file(fallback_lease_file) |
537 | + except IOError as ex: |
538 | + LOG.error("Failed to read %s: %s", fallback_lease_file, ex) |
539 | + return None |
540 | + |
541 | LOG.debug("content is %s", content) |
542 | option_name = _get_dhcp_endpoint_option_name() |
543 | for line in content.splitlines(): |
544 | @@ -372,9 +380,9 @@ class WALinuxAgentShim(object): |
545 | fallback_lease_file) |
546 | value = WALinuxAgentShim._get_value_from_leases_file( |
547 | fallback_lease_file) |
548 | - |
549 | if value is None: |
550 | - raise ValueError('No endpoint found.') |
551 | + LOG.warning("No lease found; using default endpoint") |
552 | + value = DEFAULT_WIRESERVER_ENDPOINT |
553 | |
554 | endpoint_ip_address = WALinuxAgentShim.get_ip_from_lease_value(value) |
555 | LOG.debug('Azure endpoint found at %s', endpoint_ip_address) |
556 | diff --git a/cloudinit/util.py b/cloudinit/util.py |
557 | index 385f231..ea4199c 100644 |
558 | --- a/cloudinit/util.py |
559 | +++ b/cloudinit/util.py |
560 | @@ -1679,7 +1679,7 @@ def mounts(): |
561 | return mounted |
562 | |
563 | |
564 | -def mount_cb(device, callback, data=None, rw=False, mtype=None, sync=True, |
565 | +def mount_cb(device, callback, data=None, mtype=None, |
566 | update_env_for_mount=None): |
567 | """ |
568 | Mount the device, call method 'callback' passing the directory |
569 | @@ -1726,18 +1726,7 @@ def mount_cb(device, callback, data=None, rw=False, mtype=None, sync=True, |
570 | for mtype in mtypes: |
571 | mountpoint = None |
572 | try: |
573 | - mountcmd = ['mount'] |
574 | - mountopts = [] |
575 | - if rw: |
576 | - mountopts.append('rw') |
577 | - else: |
578 | - mountopts.append('ro') |
579 | - if sync: |
580 | - # This seems like the safe approach to do |
581 | - # (ie where this is on by default) |
582 | - mountopts.append("sync") |
583 | - if mountopts: |
584 | - mountcmd.extend(["-o", ",".join(mountopts)]) |
585 | + mountcmd = ['mount', '-o', 'ro'] |
586 | if mtype: |
587 | mountcmd.extend(['-t', mtype]) |
588 | mountcmd.append(device) |
589 | diff --git a/cloudinit/version.py b/cloudinit/version.py |
590 | index a2c5d43..ddcd436 100644 |
591 | --- a/cloudinit/version.py |
592 | +++ b/cloudinit/version.py |
593 | @@ -4,7 +4,7 @@ |
594 | # |
595 | # This file is part of cloud-init. See LICENSE file for license information. |
596 | |
597 | -__VERSION__ = "18.5" |
598 | +__VERSION__ = "19.1" |
599 | _PACKAGED_VERSION = '@@PACKAGED_VERSION@@' |
600 | |
601 | FEATURES = [ |
602 | diff --git a/debian/changelog b/debian/changelog |
603 | index 0630854..8379093 100644 |
604 | --- a/debian/changelog |
605 | +++ b/debian/changelog |
606 | @@ -1,3 +1,30 @@ |
607 | +cloud-init (19.1-1-gbaa47854-0ubuntu1~19.04.1) disco; urgency=medium |
608 | + |
609 | + * New upstream snapshot. |
610 | + - Azure: Return static fallback address as if failed to find endpoint |
611 | + [Jason Zions (MSFT)] |
612 | + - release 19.1 (LP: #1828479) |
613 | + - freebsd: add chpasswd pkg in the image [Gonéri Le Bouder] |
614 | + - tests: add Eoan release [Paride Legovini] |
615 | + - cc_mounts: check if mount -a on no-change fstab path |
616 | + [Jason Zions (MSFT)] (LP: #1825596) |
617 | + - replace remaining occurrences of LOG.warn |
618 | + - DataSourceAzure: Adjust timeout for polling IMDS [Anh Vo] |
619 | + - Azure: Changes to the Hyper-V KVP Reporter [Anh Vo] |
620 | + - git tests: no longer show warning about safe yaml. [Scott Moser] |
621 | + - tools/read-version: handle errors [Chad Miller] |
622 | + - net/sysconfig: only indicate available on known sysconfig distros |
623 | + (LP: #1819994) |
624 | + - packages: update rpm specs for new bash completion path (LP: #1825444) |
625 | + - test_azure: mock util.SeLinuxGuard where needed |
626 | + [Jason Zions (MSFT)] (LP: #1825253) |
627 | + - setup.py: install bash completion script in new location |
628 | + - mount_cb: do not pass sync and rw options to mount |
629 | + [Gonéri Le Bouder] (LP: #1645824) |
630 | + - cc_apt_configure: fix typo in apt documentation [Dominic Schlegel] |
631 | + |
632 | + -- Chad Smith <chad.smith@canonical.com> Fri, 10 May 2019 21:11:57 -0600 |
633 | + |
634 | cloud-init (18.5-62-g6322c2dd-0ubuntu1) disco; urgency=medium |
635 | |
636 | * New upstream snapshot. |
637 | diff --git a/packages/redhat/cloud-init.spec.in b/packages/redhat/cloud-init.spec.in |
638 | index 6b2022b..057a578 100644 |
639 | --- a/packages/redhat/cloud-init.spec.in |
640 | +++ b/packages/redhat/cloud-init.spec.in |
641 | @@ -205,7 +205,9 @@ fi |
642 | %dir %{_sysconfdir}/cloud/templates |
643 | %config(noreplace) %{_sysconfdir}/cloud/templates/* |
644 | %config(noreplace) %{_sysconfdir}/rsyslog.d/21-cloudinit.conf |
645 | -%{_sysconfdir}/bash_completion.d/cloud-init |
646 | + |
647 | +# Bash completion script |
648 | +%{_datadir}/bash-completion/completions/cloud-init |
649 | |
650 | %{_libexecdir}/%{name} |
651 | %dir %{_sharedstatedir}/cloud |
652 | diff --git a/packages/suse/cloud-init.spec.in b/packages/suse/cloud-init.spec.in |
653 | index 26894b3..004b875 100644 |
654 | --- a/packages/suse/cloud-init.spec.in |
655 | +++ b/packages/suse/cloud-init.spec.in |
656 | @@ -120,7 +120,9 @@ version_pys=$(cd "%{buildroot}" && find . -name version.py -type f) |
657 | %config(noreplace) %{_sysconfdir}/cloud/cloud.cfg.d/README |
658 | %dir %{_sysconfdir}/cloud/templates |
659 | %config(noreplace) %{_sysconfdir}/cloud/templates/* |
660 | -%{_sysconfdir}/bash_completion.d/cloud-init |
661 | + |
662 | +# Bash completion script |
663 | +%{_datadir}/bash-completion/completions/cloud-init |
664 | |
665 | %{_sysconfdir}/dhcp/dhclient-exit-hooks.d/hook-dhclient |
666 | %{_sysconfdir}/NetworkManager/dispatcher.d/hook-network-manager |
667 | diff --git a/setup.py b/setup.py |
668 | index 186e215..fcaf26f 100755 |
669 | --- a/setup.py |
670 | +++ b/setup.py |
671 | @@ -245,13 +245,14 @@ if not in_virtualenv(): |
672 | INITSYS_ROOTS[k] = "/" + INITSYS_ROOTS[k] |
673 | |
674 | data_files = [ |
675 | - (ETC + '/bash_completion.d', ['bash_completion/cloud-init']), |
676 | (ETC + '/cloud', [render_tmpl("config/cloud.cfg.tmpl")]), |
677 | (ETC + '/cloud/cloud.cfg.d', glob('config/cloud.cfg.d/*')), |
678 | (ETC + '/cloud/templates', glob('templates/*')), |
679 | (USR_LIB_EXEC + '/cloud-init', ['tools/ds-identify', |
680 | 'tools/uncloud-init', |
681 | 'tools/write-ssh-key-fingerprints']), |
682 | + (USR + '/share/bash-completion/completions', |
683 | + ['bash_completion/cloud-init']), |
684 | (USR + '/share/doc/cloud-init', [f for f in glob('doc/*') if is_f(f)]), |
685 | (USR + '/share/doc/cloud-init/examples', |
686 | [f for f in glob('doc/examples/*') if is_f(f)]), |
687 | diff --git a/tests/cloud_tests/releases.yaml b/tests/cloud_tests/releases.yaml |
688 | index ec5da72..924ad95 100644 |
689 | --- a/tests/cloud_tests/releases.yaml |
690 | +++ b/tests/cloud_tests/releases.yaml |
691 | @@ -129,6 +129,22 @@ features: |
692 | |
693 | releases: |
694 | # UBUNTU ================================================================= |
695 | + eoan: |
696 | + # EOL: Jul 2020 |
697 | + default: |
698 | + enabled: true |
699 | + release: eoan |
700 | + version: 19.10 |
701 | + os: ubuntu |
702 | + feature_groups: |
703 | + - base |
704 | + - debian_base |
705 | + - ubuntu_specific |
706 | + lxd: |
707 | + sstreams_server: https://cloud-images.ubuntu.com/daily |
708 | + alias: eoan |
709 | + setup_overrides: null |
710 | + override_templates: false |
711 | disco: |
712 | # EOL: Jan 2020 |
713 | default: |
714 | diff --git a/tests/unittests/test_datasource/test_azure.py b/tests/unittests/test_datasource/test_azure.py |
715 | index 53c56cd..427ab7e 100644 |
716 | --- a/tests/unittests/test_datasource/test_azure.py |
717 | +++ b/tests/unittests/test_datasource/test_azure.py |
718 | @@ -163,7 +163,8 @@ class TestGetMetadataFromIMDS(HttprettyTestCase): |
719 | |
720 | m_readurl.assert_called_with( |
721 | self.network_md_url, exception_cb=mock.ANY, |
722 | - headers={'Metadata': 'true'}, retries=2, timeout=1) |
723 | + headers={'Metadata': 'true'}, retries=2, |
724 | + timeout=dsaz.IMDS_TIMEOUT_IN_SECONDS) |
725 | |
726 | @mock.patch('cloudinit.url_helper.time.sleep') |
727 | @mock.patch(MOCKPATH + 'net.is_up') |
728 | @@ -1375,12 +1376,15 @@ class TestCanDevBeReformatted(CiTestCase): |
729 | self._domock(p + "util.mount_cb", 'm_mount_cb') |
730 | self._domock(p + "os.path.realpath", 'm_realpath') |
731 | self._domock(p + "os.path.exists", 'm_exists') |
732 | + self._domock(p + "util.SeLinuxGuard", 'm_selguard') |
733 | |
734 | self.m_exists.side_effect = lambda p: p in bypath |
735 | self.m_realpath.side_effect = realpath |
736 | self.m_has_ntfs_filesystem.side_effect = has_ntfs_fs |
737 | self.m_mount_cb.side_effect = mount_cb |
738 | self.m_partitions_on_device.side_effect = partitions_on_device |
739 | + self.m_selguard.__enter__ = mock.Mock(return_value=False) |
740 | + self.m_selguard.__exit__ = mock.Mock() |
741 | |
742 | def test_three_partitions_is_false(self): |
743 | """A disk with 3 partitions can not be formatted.""" |
744 | @@ -1788,7 +1792,8 @@ class TestAzureDataSourcePreprovisioning(CiTestCase): |
745 | headers={'Metadata': 'true', |
746 | 'User-Agent': |
747 | 'Cloud-Init/%s' % vs() |
748 | - }, method='GET', timeout=1, |
749 | + }, method='GET', |
750 | + timeout=dsaz.IMDS_TIMEOUT_IN_SECONDS, |
751 | url=full_url)]) |
752 | self.assertEqual(m_dhcp.call_count, 2) |
753 | m_net.assert_any_call( |
754 | @@ -1825,7 +1830,9 @@ class TestAzureDataSourcePreprovisioning(CiTestCase): |
755 | headers={'Metadata': 'true', |
756 | 'User-Agent': |
757 | 'Cloud-Init/%s' % vs()}, |
758 | - method='GET', timeout=1, url=full_url)]) |
759 | + method='GET', |
760 | + timeout=dsaz.IMDS_TIMEOUT_IN_SECONDS, |
761 | + url=full_url)]) |
762 | self.assertEqual(m_dhcp.call_count, 2) |
763 | m_net.assert_any_call( |
764 | broadcast='192.168.2.255', interface='eth9', ip='192.168.2.9', |
765 | diff --git a/tests/unittests/test_datasource/test_azure_helper.py b/tests/unittests/test_datasource/test_azure_helper.py |
766 | index 0255616..bd006ab 100644 |
767 | --- a/tests/unittests/test_datasource/test_azure_helper.py |
768 | +++ b/tests/unittests/test_datasource/test_azure_helper.py |
769 | @@ -67,12 +67,17 @@ class TestFindEndpoint(CiTestCase): |
770 | self.networkd_leases.return_value = None |
771 | |
772 | def test_missing_file(self): |
773 | - self.assertRaises(ValueError, wa_shim.find_endpoint) |
774 | + """wa_shim find_endpoint uses default endpoint if leasefile not found |
775 | + """ |
776 | + self.assertEqual(wa_shim.find_endpoint(), "168.63.129.16") |
777 | |
778 | def test_missing_special_azure_line(self): |
779 | + """wa_shim find_endpoint uses default endpoint if leasefile is found |
780 | + but does not contain DHCP Option 245 (whose value is the endpoint) |
781 | + """ |
782 | self.load_file.return_value = '' |
783 | self.dhcp_options.return_value = {'eth0': {'key': 'value'}} |
784 | - self.assertRaises(ValueError, wa_shim.find_endpoint) |
785 | + self.assertEqual(wa_shim.find_endpoint(), "168.63.129.16") |
786 | |
787 | @staticmethod |
788 | def _build_lease_content(encoded_address): |
789 | diff --git a/tests/unittests/test_handler/test_handler_mounts.py b/tests/unittests/test_handler/test_handler_mounts.py |
790 | index 8fea6c2..0fb160b 100644 |
791 | --- a/tests/unittests/test_handler/test_handler_mounts.py |
792 | +++ b/tests/unittests/test_handler/test_handler_mounts.py |
793 | @@ -154,7 +154,15 @@ class TestFstabHandling(test_helpers.FilesystemMockingTestCase): |
794 | return_value=True) |
795 | |
796 | self.add_patch('cloudinit.config.cc_mounts.util.subp', |
797 | - 'mock_util_subp') |
798 | + 'm_util_subp') |
799 | + |
800 | + self.add_patch('cloudinit.config.cc_mounts.util.mounts', |
801 | + 'mock_util_mounts', |
802 | + return_value={ |
803 | + '/dev/sda1': {'fstype': 'ext4', |
804 | + 'mountpoint': '/', |
805 | + 'opts': 'rw,relatime,discard' |
806 | + }}) |
807 | |
808 | self.mock_cloud = mock.Mock() |
809 | self.mock_log = mock.Mock() |
810 | @@ -230,4 +238,24 @@ class TestFstabHandling(test_helpers.FilesystemMockingTestCase): |
811 | fstab_new_content = fd.read() |
812 | self.assertEqual(fstab_expected_content, fstab_new_content) |
813 | |
814 | + def test_no_change_fstab_sets_needs_mount_all(self): |
815 | + '''verify unchanged fstab entries are mounted if not call mount -a''' |
816 | + fstab_original_content = ( |
817 | + 'LABEL=cloudimg-rootfs / ext4 defaults 0 0\n' |
818 | + 'LABEL=UEFI /boot/efi vfat defaults 0 0\n' |
819 | + '/dev/vdb /mnt auto defaults,noexec,comment=cloudconfig 0 2\n' |
820 | + ) |
821 | + fstab_expected_content = fstab_original_content |
822 | + cc = {'mounts': [ |
823 | + ['/dev/vdb', '/mnt', 'auto', 'defaults,noexec']]} |
824 | + with open(cc_mounts.FSTAB_PATH, 'w') as fd: |
825 | + fd.write(fstab_original_content) |
826 | + with open(cc_mounts.FSTAB_PATH, 'r') as fd: |
827 | + fstab_new_content = fd.read() |
828 | + self.assertEqual(fstab_expected_content, fstab_new_content) |
829 | + cc_mounts.handle(None, cc, self.mock_cloud, self.mock_log, []) |
830 | + self.m_util_subp.assert_has_calls([ |
831 | + mock.call(['mount', '-a']), |
832 | + mock.call(['systemctl', 'daemon-reload'])]) |
833 | + |
834 | # vi: ts=4 expandtab |
835 | diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py |
836 | index fd03deb..e85e964 100644 |
837 | --- a/tests/unittests/test_net.py |
838 | +++ b/tests/unittests/test_net.py |
839 | @@ -9,6 +9,7 @@ from cloudinit.net import ( |
840 | from cloudinit.sources.helpers import openstack |
841 | from cloudinit import temp_utils |
842 | from cloudinit import util |
843 | +from cloudinit import safeyaml as yaml |
844 | |
845 | from cloudinit.tests.helpers import ( |
846 | CiTestCase, FilesystemMockingTestCase, dir2dict, mock, populate_dir) |
847 | @@ -21,7 +22,7 @@ import json |
848 | import os |
849 | import re |
850 | import textwrap |
851 | -import yaml |
852 | +from yaml.serializer import Serializer |
853 | |
854 | |
855 | DHCP_CONTENT_1 = """ |
856 | @@ -3269,9 +3270,12 @@ class TestNetplanPostcommands(CiTestCase): |
857 | mock_netplan_generate.assert_called_with(run=True) |
858 | mock_net_setup_link.assert_called_with(run=True) |
859 | |
860 | + @mock.patch('cloudinit.util.SeLinuxGuard') |
861 | @mock.patch.object(netplan, "get_devicelist") |
862 | @mock.patch('cloudinit.util.subp') |
863 | - def test_netplan_postcmds(self, mock_subp, mock_devlist): |
864 | + def test_netplan_postcmds(self, mock_subp, mock_devlist, mock_sel): |
865 | + mock_sel.__enter__ = mock.Mock(return_value=False) |
866 | + mock_sel.__exit__ = mock.Mock() |
867 | mock_devlist.side_effect = [['lo']] |
868 | tmp_dir = self.tmp_dir() |
869 | ns = network_state.parse_net_config_data(self.mycfg, |
870 | @@ -3572,7 +3576,7 @@ class TestNetplanRoundTrip(CiTestCase): |
871 | # now look for any alias, avoid rendering them entirely |
872 | # generate the first anchor string using the template |
873 | # as of this writing, looks like "&id001" |
874 | - anchor = r'&' + yaml.serializer.Serializer.ANCHOR_TEMPLATE % 1 |
875 | + anchor = r'&' + Serializer.ANCHOR_TEMPLATE % 1 |
876 | found_alias = re.search(anchor, content, re.MULTILINE) |
877 | if found_alias: |
878 | msg = "Error at: %s\nContent:\n%s" % (found_alias, content) |
879 | @@ -3826,6 +3830,41 @@ class TestNetRenderers(CiTestCase): |
880 | self.assertRaises(net.RendererNotFoundError, renderers.select, |
881 | priority=['sysconfig', 'eni']) |
882 | |
883 | + @mock.patch("cloudinit.net.renderers.netplan.available") |
884 | + @mock.patch("cloudinit.net.renderers.sysconfig.available_sysconfig") |
885 | + @mock.patch("cloudinit.net.renderers.sysconfig.available_nm") |
886 | + @mock.patch("cloudinit.net.renderers.eni.available") |
887 | + @mock.patch("cloudinit.net.renderers.sysconfig.util.get_linux_distro") |
888 | + def test_sysconfig_selected_on_sysconfig_enabled_distros(self, m_distro, |
889 | + m_eni, m_sys_nm, |
890 | + m_sys_scfg, |
891 | + m_netplan): |
892 | + """sysconfig only selected on specific distros (rhel/sles).""" |
893 | + |
894 | + # Ubuntu with Network-Manager installed |
895 | + m_eni.return_value = False # no ifupdown (ifquery) |
896 | + m_sys_scfg.return_value = False # no sysconfig/ifup/ifdown |
897 | + m_sys_nm.return_value = True # network-manager is installed |
898 | + m_netplan.return_value = True # netplan is installed |
899 | + m_distro.return_value = ('ubuntu', None, None) |
900 | + self.assertEqual('netplan', renderers.select(priority=None)[0]) |
901 | + |
902 | + # Centos with Network-Manager installed |
903 | + m_eni.return_value = False # no ifupdown (ifquery) |
904 | + m_sys_scfg.return_value = False # no sysconfig/ifup/ifdown |
905 | + m_sys_nm.return_value = True # network-manager is installed |
906 | + m_netplan.return_value = False # netplan is not installed |
907 | + m_distro.return_value = ('centos', None, None) |
908 | + self.assertEqual('sysconfig', renderers.select(priority=None)[0]) |
909 | + |
910 | + # OpenSuse with Network-Manager installed |
911 | + m_eni.return_value = False # no ifupdown (ifquery) |
912 | + m_sys_scfg.return_value = False # no sysconfig/ifup/ifdown |
913 | + m_sys_nm.return_value = True # network-manager is installed |
914 | + m_netplan.return_value = False # netplan is not installed |
915 | + m_distro.return_value = ('opensuse', None, None) |
916 | + self.assertEqual('sysconfig', renderers.select(priority=None)[0]) |
917 | + |
918 | |
919 | class TestGetInterfaces(CiTestCase): |
920 | _data = {'bonds': ['bond1'], |
921 | diff --git a/tests/unittests/test_reporting_hyperv.py b/tests/unittests/test_reporting_hyperv.py |
922 | old mode 100644 |
923 | new mode 100755 |
924 | index 2e64c6c..d01ed5b |
925 | --- a/tests/unittests/test_reporting_hyperv.py |
926 | +++ b/tests/unittests/test_reporting_hyperv.py |
927 | @@ -1,10 +1,12 @@ |
928 | # This file is part of cloud-init. See LICENSE file for license information. |
929 | |
930 | from cloudinit.reporting import events |
931 | -from cloudinit.reporting import handlers |
932 | +from cloudinit.reporting.handlers import HyperVKvpReportingHandler |
933 | |
934 | import json |
935 | import os |
936 | +import struct |
937 | +import time |
938 | |
939 | from cloudinit import util |
940 | from cloudinit.tests.helpers import CiTestCase |
941 | @@ -13,7 +15,7 @@ from cloudinit.tests.helpers import CiTestCase |
942 | class TestKvpEncoding(CiTestCase): |
943 | def test_encode_decode(self): |
944 | kvp = {'key': 'key1', 'value': 'value1'} |
945 | - kvp_reporting = handlers.HyperVKvpReportingHandler() |
946 | + kvp_reporting = HyperVKvpReportingHandler() |
947 | data = kvp_reporting._encode_kvp_item(kvp['key'], kvp['value']) |
948 | self.assertEqual(len(data), kvp_reporting.HV_KVP_RECORD_SIZE) |
949 | decoded_kvp = kvp_reporting._decode_kvp_item(data) |
950 | @@ -26,57 +28,9 @@ class TextKvpReporter(CiTestCase): |
951 | self.tmp_file_path = self.tmp_path('kvp_pool_file') |
952 | util.ensure_file(self.tmp_file_path) |
953 | |
954 | - def test_event_type_can_be_filtered(self): |
955 | - reporter = handlers.HyperVKvpReportingHandler( |
956 | - kvp_file_path=self.tmp_file_path, |
957 | - event_types=['foo', 'bar']) |
958 | - |
959 | - reporter.publish_event( |
960 | - events.ReportingEvent('foo', 'name', 'description')) |
961 | - reporter.publish_event( |
962 | - events.ReportingEvent('some_other', 'name', 'description3')) |
963 | - reporter.q.join() |
964 | - |
965 | - kvps = list(reporter._iterate_kvps(0)) |
966 | - self.assertEqual(1, len(kvps)) |
967 | - |
968 | - reporter.publish_event( |
969 | - events.ReportingEvent('bar', 'name', 'description2')) |
970 | - reporter.q.join() |
971 | - kvps = list(reporter._iterate_kvps(0)) |
972 | - self.assertEqual(2, len(kvps)) |
973 | - |
974 | - self.assertIn('foo', kvps[0]['key']) |
975 | - self.assertIn('bar', kvps[1]['key']) |
976 | - self.assertNotIn('some_other', kvps[0]['key']) |
977 | - self.assertNotIn('some_other', kvps[1]['key']) |
978 | - |
979 | - def test_events_are_over_written(self): |
980 | - reporter = handlers.HyperVKvpReportingHandler( |
981 | - kvp_file_path=self.tmp_file_path) |
982 | - |
983 | - self.assertEqual(0, len(list(reporter._iterate_kvps(0)))) |
984 | - |
985 | - reporter.publish_event( |
986 | - events.ReportingEvent('foo', 'name1', 'description')) |
987 | - reporter.publish_event( |
988 | - events.ReportingEvent('foo', 'name2', 'description')) |
989 | - reporter.q.join() |
990 | - self.assertEqual(2, len(list(reporter._iterate_kvps(0)))) |
991 | - |
992 | - reporter2 = handlers.HyperVKvpReportingHandler( |
993 | - kvp_file_path=self.tmp_file_path) |
994 | - reporter2.incarnation_no = reporter.incarnation_no + 1 |
995 | - reporter2.publish_event( |
996 | - events.ReportingEvent('foo', 'name3', 'description')) |
997 | - reporter2.q.join() |
998 | - |
999 | - self.assertEqual(2, len(list(reporter2._iterate_kvps(0)))) |
1000 | - |
1001 | def test_events_with_higher_incarnation_not_over_written(self): |
1002 | - reporter = handlers.HyperVKvpReportingHandler( |
1003 | + reporter = HyperVKvpReportingHandler( |
1004 | kvp_file_path=self.tmp_file_path) |
1005 | - |
1006 | self.assertEqual(0, len(list(reporter._iterate_kvps(0)))) |
1007 | |
1008 | reporter.publish_event( |
1009 | @@ -86,7 +40,7 @@ class TextKvpReporter(CiTestCase): |
1010 | reporter.q.join() |
1011 | self.assertEqual(2, len(list(reporter._iterate_kvps(0)))) |
1012 | |
1013 | - reporter3 = handlers.HyperVKvpReportingHandler( |
1014 | + reporter3 = HyperVKvpReportingHandler( |
1015 | kvp_file_path=self.tmp_file_path) |
1016 | reporter3.incarnation_no = reporter.incarnation_no - 1 |
1017 | reporter3.publish_event( |
1018 | @@ -95,7 +49,7 @@ class TextKvpReporter(CiTestCase): |
1019 | self.assertEqual(3, len(list(reporter3._iterate_kvps(0)))) |
1020 | |
1021 | def test_finish_event_result_is_logged(self): |
1022 | - reporter = handlers.HyperVKvpReportingHandler( |
1023 | + reporter = HyperVKvpReportingHandler( |
1024 | kvp_file_path=self.tmp_file_path) |
1025 | reporter.publish_event( |
1026 | events.FinishReportingEvent('name2', 'description1', |
1027 | @@ -105,7 +59,7 @@ class TextKvpReporter(CiTestCase): |
1028 | |
1029 | def test_file_operation_issue(self): |
1030 | os.remove(self.tmp_file_path) |
1031 | - reporter = handlers.HyperVKvpReportingHandler( |
1032 | + reporter = HyperVKvpReportingHandler( |
1033 | kvp_file_path=self.tmp_file_path) |
1034 | reporter.publish_event( |
1035 | events.FinishReportingEvent('name2', 'description1', |
1036 | @@ -113,7 +67,7 @@ class TextKvpReporter(CiTestCase): |
1037 | reporter.q.join() |
1038 | |
1039 | def test_event_very_long(self): |
1040 | - reporter = handlers.HyperVKvpReportingHandler( |
1041 | + reporter = HyperVKvpReportingHandler( |
1042 | kvp_file_path=self.tmp_file_path) |
1043 | description = 'ab' * reporter.HV_KVP_EXCHANGE_MAX_VALUE_SIZE |
1044 | long_event = events.FinishReportingEvent( |
1045 | @@ -132,3 +86,43 @@ class TextKvpReporter(CiTestCase): |
1046 | self.assertEqual(msg_slice['msg_i'], i) |
1047 | full_description += msg_slice['msg'] |
1048 | self.assertEqual(description, full_description) |
1049 | + |
1050 | + def test_not_truncate_kvp_file_modified_after_boot(self): |
1051 | + with open(self.tmp_file_path, "wb+") as f: |
1052 | + kvp = {'key': 'key1', 'value': 'value1'} |
1053 | + data = (struct.pack("%ds%ds" % ( |
1054 | + HyperVKvpReportingHandler.HV_KVP_EXCHANGE_MAX_KEY_SIZE, |
1055 | + HyperVKvpReportingHandler.HV_KVP_EXCHANGE_MAX_VALUE_SIZE), |
1056 | + kvp['key'].encode('utf-8'), kvp['value'].encode('utf-8'))) |
1057 | + f.write(data) |
1058 | + cur_time = time.time() |
1059 | + os.utime(self.tmp_file_path, (cur_time, cur_time)) |
1060 | + |
1061 | + # reset this because the unit test framework |
1062 | + # has already polluted the class variable |
1063 | + HyperVKvpReportingHandler._already_truncated_pool_file = False |
1064 | + |
1065 | + reporter = HyperVKvpReportingHandler(kvp_file_path=self.tmp_file_path) |
1066 | + kvps = list(reporter._iterate_kvps(0)) |
1067 | + self.assertEqual(1, len(kvps)) |
1068 | + |
1069 | + def test_truncate_stale_kvp_file(self): |
1070 | + with open(self.tmp_file_path, "wb+") as f: |
1071 | + kvp = {'key': 'key1', 'value': 'value1'} |
1072 | + data = (struct.pack("%ds%ds" % ( |
1073 | + HyperVKvpReportingHandler.HV_KVP_EXCHANGE_MAX_KEY_SIZE, |
1074 | + HyperVKvpReportingHandler.HV_KVP_EXCHANGE_MAX_VALUE_SIZE), |
1075 | + kvp['key'].encode('utf-8'), kvp['value'].encode('utf-8'))) |
1076 | + f.write(data) |
1077 | + |
1078 | + # set the time ways back to make it look like |
1079 | + # we had an old kvp file |
1080 | + os.utime(self.tmp_file_path, (1000000, 1000000)) |
1081 | + |
1082 | + # reset this because the unit test framework |
1083 | + # has already polluted the class variable |
1084 | + HyperVKvpReportingHandler._already_truncated_pool_file = False |
1085 | + |
1086 | + reporter = HyperVKvpReportingHandler(kvp_file_path=self.tmp_file_path) |
1087 | + kvps = list(reporter._iterate_kvps(0)) |
1088 | + self.assertEqual(0, len(kvps)) |
1089 | diff --git a/tools/build-on-freebsd b/tools/build-on-freebsd |
1090 | index d23fde2..dc3b974 100755 |
1091 | --- a/tools/build-on-freebsd |
1092 | +++ b/tools/build-on-freebsd |
1093 | @@ -9,6 +9,7 @@ fail() { echo "FAILED:" "$@" 1>&2; exit 1; } |
1094 | depschecked=/tmp/c-i.dependencieschecked |
1095 | pkgs=" |
1096 | bash |
1097 | + chpasswd |
1098 | dmidecode |
1099 | e2fsprogs |
1100 | py27-Jinja2 |
1101 | @@ -17,6 +18,7 @@ pkgs=" |
1102 | py27-configobj |
1103 | py27-jsonpatch |
1104 | py27-jsonpointer |
1105 | + py27-jsonschema |
1106 | py27-oauthlib |
1107 | py27-requests |
1108 | py27-serial |
1109 | @@ -28,12 +30,9 @@ pkgs=" |
1110 | [ -f "$depschecked" ] || pkg install ${pkgs} || fail "install packages" |
1111 | touch $depschecked |
1112 | |
1113 | -# Required but unavailable port/pkg: py27-jsonpatch py27-jsonpointer |
1114 | -# Luckily, the install step will take care of this by installing it from pypi... |
1115 | - |
1116 | # Build the code and install in /usr/local/: |
1117 | -python setup.py build |
1118 | -python setup.py install -O1 --skip-build --prefix /usr/local/ --init-system sysvinit_freebsd |
1119 | +python2.7 setup.py build |
1120 | +python2.7 setup.py install -O1 --skip-build --prefix /usr/local/ --init-system sysvinit_freebsd |
1121 | |
1122 | # Enable cloud-init in /etc/rc.conf: |
1123 | sed -i.bak -e "/cloudinit_enable=.*/d" /etc/rc.conf |
1124 | diff --git a/tools/read-version b/tools/read-version |
1125 | index e69c2ce..6dca659 100755 |
1126 | --- a/tools/read-version |
1127 | +++ b/tools/read-version |
1128 | @@ -71,9 +71,12 @@ if is_gitdir(_tdir) and which("git"): |
1129 | flags = ['--tags'] |
1130 | cmd = ['git', 'describe', '--abbrev=8', '--match=[0-9]*'] + flags |
1131 | |
1132 | - version = tiny_p(cmd).strip() |
1133 | + try: |
1134 | + version = tiny_p(cmd).strip() |
1135 | + except RuntimeError: |
1136 | + version = None |
1137 | |
1138 | - if not version.startswith(src_version): |
1139 | + if version is None or not version.startswith(src_version): |
1140 | sys.stderr.write("git describe version (%s) differs from " |
1141 | "cloudinit.version (%s)\n" % (version, src_version)) |
1142 | sys.stderr.write( |
PASSED: Continuous integration, rev:59835f9211e 70101bde10c63c7 e84cc6a74c930c /jenkins. ubuntu. com/server/ job/cloud- init-ci/ 720/
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild: /jenkins. ubuntu. com/server/ job/cloud- init-ci/ 720/rebuild
https:/