Merge lp:~smoser/cloud-init/trunk.fix-networking into lp:~cloud-init-dev/cloud-init/trunk
- trunk.fix-networking
- Merge into trunk
Status: | Merged |
---|---|
Merged at revision: | 1225 |
Proposed branch: | lp:~smoser/cloud-init/trunk.fix-networking |
Merge into: | lp:~cloud-init-dev/cloud-init/trunk |
Diff against target: |
2765 lines (+1142/-688) 22 files modified
ChangeLog (+2/-0) bin/cloud-init (+38/-22) cloudinit/config/cc_emit_upstart.py (+1/-1) cloudinit/config/cc_lxd.py (+2/-1) cloudinit/distros/__init__.py (+6/-0) cloudinit/helpers.py (+12/-10) cloudinit/net/__init__.py (+221/-1) cloudinit/sources/DataSourceCloudSigma.py (+5/-14) cloudinit/sources/DataSourceConfigDrive.py (+70/-86) cloudinit/sources/DataSourceNoCloud.py (+35/-45) cloudinit/sources/DataSourceOpenNebula.py (+9/-35) cloudinit/sources/DataSourceOpenStack.py (+2/-7) cloudinit/sources/DataSourceSmartOS.py (+420/-173) cloudinit/sources/__init__.py (+33/-0) cloudinit/sources/helpers/openstack.py (+11/-7) cloudinit/stages.py (+70/-25) setup.py (+0/-1) systemd/cloud-init-generator (+0/-3) tests/unittests/test_datasource/test_configdrive.py (+77/-3) tests/unittests/test_datasource/test_smartos.py (+128/-174) udev/79-cloud-init-net-wait.rules (+0/-10) udev/cloud-init-wait (+0/-70) |
To merge this branch: | bzr merge lp:~smoser/cloud-init/trunk.fix-networking |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
cloud-init Commiters | Pending | ||
Review via email: mp+296272@code.launchpad.net |
Commit message
Description of the change
I'll improve this description later.
DataSource Mode (dsmode) is present in many datasources in cloud-init.
dsmode was originally added to cloud-init to specify when this datasource
should be 'realized'.
cloud-init has 4 stages of boot.
a.) cloud-init --local . network is guaranteed not present.
b.) cloud-init (--network). network is guaranteed present.
c.) cloud-config
d.) cloud-init final
'init_modules' [1] are run "as early as possible". And as such, are executed
in either 'a' or 'b' based on the datasource. However, executing them means
that user-data has been fully consumed. User-data and vendor-data may have
'#include http://
are an example of the things run in init_modules.
The 'dsmode' was a way for a user to indicate that init_modules
should run at 'a' (dsmode=local) or 'b' (dsmode=net) directly.
Things were further confused when a datasource could provide networking
configuration. Then, we needed to apply the networking config at 'a'
but if the user had provided boothooks that expected networking, then the
init_modules would need to be executed at 'b'. The config drive datasource
hacked its way through this and applies networking if *it* detects it is
a new instance.
== Suggested Change ==
The plan is to
1. incorporate 'dsmode' into DataSource superclass
2. make all existing datasources default to network
3. apply any networking configuration from a datasource on first boot only
apply_
for bug 1579130.
4. run init_modules at cloud-init (network) time frame unless datasource
is 'local'.
5. Datasources can provide a 'first_boot' method that will be called when
a new instance_id is found. This will allow the config drive's write_files
to be applied once.
Over all, this will very much simplify things. We'll no longer have
2 sources like DataSourceNoCloud and DataSourceNoClo
have one source with a dsmode.
== Concerns ==
Some things have odd reliance on dsmode. For example, OpenNebula's get_hostname
uses it to determine if it should do a lookup of an ip address.
== Bugs to fix here ==
http://
http://
http://
httpP/
- 1241. By Scott Moser
-
revert unintended change to tox.ini
- 1242. By Scott Moser
-
SmartOS: datasource improvements, support for networking information.
This adds support for reading networking information from the
SmartOS metadata service and applying. - 1243. By Scott Moser
-
fix log message in emit_upstart
- 1244. By Scott Moser
-
fix tox
- 1245. By Scott Moser
-
openstack: support decoding when reading files, use that for network_config
The network config file is /etc/network/
interfaces formated.
We will decode that here so that the user can expect that it is
a string. The issue was that it was bytes but convert_eni_data
was expecting a string. - 1246. By Scott Moser
-
eni parsing: support 'ether' in hwaddress, netmask and broadcast
this adds ability to support ENI that has:
hwadress ether 36:4c:e1:3b:14:31
or
hwaddress 36:4c:e1:3b:14:31the former is written by openstack (at least on dreamhost).
Also, in the conversion of eni to network config support broadcast
and netmask. - 1247. By Scott Moser
-
smartos: do not raise error when not on smartos
if get_smartos_
environ( ) returned a None, then
the datasoure would raise a ValueError when get_data was called.
Fix that. - 1248. By Scott Moser
-
fix untested previous change to smartos
- 1249. By Scott Moser
-
re-add the 'Net' classes for datasources
When the .pkl file is loaded, the module that it is loaded
from must have the same symbol. Ie, if booted once and got
DataSourceConfigDriveNet
then upgraded and rebooted, then next boot would show
Can't get attribute 'DataSourceConfigDriveNet' - 1250. By Scott Moser
-
merge with trunk
Scott Moser (smoser) wrote : | # |
Scott Moser (smoser) wrote : | # |
the one change left here is i think to make openstack config drive datasource not use 'id' as the nic namne as those are arbitrary.
Scott Moser (smoser) wrote : | # |
this is available at https:/
- 1251. By Scott Moser
-
ConfigDrive: do not use 'id' on a link for the device name
'id' on a link in the openstack spec should be "Generic, generated ID".
current implementation was to use the host's name for the host
side nic. Which provided names like 'tap-adfasdffd'.We do not want to name devices like that as its quite unexpected
and non user friendly. So here we use the system name for any
nic that is present, but then require that the nics found also
be present at the time of rendering.The end result is that if the system boots with net.ifnames=0
then it will get 'eth0' like names. and if it boots without net.ifnames
then it will get enp0s1 like names. - 1252. By Scott Moser
-
lxd: fix log messsage
Joshua Harlow (harlowja) wrote : | # |
Can u add a part of the description around 'def rename_interfaces' so that people know why this is needed, be much appreciated :)
- 1253. By Scott Moser
-
avoid rendering 'lo' twice by not writing it in network config.
- 1254. By Scott Moser
-
fix issue with routes on subnets not getting rendered
- 1255. By Scott Moser
-
config drive conversion: recognize 'bridge' as a physical type, fix mtu
the network json in openstack provides a type of 'bridge' when
the underlying (host) type is a bridge. Silly, but we need to
consider that a physical device as it will be for us.also, the 'mtu' will appear on the link, not on the route
Preview Diff
1 | === modified file 'ChangeLog' |
2 | --- ChangeLog 2016-06-02 16:32:14 +0000 |
3 | +++ ChangeLog 2016-06-03 19:07:08 +0000 |
4 | @@ -113,6 +113,8 @@ |
5 | - settings on the kernel command line (cc:) override all local settings |
6 | rather than only those in /etc/cloud/cloud.cfg (LP: #1582323) |
7 | - Improve merging documentation [Daniel Watkins] |
8 | + - SmartOS: datasource improvements and support for metadata service |
9 | + providing networking information. |
10 | |
11 | 0.7.6: |
12 | - open 0.7.6 |
13 | |
14 | === modified file 'bin/cloud-init' |
15 | --- bin/cloud-init 2016-04-15 17:54:05 +0000 |
16 | +++ bin/cloud-init 2016-06-03 19:07:08 +0000 |
17 | @@ -211,27 +211,27 @@ |
18 | util.logexc(LOG, "Failed to initialize, likely bad things to come!") |
19 | # Stage 4 |
20 | path_helper = init.paths |
21 | - if not args.local: |
22 | + mode = sources.DSMODE_LOCAL if args.local else sources.DSMODE_NETWORK |
23 | + |
24 | + if mode == sources.DSMODE_NETWORK: |
25 | existing = "trust" |
26 | sys.stderr.write("%s\n" % (netinfo.debug_info())) |
27 | LOG.debug(("Checking to see if files that we need already" |
28 | " exist from a previous run that would allow us" |
29 | " to stop early.")) |
30 | + # no-net is written by upstart cloud-init-nonet when network failed |
31 | + # to come up |
32 | stop_files = [ |
33 | os.path.join(path_helper.get_cpath("data"), "no-net"), |
34 | - path_helper.get_ipath_cur("obj_pkl"), |
35 | ] |
36 | existing_files = [] |
37 | for fn in stop_files: |
38 | - try: |
39 | - c = util.load_file(fn) |
40 | - if len(c): |
41 | - existing_files.append((fn, len(c))) |
42 | - except Exception: |
43 | - pass |
44 | + if os.path.isfile(fn): |
45 | + existing_files.append(fn) |
46 | + |
47 | if existing_files: |
48 | - LOG.debug("Exiting early due to the existence of %s files", |
49 | - existing_files) |
50 | + LOG.debug("[%s] Exiting. stop file %s existed", |
51 | + mode, existing_files) |
52 | return (None, []) |
53 | else: |
54 | LOG.debug("Execution continuing, no previous run detected that" |
55 | @@ -248,34 +248,50 @@ |
56 | # Stage 5 |
57 | try: |
58 | init.fetch(existing=existing) |
59 | + # if in network mode, and the datasource is local |
60 | + # then work was done at that stage. |
61 | + if mode == sources.DSMODE_NETWORK and init.datasource.dsmode != mode: |
62 | + LOG.debug("[%s] Exiting. datasource %s in local mode", |
63 | + mode, init.datasource) |
64 | + return (None, []) |
65 | except sources.DataSourceNotFoundException: |
66 | # In the case of 'cloud-init init' without '--local' it is a bit |
67 | # more likely that the user would consider it failure if nothing was |
68 | # found. When using upstart it will also mentions job failure |
69 | # in console log if exit code is != 0. |
70 | - if args.local: |
71 | + if mode == sources.DSMODE_LOCAL: |
72 | LOG.debug("No local datasource found") |
73 | else: |
74 | util.logexc(LOG, ("No instance datasource found!" |
75 | " Likely bad things to come!")) |
76 | if not args.force: |
77 | - init.apply_network_config() |
78 | - if args.local: |
79 | + init.apply_network_config(bring_up=not args.local) |
80 | + LOG.debug("[%s] Exiting without datasource in local mode", mode) |
81 | + if mode == sources.DSMODE_LOCAL: |
82 | return (None, []) |
83 | else: |
84 | return (None, ["No instance datasource found."]) |
85 | - |
86 | - if args.local: |
87 | - if not init.ds_restored: |
88 | - # if local mode and the datasource was not restored from cache |
89 | - # (this is not first boot) then apply networking. |
90 | - init.apply_network_config() |
91 | else: |
92 | - LOG.debug("skipping networking config from restored datasource.") |
93 | + LOG.debug("[%s] barreling on in force mode without datasource", |
94 | + mode) |
95 | |
96 | # Stage 6 |
97 | iid = init.instancify() |
98 | - LOG.debug("%s will now be targeting instance id: %s", name, iid) |
99 | + LOG.debug("[%s] %s will now be targeting instance id: %s. new=%s", |
100 | + mode, name, iid, init.is_new_instance()) |
101 | + |
102 | + init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL)) |
103 | + |
104 | + if mode == sources.DSMODE_LOCAL: |
105 | + if init.datasource.dsmode != mode: |
106 | + LOG.debug("[%s] Exiting. datasource %s not in local mode.", |
107 | + mode, init.datasource) |
108 | + return (init.datasource, []) |
109 | + else: |
110 | + LOG.debug("[%s] %s is in local mode, will apply init modules now.", |
111 | + mode, init.datasource) |
112 | + |
113 | + # update fully realizes user-data (pulling in #include if necessary) |
114 | init.update() |
115 | # Stage 7 |
116 | try: |
117 | @@ -528,7 +544,7 @@ |
118 | v1[mode]['errors'] = [str(e) for e in errors] |
119 | |
120 | except Exception as e: |
121 | - util.logexc(LOG, "failed of stage %s", mode) |
122 | + util.logexc(LOG, "failed stage %s", mode) |
123 | print_exc("failed run of stage %s" % mode) |
124 | v1[mode]['errors'] = [str(e)] |
125 | |
126 | |
127 | === modified file 'cloudinit/config/cc_emit_upstart.py' |
128 | --- cloudinit/config/cc_emit_upstart.py 2016-05-12 17:56:26 +0000 |
129 | +++ cloudinit/config/cc_emit_upstart.py 2016-06-03 19:07:08 +0000 |
130 | @@ -56,7 +56,7 @@ |
131 | event_names = ['cloud-config'] |
132 | |
133 | if not is_upstart_system(): |
134 | - log.debug("not upstart system, '%s' disabled") |
135 | + log.debug("not upstart system, '%s' disabled", name) |
136 | return |
137 | |
138 | cfgpath = cloud.paths.get_ipath_cur("cloud_config") |
139 | |
140 | === modified file 'cloudinit/config/cc_lxd.py' |
141 | --- cloudinit/config/cc_lxd.py 2016-05-12 17:56:26 +0000 |
142 | +++ cloudinit/config/cc_lxd.py 2016-06-03 19:07:08 +0000 |
143 | @@ -52,7 +52,8 @@ |
144 | # Get config |
145 | lxd_cfg = cfg.get('lxd') |
146 | if not lxd_cfg: |
147 | - log.debug("Skipping module named %s, not present or disabled by cfg") |
148 | + log.debug("Skipping module named %s, not present or disabled by cfg", |
149 | + name) |
150 | return |
151 | if not isinstance(lxd_cfg, dict): |
152 | log.warn("lxd config must be a dictionary. found a '%s'", |
153 | |
154 | === modified file 'cloudinit/distros/__init__.py' |
155 | --- cloudinit/distros/__init__.py 2016-05-12 17:56:26 +0000 |
156 | +++ cloudinit/distros/__init__.py 2016-06-03 19:07:08 +0000 |
157 | @@ -31,6 +31,7 @@ |
158 | |
159 | from cloudinit import importer |
160 | from cloudinit import log as logging |
161 | +from cloudinit import net |
162 | from cloudinit import ssh_util |
163 | from cloudinit import type_utils |
164 | from cloudinit import util |
165 | @@ -128,6 +129,8 @@ |
166 | mirror_info=arch_info) |
167 | |
168 | def apply_network(self, settings, bring_up=True): |
169 | + # this applies network where 'settings' is interfaces(5) style |
170 | + # it is obsolete compared to apply_network_config |
171 | # Write it out |
172 | dev_names = self._write_network(settings) |
173 | # Now try to bring them up |
174 | @@ -143,6 +146,9 @@ |
175 | return self._bring_up_interfaces(dev_names) |
176 | return False |
177 | |
178 | + def apply_network_config_names(self, netconfig): |
179 | + net.apply_network_config_names(netconfig) |
180 | + |
181 | @abc.abstractmethod |
182 | def apply_locale(self, locale, out_fn=None): |
183 | raise NotImplementedError() |
184 | |
185 | === modified file 'cloudinit/helpers.py' |
186 | --- cloudinit/helpers.py 2016-05-12 17:56:26 +0000 |
187 | +++ cloudinit/helpers.py 2016-06-03 19:07:08 +0000 |
188 | @@ -328,6 +328,7 @@ |
189 | self.cfgs = path_cfgs |
190 | # Populate all the initial paths |
191 | self.cloud_dir = path_cfgs.get('cloud_dir', '/var/lib/cloud') |
192 | + self.run_dir = path_cfgs.get('run_dir', '/run/cloud-init') |
193 | self.instance_link = os.path.join(self.cloud_dir, 'instance') |
194 | self.boot_finished = os.path.join(self.instance_link, "boot-finished") |
195 | self.upstart_conf_d = path_cfgs.get('upstart_dir') |
196 | @@ -349,26 +350,19 @@ |
197 | "data": "data", |
198 | "vendordata_raw": "vendor-data.txt", |
199 | "vendordata": "vendor-data.txt.i", |
200 | + "instance_id": ".instance-id", |
201 | } |
202 | # Set when a datasource becomes active |
203 | self.datasource = ds |
204 | |
205 | # get_ipath_cur: get the current instance path for an item |
206 | def get_ipath_cur(self, name=None): |
207 | - ipath = self.instance_link |
208 | - add_on = self.lookups.get(name) |
209 | - if add_on: |
210 | - ipath = os.path.join(ipath, add_on) |
211 | - return ipath |
212 | + return self._get_path(self.instance_link, name) |
213 | |
214 | # get_cpath : get the "clouddir" (/var/lib/cloud/<name>) |
215 | # for a name in dirmap |
216 | def get_cpath(self, name=None): |
217 | - cpath = self.cloud_dir |
218 | - add_on = self.lookups.get(name) |
219 | - if add_on: |
220 | - cpath = os.path.join(cpath, add_on) |
221 | - return cpath |
222 | + return self._get_path(self.cloud_dir, name) |
223 | |
224 | # _get_ipath : get the instance path for a name in pathmap |
225 | # (/var/lib/cloud/instances/<instance>/<name>) |
226 | @@ -397,6 +391,14 @@ |
227 | else: |
228 | return ipath |
229 | |
230 | + def _get_path(self, base, name=None): |
231 | + if name is None: |
232 | + return base |
233 | + return os.path.join(base, self.lookups[name]) |
234 | + |
235 | + def get_runpath(self, name=None): |
236 | + return self._get_path(self.run_dir, name) |
237 | + |
238 | |
239 | # This config parser will not throw when sections don't exist |
240 | # and you are setting values on those sections which is useful |
241 | |
242 | === modified file 'cloudinit/net/__init__.py' |
243 | --- cloudinit/net/__init__.py 2016-05-12 17:56:26 +0000 |
244 | +++ cloudinit/net/__init__.py 2016-06-03 19:07:08 +0000 |
245 | @@ -201,7 +201,11 @@ |
246 | ifaces[iface]['method'] = method |
247 | currif = iface |
248 | elif option == "hwaddress": |
249 | - ifaces[currif]['hwaddress'] = split[1] |
250 | + if split[1] == "ether": |
251 | + val = split[2] |
252 | + else: |
253 | + val = split[1] |
254 | + ifaces[currif]['hwaddress'] = val |
255 | elif option in NET_CONFIG_OPTIONS: |
256 | ifaces[currif][option] = split[1] |
257 | elif option in NET_CONFIG_COMMANDS: |
258 | @@ -570,6 +574,8 @@ |
259 | content += iface_start_entry(iface, index) |
260 | content += iface_add_subnet(iface, subnet) |
261 | content += iface_add_attrs(iface) |
262 | + for route in subnet.get('routes', []): |
263 | + content += render_route(route, indent=" ") |
264 | else: |
265 | # ifenslave docs say to auto the slave devices |
266 | if 'bond-master' in iface: |
267 | @@ -768,4 +774,218 @@ |
268 | return config_from_klibc_net_cfg(files=files, mac_addrs=mac_addrs) |
269 | |
270 | |
271 | +def convert_eni_data(eni_data): |
272 | + # return a network config representation of what is in eni_data |
273 | + ifaces = {} |
274 | + parse_deb_config_data(ifaces, eni_data, src_dir=None, src_path=None) |
275 | + return _ifaces_to_net_config_data(ifaces) |
276 | + |
277 | + |
278 | +def _ifaces_to_net_config_data(ifaces): |
279 | + """Return network config that represents the ifaces data provided. |
280 | + ifaces = parse_deb_config("/etc/network/interfaces") |
281 | + config = ifaces_to_net_config_data(ifaces) |
282 | + state = parse_net_config_data(config).""" |
283 | + devs = {} |
284 | + for name, data in ifaces.items(): |
285 | + # devname is 'eth0' for name='eth0:1' |
286 | + devname = name.partition(":")[0] |
287 | + if devname == "lo": |
288 | + # currently provding 'lo' in network config results in duplicate |
289 | + # entries. in rendered interfaces file. so skip it. |
290 | + continue |
291 | + if devname not in devs: |
292 | + devs[devname] = {'type': 'physical', 'name': devname, |
293 | + 'subnets': []} |
294 | + # this isnt strictly correct, but some might specify |
295 | + # hwaddress on a nic for matching / declaring name. |
296 | + if 'hwaddress' in data: |
297 | + devs[devname]['mac_address'] = data['hwaddress'] |
298 | + subnet = {'_orig_eni_name': name, 'type': data['method']} |
299 | + if data.get('auto'): |
300 | + subnet['control'] = 'auto' |
301 | + else: |
302 | + subnet['control'] = 'manual' |
303 | + |
304 | + if data.get('method') == 'static': |
305 | + subnet['address'] = data['address'] |
306 | + |
307 | + for copy_key in ('netmask', 'gateway', 'broadcast'): |
308 | + if copy_key in data: |
309 | + subnet[copy_key] = data[copy_key] |
310 | + |
311 | + if 'dns' in data: |
312 | + for n in ('nameservers', 'search'): |
313 | + if n in data['dns'] and data['dns'][n]: |
314 | + subnet['dns_' + n] = data['dns'][n] |
315 | + devs[devname]['subnets'].append(subnet) |
316 | + |
317 | + return {'version': 1, |
318 | + 'config': [devs[d] for d in sorted(devs)]} |
319 | + |
320 | + |
321 | +def apply_network_config_names(netcfg, strict_present=True, strict_busy=True): |
322 | + """read the network config and rename devices accordingly. |
323 | + if strict_present is false, then do not raise exception if no devices |
324 | + match. if strict_busy is false, then do not raise exception if the |
325 | + device cannot be renamed because it is currently configured.""" |
326 | + renames = [] |
327 | + for ent in netcfg.get('config', {}): |
328 | + if ent.get('type') != 'physical': |
329 | + continue |
330 | + mac = ent.get('mac_address') |
331 | + name = ent.get('name') |
332 | + if not mac: |
333 | + continue |
334 | + renames.append([mac, name]) |
335 | + |
336 | + return rename_interfaces(renames) |
337 | + |
338 | + |
339 | +def _get_current_rename_info(check_downable=True): |
340 | + """Collect information necessary for rename_interfaces.""" |
341 | + names = get_devicelist() |
342 | + bymac = {} |
343 | + for n in names: |
344 | + bymac[get_interface_mac(n)] = { |
345 | + 'name': n, 'up': is_up(n), 'downable': None} |
346 | + |
347 | + if check_downable: |
348 | + nmatch = re.compile(r"[0-9]+:\s+(\w+)[@:]") |
349 | + ipv6, _err = util.subp(['ip', '-6', 'addr', 'show', 'permanent', |
350 | + 'scope', 'global'], capture=True) |
351 | + ipv4, _err = util.subp(['ip', '-4', 'addr', 'show'], capture=True) |
352 | + |
353 | + nics_with_addresses = set() |
354 | + for bytes_out in (ipv6, ipv4): |
355 | + nics_with_addresses.update(nmatch.findall(bytes_out)) |
356 | + |
357 | + for d in bymac.values(): |
358 | + d['downable'] = (d['up'] is False or |
359 | + d['name'] not in nics_with_addresses) |
360 | + |
361 | + return bymac |
362 | + |
363 | + |
364 | +def rename_interfaces(renames, strict_present=True, strict_busy=True, |
365 | + current_info=None): |
366 | + if current_info is None: |
367 | + current_info = _get_current_rename_info() |
368 | + |
369 | + cur_bymac = {} |
370 | + for mac, data in current_info.items(): |
371 | + cur = data.copy() |
372 | + cur['mac'] = mac |
373 | + cur_bymac[mac] = cur |
374 | + |
375 | + def update_byname(bymac): |
376 | + return {data['name']: data for data in bymac.values()} |
377 | + |
378 | + def rename(cur, new): |
379 | + util.subp(["ip", "link", "set", cur, "name", new], capture=True) |
380 | + |
381 | + def down(name): |
382 | + util.subp(["ip", "link", "set", name, "down"], capture=True) |
383 | + |
384 | + def up(name): |
385 | + util.subp(["ip", "link", "set", name, "up"], capture=True) |
386 | + |
387 | + ops = [] |
388 | + errors = [] |
389 | + ups = [] |
390 | + cur_byname = update_byname(cur_bymac) |
391 | + tmpname_fmt = "cirename%d" |
392 | + tmpi = -1 |
393 | + |
394 | + for mac, new_name in renames: |
395 | + cur = cur_bymac.get(mac, {}) |
396 | + cur_name = cur.get('name') |
397 | + cur_ops = [] |
398 | + if cur_name == new_name: |
399 | + # nothing to do |
400 | + continue |
401 | + |
402 | + if not cur_name: |
403 | + if strict_present: |
404 | + errors.append( |
405 | + "[nic not present] Cannot rename mac=%s to %s" |
406 | + ", not available." % (mac, new_name)) |
407 | + continue |
408 | + |
409 | + if cur['up']: |
410 | + msg = "[busy] Error renaming mac=%s from %s to %s" |
411 | + if not cur['downable']: |
412 | + if strict_busy: |
413 | + errors.append(msg % (mac, cur_name, new_name)) |
414 | + continue |
415 | + cur['up'] = False |
416 | + cur_ops.append(("down", mac, new_name, (cur_name,))) |
417 | + ups.append(("up", mac, new_name, (new_name,))) |
418 | + |
419 | + if new_name in cur_byname: |
420 | + target = cur_byname[new_name] |
421 | + if target['up']: |
422 | + msg = "[busy-target] Error renaming mac=%s from %s to %s." |
423 | + if not target['downable']: |
424 | + if strict_busy: |
425 | + errors.append(msg % (mac, cur_name, new_name)) |
426 | + continue |
427 | + else: |
428 | + cur_ops.append(("down", mac, new_name, (new_name,))) |
429 | + |
430 | + tmp_name = None |
431 | + while tmp_name is None or tmp_name in cur_byname: |
432 | + tmpi += 1 |
433 | + tmp_name = tmpname_fmt % tmpi |
434 | + |
435 | + cur_ops.append(("rename", mac, new_name, (new_name, tmp_name))) |
436 | + target['name'] = tmp_name |
437 | + cur_byname = update_byname(cur_bymac) |
438 | + if target['up']: |
439 | + ups.append(("up", mac, new_name, (tmp_name,))) |
440 | + |
441 | + cur_ops.append(("rename", mac, new_name, (cur['name'], new_name))) |
442 | + cur['name'] = new_name |
443 | + cur_byname = update_byname(cur_bymac) |
444 | + ops += cur_ops |
445 | + |
446 | + opmap = {'rename': rename, 'down': down, 'up': up} |
447 | + |
448 | + if len(ops) + len(ups) == 0: |
449 | + if len(errors): |
450 | + LOG.debug("unable to do any work for renaming of %s", renames) |
451 | + else: |
452 | + LOG.debug("no work necessary for renaming of %s", renames) |
453 | + else: |
454 | + LOG.debug("achieving renaming of %s with ops %s", renames, ops + ups) |
455 | + |
456 | + for op, mac, new_name, params in ops + ups: |
457 | + try: |
458 | + opmap.get(op)(*params) |
459 | + except Exception as e: |
460 | + errors.append( |
461 | + "[unknown] Error performing %s%s for %s, %s: %s" % |
462 | + (op, params, mac, new_name, e)) |
463 | + |
464 | + if len(errors): |
465 | + raise Exception('\n'.join(errors)) |
466 | + |
467 | + |
468 | +def get_interface_mac(ifname): |
469 | + """Returns the string value of an interface's MAC Address""" |
470 | + return read_sys_net(ifname, "address", enoent=False) |
471 | + |
472 | + |
473 | +def get_interfaces_by_mac(devs=None): |
474 | + """Build a dictionary of tuples {mac: name}""" |
475 | + if devs is None: |
476 | + devs = get_devicelist() |
477 | + ret = {} |
478 | + for name in devs: |
479 | + mac = get_interface_mac(name) |
480 | + # some devices may not have a mac (tun0) |
481 | + if mac: |
482 | + ret[mac] = name |
483 | + return ret |
484 | + |
485 | # vi: ts=4 expandtab syntax=python |
486 | |
487 | === modified file 'cloudinit/sources/DataSourceCloudSigma.py' |
488 | --- cloudinit/sources/DataSourceCloudSigma.py 2016-05-12 17:56:26 +0000 |
489 | +++ cloudinit/sources/DataSourceCloudSigma.py 2016-06-03 19:07:08 +0000 |
490 | @@ -27,8 +27,6 @@ |
491 | |
492 | LOG = logging.getLogger(__name__) |
493 | |
494 | -VALID_DSMODES = ("local", "net", "disabled") |
495 | - |
496 | |
497 | class DataSourceCloudSigma(sources.DataSource): |
498 | """ |
499 | @@ -38,7 +36,6 @@ |
500 | http://cloudsigma-docs.readthedocs.org/en/latest/server_context.html |
501 | """ |
502 | def __init__(self, sys_cfg, distro, paths): |
503 | - self.dsmode = 'local' |
504 | self.cepko = Cepko() |
505 | self.ssh_public_key = '' |
506 | sources.DataSource.__init__(self, sys_cfg, distro, paths) |
507 | @@ -84,11 +81,9 @@ |
508 | LOG.debug("CloudSigma: Unable to read from serial port") |
509 | return False |
510 | |
511 | - dsmode = server_meta.get('cloudinit-dsmode', self.dsmode) |
512 | - if dsmode not in VALID_DSMODES: |
513 | - LOG.warn("Invalid dsmode %s, assuming default of 'net'", dsmode) |
514 | - dsmode = 'net' |
515 | - if dsmode == "disabled" or dsmode != self.dsmode: |
516 | + self.dsmode = self._determine_dsmode( |
517 | + [server_meta.get('cloudinit-dsmode')]) |
518 | + if dsmode == sources.DSMODE_DISABLED: |
519 | return False |
520 | |
521 | base64_fields = server_meta.get('base64_fields', '').split(',') |
522 | @@ -120,17 +115,13 @@ |
523 | return self.metadata['uuid'] |
524 | |
525 | |
526 | -class DataSourceCloudSigmaNet(DataSourceCloudSigma): |
527 | - def __init__(self, sys_cfg, distro, paths): |
528 | - DataSourceCloudSigma.__init__(self, sys_cfg, distro, paths) |
529 | - self.dsmode = 'net' |
530 | - |
531 | +# Legacy: Must be present in case we load an old pkl object |
532 | +DataSourceCloudSigmaNet = DataSourceCloudSigma |
533 | |
534 | # Used to match classes to dependencies. Since this datasource uses the serial |
535 | # port network is not really required, so it's okay to load without it, too. |
536 | datasources = [ |
537 | (DataSourceCloudSigma, (sources.DEP_FILESYSTEM)), |
538 | - (DataSourceCloudSigmaNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)), |
539 | ] |
540 | |
541 | |
542 | |
543 | === modified file 'cloudinit/sources/DataSourceConfigDrive.py' |
544 | --- cloudinit/sources/DataSourceConfigDrive.py 2016-04-29 13:04:36 +0000 |
545 | +++ cloudinit/sources/DataSourceConfigDrive.py 2016-06-03 19:07:08 +0000 |
546 | @@ -22,6 +22,7 @@ |
547 | import os |
548 | |
549 | from cloudinit import log as logging |
550 | +from cloudinit import net |
551 | from cloudinit import sources |
552 | from cloudinit import util |
553 | |
554 | @@ -35,7 +36,6 @@ |
555 | DEFAULT_METADATA = { |
556 | "instance-id": DEFAULT_IID, |
557 | } |
558 | -VALID_DSMODES = ("local", "net", "pass", "disabled") |
559 | FS_TYPES = ('vfat', 'iso9660') |
560 | LABEL_TYPES = ('config-2',) |
561 | POSSIBLE_MOUNTS = ('sr', 'cd') |
562 | @@ -47,12 +47,12 @@ |
563 | def __init__(self, sys_cfg, distro, paths): |
564 | super(DataSourceConfigDrive, self).__init__(sys_cfg, distro, paths) |
565 | self.source = None |
566 | - self.dsmode = 'local' |
567 | self.seed_dir = os.path.join(paths.seed_dir, 'config_drive') |
568 | self.version = None |
569 | self.ec2_metadata = None |
570 | self._network_config = None |
571 | self.network_json = None |
572 | + self.network_eni = None |
573 | self.files = {} |
574 | |
575 | def __str__(self): |
576 | @@ -98,38 +98,22 @@ |
577 | |
578 | md = results.get('metadata', {}) |
579 | md = util.mergemanydict([md, DEFAULT_METADATA]) |
580 | - user_dsmode = results.get('dsmode', None) |
581 | - if user_dsmode not in VALID_DSMODES + (None,): |
582 | - LOG.warn("User specified invalid mode: %s", user_dsmode) |
583 | - user_dsmode = None |
584 | - |
585 | - dsmode = get_ds_mode(cfgdrv_ver=results['version'], |
586 | - ds_cfg=self.ds_cfg.get('dsmode'), |
587 | - user=user_dsmode) |
588 | - |
589 | - if dsmode == "disabled": |
590 | - # most likely user specified |
591 | + |
592 | + self.dsmode = self._determine_dsmode( |
593 | + [results.get('dsmode'), self.ds_cfg.get('dsmode'), |
594 | + sources.DSMODE_PASS if results['version'] == 1 else None]) |
595 | + |
596 | + if self.dsmode == sources.DSMODE_DISABLED: |
597 | return False |
598 | |
599 | - # TODO(smoser): fix this, its dirty. |
600 | - # we want to do some things (writing files and network config) |
601 | - # only on first boot, and even then, we want to do so in the |
602 | - # local datasource (so they happen earlier) even if the configured |
603 | - # dsmode is 'net' or 'pass'. To do this, we check the previous |
604 | - # instance-id |
605 | + # This is legacy and sneaky. If dsmode is 'pass' then write |
606 | + # 'injected files' and apply legacy ENI network format. |
607 | prev_iid = get_previous_iid(self.paths) |
608 | cur_iid = md['instance-id'] |
609 | - if prev_iid != cur_iid and self.dsmode == "local": |
610 | + if prev_iid != cur_iid and self.dsmode == sources.DSMODE_PASS: |
611 | on_first_boot(results, distro=self.distro) |
612 | - |
613 | - # dsmode != self.dsmode here if: |
614 | - # * dsmode = "pass", pass means it should only copy files and then |
615 | - # pass to another datasource |
616 | - # * dsmode = "net" and self.dsmode = "local" |
617 | - # so that user boothooks would be applied with network, the |
618 | - # local datasource just gets out of the way, and lets the net claim |
619 | - if dsmode != self.dsmode: |
620 | - LOG.debug("%s: not claiming datasource, dsmode=%s", self, dsmode) |
621 | + LOG.debug("%s: not claiming datasource, dsmode=%s", self, |
622 | + self.dsmode) |
623 | return False |
624 | |
625 | self.source = found |
626 | @@ -147,12 +131,11 @@ |
627 | LOG.warn("Invalid content in vendor-data: %s", e) |
628 | self.vendordata_raw = None |
629 | |
630 | - try: |
631 | - self.network_json = results.get('networkdata') |
632 | - except ValueError as e: |
633 | - LOG.warn("Invalid content in network-data: %s", e) |
634 | - self.network_json = None |
635 | - |
636 | + # network_config is an /etc/network/interfaces formated file and is |
637 | + # obsolete compared to networkdata (from network_data.json) but both |
638 | + # might be present. |
639 | + self.network_eni = results.get("network_config") |
640 | + self.network_json = results.get('networkdata') |
641 | return True |
642 | |
643 | def check_instance_id(self, sys_cfg): |
644 | @@ -163,41 +146,16 @@ |
645 | def network_config(self): |
646 | if self._network_config is None: |
647 | if self.network_json is not None: |
648 | + LOG.debug("network config provided via network_json") |
649 | self._network_config = convert_network_data(self.network_json) |
650 | + elif self.network_eni is not None: |
651 | + self._network_config = net.convert_eni_data(self.network_eni) |
652 | + LOG.debug("network config provided via converted eni data") |
653 | + else: |
654 | + LOG.debug("no network configuration available") |
655 | return self._network_config |
656 | |
657 | |
658 | -class DataSourceConfigDriveNet(DataSourceConfigDrive): |
659 | - def __init__(self, sys_cfg, distro, paths): |
660 | - DataSourceConfigDrive.__init__(self, sys_cfg, distro, paths) |
661 | - self.dsmode = 'net' |
662 | - |
663 | - |
664 | -def get_ds_mode(cfgdrv_ver, ds_cfg=None, user=None): |
665 | - """Determine what mode should be used. |
666 | - valid values are 'pass', 'disabled', 'local', 'net' |
667 | - """ |
668 | - # user passed data trumps everything |
669 | - if user is not None: |
670 | - return user |
671 | - |
672 | - if ds_cfg is not None: |
673 | - return ds_cfg |
674 | - |
675 | - # at config-drive version 1, the default behavior was pass. That |
676 | - # meant to not use use it as primary data source, but expect a ec2 metadata |
677 | - # source. for version 2, we default to 'net', which means |
678 | - # the DataSourceConfigDriveNet, would be used. |
679 | - # |
680 | - # this could change in the future. If there was definitive metadata |
681 | - # that indicated presense of an openstack metadata service, then |
682 | - # we could change to 'pass' by default also. The motivation for that |
683 | - # would be 'cloud-init query' as the web service could be more dynamic |
684 | - if cfgdrv_ver == 1: |
685 | - return "pass" |
686 | - return "net" |
687 | - |
688 | - |
689 | def read_config_drive(source_dir): |
690 | reader = openstack.ConfigDriveReader(source_dir) |
691 | finders = [ |
692 | @@ -231,9 +189,12 @@ |
693 | % (type(data))) |
694 | net_conf = data.get("network_config", '') |
695 | if net_conf and distro: |
696 | - LOG.debug("Updating network interfaces from config drive") |
697 | + LOG.warn("Updating network interfaces from config drive") |
698 | distro.apply_network(net_conf) |
699 | - files = data.get('files', {}) |
700 | + write_injected_files(data.get('files')) |
701 | + |
702 | + |
703 | +def write_injected_files(files): |
704 | if files: |
705 | LOG.debug("Writing %s injected files", len(files)) |
706 | for (filename, content) in files.items(): |
707 | @@ -293,20 +254,8 @@ |
708 | return devices |
709 | |
710 | |
711 | -# Used to match classes to dependencies |
712 | -datasources = [ |
713 | - (DataSourceConfigDrive, (sources.DEP_FILESYSTEM, )), |
714 | - (DataSourceConfigDriveNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)), |
715 | -] |
716 | - |
717 | - |
718 | -# Return a list of data sources that match this set of dependencies |
719 | -def get_datasource_list(depends): |
720 | - return sources.list_from_depends(depends, datasources) |
721 | - |
722 | - |
723 | # Convert OpenStack ConfigDrive NetworkData json to network_config yaml |
724 | -def convert_network_data(network_json=None): |
725 | +def convert_network_data(network_json=None, known_macs=None): |
726 | """Return a dictionary of network_config by parsing provided |
727 | OpenStack ConfigDrive NetworkData json format |
728 | |
729 | @@ -344,6 +293,7 @@ |
730 | 'mac_address', |
731 | 'subnets', |
732 | 'params', |
733 | + 'mtu', |
734 | ], |
735 | 'subnet': [ |
736 | 'type', |
737 | @@ -353,7 +303,6 @@ |
738 | 'metric', |
739 | 'gateway', |
740 | 'pointopoint', |
741 | - 'mtu', |
742 | 'scope', |
743 | 'dns_nameservers', |
744 | 'dns_search', |
745 | @@ -370,9 +319,15 @@ |
746 | subnets = [] |
747 | cfg = {k: v for k, v in link.items() |
748 | if k in valid_keys['physical']} |
749 | - cfg.update({'name': link['id']}) |
750 | - for network in [net for net in networks |
751 | - if net['link'] == link['id']]: |
752 | + # 'name' is not in openstack spec yet, but we will support it if it is |
753 | + # present. The 'id' in the spec is currently implemented as the host |
754 | + # nic's name, meaning something like 'tap-adfasdffd'. We do not want |
755 | + # to name guest devices with such ugly names. |
756 | + if 'name' in link: |
757 | + cfg['name'] = link['name'] |
758 | + |
759 | + for network in [n for n in networks |
760 | + if n['link'] == link['id']]: |
761 | subnet = {k: v for k, v in network.items() |
762 | if k in valid_keys['subnet']} |
763 | if 'dhcp' in network['type']: |
764 | @@ -387,7 +342,7 @@ |
765 | }) |
766 | subnets.append(subnet) |
767 | cfg.update({'subnets': subnets}) |
768 | - if link['type'] in ['ethernet', 'vif', 'ovs', 'phy']: |
769 | + if link['type'] in ['ethernet', 'vif', 'ovs', 'phy', 'bridge']: |
770 | cfg.update({ |
771 | 'type': 'physical', |
772 | 'mac_address': link['ethernet_mac_address']}) |
773 | @@ -416,9 +371,38 @@ |
774 | |
775 | config.append(cfg) |
776 | |
777 | + need_names = [d for d in config |
778 | + if d.get('type') == 'physical' and 'name' not in d] |
779 | + |
780 | + if need_names: |
781 | + if known_macs is None: |
782 | + known_macs = net.get_interfaces_by_mac() |
783 | + |
784 | + for d in need_names: |
785 | + mac = d.get('mac_address') |
786 | + if not mac: |
787 | + raise ValueError("No mac_address or name entry for %s" % d) |
788 | + if mac not in known_macs: |
789 | + raise ValueError("Unable to find a system nic for %s" % d) |
790 | + d['name'] = known_macs[mac] |
791 | + |
792 | for service in services: |
793 | cfg = service |
794 | cfg.update({'type': 'nameserver'}) |
795 | config.append(cfg) |
796 | |
797 | return {'version': 1, 'config': config} |
798 | + |
799 | + |
800 | +# Legacy: Must be present in case we load an old pkl object |
801 | +DataSourceConfigDriveNet = DataSourceConfigDrive |
802 | + |
803 | +# Used to match classes to dependencies |
804 | +datasources = [ |
805 | + (DataSourceConfigDrive, (sources.DEP_FILESYSTEM, )), |
806 | +] |
807 | + |
808 | + |
809 | +# Return a list of data sources that match this set of dependencies |
810 | +def get_datasource_list(depends): |
811 | + return sources.list_from_depends(depends, datasources) |
812 | |
813 | === modified file 'cloudinit/sources/DataSourceNoCloud.py' |
814 | --- cloudinit/sources/DataSourceNoCloud.py 2016-05-12 17:56:26 +0000 |
815 | +++ cloudinit/sources/DataSourceNoCloud.py 2016-06-03 19:07:08 +0000 |
816 | @@ -24,6 +24,7 @@ |
817 | import os |
818 | |
819 | from cloudinit import log as logging |
820 | +from cloudinit import net |
821 | from cloudinit import sources |
822 | from cloudinit import util |
823 | |
824 | @@ -35,7 +36,6 @@ |
825 | sources.DataSource.__init__(self, sys_cfg, distro, paths) |
826 | self.dsmode = 'local' |
827 | self.seed = None |
828 | - self.cmdline_id = "ds=nocloud" |
829 | self.seed_dirs = [os.path.join(paths.seed_dir, 'nocloud'), |
830 | os.path.join(paths.seed_dir, 'nocloud-net')] |
831 | self.seed_dir = None |
832 | @@ -58,7 +58,7 @@ |
833 | try: |
834 | # Parse the kernel command line, getting data passed in |
835 | md = {} |
836 | - if parse_cmdline_data(self.cmdline_id, md): |
837 | + if load_cmdline_data(md): |
838 | found.append("cmdline") |
839 | mydata = _merge_new_seed(mydata, {'meta-data': md}) |
840 | except Exception: |
841 | @@ -123,12 +123,6 @@ |
842 | |
843 | mydata = _merge_new_seed(mydata, seeded) |
844 | |
845 | - # For seed from a device, the default mode is 'net'. |
846 | - # that is more likely to be what is desired. If they want |
847 | - # dsmode of local, then they must specify that. |
848 | - if 'dsmode' not in mydata['meta-data']: |
849 | - mydata['meta-data']['dsmode'] = "net" |
850 | - |
851 | LOG.debug("Using data from %s", dev) |
852 | found.append(dev) |
853 | break |
854 | @@ -144,7 +138,6 @@ |
855 | if len(found) == 0: |
856 | return False |
857 | |
858 | - seeded_network = None |
859 | # The special argument "seedfrom" indicates we should |
860 | # attempt to seed the userdata / metadata from its value |
861 | # its primarily value is in allowing the user to type less |
862 | @@ -160,10 +153,6 @@ |
863 | LOG.debug("Seed from %s not supported by %s", seedfrom, self) |
864 | return False |
865 | |
866 | - if (mydata['meta-data'].get('network-interfaces') or |
867 | - mydata.get('network-config')): |
868 | - seeded_network = self.dsmode |
869 | - |
870 | # This could throw errors, but the user told us to do it |
871 | # so if errors are raised, let them raise |
872 | (md_seed, ud) = util.read_seeded(seedfrom, timeout=None) |
873 | @@ -179,35 +168,21 @@ |
874 | mydata['meta-data'] = util.mergemanydict([mydata['meta-data'], |
875 | defaults]) |
876 | |
877 | - netdata = {'format': None, 'data': None} |
878 | - if mydata['meta-data'].get('network-interfaces'): |
879 | - netdata['format'] = 'interfaces' |
880 | - netdata['data'] = mydata['meta-data']['network-interfaces'] |
881 | - elif mydata.get('network-config'): |
882 | - netdata['format'] = 'network-config' |
883 | - netdata['data'] = mydata['network-config'] |
884 | - |
885 | - # if this is the local datasource or 'seedfrom' was used |
886 | - # and the source of the seed was self.dsmode. |
887 | - # Then see if there is network config to apply. |
888 | - # note this is obsolete network-interfaces style seeding. |
889 | - if self.dsmode in ("local", seeded_network): |
890 | - if mydata['meta-data'].get('network-interfaces'): |
891 | - LOG.debug("Updating network interfaces from %s", self) |
892 | - self.distro.apply_network( |
893 | - mydata['meta-data']['network-interfaces']) |
894 | - |
895 | - if mydata['meta-data']['dsmode'] == self.dsmode: |
896 | - self.seed = ",".join(found) |
897 | - self.metadata = mydata['meta-data'] |
898 | - self.userdata_raw = mydata['user-data'] |
899 | - self.vendordata_raw = mydata['vendor-data'] |
900 | - self._network_config = mydata['network-config'] |
901 | - return True |
902 | - |
903 | - LOG.debug("%s: not claiming datasource, dsmode=%s", self, |
904 | - mydata['meta-data']['dsmode']) |
905 | - return False |
906 | + self.dsmode = self._determine_dsmode( |
907 | + [mydata['meta-data'].get('dsmode')]) |
908 | + |
909 | + if self.dsmode == sources.DSMODE_DISABLED: |
910 | + LOG.debug("%s: not claiming datasource, dsmode=%s", self, |
911 | + self.dsmode) |
912 | + return False |
913 | + |
914 | + self.seed = ",".join(found) |
915 | + self.metadata = mydata['meta-data'] |
916 | + self.userdata_raw = mydata['user-data'] |
917 | + self.vendordata_raw = mydata['vendor-data'] |
918 | + self._network_config = mydata['network-config'] |
919 | + self._network_eni = mydata['meta-data'].get('network-interfaces') |
920 | + return True |
921 | |
922 | def check_instance_id(self, sys_cfg): |
923 | # quickly (local check only) if self.instance_id is still valid |
924 | @@ -227,6 +202,9 @@ |
925 | |
926 | @property |
927 | def network_config(self): |
928 | + if self._network_config is None: |
929 | + if self.network_eni is not None: |
930 | + self._network_config = net.convert_eni_data(self.network_eni) |
931 | return self._network_config |
932 | |
933 | |
934 | @@ -254,8 +232,22 @@ |
935 | return None |
936 | |
937 | |
938 | +def load_cmdline_data(fill, cmdline=None): |
939 | + pairs = [("ds=nocloud", sources.DSMODE_LOCAL), |
940 | + ("ds=nocloud-net", sources.DSMODE_NETWORK)] |
941 | + for idstr, dsmode in pairs: |
942 | + if parse_cmdline_data(idstr, fill, cmdline): |
943 | + # if dsmode was explicitly in the commanad line, then |
944 | + # prefer it to the dsmode based on the command line id |
945 | + if 'dsmode' not in fill: |
946 | + fill['dsmode'] = dsmode |
947 | + return True |
948 | + return False |
949 | + |
950 | + |
951 | # Returns true or false indicating if cmdline indicated |
952 | -# that this module should be used |
953 | +# that this module should be used. Updates dictionary 'fill' |
954 | +# with data that was found. |
955 | # Example cmdline: |
956 | # root=LABEL=uec-rootfs ro ds=nocloud |
957 | def parse_cmdline_data(ds_id, fill, cmdline=None): |
958 | @@ -319,9 +311,7 @@ |
959 | class DataSourceNoCloudNet(DataSourceNoCloud): |
960 | def __init__(self, sys_cfg, distro, paths): |
961 | DataSourceNoCloud.__init__(self, sys_cfg, distro, paths) |
962 | - self.cmdline_id = "ds=nocloud-net" |
963 | self.supported_seed_starts = ("http://", "https://", "ftp://") |
964 | - self.dsmode = "net" |
965 | |
966 | |
967 | # Used to match classes to dependencies |
968 | |
969 | === modified file 'cloudinit/sources/DataSourceOpenNebula.py' |
970 | --- cloudinit/sources/DataSourceOpenNebula.py 2016-03-04 06:45:58 +0000 |
971 | +++ cloudinit/sources/DataSourceOpenNebula.py 2016-06-03 19:07:08 +0000 |
972 | @@ -37,16 +37,13 @@ |
973 | LOG = logging.getLogger(__name__) |
974 | |
975 | DEFAULT_IID = "iid-dsopennebula" |
976 | -DEFAULT_MODE = 'net' |
977 | DEFAULT_PARSEUSER = 'nobody' |
978 | CONTEXT_DISK_FILES = ["context.sh"] |
979 | -VALID_DSMODES = ("local", "net", "disabled") |
980 | |
981 | |
982 | class DataSourceOpenNebula(sources.DataSource): |
983 | def __init__(self, sys_cfg, distro, paths): |
984 | sources.DataSource.__init__(self, sys_cfg, distro, paths) |
985 | - self.dsmode = 'local' |
986 | self.seed = None |
987 | self.seed_dir = os.path.join(paths.seed_dir, 'opennebula') |
988 | |
989 | @@ -93,52 +90,27 @@ |
990 | md = util.mergemanydict([md, defaults]) |
991 | |
992 | # check for valid user specified dsmode |
993 | - user_dsmode = results['metadata'].get('DSMODE', None) |
994 | - if user_dsmode not in VALID_DSMODES + (None,): |
995 | - LOG.warn("user specified invalid mode: %s", user_dsmode) |
996 | - user_dsmode = None |
997 | - |
998 | - # decide dsmode |
999 | - if user_dsmode: |
1000 | - dsmode = user_dsmode |
1001 | - elif self.ds_cfg.get('dsmode'): |
1002 | - dsmode = self.ds_cfg.get('dsmode') |
1003 | - else: |
1004 | - dsmode = DEFAULT_MODE |
1005 | - |
1006 | - if dsmode == "disabled": |
1007 | - # most likely user specified |
1008 | - return False |
1009 | - |
1010 | - # apply static network configuration only in 'local' dsmode |
1011 | - if ('network-interfaces' in results and self.dsmode == "local"): |
1012 | - LOG.debug("Updating network interfaces from %s", self) |
1013 | - self.distro.apply_network(results['network-interfaces']) |
1014 | - |
1015 | - if dsmode != self.dsmode: |
1016 | - LOG.debug("%s: not claiming datasource, dsmode=%s", self, dsmode) |
1017 | + self.dsmode = self._determine_dsmode( |
1018 | + [results.get('DSMODE'), self.ds_cfg.get('dsmode')]) |
1019 | + |
1020 | + if self.dsmode == sources.DSMODE_DISABLED: |
1021 | return False |
1022 | |
1023 | self.seed = seed |
1024 | + self.network_eni = results.get("network_config") |
1025 | self.metadata = md |
1026 | self.userdata_raw = results.get('userdata') |
1027 | return True |
1028 | |
1029 | def get_hostname(self, fqdn=False, resolve_ip=None): |
1030 | if resolve_ip is None: |
1031 | - if self.dsmode == 'net': |
1032 | + if self.dsmode == sources.DSMODE_NET: |
1033 | resolve_ip = True |
1034 | else: |
1035 | resolve_ip = False |
1036 | return sources.DataSource.get_hostname(self, fqdn, resolve_ip) |
1037 | |
1038 | |
1039 | -class DataSourceOpenNebulaNet(DataSourceOpenNebula): |
1040 | - def __init__(self, sys_cfg, distro, paths): |
1041 | - DataSourceOpenNebula.__init__(self, sys_cfg, distro, paths) |
1042 | - self.dsmode = 'net' |
1043 | - |
1044 | - |
1045 | class NonContextDiskDir(Exception): |
1046 | pass |
1047 | |
1048 | @@ -443,10 +415,12 @@ |
1049 | return results |
1050 | |
1051 | |
1052 | +# Legacy: Must be present in case we load an old pkl object |
1053 | +DataSourceOpenNebulaNet = DataSourceOpenNebula |
1054 | + |
1055 | # Used to match classes to dependencies |
1056 | datasources = [ |
1057 | (DataSourceOpenNebula, (sources.DEP_FILESYSTEM, )), |
1058 | - (DataSourceOpenNebulaNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)), |
1059 | ] |
1060 | |
1061 | |
1062 | |
1063 | === modified file 'cloudinit/sources/DataSourceOpenStack.py' |
1064 | --- cloudinit/sources/DataSourceOpenStack.py 2016-05-16 23:08:19 +0000 |
1065 | +++ cloudinit/sources/DataSourceOpenStack.py 2016-06-03 19:07:08 +0000 |
1066 | @@ -33,13 +33,11 @@ |
1067 | DEFAULT_METADATA = { |
1068 | "instance-id": DEFAULT_IID, |
1069 | } |
1070 | -VALID_DSMODES = ("net", "disabled") |
1071 | |
1072 | |
1073 | class DataSourceOpenStack(openstack.SourceMixin, sources.DataSource): |
1074 | def __init__(self, sys_cfg, distro, paths): |
1075 | super(DataSourceOpenStack, self).__init__(sys_cfg, distro, paths) |
1076 | - self.dsmode = 'net' |
1077 | self.metadata_address = None |
1078 | self.ssl_details = util.fetch_ssl_details(self.paths) |
1079 | self.version = None |
1080 | @@ -125,11 +123,8 @@ |
1081 | self.metadata_address) |
1082 | return False |
1083 | |
1084 | - user_dsmode = results.get('dsmode', None) |
1085 | - if user_dsmode not in VALID_DSMODES + (None,): |
1086 | - LOG.warn("User specified invalid mode: %s", user_dsmode) |
1087 | - user_dsmode = None |
1088 | - if user_dsmode == 'disabled': |
1089 | + self.dsmode = self._determine_dsmode([results.get('dsmode')]) |
1090 | + if self.dsmode == sources.DSMODE_DISABLED: |
1091 | return False |
1092 | |
1093 | md = results.get('metadata', {}) |
1094 | |
1095 | === modified file 'cloudinit/sources/DataSourceSmartOS.py' |
1096 | --- cloudinit/sources/DataSourceSmartOS.py 2016-04-12 16:57:50 +0000 |
1097 | +++ cloudinit/sources/DataSourceSmartOS.py 2016-06-03 19:07:08 +0000 |
1098 | @@ -32,13 +32,13 @@ |
1099 | # http://us-east.manta.joyent.com/jmc/public/mdata/datadict.html |
1100 | # Comments with "@datadictionary" are snippets of the definition |
1101 | |
1102 | +import base64 |
1103 | import binascii |
1104 | -import contextlib |
1105 | +import json |
1106 | import os |
1107 | import random |
1108 | import re |
1109 | import socket |
1110 | -import stat |
1111 | |
1112 | import serial |
1113 | |
1114 | @@ -64,14 +64,36 @@ |
1115 | 'operator-script': ('sdc:operator-script', False), |
1116 | } |
1117 | |
1118 | +SMARTOS_ATTRIB_JSON = { |
1119 | + # Cloud-init Key : (SmartOS Key known JSON) |
1120 | + 'network-data': 'sdc:nics', |
1121 | +} |
1122 | + |
1123 | +SMARTOS_ENV_LX_BRAND = "lx-brand" |
1124 | +SMARTOS_ENV_KVM = "kvm" |
1125 | + |
1126 | DS_NAME = 'SmartOS' |
1127 | DS_CFG_PATH = ['datasource', DS_NAME] |
1128 | +NO_BASE64_DECODE = [ |
1129 | + 'iptables_disable', |
1130 | + 'motd_sys_info', |
1131 | + 'root_authorized_keys', |
1132 | + 'sdc:datacenter_name', |
1133 | + 'sdc:uuid' |
1134 | + 'user-data', |
1135 | + 'user-script', |
1136 | +] |
1137 | + |
1138 | +METADATA_SOCKFILE = '/native/.zonecontrol/metadata.sock' |
1139 | +SERIAL_DEVICE = '/dev/ttyS1' |
1140 | +SERIAL_TIMEOUT = 60 |
1141 | + |
1142 | # BUILT-IN DATASOURCE CONFIGURATION |
1143 | # The following is the built-in configuration. If the values |
1144 | # are not set via the system configuration, then these default |
1145 | # will be used: |
1146 | # serial_device: which serial device to use for the meta-data |
1147 | -# seed_timeout: how long to wait on the device |
1148 | +# serial_timeout: how long to wait on the device |
1149 | # no_base64_decode: values which are not base64 encoded and |
1150 | # are fetched directly from SmartOS, not meta-data values |
1151 | # base64_keys: meta-data keys that are delivered in base64 |
1152 | @@ -81,16 +103,10 @@ |
1153 | # fs_setup: describes how to format the ephemeral drive |
1154 | # |
1155 | BUILTIN_DS_CONFIG = { |
1156 | - 'serial_device': '/dev/ttyS1', |
1157 | - 'metadata_sockfile': '/native/.zonecontrol/metadata.sock', |
1158 | - 'seed_timeout': 60, |
1159 | - 'no_base64_decode': ['root_authorized_keys', |
1160 | - 'motd_sys_info', |
1161 | - 'iptables_disable', |
1162 | - 'user-data', |
1163 | - 'user-script', |
1164 | - 'sdc:datacenter_name', |
1165 | - 'sdc:uuid'], |
1166 | + 'serial_device': SERIAL_DEVICE, |
1167 | + 'serial_timeout': SERIAL_TIMEOUT, |
1168 | + 'metadata_sockfile': METADATA_SOCKFILE, |
1169 | + 'no_base64_decode': NO_BASE64_DECODE, |
1170 | 'base64_keys': [], |
1171 | 'base64_all': False, |
1172 | 'disk_aliases': {'ephemeral0': '/dev/vdb'}, |
1173 | @@ -154,59 +170,41 @@ |
1174 | |
1175 | |
1176 | class DataSourceSmartOS(sources.DataSource): |
1177 | + _unset = "_unset" |
1178 | + smartos_type = _unset |
1179 | + md_client = _unset |
1180 | + |
1181 | def __init__(self, sys_cfg, distro, paths): |
1182 | sources.DataSource.__init__(self, sys_cfg, distro, paths) |
1183 | - self.is_smartdc = None |
1184 | self.ds_cfg = util.mergemanydict([ |
1185 | self.ds_cfg, |
1186 | util.get_cfg_by_path(sys_cfg, DS_CFG_PATH, {}), |
1187 | BUILTIN_DS_CONFIG]) |
1188 | |
1189 | self.metadata = {} |
1190 | + self.network_data = None |
1191 | + self._network_config = None |
1192 | |
1193 | - # SDC LX-Brand Zones lack dmidecode (no /dev/mem) but |
1194 | - # report 'BrandZ virtual linux' as the kernel version |
1195 | - if os.uname()[3].lower() == 'brandz virtual linux': |
1196 | - LOG.debug("Host is SmartOS, guest in Zone") |
1197 | - self.is_smartdc = True |
1198 | - self.smartos_type = 'lx-brand' |
1199 | - self.cfg = {} |
1200 | - self.seed = self.ds_cfg.get("metadata_sockfile") |
1201 | - else: |
1202 | - self.is_smartdc = True |
1203 | - self.smartos_type = 'kvm' |
1204 | - self.seed = self.ds_cfg.get("serial_device") |
1205 | - self.cfg = BUILTIN_CLOUD_CONFIG |
1206 | - self.seed_timeout = self.ds_cfg.get("serial_timeout") |
1207 | - self.smartos_no_base64 = self.ds_cfg.get('no_base64_decode') |
1208 | - self.b64_keys = self.ds_cfg.get('base64_keys') |
1209 | - self.b64_all = self.ds_cfg.get('base64_all') |
1210 | self.script_base_d = os.path.join(self.paths.get_cpath("scripts")) |
1211 | |
1212 | + self._init() |
1213 | + |
1214 | def __str__(self): |
1215 | root = sources.DataSource.__str__(self) |
1216 | - return "%s [seed=%s]" % (root, self.seed) |
1217 | - |
1218 | - def _get_seed_file_object(self): |
1219 | - if not self.seed: |
1220 | - raise AttributeError("seed device is not set") |
1221 | - |
1222 | - if self.smartos_type == 'lx-brand': |
1223 | - if not stat.S_ISSOCK(os.stat(self.seed).st_mode): |
1224 | - LOG.debug("Seed %s is not a socket", self.seed) |
1225 | - return None |
1226 | - sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) |
1227 | - sock.connect(self.seed) |
1228 | - return sock.makefile('rwb') |
1229 | - else: |
1230 | - if not stat.S_ISCHR(os.stat(self.seed).st_mode): |
1231 | - LOG.debug("Seed %s is not a character device") |
1232 | - return None |
1233 | - ser = serial.Serial(self.seed, timeout=self.seed_timeout) |
1234 | - if not ser.isOpen(): |
1235 | - raise SystemError("Unable to open %s" % self.seed) |
1236 | - return ser |
1237 | - return None |
1238 | + return "%s [client=%s]" % (root, self.md_client) |
1239 | + |
1240 | + def _init(self): |
1241 | + if self.smartos_type == self._unset: |
1242 | + self.smartos_type = get_smartos_environ() |
1243 | + if self.smartos_type is None: |
1244 | + self.md_client = None |
1245 | + |
1246 | + if self.md_client == self._unset: |
1247 | + self.md_client = jmc_client_factory( |
1248 | + smartos_type=self.smartos_type, |
1249 | + metadata_sockfile=self.ds_cfg['metadata_sockfile'], |
1250 | + serial_device=self.ds_cfg['serial_device'], |
1251 | + serial_timeout=self.ds_cfg['serial_timeout']) |
1252 | |
1253 | def _set_provisioned(self): |
1254 | '''Mark the instance provisioning state as successful. |
1255 | @@ -225,50 +223,26 @@ |
1256 | '/'.join([svc_path, 'provision_success'])) |
1257 | |
1258 | def get_data(self): |
1259 | + self._init() |
1260 | + |
1261 | md = {} |
1262 | ud = "" |
1263 | |
1264 | - if not device_exists(self.seed): |
1265 | - LOG.debug("No metadata device '%s' found for SmartOS datasource", |
1266 | - self.seed) |
1267 | - return False |
1268 | - |
1269 | - uname_arch = os.uname()[4] |
1270 | - if uname_arch.startswith("arm") or uname_arch == "aarch64": |
1271 | - # Disabling because dmidcode in dmi_data() crashes kvm process |
1272 | - LOG.debug("Disabling SmartOS datasource on arm (LP: #1243287)") |
1273 | - return False |
1274 | - |
1275 | - # SDC KVM instances will provide dmi data, LX-brand does not |
1276 | - if self.smartos_type == 'kvm': |
1277 | - dmi_info = dmi_data() |
1278 | - if dmi_info is None: |
1279 | - LOG.debug("No dmidata utility found") |
1280 | - return False |
1281 | - |
1282 | - system_type = dmi_info |
1283 | - if 'smartdc' not in system_type.lower(): |
1284 | - LOG.debug("Host is not on SmartOS. system_type=%s", |
1285 | - system_type) |
1286 | - return False |
1287 | - LOG.debug("Host is SmartOS, guest in KVM") |
1288 | - |
1289 | - seed_obj = self._get_seed_file_object() |
1290 | - if seed_obj is None: |
1291 | - LOG.debug('Seed file object not found.') |
1292 | - return False |
1293 | - with contextlib.closing(seed_obj) as seed: |
1294 | - b64_keys = self.query('base64_keys', seed, strip=True, b64=False) |
1295 | - if b64_keys is not None: |
1296 | - self.b64_keys = [k.strip() for k in str(b64_keys).split(',')] |
1297 | - |
1298 | - b64_all = self.query('base64_all', seed, strip=True, b64=False) |
1299 | - if b64_all is not None: |
1300 | - self.b64_all = util.is_true(b64_all) |
1301 | - |
1302 | - for ci_noun, attribute in SMARTOS_ATTRIB_MAP.items(): |
1303 | - smartos_noun, strip = attribute |
1304 | - md[ci_noun] = self.query(smartos_noun, seed, strip=strip) |
1305 | + if not self.smartos_type: |
1306 | + LOG.debug("Not running on smartos") |
1307 | + return False |
1308 | + |
1309 | + if not self.md_client.exists(): |
1310 | + LOG.debug("No metadata device '%r' found for SmartOS datasource", |
1311 | + self.md_client) |
1312 | + return False |
1313 | + |
1314 | + for ci_noun, attribute in SMARTOS_ATTRIB_MAP.items(): |
1315 | + smartos_noun, strip = attribute |
1316 | + md[ci_noun] = self.md_client.get(smartos_noun, strip=strip) |
1317 | + |
1318 | + for ci_noun, smartos_noun in SMARTOS_ATTRIB_JSON.items(): |
1319 | + md[ci_noun] = self.md_client.get_json(smartos_noun) |
1320 | |
1321 | # @datadictionary: This key may contain a program that is written |
1322 | # to a file in the filesystem of the guest on each boot and then |
1323 | @@ -318,6 +292,7 @@ |
1324 | self.metadata = util.mergemanydict([md, self.metadata]) |
1325 | self.userdata_raw = ud |
1326 | self.vendordata_raw = md['vendor-data'] |
1327 | + self.network_data = md['network-data'] |
1328 | |
1329 | self._set_provisioned() |
1330 | return True |
1331 | @@ -326,69 +301,20 @@ |
1332 | return self.ds_cfg['disk_aliases'].get(name) |
1333 | |
1334 | def get_config_obj(self): |
1335 | - return self.cfg |
1336 | + if self.smartos_type == SMARTOS_ENV_KVM: |
1337 | + return BUILTIN_CLOUD_CONFIG |
1338 | + return {} |
1339 | |
1340 | def get_instance_id(self): |
1341 | return self.metadata['instance-id'] |
1342 | |
1343 | - def query(self, noun, seed_file, strip=False, default=None, b64=None): |
1344 | - if b64 is None: |
1345 | - if noun in self.smartos_no_base64: |
1346 | - b64 = False |
1347 | - elif self.b64_all or noun in self.b64_keys: |
1348 | - b64 = True |
1349 | - |
1350 | - return self._query_data(noun, seed_file, strip=strip, |
1351 | - default=default, b64=b64) |
1352 | - |
1353 | - def _query_data(self, noun, seed_file, strip=False, |
1354 | - default=None, b64=None): |
1355 | - """Makes a request via "GET <NOUN>" |
1356 | - |
1357 | - In the response, the first line is the status, while subsequent |
1358 | - lines are is the value. A blank line with a "." is used to |
1359 | - indicate end of response. |
1360 | - |
1361 | - If the response is expected to be base64 encoded, then set |
1362 | - b64encoded to true. Unfortantely, there is no way to know if |
1363 | - something is 100% encoded, so this method relies on being told |
1364 | - if the data is base64 or not. |
1365 | - """ |
1366 | - |
1367 | - if not noun: |
1368 | - return False |
1369 | - |
1370 | - response = JoyentMetadataClient(seed_file).get_metadata(noun) |
1371 | - |
1372 | - if response is None: |
1373 | - return default |
1374 | - |
1375 | - if b64 is None: |
1376 | - b64 = self._query_data('b64-%s' % noun, seed_file, b64=False, |
1377 | - default=False, strip=True) |
1378 | - b64 = util.is_true(b64) |
1379 | - |
1380 | - resp = None |
1381 | - if b64 or strip: |
1382 | - resp = "".join(response).rstrip() |
1383 | - else: |
1384 | - resp = "".join(response) |
1385 | - |
1386 | - if b64: |
1387 | - try: |
1388 | - return util.b64d(resp) |
1389 | - # Bogus input produces different errors in Python 2 and 3; |
1390 | - # catch both. |
1391 | - except (TypeError, binascii.Error): |
1392 | - LOG.warn("Failed base64 decoding key '%s'", noun) |
1393 | - return resp |
1394 | - |
1395 | - return resp |
1396 | - |
1397 | - |
1398 | -def device_exists(device): |
1399 | - """Symplistic method to determine if the device exists or not""" |
1400 | - return os.path.exists(device) |
1401 | + @property |
1402 | + def network_config(self): |
1403 | + if self._network_config is None: |
1404 | + if self.network_data is not None: |
1405 | + self._network_config = ( |
1406 | + convert_smartos_network_data(self.network_data)) |
1407 | + return self._network_config |
1408 | |
1409 | |
1410 | class JoyentMetadataFetchException(Exception): |
1411 | @@ -407,8 +333,11 @@ |
1412 | r' (?P<body>(?P<request_id>[0-9a-f]+) (?P<status>SUCCESS|NOTFOUND)' |
1413 | r'( (?P<payload>.+))?)') |
1414 | |
1415 | - def __init__(self, metasource): |
1416 | - self.metasource = metasource |
1417 | + def __init__(self, smartos_type=None, fp=None): |
1418 | + if smartos_type is None: |
1419 | + smartos_type = get_smartos_environ() |
1420 | + self.smartos_type = smartos_type |
1421 | + self.fp = fp |
1422 | |
1423 | def _checksum(self, body): |
1424 | return '{0:08x}'.format( |
1425 | @@ -436,37 +365,229 @@ |
1426 | LOG.debug('Value "%s" found.', value) |
1427 | return value |
1428 | |
1429 | - def get_metadata(self, metadata_key): |
1430 | - LOG.debug('Fetching metadata key "%s"...', metadata_key) |
1431 | + def request(self, rtype, param=None): |
1432 | request_id = '{0:08x}'.format(random.randint(0, 0xffffffff)) |
1433 | - message_body = '{0} GET {1}'.format(request_id, |
1434 | - util.b64e(metadata_key)) |
1435 | + message_body = ' '.join((request_id, rtype,)) |
1436 | + if param: |
1437 | + message_body += ' ' + base64.b64encode(param.encode()).decode() |
1438 | msg = 'V2 {0} {1} {2}\n'.format( |
1439 | len(message_body), self._checksum(message_body), message_body) |
1440 | LOG.debug('Writing "%s" to metadata transport.', msg) |
1441 | - self.metasource.write(msg.encode('ascii')) |
1442 | - self.metasource.flush() |
1443 | + |
1444 | + need_close = False |
1445 | + if not self.fp: |
1446 | + self.open_transport() |
1447 | + need_close = True |
1448 | + |
1449 | + self.fp.write(msg.encode('ascii')) |
1450 | + self.fp.flush() |
1451 | |
1452 | response = bytearray() |
1453 | - response.extend(self.metasource.read(1)) |
1454 | + response.extend(self.fp.read(1)) |
1455 | while response[-1:] != b'\n': |
1456 | - response.extend(self.metasource.read(1)) |
1457 | + response.extend(self.fp.read(1)) |
1458 | + |
1459 | + if need_close: |
1460 | + self.close_transport() |
1461 | + |
1462 | response = response.rstrip().decode('ascii') |
1463 | LOG.debug('Read "%s" from metadata transport.', response) |
1464 | |
1465 | if 'SUCCESS' not in response: |
1466 | return None |
1467 | |
1468 | - return self._get_value_from_frame(request_id, response) |
1469 | - |
1470 | - |
1471 | -def dmi_data(): |
1472 | - sys_type = util.read_dmi_data("system-product-name") |
1473 | - |
1474 | - if not sys_type: |
1475 | + value = self._get_value_from_frame(request_id, response) |
1476 | + return value |
1477 | + |
1478 | + def get(self, key, default=None, strip=False): |
1479 | + result = self.request(rtype='GET', param=key) |
1480 | + if result is None: |
1481 | + return default |
1482 | + if result and strip: |
1483 | + result = result.strip() |
1484 | + return result |
1485 | + |
1486 | + def get_json(self, key, default=None): |
1487 | + result = self.get(key, default=default) |
1488 | + if result is None: |
1489 | + return default |
1490 | + return json.loads(result) |
1491 | + |
1492 | + def list(self): |
1493 | + result = self.request(rtype='KEYS') |
1494 | + if result: |
1495 | + result = result.split('\n') |
1496 | + return result |
1497 | + |
1498 | + def put(self, key, val): |
1499 | + param = b' '.join([base64.b64encode(i.encode()) |
1500 | + for i in (key, val)]).decode() |
1501 | + return self.request(rtype='PUT', param=param) |
1502 | + |
1503 | + def delete(self, key): |
1504 | + return self.request(rtype='DELETE', param=key) |
1505 | + |
1506 | + def close_transport(self): |
1507 | + if self.fp: |
1508 | + self.fp.close() |
1509 | + self.fp = None |
1510 | + |
1511 | + def __enter__(self): |
1512 | + if self.fp: |
1513 | + return self |
1514 | + self.open_transport() |
1515 | + return self |
1516 | + |
1517 | + def __exit__(self, exc_type, exc_value, traceback): |
1518 | + self.close_transport() |
1519 | + return |
1520 | + |
1521 | + def open_transport(self): |
1522 | + raise NotImplementedError |
1523 | + |
1524 | + |
1525 | +class JoyentMetadataSocketClient(JoyentMetadataClient): |
1526 | + def __init__(self, socketpath): |
1527 | + self.socketpath = socketpath |
1528 | + |
1529 | + def open_transport(self): |
1530 | + sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) |
1531 | + sock.connect(self.socketpath) |
1532 | + self.fp = sock.makefile('rwb') |
1533 | + |
1534 | + def exists(self): |
1535 | + return os.path.exists(self.socketpath) |
1536 | + |
1537 | + def __repr__(self): |
1538 | + return "%s(socketpath=%s)" % (self.__class__.__name__, self.socketpath) |
1539 | + |
1540 | + |
1541 | +class JoyentMetadataSerialClient(JoyentMetadataClient): |
1542 | + def __init__(self, device, timeout=10, smartos_type=None): |
1543 | + super(JoyentMetadataSerialClient, self).__init__(smartos_type) |
1544 | + self.device = device |
1545 | + self.timeout = timeout |
1546 | + |
1547 | + def exists(self): |
1548 | + return os.path.exists(self.device) |
1549 | + |
1550 | + def open_transport(self): |
1551 | + ser = serial.Serial(self.device, timeout=self.timeout) |
1552 | + if not ser.isOpen(): |
1553 | + raise SystemError("Unable to open %s" % self.device) |
1554 | + self.fp = ser |
1555 | + |
1556 | + def __repr__(self): |
1557 | + return "%s(device=%s, timeout=%s)" % ( |
1558 | + self.__class__.__name__, self.device, self.timeout) |
1559 | + |
1560 | + |
1561 | +class JoyentMetadataLegacySerialClient(JoyentMetadataSerialClient): |
1562 | + """V1 of the protocol was not safe for all values. |
1563 | + Thus, we allowed the user to pass values in as base64 encoded. |
1564 | + Users may still reasonably expect to be able to send base64 data |
1565 | + and have it transparently decoded. So even though the V2 format is |
1566 | + now used, and is safe (using base64 itself), we keep legacy support. |
1567 | + |
1568 | + The way for a user to do this was: |
1569 | + a.) specify 'base64_keys' key whose value is a comma delimited |
1570 | + list of keys that were base64 encoded. |
1571 | + b.) base64_all: string interpreted as a boolean that indicates |
1572 | + if all keys are base64 encoded. |
1573 | + c.) set a key named b64-<keyname> with a boolean indicating that |
1574 | + <keyname> is base64 encoded.""" |
1575 | + |
1576 | + def __init__(self, device, timeout=10, smartos_type=None): |
1577 | + s = super(JoyentMetadataLegacySerialClient, self) |
1578 | + s.__init__(device, timeout, smartos_type) |
1579 | + self.base64_keys = None |
1580 | + self.base64_all = None |
1581 | + |
1582 | + def _init_base64_keys(self, reset=False): |
1583 | + if reset: |
1584 | + self.base64_keys = None |
1585 | + self.base64_all = None |
1586 | + |
1587 | + keys = None |
1588 | + if self.base64_all is None: |
1589 | + keys = self.list() |
1590 | + if 'base64_all' in keys: |
1591 | + self.base64_all = util.is_true(self._get("base64_all")) |
1592 | + else: |
1593 | + self.base64_all = False |
1594 | + |
1595 | + if self.base64_all: |
1596 | + # short circuit if base64_all is true |
1597 | + return |
1598 | + |
1599 | + if self.base64_keys is None: |
1600 | + if keys is None: |
1601 | + keys = self.list() |
1602 | + b64_keys = set() |
1603 | + if 'base64_keys' in keys: |
1604 | + b64_keys = set(self._get("base64_keys").split(",")) |
1605 | + |
1606 | + # now add any b64-<keyname> that has a true value |
1607 | + for key in [k[3:] for k in keys if k.startswith("b64-")]: |
1608 | + if util.is_true(self._get(key)): |
1609 | + b64_keys.add(key) |
1610 | + else: |
1611 | + if key in b64_keys: |
1612 | + b64_keys.remove(key) |
1613 | + |
1614 | + self.base64_keys = b64_keys |
1615 | + |
1616 | + def _get(self, key, default=None, strip=False): |
1617 | + return (super(JoyentMetadataLegacySerialClient, self). |
1618 | + get(key, default=default, strip=strip)) |
1619 | + |
1620 | + def is_b64_encoded(self, key, reset=False): |
1621 | + if key in NO_BASE64_DECODE: |
1622 | + return False |
1623 | + |
1624 | + self._init_base64_keys(reset=reset) |
1625 | + if self.base64_all: |
1626 | + return True |
1627 | + |
1628 | + return key in self.base64_keys |
1629 | + |
1630 | + def get(self, key, default=None, strip=False): |
1631 | + mdefault = object() |
1632 | + val = self._get(key, strip=False, default=mdefault) |
1633 | + if val is mdefault: |
1634 | + return default |
1635 | + |
1636 | + if self.is_b64_encoded(key): |
1637 | + try: |
1638 | + val = base64.b64decode(val.encode()).decode() |
1639 | + # Bogus input produces different errors in Python 2 and 3 |
1640 | + except (TypeError, binascii.Error): |
1641 | + LOG.warn("Failed base64 decoding key '%s': %s", key, val) |
1642 | + |
1643 | + if strip: |
1644 | + val = val.strip() |
1645 | + |
1646 | + return val |
1647 | + |
1648 | + |
1649 | +def jmc_client_factory( |
1650 | + smartos_type=None, metadata_sockfile=METADATA_SOCKFILE, |
1651 | + serial_device=SERIAL_DEVICE, serial_timeout=SERIAL_TIMEOUT, |
1652 | + uname_version=None): |
1653 | + |
1654 | + if smartos_type is None: |
1655 | + smartos_type = get_smartos_environ(uname_version) |
1656 | + |
1657 | + if smartos_type is None: |
1658 | return None |
1659 | + elif smartos_type == SMARTOS_ENV_KVM: |
1660 | + return JoyentMetadataLegacySerialClient( |
1661 | + device=serial_device, timeout=serial_timeout, |
1662 | + smartos_type=smartos_type) |
1663 | + elif smartos_type == SMARTOS_ENV_LX_BRAND: |
1664 | + return JoyentMetadataSocketClient(socketpath=metadata_sockfile) |
1665 | |
1666 | - return sys_type |
1667 | + raise ValueError("Unknown value for smartos_type: %s" % smartos_type) |
1668 | |
1669 | |
1670 | def write_boot_content(content, content_f, link=None, shebang=False, |
1671 | @@ -522,15 +643,141 @@ |
1672 | util.ensure_dir(os.path.dirname(link)) |
1673 | os.symlink(content_f, link) |
1674 | except IOError as e: |
1675 | - util.logexc(LOG, "failed establishing content link", e) |
1676 | + util.logexc(LOG, "failed establishing content link: %s", e) |
1677 | + |
1678 | + |
1679 | +def get_smartos_environ(uname_version=None, product_name=None, |
1680 | + uname_arch=None): |
1681 | + uname = os.uname() |
1682 | + if uname_arch is None: |
1683 | + uname_arch = uname[4] |
1684 | + |
1685 | + if uname_arch.startswith("arm") or uname_arch == "aarch64": |
1686 | + return None |
1687 | + |
1688 | + # SDC LX-Brand Zones lack dmidecode (no /dev/mem) but |
1689 | + # report 'BrandZ virtual linux' as the kernel version |
1690 | + if uname_version is None: |
1691 | + uname_version = uname[3] |
1692 | + if uname_version.lower() == 'brandz virtual linux': |
1693 | + return SMARTOS_ENV_LX_BRAND |
1694 | + |
1695 | + if product_name is None: |
1696 | + system_type = util.read_dmi_data("system-product-name") |
1697 | + else: |
1698 | + system_type = product_name |
1699 | + |
1700 | + if system_type and 'smartdc' in system_type.lower(): |
1701 | + return SMARTOS_ENV_KVM |
1702 | + |
1703 | + return None |
1704 | + |
1705 | + |
1706 | +# Covert SMARTOS 'sdc:nics' data to network_config yaml |
1707 | +def convert_smartos_network_data(network_data=None): |
1708 | + """Return a dictionary of network_config by parsing provided |
1709 | + SMARTOS sdc:nics configuration data |
1710 | + |
1711 | + sdc:nics data is a dictionary of properties of a nic and the ip |
1712 | + configuration desired. Additional nic dictionaries are appended |
1713 | + to the list. |
1714 | + |
1715 | + Converting the format is straightforward though it does include |
1716 | + duplicate information as well as data which appears to be relevant |
1717 | + to the hostOS rather than the guest. |
1718 | + |
1719 | + For each entry in the nics list returned from query sdc:nics, we |
1720 | + create a type: physical entry, and extract the interface properties: |
1721 | + 'mac' -> 'mac_address', 'mtu', 'interface' -> 'name'. The remaining |
1722 | + keys are related to ip configuration. For each ip in the 'ips' list |
1723 | + we create a subnet entry under 'subnets' pairing the ip to a one in |
1724 | + the 'gateways' list. |
1725 | + """ |
1726 | + |
1727 | + valid_keys = { |
1728 | + 'physical': [ |
1729 | + 'mac_address', |
1730 | + 'mtu', |
1731 | + 'name', |
1732 | + 'params', |
1733 | + 'subnets', |
1734 | + 'type', |
1735 | + ], |
1736 | + 'subnet': [ |
1737 | + 'address', |
1738 | + 'broadcast', |
1739 | + 'dns_nameservers', |
1740 | + 'dns_search', |
1741 | + 'gateway', |
1742 | + 'metric', |
1743 | + 'netmask', |
1744 | + 'pointopoint', |
1745 | + 'routes', |
1746 | + 'scope', |
1747 | + 'type', |
1748 | + ], |
1749 | + } |
1750 | + |
1751 | + config = [] |
1752 | + for nic in network_data: |
1753 | + cfg = {k: v for k, v in nic.items() |
1754 | + if k in valid_keys['physical']} |
1755 | + cfg.update({ |
1756 | + 'type': 'physical', |
1757 | + 'name': nic['interface']}) |
1758 | + if 'mac' in nic: |
1759 | + cfg.update({'mac_address': nic['mac']}) |
1760 | + |
1761 | + subnets = [] |
1762 | + for ip, gw in zip(nic['ips'], nic['gateways']): |
1763 | + subnet = {k: v for k, v in nic.items() |
1764 | + if k in valid_keys['subnet']} |
1765 | + subnet.update({ |
1766 | + 'type': 'static', |
1767 | + 'address': ip, |
1768 | + 'gateway': gw, |
1769 | + }) |
1770 | + subnets.append(subnet) |
1771 | + cfg.update({'subnets': subnets}) |
1772 | + config.append(cfg) |
1773 | + |
1774 | + return {'version': 1, 'config': config} |
1775 | |
1776 | |
1777 | # Used to match classes to dependencies |
1778 | datasources = [ |
1779 | - (DataSourceSmartOS, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)), |
1780 | + (DataSourceSmartOS, (sources.DEP_FILESYSTEM, )), |
1781 | ] |
1782 | |
1783 | |
1784 | # Return a list of data sources that match this set of dependencies |
1785 | def get_datasource_list(depends): |
1786 | return sources.list_from_depends(depends, datasources) |
1787 | + |
1788 | + |
1789 | +if __name__ == "__main__": |
1790 | + import sys |
1791 | + jmc = jmc_client_factory() |
1792 | + if jmc is None: |
1793 | + print("Do not appear to be on smartos.") |
1794 | + sys.exit(1) |
1795 | + if len(sys.argv) == 1: |
1796 | + keys = (list(SMARTOS_ATTRIB_JSON.keys()) + |
1797 | + list(SMARTOS_ATTRIB_MAP.keys())) |
1798 | + else: |
1799 | + keys = sys.argv[1:] |
1800 | + |
1801 | + data = {} |
1802 | + for key in keys: |
1803 | + if key in SMARTOS_ATTRIB_JSON: |
1804 | + keyname = SMARTOS_ATTRIB_JSON[key] |
1805 | + data[key] = jmc.get_json(keyname) |
1806 | + else: |
1807 | + if key in SMARTOS_ATTRIB_MAP: |
1808 | + keyname, strip = SMARTOS_ATTRIB_MAP[key] |
1809 | + else: |
1810 | + keyname, strip = (key, False) |
1811 | + val = jmc.get(keyname, strip=strip) |
1812 | + data[key] = jmc.get(keyname, strip=strip) |
1813 | + |
1814 | + print(json.dumps(data, indent=1)) |
1815 | |
1816 | === modified file 'cloudinit/sources/__init__.py' |
1817 | --- cloudinit/sources/__init__.py 2016-05-12 17:56:26 +0000 |
1818 | +++ cloudinit/sources/__init__.py 2016-06-03 19:07:08 +0000 |
1819 | @@ -34,6 +34,13 @@ |
1820 | from cloudinit.filters import launch_index |
1821 | from cloudinit.reporting import events |
1822 | |
1823 | +DSMODE_DISABLED = "disabled" |
1824 | +DSMODE_LOCAL = "local" |
1825 | +DSMODE_NETWORK = "net" |
1826 | +DSMODE_PASS = "pass" |
1827 | + |
1828 | +VALID_DSMODES = [DSMODE_DISABLED, DSMODE_LOCAL, DSMODE_NETWORK] |
1829 | + |
1830 | DEP_FILESYSTEM = "FILESYSTEM" |
1831 | DEP_NETWORK = "NETWORK" |
1832 | DS_PREFIX = 'DataSource' |
1833 | @@ -57,6 +64,7 @@ |
1834 | self.userdata_raw = None |
1835 | self.vendordata = None |
1836 | self.vendordata_raw = None |
1837 | + self.dsmode = DSMODE_NETWORK |
1838 | |
1839 | # find the datasource config name. |
1840 | # remove 'DataSource' from classname on front, and remove 'Net' on end. |
1841 | @@ -223,10 +231,35 @@ |
1842 | # quickly (local check only) if self.instance_id is still |
1843 | return False |
1844 | |
1845 | + @staticmethod |
1846 | + def _determine_dsmode(candidates, default=None, valid=None): |
1847 | + # return the first candidate that is non None, warn if not valid |
1848 | + if default is None: |
1849 | + default = DSMODE_NETWORK |
1850 | + |
1851 | + if valid is None: |
1852 | + valid = VALID_DSMODES |
1853 | + |
1854 | + for candidate in candidates: |
1855 | + if candidate is None: |
1856 | + continue |
1857 | + if candidate in valid: |
1858 | + return candidate |
1859 | + else: |
1860 | + LOG.warn("invalid dsmode '%s', using default=%s", |
1861 | + candidate, default) |
1862 | + return default |
1863 | + |
1864 | + return default |
1865 | + |
1866 | @property |
1867 | def network_config(self): |
1868 | return None |
1869 | |
1870 | + @property |
1871 | + def first_instance_boot(self): |
1872 | + return |
1873 | + |
1874 | |
1875 | def normalize_pubkey_data(pubkey_data): |
1876 | keys = [] |
1877 | |
1878 | === modified file 'cloudinit/sources/helpers/openstack.py' |
1879 | --- cloudinit/sources/helpers/openstack.py 2016-05-12 17:56:26 +0000 |
1880 | +++ cloudinit/sources/helpers/openstack.py 2016-06-03 19:07:08 +0000 |
1881 | @@ -190,14 +190,14 @@ |
1882 | versions_available) |
1883 | return selected_version |
1884 | |
1885 | - def _read_content_path(self, item): |
1886 | + def _read_content_path(self, item, decode=False): |
1887 | path = item.get('content_path', '').lstrip("/") |
1888 | path_pieces = path.split("/") |
1889 | valid_pieces = [p for p in path_pieces if len(p)] |
1890 | if not valid_pieces: |
1891 | raise BrokenMetadata("Item %s has no valid content path" % (item)) |
1892 | path = self._path_join(self.base_path, "openstack", *path_pieces) |
1893 | - return self._path_read(path) |
1894 | + return self._path_read(path, decode=decode) |
1895 | |
1896 | def read_v2(self): |
1897 | """Reads a version 2 formatted location. |
1898 | @@ -298,7 +298,8 @@ |
1899 | net_item = metadata.get("network_config", None) |
1900 | if net_item: |
1901 | try: |
1902 | - results['network_config'] = self._read_content_path(net_item) |
1903 | + content = self._read_content_path(net_item, decode=True) |
1904 | + results['network_config'] = content |
1905 | except IOError as e: |
1906 | raise BrokenMetadata("Failed to read network" |
1907 | " configuration: %s" % (e)) |
1908 | @@ -333,8 +334,8 @@ |
1909 | components = [base] + list(add_ons) |
1910 | return os.path.join(*components) |
1911 | |
1912 | - def _path_read(self, path): |
1913 | - return util.load_file(path, decode=False) |
1914 | + def _path_read(self, path, decode=False): |
1915 | + return util.load_file(path, decode=decode) |
1916 | |
1917 | def _fetch_available_versions(self): |
1918 | if self._versions is None: |
1919 | @@ -446,7 +447,7 @@ |
1920 | self._versions = found |
1921 | return self._versions |
1922 | |
1923 | - def _path_read(self, path): |
1924 | + def _path_read(self, path, decode=False): |
1925 | |
1926 | def should_retry_cb(_request_args, cause): |
1927 | try: |
1928 | @@ -463,7 +464,10 @@ |
1929 | ssl_details=self.ssl_details, |
1930 | timeout=self.timeout, |
1931 | exception_cb=should_retry_cb) |
1932 | - return response.contents |
1933 | + if decode: |
1934 | + return response.contents.decode() |
1935 | + else: |
1936 | + return response.contents |
1937 | |
1938 | def _path_join(self, base, *add_ons): |
1939 | return url_helper.combine_url(base, *add_ons) |
1940 | |
1941 | === modified file 'cloudinit/stages.py' |
1942 | --- cloudinit/stages.py 2016-05-26 13:02:17 +0000 |
1943 | +++ cloudinit/stages.py 2016-06-03 19:07:08 +0000 |
1944 | @@ -52,6 +52,7 @@ |
1945 | LOG = logging.getLogger(__name__) |
1946 | |
1947 | NULL_DATA_SOURCE = None |
1948 | +NO_PREVIOUS_INSTANCE_ID = "NO_PREVIOUS_INSTANCE_ID" |
1949 | |
1950 | |
1951 | class Init(object): |
1952 | @@ -67,6 +68,7 @@ |
1953 | # Changed only when a fetch occurs |
1954 | self.datasource = NULL_DATA_SOURCE |
1955 | self.ds_restored = False |
1956 | + self._previous_iid = None |
1957 | |
1958 | if reporter is None: |
1959 | reporter = events.ReportEventStack( |
1960 | @@ -213,6 +215,31 @@ |
1961 | cfg_list = self.cfg.get('datasource_list') or [] |
1962 | return (cfg_list, pkg_list) |
1963 | |
1964 | + def _restore_from_checked_cache(self, existing): |
1965 | + if existing not in ("check", "trust"): |
1966 | + raise ValueError("Unexpected value for existing: %s" % existing) |
1967 | + |
1968 | + ds = self._restore_from_cache() |
1969 | + if not ds: |
1970 | + return (None, "no cache found") |
1971 | + |
1972 | + run_iid_fn = self.paths.get_runpath('instance_id') |
1973 | + if os.path.exists(run_iid_fn): |
1974 | + run_iid = util.load_file(run_iid_fn).strip() |
1975 | + else: |
1976 | + run_iid = None |
1977 | + |
1978 | + if run_iid == ds.get_instance_id(): |
1979 | + return (ds, "restored from cache with run check: %s" % ds) |
1980 | + elif existing == "trust": |
1981 | + return (ds, "restored from cache: %s" % ds) |
1982 | + else: |
1983 | + if (hasattr(ds, 'check_instance_id') and |
1984 | + ds.check_instance_id(self.cfg)): |
1985 | + return (ds, "restored from checked cache: %s" % ds) |
1986 | + else: |
1987 | + return (None, "cache invalid in datasource: %s" % ds) |
1988 | + |
1989 | def _get_data_source(self, existing): |
1990 | if self.datasource is not NULL_DATA_SOURCE: |
1991 | return self.datasource |
1992 | @@ -221,19 +248,9 @@ |
1993 | name="check-cache", |
1994 | description="attempting to read from cache [%s]" % existing, |
1995 | parent=self.reporter) as myrep: |
1996 | - ds = self._restore_from_cache() |
1997 | - if ds and existing == "trust": |
1998 | - myrep.description = "restored from cache: %s" % ds |
1999 | - elif ds and existing == "check": |
2000 | - if (hasattr(ds, 'check_instance_id') and |
2001 | - ds.check_instance_id(self.cfg)): |
2002 | - myrep.description = "restored from checked cache: %s" % ds |
2003 | - else: |
2004 | - myrep.description = "cache invalid in datasource: %s" % ds |
2005 | - ds = None |
2006 | - else: |
2007 | - myrep.description = "no cache found" |
2008 | |
2009 | + ds, desc = self._restore_from_checked_cache(existing) |
2010 | + myrep.description = desc |
2011 | self.ds_restored = bool(ds) |
2012 | LOG.debug(myrep.description) |
2013 | |
2014 | @@ -301,23 +318,41 @@ |
2015 | |
2016 | # What the instance id was and is... |
2017 | iid = self.datasource.get_instance_id() |
2018 | - previous_iid = None |
2019 | iid_fn = os.path.join(dp, 'instance-id') |
2020 | - try: |
2021 | - previous_iid = util.load_file(iid_fn).strip() |
2022 | - except Exception: |
2023 | - pass |
2024 | - if not previous_iid: |
2025 | - previous_iid = iid |
2026 | + |
2027 | + previous_iid = self.previous_iid() |
2028 | util.write_file(iid_fn, "%s\n" % iid) |
2029 | + util.write_file(self.paths.get_runpath('instance_id'), "%s\n" % iid) |
2030 | util.write_file(os.path.join(dp, 'previous-instance-id'), |
2031 | "%s\n" % (previous_iid)) |
2032 | + |
2033 | + self._write_to_cache() |
2034 | # Ensure needed components are regenerated |
2035 | # after change of instance which may cause |
2036 | # change of configuration |
2037 | self._reset() |
2038 | return iid |
2039 | |
2040 | + def previous_iid(self): |
2041 | + if self._previous_iid is not None: |
2042 | + return self._previous_iid |
2043 | + |
2044 | + dp = self.paths.get_cpath('data') |
2045 | + iid_fn = os.path.join(dp, 'instance-id') |
2046 | + try: |
2047 | + self._previous_iid = util.load_file(iid_fn).strip() |
2048 | + except Exception: |
2049 | + self._previous_iid = NO_PREVIOUS_INSTANCE_ID |
2050 | + |
2051 | + LOG.debug("previous iid found to be %s", self._previous_iid) |
2052 | + return self._previous_iid |
2053 | + |
2054 | + def is_new_instance(self): |
2055 | + previous = self.previous_iid() |
2056 | + ret = (previous == NO_PREVIOUS_INSTANCE_ID or |
2057 | + previous != self.datasource.get_instance_id()) |
2058 | + return ret |
2059 | + |
2060 | def fetch(self, existing="check"): |
2061 | return self._get_data_source(existing=existing) |
2062 | |
2063 | @@ -332,8 +367,6 @@ |
2064 | reporter=self.reporter) |
2065 | |
2066 | def update(self): |
2067 | - if not self._write_to_cache(): |
2068 | - return |
2069 | self._store_userdata() |
2070 | self._store_vendordata() |
2071 | |
2072 | @@ -593,15 +626,27 @@ |
2073 | return (ncfg, loc) |
2074 | return (net.generate_fallback_config(), "fallback") |
2075 | |
2076 | - def apply_network_config(self): |
2077 | + def apply_network_config(self, bring_up): |
2078 | netcfg, src = self._find_networking_config() |
2079 | if netcfg is None: |
2080 | LOG.info("network config is disabled by %s", src) |
2081 | return |
2082 | |
2083 | - LOG.info("Applying network configuration from %s: %s", src, netcfg) |
2084 | - try: |
2085 | - return self.distro.apply_network_config(netcfg) |
2086 | + try: |
2087 | + LOG.debug("applying net config names for %s" % netcfg) |
2088 | + self.distro.apply_network_config_names(netcfg) |
2089 | + except Exception as e: |
2090 | + LOG.warn("Failed to rename devices: %s", e) |
2091 | + |
2092 | + if (self.datasource is not NULL_DATA_SOURCE and |
2093 | + not self.is_new_instance()): |
2094 | + LOG.debug("not a new instance. network config is not applied.") |
2095 | + return |
2096 | + |
2097 | + LOG.info("Applying network configuration from %s bringup=%s: %s", |
2098 | + src, bring_up, netcfg) |
2099 | + try: |
2100 | + return self.distro.apply_network_config(netcfg, bring_up=bring_up) |
2101 | except NotImplementedError: |
2102 | LOG.warn("distro '%s' does not implement apply_network_config. " |
2103 | "networking may not be configured properly." % |
2104 | |
2105 | === modified file 'setup.py' |
2106 | --- setup.py 2016-05-12 20:49:10 +0000 |
2107 | +++ setup.py 2016-06-03 19:07:08 +0000 |
2108 | @@ -184,7 +184,6 @@ |
2109 | (USR + '/share/doc/cloud-init/examples/seed', |
2110 | [f for f in glob('doc/examples/seed/*') if is_f(f)]), |
2111 | (LIB + '/udev/rules.d', [f for f in glob('udev/*.rules')]), |
2112 | - (LIB + '/udev', ['udev/cloud-init-wait']), |
2113 | ] |
2114 | # Use a subclass for install that handles |
2115 | # adding on the right init system configuration files |
2116 | |
2117 | === modified file 'systemd/cloud-init-generator' |
2118 | --- systemd/cloud-init-generator 2016-03-19 00:40:54 +0000 |
2119 | +++ systemd/cloud-init-generator 2016-06-03 19:07:08 +0000 |
2120 | @@ -107,9 +107,6 @@ |
2121 | "ln $CLOUD_SYSTEM_TARGET $link_path" |
2122 | fi |
2123 | fi |
2124 | - # this touches /run/cloud-init/enabled, which is read by |
2125 | - # udev/cloud-init-wait. If not present, it will exit quickly. |
2126 | - touch "$LOG_D/$ENABLE" |
2127 | elif [ "$result" = "$DISABLE" ]; then |
2128 | if [ -f "$link_path" ]; then |
2129 | if rm -f "$link_path"; then |
2130 | |
2131 | === modified file 'tests/unittests/test_datasource/test_configdrive.py' |
2132 | --- tests/unittests/test_datasource/test_configdrive.py 2016-05-12 20:43:11 +0000 |
2133 | +++ tests/unittests/test_datasource/test_configdrive.py 2016-06-03 19:07:08 +0000 |
2134 | @@ -15,6 +15,7 @@ |
2135 | from contextlib2 import ExitStack |
2136 | |
2137 | from cloudinit import helpers |
2138 | +from cloudinit import net |
2139 | from cloudinit import settings |
2140 | from cloudinit.sources import DataSourceConfigDrive as ds |
2141 | from cloudinit.sources.helpers import openstack |
2142 | @@ -73,7 +74,7 @@ |
2143 | 'type': 'ovs', 'mtu': None, 'id': 'tap2f88d109-5b'}, |
2144 | {'vif_id': '1a5382f8-04c5-4d75-ab98-d666c1ef52cc', |
2145 | 'ethernet_mac_address': 'fa:16:3e:05:30:fe', |
2146 | - 'type': 'ovs', 'mtu': None, 'id': 'tap1a5382f8-04'} |
2147 | + 'type': 'ovs', 'mtu': None, 'id': 'tap1a5382f8-04', 'name': 'nic0'} |
2148 | ], |
2149 | 'networks': [ |
2150 | {'link': 'tap2ecc7709-b3', 'type': 'ipv4_dhcp', |
2151 | @@ -88,6 +89,34 @@ |
2152 | ] |
2153 | } |
2154 | |
2155 | +NETWORK_DATA_2 = { |
2156 | + "services": [ |
2157 | + {"type": "dns", "address": "1.1.1.191"}, |
2158 | + {"type": "dns", "address": "1.1.1.4"}], |
2159 | + "networks": [ |
2160 | + {"network_id": "d94bbe94-7abc-48d4-9c82-4628ea26164a", "type": "ipv4", |
2161 | + "netmask": "255.255.255.248", "link": "eth0", |
2162 | + "routes": [{"netmask": "0.0.0.0", "network": "0.0.0.0", |
2163 | + "gateway": "2.2.2.9"}], |
2164 | + "ip_address": "2.2.2.10", "id": "network0-ipv4"}, |
2165 | + {"network_id": "ca447c83-6409-499b-aaef-6ad1ae995348", "type": "ipv4", |
2166 | + "netmask": "255.255.255.224", "link": "eth1", |
2167 | + "routes": [], "ip_address": "3.3.3.24", "id": "network1-ipv4"}], |
2168 | + "links": [ |
2169 | + {"ethernet_mac_address": "fa:16:3e:dd:50:9a", "mtu": 1500, |
2170 | + "type": "vif", "id": "eth0", "vif_id": "vif-foo1"}, |
2171 | + {"ethernet_mac_address": "fa:16:3e:a8:14:69", "mtu": 1500, |
2172 | + "type": "vif", "id": "eth1", "vif_id": "vif-foo2"}] |
2173 | +} |
2174 | + |
2175 | + |
2176 | +KNOWN_MACS = { |
2177 | + 'fa:16:3e:69:b0:58': 'enp0s1', |
2178 | + 'fa:16:3e:d4:57:ad': 'enp0s2', |
2179 | + 'fa:16:3e:dd:50:9a': 'foo1', |
2180 | + 'fa:16:3e:a8:14:69': 'foo2', |
2181 | +} |
2182 | + |
2183 | CFG_DRIVE_FILES_V2 = { |
2184 | 'ec2/2009-04-04/meta-data.json': json.dumps(EC2_META), |
2185 | 'ec2/2009-04-04/user-data': USER_DATA, |
2186 | @@ -365,10 +394,54 @@ |
2187 | """Verify that network_data is converted and present on ds object.""" |
2188 | populate_dir(self.tmp, CFG_DRIVE_FILES_V2) |
2189 | myds = cfg_ds_from_dir(self.tmp) |
2190 | - network_config = ds.convert_network_data(NETWORK_DATA) |
2191 | + network_config = ds.convert_network_data(NETWORK_DATA, |
2192 | + known_macs=KNOWN_MACS) |
2193 | self.assertEqual(myds.network_config, network_config) |
2194 | |
2195 | |
2196 | +class TestConvertNetworkData(TestCase): |
2197 | + def _getnames_in_config(self, ncfg): |
2198 | + return set([n['name'] for n in ncfg['config'] |
2199 | + if n['type'] == 'physical']) |
2200 | + |
2201 | + def test_conversion_fills_names(self): |
2202 | + ncfg = ds.convert_network_data(NETWORK_DATA, known_macs=KNOWN_MACS) |
2203 | + expected = set(['nic0', 'enp0s1', 'enp0s2']) |
2204 | + found = self._getnames_in_config(ncfg) |
2205 | + self.assertEqual(found, expected) |
2206 | + |
2207 | + @mock.patch('cloudinit.net.get_interfaces_by_mac') |
2208 | + def test_convert_reads_system_prefers_name(self, get_interfaces_by_mac): |
2209 | + macs = KNOWN_MACS.copy() |
2210 | + macs.update({'fa:16:3e:05:30:fe': 'foonic1', |
2211 | + 'fa:16:3e:69:b0:58': 'ens1'}) |
2212 | + get_interfaces_by_mac.return_value = macs |
2213 | + |
2214 | + ncfg = ds.convert_network_data(NETWORK_DATA) |
2215 | + expected = set(['nic0', 'ens1', 'enp0s2']) |
2216 | + found = self._getnames_in_config(ncfg) |
2217 | + self.assertEqual(found, expected) |
2218 | + |
2219 | + def test_convert_raises_value_error_on_missing_name(self): |
2220 | + macs = {'aa:aa:aa:aa:aa:00': 'ens1'} |
2221 | + self.assertRaises(ValueError, ds.convert_network_data, |
2222 | + NETWORK_DATA, known_macs=macs) |
2223 | + |
2224 | + def test_conversion_with_route(self): |
2225 | + ncfg = ds.convert_network_data(NETWORK_DATA_2, known_macs=KNOWN_MACS) |
2226 | + # not the best test, but see that we get a route in the |
2227 | + # network config and that it gets rendered to an ENI file |
2228 | + routes = [] |
2229 | + for n in ncfg['config']: |
2230 | + for s in n.get('subnets', []): |
2231 | + routes.extend(s.get('routes', [])) |
2232 | + self.assertIn( |
2233 | + {'network': '0.0.0.0', 'netmask': '0.0.0.0', 'gateway': '2.2.2.9'}, |
2234 | + routes) |
2235 | + eni = net.render_interfaces(net.parse_net_config_data(ncfg)) |
2236 | + self.assertIn("route add default gw 2.2.2.9", eni) |
2237 | + |
2238 | + |
2239 | def cfg_ds_from_dir(seed_d): |
2240 | found = ds.read_config_drive(seed_d) |
2241 | cfg_ds = ds.DataSourceConfigDrive(settings.CFG_BUILTIN, None, |
2242 | @@ -387,7 +460,8 @@ |
2243 | cfg_ds.userdata_raw = results.get('userdata') |
2244 | cfg_ds.version = results.get('version') |
2245 | cfg_ds.network_json = results.get('networkdata') |
2246 | - cfg_ds._network_config = ds.convert_network_data(cfg_ds.network_json) |
2247 | + cfg_ds._network_config = ds.convert_network_data( |
2248 | + cfg_ds.network_json, known_macs=KNOWN_MACS) |
2249 | |
2250 | |
2251 | def populate_dir(seed_dir, files): |
2252 | |
2253 | === modified file 'tests/unittests/test_datasource/test_smartos.py' |
2254 | --- tests/unittests/test_datasource/test_smartos.py 2016-05-12 20:43:11 +0000 |
2255 | +++ tests/unittests/test_datasource/test_smartos.py 2016-06-03 19:07:08 +0000 |
2256 | @@ -25,6 +25,7 @@ |
2257 | from __future__ import print_function |
2258 | |
2259 | from binascii import crc32 |
2260 | +import json |
2261 | import os |
2262 | import os.path |
2263 | import re |
2264 | @@ -40,12 +41,49 @@ |
2265 | from cloudinit.sources import DataSourceSmartOS |
2266 | from cloudinit.util import b64e |
2267 | |
2268 | -from .. import helpers |
2269 | +from ..helpers import mock, FilesystemMockingTestCase, TestCase |
2270 | |
2271 | -try: |
2272 | - from unittest import mock |
2273 | -except ImportError: |
2274 | - import mock |
2275 | +SDC_NICS = json.loads(""" |
2276 | +[ |
2277 | + { |
2278 | + "nic_tag": "external", |
2279 | + "primary": true, |
2280 | + "mtu": 1500, |
2281 | + "model": "virtio", |
2282 | + "gateway": "8.12.42.1", |
2283 | + "netmask": "255.255.255.0", |
2284 | + "ip": "8.12.42.102", |
2285 | + "network_uuid": "992fc7ce-6aac-4b74-aed6-7b9d2c6c0bfe", |
2286 | + "gateways": [ |
2287 | + "8.12.42.1" |
2288 | + ], |
2289 | + "vlan_id": 324, |
2290 | + "mac": "90:b8:d0:f5:e4:f5", |
2291 | + "interface": "net0", |
2292 | + "ips": [ |
2293 | + "8.12.42.102/24" |
2294 | + ] |
2295 | + }, |
2296 | + { |
2297 | + "nic_tag": "sdc_overlay/16187209", |
2298 | + "gateway": "192.168.128.1", |
2299 | + "model": "virtio", |
2300 | + "mac": "90:b8:d0:a5:ff:cd", |
2301 | + "netmask": "255.255.252.0", |
2302 | + "ip": "192.168.128.93", |
2303 | + "network_uuid": "4cad71da-09bc-452b-986d-03562a03a0a9", |
2304 | + "gateways": [ |
2305 | + "192.168.128.1" |
2306 | + ], |
2307 | + "vlan_id": 2, |
2308 | + "mtu": 8500, |
2309 | + "interface": "net1", |
2310 | + "ips": [ |
2311 | + "192.168.128.93/22" |
2312 | + ] |
2313 | + } |
2314 | +] |
2315 | +""") |
2316 | |
2317 | MOCK_RETURNS = { |
2318 | 'hostname': 'test-host', |
2319 | @@ -60,79 +98,66 @@ |
2320 | 'sdc:vendor-data': '\n'.join(['VENDOR_DATA', '']), |
2321 | 'user-data': '\n'.join(['something', '']), |
2322 | 'user-script': '\n'.join(['/bin/true', '']), |
2323 | + 'sdc:nics': json.dumps(SDC_NICS), |
2324 | } |
2325 | |
2326 | DMI_DATA_RETURN = 'smartdc' |
2327 | |
2328 | |
2329 | -def get_mock_client(mockdata): |
2330 | - class MockMetadataClient(object): |
2331 | - |
2332 | - def __init__(self, serial): |
2333 | - pass |
2334 | - |
2335 | - def get_metadata(self, metadata_key): |
2336 | - return mockdata.get(metadata_key) |
2337 | - return MockMetadataClient |
2338 | - |
2339 | - |
2340 | -class TestSmartOSDataSource(helpers.FilesystemMockingTestCase): |
2341 | +class PsuedoJoyentClient(object): |
2342 | + def __init__(self, data=None): |
2343 | + if data is None: |
2344 | + data = MOCK_RETURNS.copy() |
2345 | + self.data = data |
2346 | + return |
2347 | + |
2348 | + def get(self, key, default=None, strip=False): |
2349 | + if key in self.data: |
2350 | + r = self.data[key] |
2351 | + if strip: |
2352 | + r = r.strip() |
2353 | + else: |
2354 | + r = default |
2355 | + return r |
2356 | + |
2357 | + def get_json(self, key, default=None): |
2358 | + result = self.get(key, default=default) |
2359 | + if result is None: |
2360 | + return default |
2361 | + return json.loads(result) |
2362 | + |
2363 | + def exists(self): |
2364 | + return True |
2365 | + |
2366 | + |
2367 | +class TestSmartOSDataSource(FilesystemMockingTestCase): |
2368 | def setUp(self): |
2369 | super(TestSmartOSDataSource, self).setUp() |
2370 | |
2371 | + dsmos = 'cloudinit.sources.DataSourceSmartOS' |
2372 | + patcher = mock.patch(dsmos + ".jmc_client_factory") |
2373 | + self.jmc_cfact = patcher.start() |
2374 | + self.addCleanup(patcher.stop) |
2375 | + patcher = mock.patch(dsmos + ".get_smartos_environ") |
2376 | + self.get_smartos_environ = patcher.start() |
2377 | + self.addCleanup(patcher.stop) |
2378 | + |
2379 | self.tmp = tempfile.mkdtemp() |
2380 | self.addCleanup(shutil.rmtree, self.tmp) |
2381 | + self.paths = c_helpers.Paths({'cloud_dir': self.tmp}) |
2382 | + |
2383 | self.legacy_user_d = tempfile.mkdtemp() |
2384 | - self.addCleanup(shutil.rmtree, self.legacy_user_d) |
2385 | - |
2386 | - # If you should want to watch the logs... |
2387 | - self._log = None |
2388 | - self._log_file = None |
2389 | - self._log_handler = None |
2390 | - |
2391 | - # patch cloud_dir, so our 'seed_dir' is guaranteed empty |
2392 | - self.paths = c_helpers.Paths({'cloud_dir': self.tmp}) |
2393 | - |
2394 | - self.unapply = [] |
2395 | - super(TestSmartOSDataSource, self).setUp() |
2396 | + self.orig_lud = DataSourceSmartOS.LEGACY_USER_D |
2397 | + DataSourceSmartOS.LEGACY_USER_D = self.legacy_user_d |
2398 | |
2399 | def tearDown(self): |
2400 | - helpers.FilesystemMockingTestCase.tearDown(self) |
2401 | - if self._log_handler and self._log: |
2402 | - self._log.removeHandler(self._log_handler) |
2403 | - apply_patches([i for i in reversed(self.unapply)]) |
2404 | + DataSourceSmartOS.LEGACY_USER_D = self.orig_lud |
2405 | super(TestSmartOSDataSource, self).tearDown() |
2406 | |
2407 | - def _patchIn(self, root): |
2408 | - self.restore() |
2409 | - self.patchOS(root) |
2410 | - self.patchUtils(root) |
2411 | - |
2412 | - def apply_patches(self, patches): |
2413 | - ret = apply_patches(patches) |
2414 | - self.unapply += ret |
2415 | - |
2416 | - def _get_ds(self, sys_cfg=None, ds_cfg=None, mockdata=None, dmi_data=None, |
2417 | - is_lxbrand=False): |
2418 | - mod = DataSourceSmartOS |
2419 | - |
2420 | - if mockdata is None: |
2421 | - mockdata = MOCK_RETURNS |
2422 | - |
2423 | - if dmi_data is None: |
2424 | - dmi_data = DMI_DATA_RETURN |
2425 | - |
2426 | - def _dmi_data(): |
2427 | - return dmi_data |
2428 | - |
2429 | - def _os_uname(): |
2430 | - if not is_lxbrand: |
2431 | - # LP: #1243287. tests assume this runs, but running test on |
2432 | - # arm would cause them all to fail. |
2433 | - return ('LINUX', 'NODENAME', 'RELEASE', 'VERSION', 'x86_64') |
2434 | - else: |
2435 | - return ('LINUX', 'NODENAME', 'RELEASE', 'BRANDZ VIRTUAL LINUX', |
2436 | - 'X86_64') |
2437 | + def _get_ds(self, mockdata=None, mode=DataSourceSmartOS.SMARTOS_ENV_KVM, |
2438 | + sys_cfg=None, ds_cfg=None): |
2439 | + self.jmc_cfact.return_value = PsuedoJoyentClient(mockdata) |
2440 | + self.get_smartos_environ.return_value = mode |
2441 | |
2442 | if sys_cfg is None: |
2443 | sys_cfg = {} |
2444 | @@ -141,44 +166,8 @@ |
2445 | sys_cfg['datasource'] = sys_cfg.get('datasource', {}) |
2446 | sys_cfg['datasource']['SmartOS'] = ds_cfg |
2447 | |
2448 | - self.apply_patches([(mod, 'LEGACY_USER_D', self.legacy_user_d)]) |
2449 | - self.apply_patches([ |
2450 | - (mod, 'JoyentMetadataClient', get_mock_client(mockdata))]) |
2451 | - self.apply_patches([(mod, 'dmi_data', _dmi_data)]) |
2452 | - self.apply_patches([(os, 'uname', _os_uname)]) |
2453 | - self.apply_patches([(mod, 'device_exists', lambda d: True)]) |
2454 | - dsrc = mod.DataSourceSmartOS(sys_cfg, distro=None, |
2455 | - paths=self.paths) |
2456 | - self.apply_patches([(dsrc, '_get_seed_file_object', mock.MagicMock())]) |
2457 | - return dsrc |
2458 | - |
2459 | - def test_seed(self): |
2460 | - # default seed should be /dev/ttyS1 |
2461 | - dsrc = self._get_ds() |
2462 | - ret = dsrc.get_data() |
2463 | - self.assertTrue(ret) |
2464 | - self.assertEqual('kvm', dsrc.smartos_type) |
2465 | - self.assertEqual('/dev/ttyS1', dsrc.seed) |
2466 | - |
2467 | - def test_seed_lxbrand(self): |
2468 | - # default seed should be /dev/ttyS1 |
2469 | - dsrc = self._get_ds(is_lxbrand=True) |
2470 | - ret = dsrc.get_data() |
2471 | - self.assertTrue(ret) |
2472 | - self.assertEqual('lx-brand', dsrc.smartos_type) |
2473 | - self.assertEqual('/native/.zonecontrol/metadata.sock', dsrc.seed) |
2474 | - |
2475 | - def test_issmartdc(self): |
2476 | - dsrc = self._get_ds() |
2477 | - ret = dsrc.get_data() |
2478 | - self.assertTrue(ret) |
2479 | - self.assertTrue(dsrc.is_smartdc) |
2480 | - |
2481 | - def test_issmartdc_lxbrand(self): |
2482 | - dsrc = self._get_ds(is_lxbrand=True) |
2483 | - ret = dsrc.get_data() |
2484 | - self.assertTrue(ret) |
2485 | - self.assertTrue(dsrc.is_smartdc) |
2486 | + return DataSourceSmartOS.DataSourceSmartOS( |
2487 | + sys_cfg, distro=None, paths=self.paths) |
2488 | |
2489 | def test_no_base64(self): |
2490 | ds_cfg = {'no_base64_decode': ['test_var1'], 'all_base': True} |
2491 | @@ -214,58 +203,6 @@ |
2492 | self.assertEqual(MOCK_RETURNS['hostname'], |
2493 | dsrc.metadata['local-hostname']) |
2494 | |
2495 | - def test_base64_all(self): |
2496 | - # metadata provided base64_all of true |
2497 | - my_returns = MOCK_RETURNS.copy() |
2498 | - my_returns['base64_all'] = "true" |
2499 | - for k in ('hostname', 'cloud-init:user-data'): |
2500 | - my_returns[k] = b64e(my_returns[k]) |
2501 | - |
2502 | - dsrc = self._get_ds(mockdata=my_returns) |
2503 | - ret = dsrc.get_data() |
2504 | - self.assertTrue(ret) |
2505 | - self.assertEqual(MOCK_RETURNS['hostname'], |
2506 | - dsrc.metadata['local-hostname']) |
2507 | - self.assertEqual(MOCK_RETURNS['cloud-init:user-data'], |
2508 | - dsrc.userdata_raw) |
2509 | - self.assertEqual(MOCK_RETURNS['root_authorized_keys'], |
2510 | - dsrc.metadata['public-keys']) |
2511 | - self.assertEqual(MOCK_RETURNS['disable_iptables_flag'], |
2512 | - dsrc.metadata['iptables_disable']) |
2513 | - self.assertEqual(MOCK_RETURNS['enable_motd_sys_info'], |
2514 | - dsrc.metadata['motd_sys_info']) |
2515 | - |
2516 | - def test_b64_userdata(self): |
2517 | - my_returns = MOCK_RETURNS.copy() |
2518 | - my_returns['b64-cloud-init:user-data'] = "true" |
2519 | - my_returns['b64-hostname'] = "true" |
2520 | - for k in ('hostname', 'cloud-init:user-data'): |
2521 | - my_returns[k] = b64e(my_returns[k]) |
2522 | - |
2523 | - dsrc = self._get_ds(mockdata=my_returns) |
2524 | - ret = dsrc.get_data() |
2525 | - self.assertTrue(ret) |
2526 | - self.assertEqual(MOCK_RETURNS['hostname'], |
2527 | - dsrc.metadata['local-hostname']) |
2528 | - self.assertEqual(MOCK_RETURNS['cloud-init:user-data'], |
2529 | - dsrc.userdata_raw) |
2530 | - self.assertEqual(MOCK_RETURNS['root_authorized_keys'], |
2531 | - dsrc.metadata['public-keys']) |
2532 | - |
2533 | - def test_b64_keys(self): |
2534 | - my_returns = MOCK_RETURNS.copy() |
2535 | - my_returns['base64_keys'] = 'hostname,ignored' |
2536 | - for k in ('hostname',): |
2537 | - my_returns[k] = b64e(my_returns[k]) |
2538 | - |
2539 | - dsrc = self._get_ds(mockdata=my_returns) |
2540 | - ret = dsrc.get_data() |
2541 | - self.assertTrue(ret) |
2542 | - self.assertEqual(MOCK_RETURNS['hostname'], |
2543 | - dsrc.metadata['local-hostname']) |
2544 | - self.assertEqual(MOCK_RETURNS['cloud-init:user-data'], |
2545 | - dsrc.userdata_raw) |
2546 | - |
2547 | def test_userdata(self): |
2548 | dsrc = self._get_ds(mockdata=MOCK_RETURNS) |
2549 | ret = dsrc.get_data() |
2550 | @@ -275,6 +212,13 @@ |
2551 | self.assertEqual(MOCK_RETURNS['cloud-init:user-data'], |
2552 | dsrc.userdata_raw) |
2553 | |
2554 | + def test_sdc_nics(self): |
2555 | + dsrc = self._get_ds(mockdata=MOCK_RETURNS) |
2556 | + ret = dsrc.get_data() |
2557 | + self.assertTrue(ret) |
2558 | + self.assertEqual(json.loads(MOCK_RETURNS['sdc:nics']), |
2559 | + dsrc.metadata['network-data']) |
2560 | + |
2561 | def test_sdc_scripts(self): |
2562 | dsrc = self._get_ds(mockdata=MOCK_RETURNS) |
2563 | ret = dsrc.get_data() |
2564 | @@ -430,18 +374,7 @@ |
2565 | mydscfg['disk_aliases']['FOO']) |
2566 | |
2567 | |
2568 | -def apply_patches(patches): |
2569 | - ret = [] |
2570 | - for (ref, name, replace) in patches: |
2571 | - if replace is None: |
2572 | - continue |
2573 | - orig = getattr(ref, name) |
2574 | - setattr(ref, name, replace) |
2575 | - ret.append((ref, name, orig)) |
2576 | - return ret |
2577 | - |
2578 | - |
2579 | -class TestJoyentMetadataClient(helpers.FilesystemMockingTestCase): |
2580 | +class TestJoyentMetadataClient(FilesystemMockingTestCase): |
2581 | |
2582 | def setUp(self): |
2583 | super(TestJoyentMetadataClient, self).setUp() |
2584 | @@ -481,7 +414,8 @@ |
2585 | mock.Mock(return_value=self.request_id))) |
2586 | |
2587 | def _get_client(self): |
2588 | - return DataSourceSmartOS.JoyentMetadataClient(self.serial) |
2589 | + return DataSourceSmartOS.JoyentMetadataClient( |
2590 | + fp=self.serial, smartos_type=DataSourceSmartOS.SMARTOS_ENV_KVM) |
2591 | |
2592 | def assertEndsWith(self, haystack, prefix): |
2593 | self.assertTrue(haystack.endswith(prefix), |
2594 | @@ -495,7 +429,7 @@ |
2595 | |
2596 | def test_get_metadata_writes_a_single_line(self): |
2597 | client = self._get_client() |
2598 | - client.get_metadata('some_key') |
2599 | + client.get('some_key') |
2600 | self.assertEqual(1, self.serial.write.call_count) |
2601 | written_line = self.serial.write.call_args[0][0] |
2602 | print(type(written_line)) |
2603 | @@ -505,7 +439,7 @@ |
2604 | |
2605 | def _get_written_line(self, key='some_key'): |
2606 | client = self._get_client() |
2607 | - client.get_metadata(key) |
2608 | + client.get(key) |
2609 | return self.serial.write.call_args[0][0] |
2610 | |
2611 | def test_get_metadata_writes_bytes(self): |
2612 | @@ -549,32 +483,32 @@ |
2613 | |
2614 | def test_get_metadata_reads_a_line(self): |
2615 | client = self._get_client() |
2616 | - client.get_metadata('some_key') |
2617 | + client.get('some_key') |
2618 | self.assertEqual(self.metasource_data_len, self.serial.read.call_count) |
2619 | |
2620 | def test_get_metadata_returns_valid_value(self): |
2621 | client = self._get_client() |
2622 | - value = client.get_metadata('some_key') |
2623 | + value = client.get('some_key') |
2624 | self.assertEqual(self.metadata_value, value) |
2625 | |
2626 | def test_get_metadata_throws_exception_for_incorrect_length(self): |
2627 | self.response_parts['length'] = 0 |
2628 | client = self._get_client() |
2629 | self.assertRaises(DataSourceSmartOS.JoyentMetadataFetchException, |
2630 | - client.get_metadata, 'some_key') |
2631 | + client.get, 'some_key') |
2632 | |
2633 | def test_get_metadata_throws_exception_for_incorrect_crc(self): |
2634 | self.response_parts['crc'] = 'deadbeef' |
2635 | client = self._get_client() |
2636 | self.assertRaises(DataSourceSmartOS.JoyentMetadataFetchException, |
2637 | - client.get_metadata, 'some_key') |
2638 | + client.get, 'some_key') |
2639 | |
2640 | def test_get_metadata_throws_exception_for_request_id_mismatch(self): |
2641 | self.response_parts['request_id'] = 'deadbeef' |
2642 | client = self._get_client() |
2643 | client._checksum = lambda _: self.response_parts['crc'] |
2644 | self.assertRaises(DataSourceSmartOS.JoyentMetadataFetchException, |
2645 | - client.get_metadata, 'some_key') |
2646 | + client.get, 'some_key') |
2647 | |
2648 | def test_get_metadata_returns_None_if_value_not_found(self): |
2649 | self.response_parts['payload'] = '' |
2650 | @@ -582,4 +516,24 @@ |
2651 | self.response_parts['length'] = 17 |
2652 | client = self._get_client() |
2653 | client._checksum = lambda _: self.response_parts['crc'] |
2654 | - self.assertIsNone(client.get_metadata('some_key')) |
2655 | + self.assertIsNone(client.get('some_key')) |
2656 | + |
2657 | + |
2658 | +class TestNetworkConversion(TestCase): |
2659 | + |
2660 | + def test_convert_simple(self): |
2661 | + expected = { |
2662 | + 'version': 1, |
2663 | + 'config': [ |
2664 | + {'name': 'net0', 'type': 'physical', |
2665 | + 'subnets': [{'type': 'static', 'gateway': '8.12.42.1', |
2666 | + 'netmask': '255.255.255.0', |
2667 | + 'address': '8.12.42.102/24'}], |
2668 | + 'mtu': 1500, 'mac_address': '90:b8:d0:f5:e4:f5'}, |
2669 | + {'name': 'net1', 'type': 'physical', |
2670 | + 'subnets': [{'type': 'static', 'gateway': '192.168.128.1', |
2671 | + 'netmask': '255.255.252.0', |
2672 | + 'address': '192.168.128.93/22'}], |
2673 | + 'mtu': 8500, 'mac_address': '90:b8:d0:a5:ff:cd'}]} |
2674 | + found = DataSourceSmartOS.convert_smartos_network_data(SDC_NICS) |
2675 | + self.assertEqual(expected, found) |
2676 | |
2677 | === removed file 'udev/79-cloud-init-net-wait.rules' |
2678 | --- udev/79-cloud-init-net-wait.rules 2016-03-19 00:40:54 +0000 |
2679 | +++ udev/79-cloud-init-net-wait.rules 1970-01-01 00:00:00 +0000 |
2680 | @@ -1,10 +0,0 @@ |
2681 | -# cloud-init cold/hot-plug blocking mechanism |
2682 | -# this file blocks further processing of network events |
2683 | -# until cloud-init local has had a chance to read and apply network |
2684 | -SUBSYSTEM!="net", GOTO="cloudinit_naming_end" |
2685 | -ACTION!="add", GOTO="cloudinit_naming_end" |
2686 | - |
2687 | -IMPORT{program}="/lib/udev/cloud-init-wait" |
2688 | - |
2689 | -LABEL="cloudinit_naming_end" |
2690 | -# vi: ts=4 expandtab syntax=udevrules |
2691 | |
2692 | === removed file 'udev/cloud-init-wait' |
2693 | --- udev/cloud-init-wait 2016-03-29 13:11:25 +0000 |
2694 | +++ udev/cloud-init-wait 1970-01-01 00:00:00 +0000 |
2695 | @@ -1,70 +0,0 @@ |
2696 | -#!/bin/sh |
2697 | - |
2698 | -CI_NET_READY="/run/cloud-init/network-config-ready" |
2699 | -LOG="/run/cloud-init/${0##*/}.log" |
2700 | -LOG_INIT=0 |
2701 | -MAX_WAIT=60 |
2702 | -DEBUG=0 |
2703 | - |
2704 | -block_until_ready() { |
2705 | - local fname="$1" max="$2" |
2706 | - [ -f "$fname" ] && return 0 |
2707 | - # udevadm settle below will exit at the first of 3 conditions |
2708 | - # 1.) timeout 2.) file exists 3.) all in-flight udev events are processed |
2709 | - # since this is being run from a udev event, the 3 wont happen. |
2710 | - # thus, this is essentially a inotify wait or timeout on a file in /run |
2711 | - # that is created by cloud-init-local. |
2712 | - udevadm settle "--timeout=$max" "--exit-if-exists=$fname" |
2713 | -} |
2714 | - |
2715 | -log() { |
2716 | - [ -n "${LOG}" ] || return |
2717 | - [ "${DEBUG:-0}" = "0" ] && return |
2718 | - |
2719 | - if [ $LOG_INIT = 0 ]; then |
2720 | - if [ -d "${LOG%/*}" ] || mkdir -p "${LOG%/*}"; then |
2721 | - LOG_INIT=1 |
2722 | - else |
2723 | - echo "${0##*/}: WARN: log init to ${LOG%/*}" 1>&2 |
2724 | - return |
2725 | - fi |
2726 | - elif [ "$LOG_INIT" = "-1" ]; then |
2727 | - return |
2728 | - fi |
2729 | - local info="$$ $INTERFACE" |
2730 | - if [ "$DEBUG" -gt 1 ]; then |
2731 | - local up idle |
2732 | - read up idle < /proc/uptime |
2733 | - info="$$ $INTERFACE $up" |
2734 | - fi |
2735 | - echo "[$info]" "$@" >> "$LOG" |
2736 | -} |
2737 | - |
2738 | -main() { |
2739 | - local name="" readyfile="$CI_NET_READY" |
2740 | - local info="INTERFACE=${INTERFACE} ID_NET_NAME=${ID_NET_NAME}" |
2741 | - info="$info ID_NET_NAME_PATH=${ID_NET_NAME_PATH}" |
2742 | - info="$info MAC_ADDRESS=${MAC_ADDRESS}" |
2743 | - log "$info" |
2744 | - |
2745 | - ## Check to see if cloud-init.target is set. If cloud-init is |
2746 | - ## disabled we do not want to do anything. |
2747 | - if [ ! -f "/run/cloud-init/enabled" ]; then |
2748 | - log "cloud-init disabled" |
2749 | - return 0 |
2750 | - fi |
2751 | - |
2752 | - if [ "${INTERFACE#lo}" != "$INTERFACE" ]; then |
2753 | - return 0 |
2754 | - fi |
2755 | - |
2756 | - block_until_ready "$readyfile" "$MAX_WAIT" || |
2757 | - { log "failed waiting for ready on $INTERFACE"; return 1; } |
2758 | - |
2759 | - log "net config ready" |
2760 | -} |
2761 | - |
2762 | -main "$@" |
2763 | -exit |
2764 | - |
2765 | -# vi: ts=4 expandtab |
Testing I've done.
### Joyent ### .*pam_motd/ s/^/#/' /etc/pam.d/sshd
## launched a wily instance on joyent
# make login not run apt-update as painful slow
sudo sed -i '/^[^#]
# add new cloud-init to apt cloud-init- dev && sudo apt-get update && sudo apt-get dist-upgrade -qy'
sudo sh -c 'apt-add-repository -y ppa:smoser/
# un-do cloud-image local hacks for joyent on wily local/sbin/ ephemeral_ eth.sh && mv $f $f.dist && ln -sf /bin/true $f'
sudo sh -c 'f=/usr/
sudo rm -Rf /var/lib/cloud /var/log/ cloud-init*
sudo reboot
### dreamhost ### cloud-init- dev && sudo apt-get update && sudo apt-get install cloud-init' interfaces. dist ] || sudo mv /etc/network/ interfaces /etc/network/ interfaces. dist interfaces. d/*.cfg" | sudo tee /etc/network/ interfaces cloud/seed/ config_ drive *sr0.*config_ drive/s, ^,#,' /etc/fstab cloud-init* grub.d/ 50-cloudimg- settings. cfg
* started 14.04 instance and do-release-upgrade -d and also fresh 16.04
* sudo sh -c 'apt-add-repository -y ppa:smoser/
* [ -e /etc/network/
* printf "%s\n%s\n%s\n" "auto lo" "iface lo inet loopback" "source /etc/network/
* sudo umount /var/lib/
* sudo sed -i '/[^#].
* sudo rm -Rf /var/lib/cloud /var/log/
* sudo reboot
* # now verify without persistent ifnames
sudo sed -s 's,net.ifnames=0,,' /etc/default/
sudo update-grub
sudo reboot
# now should have an 'eth0' name still
### serverstack, EC2 ### cloud-init- dev && sudo apt-get update && sudo apt-get dist-upgrade -qy' cloud-init*
* clean xenial instance
* sudo sh -c 'apt-add-repository -y ppa:smoser/
* sudo reboot # [check that it works fine]
* sudo rm -Rf /var/lib/cloud /var/log/
* sudo reboot