Merge lp:~smoser/cloud-init/trunk.fix-networking into lp:~cloud-init-dev/cloud-init/trunk

Proposed by Scott Moser on 2016-06-02
Status: Merged
Merged at revision: 1225
Proposed branch: lp:~smoser/cloud-init/trunk.fix-networking
Merge into: lp:~cloud-init-dev/cloud-init/trunk
Diff against target: 2765 lines (+1142/-688)
22 files modified
ChangeLog (+2/-0)
bin/cloud-init (+38/-22)
cloudinit/config/cc_emit_upstart.py (+1/-1)
cloudinit/config/cc_lxd.py (+2/-1)
cloudinit/distros/__init__.py (+6/-0)
cloudinit/helpers.py (+12/-10)
cloudinit/net/__init__.py (+221/-1)
cloudinit/sources/DataSourceCloudSigma.py (+5/-14)
cloudinit/sources/DataSourceConfigDrive.py (+70/-86)
cloudinit/sources/DataSourceNoCloud.py (+35/-45)
cloudinit/sources/DataSourceOpenNebula.py (+9/-35)
cloudinit/sources/DataSourceOpenStack.py (+2/-7)
cloudinit/sources/DataSourceSmartOS.py (+420/-173)
cloudinit/sources/__init__.py (+33/-0)
cloudinit/sources/helpers/openstack.py (+11/-7)
cloudinit/stages.py (+70/-25)
setup.py (+0/-1)
systemd/cloud-init-generator (+0/-3)
tests/unittests/test_datasource/test_configdrive.py (+77/-3)
tests/unittests/test_datasource/test_smartos.py (+128/-174)
udev/79-cloud-init-net-wait.rules (+0/-10)
udev/cloud-init-wait (+0/-70)
To merge this branch: bzr merge lp:~smoser/cloud-init/trunk.fix-networking
Reviewer Review Type Date Requested Status
cloud-init Commiters 2016-06-02 Pending
Review via email: mp+296272@code.launchpad.net

Description of the change

I'll improve this description later.

  DataSource Mode (dsmode) is present in many datasources in cloud-init.
  dsmode was originally added to cloud-init to specify when this datasource
  should be 'realized'.

  cloud-init has 4 stages of boot.
   a.) cloud-init --local . network is guaranteed not present.
   b.) cloud-init (--network). network is guaranteed present.
   c.) cloud-config
   d.) cloud-init final

  'init_modules' [1] are run "as early as possible". And as such, are executed
  in either 'a' or 'b' based on the datasource. However, executing them means
  that user-data has been fully consumed. User-data and vendor-data may have
  '#include http://...' which then rely on the network being present. boothooks
  are an example of the things run in init_modules.

  The 'dsmode' was a way for a user to indicate that init_modules
  should run at 'a' (dsmode=local) or 'b' (dsmode=net) directly.

  Things were further confused when a datasource could provide networking
  configuration. Then, we needed to apply the networking config at 'a'
  but if the user had provided boothooks that expected networking, then the
  init_modules would need to be executed at 'b'. The config drive datasource
  hacked its way through this and applies networking if *it* detects it is
  a new instance.

  == Suggested Change ==
  The plan is to
   1. incorporate 'dsmode' into DataSource superclass
   2. make all existing datasources default to network
   3. apply any networking configuration from a datasource on first boot only
      apply_networking will always rename network devices when it runs.
      for bug 1579130.
   4. run init_modules at cloud-init (network) time frame unless datasource
      is 'local'.
   5. Datasources can provide a 'first_boot' method that will be called when
      a new instance_id is found. This will allow the config drive's write_files
      to be applied once.

  Over all, this will very much simplify things. We'll no longer have
  2 sources like DataSourceNoCloud and DataSourceNoCloudNet, but would just
  have one source with a dsmode.

  == Concerns ==
  Some things have odd reliance on dsmode. For example, OpenNebula's get_hostname
  uses it to determine if it should do a lookup of an ip address.

  == Bugs to fix here ==
  http://pad.lv/1577982 ConfigDrive: cloud-init fails to configure network from network_data.json
  http://pad.lv/1579130 need to support systemd.link renaming of devices in container
  http://pad.lv/1577844 Drop unnecessary blocking of all net udev rules
  httpP//pad.lv/1571761 zfs-import-cache.service slows boot by 60 seconds

To post a comment you must log in.
1241. By Scott Moser on 2016-06-02

revert unintended change to tox.ini

1242. By Scott Moser on 2016-06-02

SmartOS: datasource improvements, support for networking information.

This adds support for reading networking information from the
SmartOS metadata service and applying.

1243. By Scott Moser on 2016-06-02

fix log message in emit_upstart

1244. By Scott Moser on 2016-06-02

fix tox

1245. By Scott Moser on 2016-06-02

openstack: support decoding when reading files, use that for network_config

The network config file is /etc/network/interfaces formated.
We will decode that here so that the user can expect that it is
a string. The issue was that it was bytes but convert_eni_data
was expecting a string.

1246. By Scott Moser on 2016-06-02

eni parsing: support 'ether' in hwaddress, netmask and broadcast

this adds ability to support ENI that has:
 hwadress ether 36:4c:e1:3b:14:31
or
 hwaddress 36:4c:e1:3b:14:31

the former is written by openstack (at least on dreamhost).

Also, in the conversion of eni to network config support broadcast
and netmask.

1247. By Scott Moser on 2016-06-02

smartos: do not raise error when not on smartos

if get_smartos_environ() returned a None, then
the datasoure would raise a ValueError when get_data was called.
Fix that.

1248. By Scott Moser on 2016-06-02

fix untested previous change to smartos

1249. By Scott Moser on 2016-06-02

re-add the 'Net' classes for datasources

When the .pkl file is loaded, the module that it is loaded
from must have the same symbol. Ie, if booted once and got
  DataSourceConfigDriveNet
then upgraded and rebooted, then next boot would show
  Can't get attribute 'DataSourceConfigDriveNet'

1250. By Scott Moser on 2016-06-02

merge with trunk

Scott Moser (smoser) wrote :

Testing I've done.

### Joyent ###
## launched a wily instance on joyent
# make login not run apt-update as painful slow
sudo sed -i '/^[^#].*pam_motd/s/^/#/' /etc/pam.d/sshd

# add new cloud-init to apt
sudo sh -c 'apt-add-repository -y ppa:smoser/cloud-init-dev && sudo apt-get update && sudo apt-get dist-upgrade -qy'

# un-do cloud-image local hacks for joyent on wily
sudo sh -c 'f=/usr/local/sbin/ephemeral_eth.sh && mv $f $f.dist && ln -sf /bin/true $f'

sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
sudo reboot

### dreamhost ###
* started 14.04 instance and do-release-upgrade -d and also fresh 16.04
* sudo sh -c 'apt-add-repository -y ppa:smoser/cloud-init-dev && sudo apt-get update && sudo apt-get install cloud-init'
* [ -e /etc/network/interfaces.dist ] || sudo mv /etc/network/interfaces /etc/network/interfaces.dist
* printf "%s\n%s\n%s\n" "auto lo" "iface lo inet loopback" "source /etc/network/interfaces.d/*.cfg" | sudo tee /etc/network/interfaces
* sudo umount /var/lib/cloud/seed/config_drive
* sudo sed -i '/[^#].*sr0.*config_drive/s,^,#,' /etc/fstab
* sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
* sudo reboot
* # now verify without persistent ifnames
  sudo sed -s 's,net.ifnames=0,,' /etc/default/grub.d/50-cloudimg-settings.cfg
  sudo update-grub
  sudo reboot
  # now should have an 'eth0' name still

### serverstack, EC2 ###
* clean xenial instance
* sudo sh -c 'apt-add-repository -y ppa:smoser/cloud-init-dev && sudo apt-get update && sudo apt-get dist-upgrade -qy'
* sudo reboot # [check that it works fine]
* sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
* sudo reboot

Scott Moser (smoser) wrote :

the one change left here is i think to make openstack config drive datasource not use 'id' as the nic namne as those are arbitrary.

1251. By Scott Moser on 2016-06-03

ConfigDrive: do not use 'id' on a link for the device name

'id' on a link in the openstack spec should be "Generic, generated ID".
current implementation was to use the host's name for the host
side nic. Which provided names like 'tap-adfasdffd'.

We do not want to name devices like that as its quite unexpected
and non user friendly. So here we use the system name for any
nic that is present, but then require that the nics found also
be present at the time of rendering.

The end result is that if the system boots with net.ifnames=0
then it will get 'eth0' like names. and if it boots without net.ifnames
then it will get enp0s1 like names.

1252. By Scott Moser on 2016-06-03

lxd: fix log messsage

Joshua Harlow (harlowja) wrote :

Can u add a part of the description around 'def rename_interfaces' so that people know why this is needed, be much appreciated :)

1253. By Scott Moser on 2016-06-03

avoid rendering 'lo' twice by not writing it in network config.

1254. By Scott Moser on 2016-06-03

fix issue with routes on subnets not getting rendered

1255. By Scott Moser on 2016-06-03

config drive conversion: recognize 'bridge' as a physical type, fix mtu

the network json in openstack provides a type of 'bridge' when
the underlying (host) type is a bridge. Silly, but we need to
consider that a physical device as it will be for us.

also, the 'mtu' will appear on the link, not on the route

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'ChangeLog'
2--- ChangeLog 2016-06-02 16:32:14 +0000
3+++ ChangeLog 2016-06-03 19:07:08 +0000
4@@ -113,6 +113,8 @@
5 - settings on the kernel command line (cc:) override all local settings
6 rather than only those in /etc/cloud/cloud.cfg (LP: #1582323)
7 - Improve merging documentation [Daniel Watkins]
8+ - SmartOS: datasource improvements and support for metadata service
9+ providing networking information.
10
11 0.7.6:
12 - open 0.7.6
13
14=== modified file 'bin/cloud-init'
15--- bin/cloud-init 2016-04-15 17:54:05 +0000
16+++ bin/cloud-init 2016-06-03 19:07:08 +0000
17@@ -211,27 +211,27 @@
18 util.logexc(LOG, "Failed to initialize, likely bad things to come!")
19 # Stage 4
20 path_helper = init.paths
21- if not args.local:
22+ mode = sources.DSMODE_LOCAL if args.local else sources.DSMODE_NETWORK
23+
24+ if mode == sources.DSMODE_NETWORK:
25 existing = "trust"
26 sys.stderr.write("%s\n" % (netinfo.debug_info()))
27 LOG.debug(("Checking to see if files that we need already"
28 " exist from a previous run that would allow us"
29 " to stop early."))
30+ # no-net is written by upstart cloud-init-nonet when network failed
31+ # to come up
32 stop_files = [
33 os.path.join(path_helper.get_cpath("data"), "no-net"),
34- path_helper.get_ipath_cur("obj_pkl"),
35 ]
36 existing_files = []
37 for fn in stop_files:
38- try:
39- c = util.load_file(fn)
40- if len(c):
41- existing_files.append((fn, len(c)))
42- except Exception:
43- pass
44+ if os.path.isfile(fn):
45+ existing_files.append(fn)
46+
47 if existing_files:
48- LOG.debug("Exiting early due to the existence of %s files",
49- existing_files)
50+ LOG.debug("[%s] Exiting. stop file %s existed",
51+ mode, existing_files)
52 return (None, [])
53 else:
54 LOG.debug("Execution continuing, no previous run detected that"
55@@ -248,34 +248,50 @@
56 # Stage 5
57 try:
58 init.fetch(existing=existing)
59+ # if in network mode, and the datasource is local
60+ # then work was done at that stage.
61+ if mode == sources.DSMODE_NETWORK and init.datasource.dsmode != mode:
62+ LOG.debug("[%s] Exiting. datasource %s in local mode",
63+ mode, init.datasource)
64+ return (None, [])
65 except sources.DataSourceNotFoundException:
66 # In the case of 'cloud-init init' without '--local' it is a bit
67 # more likely that the user would consider it failure if nothing was
68 # found. When using upstart it will also mentions job failure
69 # in console log if exit code is != 0.
70- if args.local:
71+ if mode == sources.DSMODE_LOCAL:
72 LOG.debug("No local datasource found")
73 else:
74 util.logexc(LOG, ("No instance datasource found!"
75 " Likely bad things to come!"))
76 if not args.force:
77- init.apply_network_config()
78- if args.local:
79+ init.apply_network_config(bring_up=not args.local)
80+ LOG.debug("[%s] Exiting without datasource in local mode", mode)
81+ if mode == sources.DSMODE_LOCAL:
82 return (None, [])
83 else:
84 return (None, ["No instance datasource found."])
85-
86- if args.local:
87- if not init.ds_restored:
88- # if local mode and the datasource was not restored from cache
89- # (this is not first boot) then apply networking.
90- init.apply_network_config()
91 else:
92- LOG.debug("skipping networking config from restored datasource.")
93+ LOG.debug("[%s] barreling on in force mode without datasource",
94+ mode)
95
96 # Stage 6
97 iid = init.instancify()
98- LOG.debug("%s will now be targeting instance id: %s", name, iid)
99+ LOG.debug("[%s] %s will now be targeting instance id: %s. new=%s",
100+ mode, name, iid, init.is_new_instance())
101+
102+ init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))
103+
104+ if mode == sources.DSMODE_LOCAL:
105+ if init.datasource.dsmode != mode:
106+ LOG.debug("[%s] Exiting. datasource %s not in local mode.",
107+ mode, init.datasource)
108+ return (init.datasource, [])
109+ else:
110+ LOG.debug("[%s] %s is in local mode, will apply init modules now.",
111+ mode, init.datasource)
112+
113+ # update fully realizes user-data (pulling in #include if necessary)
114 init.update()
115 # Stage 7
116 try:
117@@ -528,7 +544,7 @@
118 v1[mode]['errors'] = [str(e) for e in errors]
119
120 except Exception as e:
121- util.logexc(LOG, "failed of stage %s", mode)
122+ util.logexc(LOG, "failed stage %s", mode)
123 print_exc("failed run of stage %s" % mode)
124 v1[mode]['errors'] = [str(e)]
125
126
127=== modified file 'cloudinit/config/cc_emit_upstart.py'
128--- cloudinit/config/cc_emit_upstart.py 2016-05-12 17:56:26 +0000
129+++ cloudinit/config/cc_emit_upstart.py 2016-06-03 19:07:08 +0000
130@@ -56,7 +56,7 @@
131 event_names = ['cloud-config']
132
133 if not is_upstart_system():
134- log.debug("not upstart system, '%s' disabled")
135+ log.debug("not upstart system, '%s' disabled", name)
136 return
137
138 cfgpath = cloud.paths.get_ipath_cur("cloud_config")
139
140=== modified file 'cloudinit/config/cc_lxd.py'
141--- cloudinit/config/cc_lxd.py 2016-05-12 17:56:26 +0000
142+++ cloudinit/config/cc_lxd.py 2016-06-03 19:07:08 +0000
143@@ -52,7 +52,8 @@
144 # Get config
145 lxd_cfg = cfg.get('lxd')
146 if not lxd_cfg:
147- log.debug("Skipping module named %s, not present or disabled by cfg")
148+ log.debug("Skipping module named %s, not present or disabled by cfg",
149+ name)
150 return
151 if not isinstance(lxd_cfg, dict):
152 log.warn("lxd config must be a dictionary. found a '%s'",
153
154=== modified file 'cloudinit/distros/__init__.py'
155--- cloudinit/distros/__init__.py 2016-05-12 17:56:26 +0000
156+++ cloudinit/distros/__init__.py 2016-06-03 19:07:08 +0000
157@@ -31,6 +31,7 @@
158
159 from cloudinit import importer
160 from cloudinit import log as logging
161+from cloudinit import net
162 from cloudinit import ssh_util
163 from cloudinit import type_utils
164 from cloudinit import util
165@@ -128,6 +129,8 @@
166 mirror_info=arch_info)
167
168 def apply_network(self, settings, bring_up=True):
169+ # this applies network where 'settings' is interfaces(5) style
170+ # it is obsolete compared to apply_network_config
171 # Write it out
172 dev_names = self._write_network(settings)
173 # Now try to bring them up
174@@ -143,6 +146,9 @@
175 return self._bring_up_interfaces(dev_names)
176 return False
177
178+ def apply_network_config_names(self, netconfig):
179+ net.apply_network_config_names(netconfig)
180+
181 @abc.abstractmethod
182 def apply_locale(self, locale, out_fn=None):
183 raise NotImplementedError()
184
185=== modified file 'cloudinit/helpers.py'
186--- cloudinit/helpers.py 2016-05-12 17:56:26 +0000
187+++ cloudinit/helpers.py 2016-06-03 19:07:08 +0000
188@@ -328,6 +328,7 @@
189 self.cfgs = path_cfgs
190 # Populate all the initial paths
191 self.cloud_dir = path_cfgs.get('cloud_dir', '/var/lib/cloud')
192+ self.run_dir = path_cfgs.get('run_dir', '/run/cloud-init')
193 self.instance_link = os.path.join(self.cloud_dir, 'instance')
194 self.boot_finished = os.path.join(self.instance_link, "boot-finished")
195 self.upstart_conf_d = path_cfgs.get('upstart_dir')
196@@ -349,26 +350,19 @@
197 "data": "data",
198 "vendordata_raw": "vendor-data.txt",
199 "vendordata": "vendor-data.txt.i",
200+ "instance_id": ".instance-id",
201 }
202 # Set when a datasource becomes active
203 self.datasource = ds
204
205 # get_ipath_cur: get the current instance path for an item
206 def get_ipath_cur(self, name=None):
207- ipath = self.instance_link
208- add_on = self.lookups.get(name)
209- if add_on:
210- ipath = os.path.join(ipath, add_on)
211- return ipath
212+ return self._get_path(self.instance_link, name)
213
214 # get_cpath : get the "clouddir" (/var/lib/cloud/<name>)
215 # for a name in dirmap
216 def get_cpath(self, name=None):
217- cpath = self.cloud_dir
218- add_on = self.lookups.get(name)
219- if add_on:
220- cpath = os.path.join(cpath, add_on)
221- return cpath
222+ return self._get_path(self.cloud_dir, name)
223
224 # _get_ipath : get the instance path for a name in pathmap
225 # (/var/lib/cloud/instances/<instance>/<name>)
226@@ -397,6 +391,14 @@
227 else:
228 return ipath
229
230+ def _get_path(self, base, name=None):
231+ if name is None:
232+ return base
233+ return os.path.join(base, self.lookups[name])
234+
235+ def get_runpath(self, name=None):
236+ return self._get_path(self.run_dir, name)
237+
238
239 # This config parser will not throw when sections don't exist
240 # and you are setting values on those sections which is useful
241
242=== modified file 'cloudinit/net/__init__.py'
243--- cloudinit/net/__init__.py 2016-05-12 17:56:26 +0000
244+++ cloudinit/net/__init__.py 2016-06-03 19:07:08 +0000
245@@ -201,7 +201,11 @@
246 ifaces[iface]['method'] = method
247 currif = iface
248 elif option == "hwaddress":
249- ifaces[currif]['hwaddress'] = split[1]
250+ if split[1] == "ether":
251+ val = split[2]
252+ else:
253+ val = split[1]
254+ ifaces[currif]['hwaddress'] = val
255 elif option in NET_CONFIG_OPTIONS:
256 ifaces[currif][option] = split[1]
257 elif option in NET_CONFIG_COMMANDS:
258@@ -570,6 +574,8 @@
259 content += iface_start_entry(iface, index)
260 content += iface_add_subnet(iface, subnet)
261 content += iface_add_attrs(iface)
262+ for route in subnet.get('routes', []):
263+ content += render_route(route, indent=" ")
264 else:
265 # ifenslave docs say to auto the slave devices
266 if 'bond-master' in iface:
267@@ -768,4 +774,218 @@
268 return config_from_klibc_net_cfg(files=files, mac_addrs=mac_addrs)
269
270
271+def convert_eni_data(eni_data):
272+ # return a network config representation of what is in eni_data
273+ ifaces = {}
274+ parse_deb_config_data(ifaces, eni_data, src_dir=None, src_path=None)
275+ return _ifaces_to_net_config_data(ifaces)
276+
277+
278+def _ifaces_to_net_config_data(ifaces):
279+ """Return network config that represents the ifaces data provided.
280+ ifaces = parse_deb_config("/etc/network/interfaces")
281+ config = ifaces_to_net_config_data(ifaces)
282+ state = parse_net_config_data(config)."""
283+ devs = {}
284+ for name, data in ifaces.items():
285+ # devname is 'eth0' for name='eth0:1'
286+ devname = name.partition(":")[0]
287+ if devname == "lo":
288+ # currently provding 'lo' in network config results in duplicate
289+ # entries. in rendered interfaces file. so skip it.
290+ continue
291+ if devname not in devs:
292+ devs[devname] = {'type': 'physical', 'name': devname,
293+ 'subnets': []}
294+ # this isnt strictly correct, but some might specify
295+ # hwaddress on a nic for matching / declaring name.
296+ if 'hwaddress' in data:
297+ devs[devname]['mac_address'] = data['hwaddress']
298+ subnet = {'_orig_eni_name': name, 'type': data['method']}
299+ if data.get('auto'):
300+ subnet['control'] = 'auto'
301+ else:
302+ subnet['control'] = 'manual'
303+
304+ if data.get('method') == 'static':
305+ subnet['address'] = data['address']
306+
307+ for copy_key in ('netmask', 'gateway', 'broadcast'):
308+ if copy_key in data:
309+ subnet[copy_key] = data[copy_key]
310+
311+ if 'dns' in data:
312+ for n in ('nameservers', 'search'):
313+ if n in data['dns'] and data['dns'][n]:
314+ subnet['dns_' + n] = data['dns'][n]
315+ devs[devname]['subnets'].append(subnet)
316+
317+ return {'version': 1,
318+ 'config': [devs[d] for d in sorted(devs)]}
319+
320+
321+def apply_network_config_names(netcfg, strict_present=True, strict_busy=True):
322+ """read the network config and rename devices accordingly.
323+ if strict_present is false, then do not raise exception if no devices
324+ match. if strict_busy is false, then do not raise exception if the
325+ device cannot be renamed because it is currently configured."""
326+ renames = []
327+ for ent in netcfg.get('config', {}):
328+ if ent.get('type') != 'physical':
329+ continue
330+ mac = ent.get('mac_address')
331+ name = ent.get('name')
332+ if not mac:
333+ continue
334+ renames.append([mac, name])
335+
336+ return rename_interfaces(renames)
337+
338+
339+def _get_current_rename_info(check_downable=True):
340+ """Collect information necessary for rename_interfaces."""
341+ names = get_devicelist()
342+ bymac = {}
343+ for n in names:
344+ bymac[get_interface_mac(n)] = {
345+ 'name': n, 'up': is_up(n), 'downable': None}
346+
347+ if check_downable:
348+ nmatch = re.compile(r"[0-9]+:\s+(\w+)[@:]")
349+ ipv6, _err = util.subp(['ip', '-6', 'addr', 'show', 'permanent',
350+ 'scope', 'global'], capture=True)
351+ ipv4, _err = util.subp(['ip', '-4', 'addr', 'show'], capture=True)
352+
353+ nics_with_addresses = set()
354+ for bytes_out in (ipv6, ipv4):
355+ nics_with_addresses.update(nmatch.findall(bytes_out))
356+
357+ for d in bymac.values():
358+ d['downable'] = (d['up'] is False or
359+ d['name'] not in nics_with_addresses)
360+
361+ return bymac
362+
363+
364+def rename_interfaces(renames, strict_present=True, strict_busy=True,
365+ current_info=None):
366+ if current_info is None:
367+ current_info = _get_current_rename_info()
368+
369+ cur_bymac = {}
370+ for mac, data in current_info.items():
371+ cur = data.copy()
372+ cur['mac'] = mac
373+ cur_bymac[mac] = cur
374+
375+ def update_byname(bymac):
376+ return {data['name']: data for data in bymac.values()}
377+
378+ def rename(cur, new):
379+ util.subp(["ip", "link", "set", cur, "name", new], capture=True)
380+
381+ def down(name):
382+ util.subp(["ip", "link", "set", name, "down"], capture=True)
383+
384+ def up(name):
385+ util.subp(["ip", "link", "set", name, "up"], capture=True)
386+
387+ ops = []
388+ errors = []
389+ ups = []
390+ cur_byname = update_byname(cur_bymac)
391+ tmpname_fmt = "cirename%d"
392+ tmpi = -1
393+
394+ for mac, new_name in renames:
395+ cur = cur_bymac.get(mac, {})
396+ cur_name = cur.get('name')
397+ cur_ops = []
398+ if cur_name == new_name:
399+ # nothing to do
400+ continue
401+
402+ if not cur_name:
403+ if strict_present:
404+ errors.append(
405+ "[nic not present] Cannot rename mac=%s to %s"
406+ ", not available." % (mac, new_name))
407+ continue
408+
409+ if cur['up']:
410+ msg = "[busy] Error renaming mac=%s from %s to %s"
411+ if not cur['downable']:
412+ if strict_busy:
413+ errors.append(msg % (mac, cur_name, new_name))
414+ continue
415+ cur['up'] = False
416+ cur_ops.append(("down", mac, new_name, (cur_name,)))
417+ ups.append(("up", mac, new_name, (new_name,)))
418+
419+ if new_name in cur_byname:
420+ target = cur_byname[new_name]
421+ if target['up']:
422+ msg = "[busy-target] Error renaming mac=%s from %s to %s."
423+ if not target['downable']:
424+ if strict_busy:
425+ errors.append(msg % (mac, cur_name, new_name))
426+ continue
427+ else:
428+ cur_ops.append(("down", mac, new_name, (new_name,)))
429+
430+ tmp_name = None
431+ while tmp_name is None or tmp_name in cur_byname:
432+ tmpi += 1
433+ tmp_name = tmpname_fmt % tmpi
434+
435+ cur_ops.append(("rename", mac, new_name, (new_name, tmp_name)))
436+ target['name'] = tmp_name
437+ cur_byname = update_byname(cur_bymac)
438+ if target['up']:
439+ ups.append(("up", mac, new_name, (tmp_name,)))
440+
441+ cur_ops.append(("rename", mac, new_name, (cur['name'], new_name)))
442+ cur['name'] = new_name
443+ cur_byname = update_byname(cur_bymac)
444+ ops += cur_ops
445+
446+ opmap = {'rename': rename, 'down': down, 'up': up}
447+
448+ if len(ops) + len(ups) == 0:
449+ if len(errors):
450+ LOG.debug("unable to do any work for renaming of %s", renames)
451+ else:
452+ LOG.debug("no work necessary for renaming of %s", renames)
453+ else:
454+ LOG.debug("achieving renaming of %s with ops %s", renames, ops + ups)
455+
456+ for op, mac, new_name, params in ops + ups:
457+ try:
458+ opmap.get(op)(*params)
459+ except Exception as e:
460+ errors.append(
461+ "[unknown] Error performing %s%s for %s, %s: %s" %
462+ (op, params, mac, new_name, e))
463+
464+ if len(errors):
465+ raise Exception('\n'.join(errors))
466+
467+
468+def get_interface_mac(ifname):
469+ """Returns the string value of an interface's MAC Address"""
470+ return read_sys_net(ifname, "address", enoent=False)
471+
472+
473+def get_interfaces_by_mac(devs=None):
474+ """Build a dictionary of tuples {mac: name}"""
475+ if devs is None:
476+ devs = get_devicelist()
477+ ret = {}
478+ for name in devs:
479+ mac = get_interface_mac(name)
480+ # some devices may not have a mac (tun0)
481+ if mac:
482+ ret[mac] = name
483+ return ret
484+
485 # vi: ts=4 expandtab syntax=python
486
487=== modified file 'cloudinit/sources/DataSourceCloudSigma.py'
488--- cloudinit/sources/DataSourceCloudSigma.py 2016-05-12 17:56:26 +0000
489+++ cloudinit/sources/DataSourceCloudSigma.py 2016-06-03 19:07:08 +0000
490@@ -27,8 +27,6 @@
491
492 LOG = logging.getLogger(__name__)
493
494-VALID_DSMODES = ("local", "net", "disabled")
495-
496
497 class DataSourceCloudSigma(sources.DataSource):
498 """
499@@ -38,7 +36,6 @@
500 http://cloudsigma-docs.readthedocs.org/en/latest/server_context.html
501 """
502 def __init__(self, sys_cfg, distro, paths):
503- self.dsmode = 'local'
504 self.cepko = Cepko()
505 self.ssh_public_key = ''
506 sources.DataSource.__init__(self, sys_cfg, distro, paths)
507@@ -84,11 +81,9 @@
508 LOG.debug("CloudSigma: Unable to read from serial port")
509 return False
510
511- dsmode = server_meta.get('cloudinit-dsmode', self.dsmode)
512- if dsmode not in VALID_DSMODES:
513- LOG.warn("Invalid dsmode %s, assuming default of 'net'", dsmode)
514- dsmode = 'net'
515- if dsmode == "disabled" or dsmode != self.dsmode:
516+ self.dsmode = self._determine_dsmode(
517+ [server_meta.get('cloudinit-dsmode')])
518+ if dsmode == sources.DSMODE_DISABLED:
519 return False
520
521 base64_fields = server_meta.get('base64_fields', '').split(',')
522@@ -120,17 +115,13 @@
523 return self.metadata['uuid']
524
525
526-class DataSourceCloudSigmaNet(DataSourceCloudSigma):
527- def __init__(self, sys_cfg, distro, paths):
528- DataSourceCloudSigma.__init__(self, sys_cfg, distro, paths)
529- self.dsmode = 'net'
530-
531+# Legacy: Must be present in case we load an old pkl object
532+DataSourceCloudSigmaNet = DataSourceCloudSigma
533
534 # Used to match classes to dependencies. Since this datasource uses the serial
535 # port network is not really required, so it's okay to load without it, too.
536 datasources = [
537 (DataSourceCloudSigma, (sources.DEP_FILESYSTEM)),
538- (DataSourceCloudSigmaNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
539 ]
540
541
542
543=== modified file 'cloudinit/sources/DataSourceConfigDrive.py'
544--- cloudinit/sources/DataSourceConfigDrive.py 2016-04-29 13:04:36 +0000
545+++ cloudinit/sources/DataSourceConfigDrive.py 2016-06-03 19:07:08 +0000
546@@ -22,6 +22,7 @@
547 import os
548
549 from cloudinit import log as logging
550+from cloudinit import net
551 from cloudinit import sources
552 from cloudinit import util
553
554@@ -35,7 +36,6 @@
555 DEFAULT_METADATA = {
556 "instance-id": DEFAULT_IID,
557 }
558-VALID_DSMODES = ("local", "net", "pass", "disabled")
559 FS_TYPES = ('vfat', 'iso9660')
560 LABEL_TYPES = ('config-2',)
561 POSSIBLE_MOUNTS = ('sr', 'cd')
562@@ -47,12 +47,12 @@
563 def __init__(self, sys_cfg, distro, paths):
564 super(DataSourceConfigDrive, self).__init__(sys_cfg, distro, paths)
565 self.source = None
566- self.dsmode = 'local'
567 self.seed_dir = os.path.join(paths.seed_dir, 'config_drive')
568 self.version = None
569 self.ec2_metadata = None
570 self._network_config = None
571 self.network_json = None
572+ self.network_eni = None
573 self.files = {}
574
575 def __str__(self):
576@@ -98,38 +98,22 @@
577
578 md = results.get('metadata', {})
579 md = util.mergemanydict([md, DEFAULT_METADATA])
580- user_dsmode = results.get('dsmode', None)
581- if user_dsmode not in VALID_DSMODES + (None,):
582- LOG.warn("User specified invalid mode: %s", user_dsmode)
583- user_dsmode = None
584-
585- dsmode = get_ds_mode(cfgdrv_ver=results['version'],
586- ds_cfg=self.ds_cfg.get('dsmode'),
587- user=user_dsmode)
588-
589- if dsmode == "disabled":
590- # most likely user specified
591+
592+ self.dsmode = self._determine_dsmode(
593+ [results.get('dsmode'), self.ds_cfg.get('dsmode'),
594+ sources.DSMODE_PASS if results['version'] == 1 else None])
595+
596+ if self.dsmode == sources.DSMODE_DISABLED:
597 return False
598
599- # TODO(smoser): fix this, its dirty.
600- # we want to do some things (writing files and network config)
601- # only on first boot, and even then, we want to do so in the
602- # local datasource (so they happen earlier) even if the configured
603- # dsmode is 'net' or 'pass'. To do this, we check the previous
604- # instance-id
605+ # This is legacy and sneaky. If dsmode is 'pass' then write
606+ # 'injected files' and apply legacy ENI network format.
607 prev_iid = get_previous_iid(self.paths)
608 cur_iid = md['instance-id']
609- if prev_iid != cur_iid and self.dsmode == "local":
610+ if prev_iid != cur_iid and self.dsmode == sources.DSMODE_PASS:
611 on_first_boot(results, distro=self.distro)
612-
613- # dsmode != self.dsmode here if:
614- # * dsmode = "pass", pass means it should only copy files and then
615- # pass to another datasource
616- # * dsmode = "net" and self.dsmode = "local"
617- # so that user boothooks would be applied with network, the
618- # local datasource just gets out of the way, and lets the net claim
619- if dsmode != self.dsmode:
620- LOG.debug("%s: not claiming datasource, dsmode=%s", self, dsmode)
621+ LOG.debug("%s: not claiming datasource, dsmode=%s", self,
622+ self.dsmode)
623 return False
624
625 self.source = found
626@@ -147,12 +131,11 @@
627 LOG.warn("Invalid content in vendor-data: %s", e)
628 self.vendordata_raw = None
629
630- try:
631- self.network_json = results.get('networkdata')
632- except ValueError as e:
633- LOG.warn("Invalid content in network-data: %s", e)
634- self.network_json = None
635-
636+ # network_config is an /etc/network/interfaces formated file and is
637+ # obsolete compared to networkdata (from network_data.json) but both
638+ # might be present.
639+ self.network_eni = results.get("network_config")
640+ self.network_json = results.get('networkdata')
641 return True
642
643 def check_instance_id(self, sys_cfg):
644@@ -163,41 +146,16 @@
645 def network_config(self):
646 if self._network_config is None:
647 if self.network_json is not None:
648+ LOG.debug("network config provided via network_json")
649 self._network_config = convert_network_data(self.network_json)
650+ elif self.network_eni is not None:
651+ self._network_config = net.convert_eni_data(self.network_eni)
652+ LOG.debug("network config provided via converted eni data")
653+ else:
654+ LOG.debug("no network configuration available")
655 return self._network_config
656
657
658-class DataSourceConfigDriveNet(DataSourceConfigDrive):
659- def __init__(self, sys_cfg, distro, paths):
660- DataSourceConfigDrive.__init__(self, sys_cfg, distro, paths)
661- self.dsmode = 'net'
662-
663-
664-def get_ds_mode(cfgdrv_ver, ds_cfg=None, user=None):
665- """Determine what mode should be used.
666- valid values are 'pass', 'disabled', 'local', 'net'
667- """
668- # user passed data trumps everything
669- if user is not None:
670- return user
671-
672- if ds_cfg is not None:
673- return ds_cfg
674-
675- # at config-drive version 1, the default behavior was pass. That
676- # meant to not use use it as primary data source, but expect a ec2 metadata
677- # source. for version 2, we default to 'net', which means
678- # the DataSourceConfigDriveNet, would be used.
679- #
680- # this could change in the future. If there was definitive metadata
681- # that indicated presense of an openstack metadata service, then
682- # we could change to 'pass' by default also. The motivation for that
683- # would be 'cloud-init query' as the web service could be more dynamic
684- if cfgdrv_ver == 1:
685- return "pass"
686- return "net"
687-
688-
689 def read_config_drive(source_dir):
690 reader = openstack.ConfigDriveReader(source_dir)
691 finders = [
692@@ -231,9 +189,12 @@
693 % (type(data)))
694 net_conf = data.get("network_config", '')
695 if net_conf and distro:
696- LOG.debug("Updating network interfaces from config drive")
697+ LOG.warn("Updating network interfaces from config drive")
698 distro.apply_network(net_conf)
699- files = data.get('files', {})
700+ write_injected_files(data.get('files'))
701+
702+
703+def write_injected_files(files):
704 if files:
705 LOG.debug("Writing %s injected files", len(files))
706 for (filename, content) in files.items():
707@@ -293,20 +254,8 @@
708 return devices
709
710
711-# Used to match classes to dependencies
712-datasources = [
713- (DataSourceConfigDrive, (sources.DEP_FILESYSTEM, )),
714- (DataSourceConfigDriveNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
715-]
716-
717-
718-# Return a list of data sources that match this set of dependencies
719-def get_datasource_list(depends):
720- return sources.list_from_depends(depends, datasources)
721-
722-
723 # Convert OpenStack ConfigDrive NetworkData json to network_config yaml
724-def convert_network_data(network_json=None):
725+def convert_network_data(network_json=None, known_macs=None):
726 """Return a dictionary of network_config by parsing provided
727 OpenStack ConfigDrive NetworkData json format
728
729@@ -344,6 +293,7 @@
730 'mac_address',
731 'subnets',
732 'params',
733+ 'mtu',
734 ],
735 'subnet': [
736 'type',
737@@ -353,7 +303,6 @@
738 'metric',
739 'gateway',
740 'pointopoint',
741- 'mtu',
742 'scope',
743 'dns_nameservers',
744 'dns_search',
745@@ -370,9 +319,15 @@
746 subnets = []
747 cfg = {k: v for k, v in link.items()
748 if k in valid_keys['physical']}
749- cfg.update({'name': link['id']})
750- for network in [net for net in networks
751- if net['link'] == link['id']]:
752+ # 'name' is not in openstack spec yet, but we will support it if it is
753+ # present. The 'id' in the spec is currently implemented as the host
754+ # nic's name, meaning something like 'tap-adfasdffd'. We do not want
755+ # to name guest devices with such ugly names.
756+ if 'name' in link:
757+ cfg['name'] = link['name']
758+
759+ for network in [n for n in networks
760+ if n['link'] == link['id']]:
761 subnet = {k: v for k, v in network.items()
762 if k in valid_keys['subnet']}
763 if 'dhcp' in network['type']:
764@@ -387,7 +342,7 @@
765 })
766 subnets.append(subnet)
767 cfg.update({'subnets': subnets})
768- if link['type'] in ['ethernet', 'vif', 'ovs', 'phy']:
769+ if link['type'] in ['ethernet', 'vif', 'ovs', 'phy', 'bridge']:
770 cfg.update({
771 'type': 'physical',
772 'mac_address': link['ethernet_mac_address']})
773@@ -416,9 +371,38 @@
774
775 config.append(cfg)
776
777+ need_names = [d for d in config
778+ if d.get('type') == 'physical' and 'name' not in d]
779+
780+ if need_names:
781+ if known_macs is None:
782+ known_macs = net.get_interfaces_by_mac()
783+
784+ for d in need_names:
785+ mac = d.get('mac_address')
786+ if not mac:
787+ raise ValueError("No mac_address or name entry for %s" % d)
788+ if mac not in known_macs:
789+ raise ValueError("Unable to find a system nic for %s" % d)
790+ d['name'] = known_macs[mac]
791+
792 for service in services:
793 cfg = service
794 cfg.update({'type': 'nameserver'})
795 config.append(cfg)
796
797 return {'version': 1, 'config': config}
798+
799+
800+# Legacy: Must be present in case we load an old pkl object
801+DataSourceConfigDriveNet = DataSourceConfigDrive
802+
803+# Used to match classes to dependencies
804+datasources = [
805+ (DataSourceConfigDrive, (sources.DEP_FILESYSTEM, )),
806+]
807+
808+
809+# Return a list of data sources that match this set of dependencies
810+def get_datasource_list(depends):
811+ return sources.list_from_depends(depends, datasources)
812
813=== modified file 'cloudinit/sources/DataSourceNoCloud.py'
814--- cloudinit/sources/DataSourceNoCloud.py 2016-05-12 17:56:26 +0000
815+++ cloudinit/sources/DataSourceNoCloud.py 2016-06-03 19:07:08 +0000
816@@ -24,6 +24,7 @@
817 import os
818
819 from cloudinit import log as logging
820+from cloudinit import net
821 from cloudinit import sources
822 from cloudinit import util
823
824@@ -35,7 +36,6 @@
825 sources.DataSource.__init__(self, sys_cfg, distro, paths)
826 self.dsmode = 'local'
827 self.seed = None
828- self.cmdline_id = "ds=nocloud"
829 self.seed_dirs = [os.path.join(paths.seed_dir, 'nocloud'),
830 os.path.join(paths.seed_dir, 'nocloud-net')]
831 self.seed_dir = None
832@@ -58,7 +58,7 @@
833 try:
834 # Parse the kernel command line, getting data passed in
835 md = {}
836- if parse_cmdline_data(self.cmdline_id, md):
837+ if load_cmdline_data(md):
838 found.append("cmdline")
839 mydata = _merge_new_seed(mydata, {'meta-data': md})
840 except Exception:
841@@ -123,12 +123,6 @@
842
843 mydata = _merge_new_seed(mydata, seeded)
844
845- # For seed from a device, the default mode is 'net'.
846- # that is more likely to be what is desired. If they want
847- # dsmode of local, then they must specify that.
848- if 'dsmode' not in mydata['meta-data']:
849- mydata['meta-data']['dsmode'] = "net"
850-
851 LOG.debug("Using data from %s", dev)
852 found.append(dev)
853 break
854@@ -144,7 +138,6 @@
855 if len(found) == 0:
856 return False
857
858- seeded_network = None
859 # The special argument "seedfrom" indicates we should
860 # attempt to seed the userdata / metadata from its value
861 # its primarily value is in allowing the user to type less
862@@ -160,10 +153,6 @@
863 LOG.debug("Seed from %s not supported by %s", seedfrom, self)
864 return False
865
866- if (mydata['meta-data'].get('network-interfaces') or
867- mydata.get('network-config')):
868- seeded_network = self.dsmode
869-
870 # This could throw errors, but the user told us to do it
871 # so if errors are raised, let them raise
872 (md_seed, ud) = util.read_seeded(seedfrom, timeout=None)
873@@ -179,35 +168,21 @@
874 mydata['meta-data'] = util.mergemanydict([mydata['meta-data'],
875 defaults])
876
877- netdata = {'format': None, 'data': None}
878- if mydata['meta-data'].get('network-interfaces'):
879- netdata['format'] = 'interfaces'
880- netdata['data'] = mydata['meta-data']['network-interfaces']
881- elif mydata.get('network-config'):
882- netdata['format'] = 'network-config'
883- netdata['data'] = mydata['network-config']
884-
885- # if this is the local datasource or 'seedfrom' was used
886- # and the source of the seed was self.dsmode.
887- # Then see if there is network config to apply.
888- # note this is obsolete network-interfaces style seeding.
889- if self.dsmode in ("local", seeded_network):
890- if mydata['meta-data'].get('network-interfaces'):
891- LOG.debug("Updating network interfaces from %s", self)
892- self.distro.apply_network(
893- mydata['meta-data']['network-interfaces'])
894-
895- if mydata['meta-data']['dsmode'] == self.dsmode:
896- self.seed = ",".join(found)
897- self.metadata = mydata['meta-data']
898- self.userdata_raw = mydata['user-data']
899- self.vendordata_raw = mydata['vendor-data']
900- self._network_config = mydata['network-config']
901- return True
902-
903- LOG.debug("%s: not claiming datasource, dsmode=%s", self,
904- mydata['meta-data']['dsmode'])
905- return False
906+ self.dsmode = self._determine_dsmode(
907+ [mydata['meta-data'].get('dsmode')])
908+
909+ if self.dsmode == sources.DSMODE_DISABLED:
910+ LOG.debug("%s: not claiming datasource, dsmode=%s", self,
911+ self.dsmode)
912+ return False
913+
914+ self.seed = ",".join(found)
915+ self.metadata = mydata['meta-data']
916+ self.userdata_raw = mydata['user-data']
917+ self.vendordata_raw = mydata['vendor-data']
918+ self._network_config = mydata['network-config']
919+ self._network_eni = mydata['meta-data'].get('network-interfaces')
920+ return True
921
922 def check_instance_id(self, sys_cfg):
923 # quickly (local check only) if self.instance_id is still valid
924@@ -227,6 +202,9 @@
925
926 @property
927 def network_config(self):
928+ if self._network_config is None:
929+ if self.network_eni is not None:
930+ self._network_config = net.convert_eni_data(self.network_eni)
931 return self._network_config
932
933
934@@ -254,8 +232,22 @@
935 return None
936
937
938+def load_cmdline_data(fill, cmdline=None):
939+ pairs = [("ds=nocloud", sources.DSMODE_LOCAL),
940+ ("ds=nocloud-net", sources.DSMODE_NETWORK)]
941+ for idstr, dsmode in pairs:
942+ if parse_cmdline_data(idstr, fill, cmdline):
943+ # if dsmode was explicitly in the commanad line, then
944+ # prefer it to the dsmode based on the command line id
945+ if 'dsmode' not in fill:
946+ fill['dsmode'] = dsmode
947+ return True
948+ return False
949+
950+
951 # Returns true or false indicating if cmdline indicated
952-# that this module should be used
953+# that this module should be used. Updates dictionary 'fill'
954+# with data that was found.
955 # Example cmdline:
956 # root=LABEL=uec-rootfs ro ds=nocloud
957 def parse_cmdline_data(ds_id, fill, cmdline=None):
958@@ -319,9 +311,7 @@
959 class DataSourceNoCloudNet(DataSourceNoCloud):
960 def __init__(self, sys_cfg, distro, paths):
961 DataSourceNoCloud.__init__(self, sys_cfg, distro, paths)
962- self.cmdline_id = "ds=nocloud-net"
963 self.supported_seed_starts = ("http://", "https://", "ftp://")
964- self.dsmode = "net"
965
966
967 # Used to match classes to dependencies
968
969=== modified file 'cloudinit/sources/DataSourceOpenNebula.py'
970--- cloudinit/sources/DataSourceOpenNebula.py 2016-03-04 06:45:58 +0000
971+++ cloudinit/sources/DataSourceOpenNebula.py 2016-06-03 19:07:08 +0000
972@@ -37,16 +37,13 @@
973 LOG = logging.getLogger(__name__)
974
975 DEFAULT_IID = "iid-dsopennebula"
976-DEFAULT_MODE = 'net'
977 DEFAULT_PARSEUSER = 'nobody'
978 CONTEXT_DISK_FILES = ["context.sh"]
979-VALID_DSMODES = ("local", "net", "disabled")
980
981
982 class DataSourceOpenNebula(sources.DataSource):
983 def __init__(self, sys_cfg, distro, paths):
984 sources.DataSource.__init__(self, sys_cfg, distro, paths)
985- self.dsmode = 'local'
986 self.seed = None
987 self.seed_dir = os.path.join(paths.seed_dir, 'opennebula')
988
989@@ -93,52 +90,27 @@
990 md = util.mergemanydict([md, defaults])
991
992 # check for valid user specified dsmode
993- user_dsmode = results['metadata'].get('DSMODE', None)
994- if user_dsmode not in VALID_DSMODES + (None,):
995- LOG.warn("user specified invalid mode: %s", user_dsmode)
996- user_dsmode = None
997-
998- # decide dsmode
999- if user_dsmode:
1000- dsmode = user_dsmode
1001- elif self.ds_cfg.get('dsmode'):
1002- dsmode = self.ds_cfg.get('dsmode')
1003- else:
1004- dsmode = DEFAULT_MODE
1005-
1006- if dsmode == "disabled":
1007- # most likely user specified
1008- return False
1009-
1010- # apply static network configuration only in 'local' dsmode
1011- if ('network-interfaces' in results and self.dsmode == "local"):
1012- LOG.debug("Updating network interfaces from %s", self)
1013- self.distro.apply_network(results['network-interfaces'])
1014-
1015- if dsmode != self.dsmode:
1016- LOG.debug("%s: not claiming datasource, dsmode=%s", self, dsmode)
1017+ self.dsmode = self._determine_dsmode(
1018+ [results.get('DSMODE'), self.ds_cfg.get('dsmode')])
1019+
1020+ if self.dsmode == sources.DSMODE_DISABLED:
1021 return False
1022
1023 self.seed = seed
1024+ self.network_eni = results.get("network_config")
1025 self.metadata = md
1026 self.userdata_raw = results.get('userdata')
1027 return True
1028
1029 def get_hostname(self, fqdn=False, resolve_ip=None):
1030 if resolve_ip is None:
1031- if self.dsmode == 'net':
1032+ if self.dsmode == sources.DSMODE_NET:
1033 resolve_ip = True
1034 else:
1035 resolve_ip = False
1036 return sources.DataSource.get_hostname(self, fqdn, resolve_ip)
1037
1038
1039-class DataSourceOpenNebulaNet(DataSourceOpenNebula):
1040- def __init__(self, sys_cfg, distro, paths):
1041- DataSourceOpenNebula.__init__(self, sys_cfg, distro, paths)
1042- self.dsmode = 'net'
1043-
1044-
1045 class NonContextDiskDir(Exception):
1046 pass
1047
1048@@ -443,10 +415,12 @@
1049 return results
1050
1051
1052+# Legacy: Must be present in case we load an old pkl object
1053+DataSourceOpenNebulaNet = DataSourceOpenNebula
1054+
1055 # Used to match classes to dependencies
1056 datasources = [
1057 (DataSourceOpenNebula, (sources.DEP_FILESYSTEM, )),
1058- (DataSourceOpenNebulaNet, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
1059 ]
1060
1061
1062
1063=== modified file 'cloudinit/sources/DataSourceOpenStack.py'
1064--- cloudinit/sources/DataSourceOpenStack.py 2016-05-16 23:08:19 +0000
1065+++ cloudinit/sources/DataSourceOpenStack.py 2016-06-03 19:07:08 +0000
1066@@ -33,13 +33,11 @@
1067 DEFAULT_METADATA = {
1068 "instance-id": DEFAULT_IID,
1069 }
1070-VALID_DSMODES = ("net", "disabled")
1071
1072
1073 class DataSourceOpenStack(openstack.SourceMixin, sources.DataSource):
1074 def __init__(self, sys_cfg, distro, paths):
1075 super(DataSourceOpenStack, self).__init__(sys_cfg, distro, paths)
1076- self.dsmode = 'net'
1077 self.metadata_address = None
1078 self.ssl_details = util.fetch_ssl_details(self.paths)
1079 self.version = None
1080@@ -125,11 +123,8 @@
1081 self.metadata_address)
1082 return False
1083
1084- user_dsmode = results.get('dsmode', None)
1085- if user_dsmode not in VALID_DSMODES + (None,):
1086- LOG.warn("User specified invalid mode: %s", user_dsmode)
1087- user_dsmode = None
1088- if user_dsmode == 'disabled':
1089+ self.dsmode = self._determine_dsmode([results.get('dsmode')])
1090+ if self.dsmode == sources.DSMODE_DISABLED:
1091 return False
1092
1093 md = results.get('metadata', {})
1094
1095=== modified file 'cloudinit/sources/DataSourceSmartOS.py'
1096--- cloudinit/sources/DataSourceSmartOS.py 2016-04-12 16:57:50 +0000
1097+++ cloudinit/sources/DataSourceSmartOS.py 2016-06-03 19:07:08 +0000
1098@@ -32,13 +32,13 @@
1099 # http://us-east.manta.joyent.com/jmc/public/mdata/datadict.html
1100 # Comments with "@datadictionary" are snippets of the definition
1101
1102+import base64
1103 import binascii
1104-import contextlib
1105+import json
1106 import os
1107 import random
1108 import re
1109 import socket
1110-import stat
1111
1112 import serial
1113
1114@@ -64,14 +64,36 @@
1115 'operator-script': ('sdc:operator-script', False),
1116 }
1117
1118+SMARTOS_ATTRIB_JSON = {
1119+ # Cloud-init Key : (SmartOS Key known JSON)
1120+ 'network-data': 'sdc:nics',
1121+}
1122+
1123+SMARTOS_ENV_LX_BRAND = "lx-brand"
1124+SMARTOS_ENV_KVM = "kvm"
1125+
1126 DS_NAME = 'SmartOS'
1127 DS_CFG_PATH = ['datasource', DS_NAME]
1128+NO_BASE64_DECODE = [
1129+ 'iptables_disable',
1130+ 'motd_sys_info',
1131+ 'root_authorized_keys',
1132+ 'sdc:datacenter_name',
1133+ 'sdc:uuid'
1134+ 'user-data',
1135+ 'user-script',
1136+]
1137+
1138+METADATA_SOCKFILE = '/native/.zonecontrol/metadata.sock'
1139+SERIAL_DEVICE = '/dev/ttyS1'
1140+SERIAL_TIMEOUT = 60
1141+
1142 # BUILT-IN DATASOURCE CONFIGURATION
1143 # The following is the built-in configuration. If the values
1144 # are not set via the system configuration, then these default
1145 # will be used:
1146 # serial_device: which serial device to use for the meta-data
1147-# seed_timeout: how long to wait on the device
1148+# serial_timeout: how long to wait on the device
1149 # no_base64_decode: values which are not base64 encoded and
1150 # are fetched directly from SmartOS, not meta-data values
1151 # base64_keys: meta-data keys that are delivered in base64
1152@@ -81,16 +103,10 @@
1153 # fs_setup: describes how to format the ephemeral drive
1154 #
1155 BUILTIN_DS_CONFIG = {
1156- 'serial_device': '/dev/ttyS1',
1157- 'metadata_sockfile': '/native/.zonecontrol/metadata.sock',
1158- 'seed_timeout': 60,
1159- 'no_base64_decode': ['root_authorized_keys',
1160- 'motd_sys_info',
1161- 'iptables_disable',
1162- 'user-data',
1163- 'user-script',
1164- 'sdc:datacenter_name',
1165- 'sdc:uuid'],
1166+ 'serial_device': SERIAL_DEVICE,
1167+ 'serial_timeout': SERIAL_TIMEOUT,
1168+ 'metadata_sockfile': METADATA_SOCKFILE,
1169+ 'no_base64_decode': NO_BASE64_DECODE,
1170 'base64_keys': [],
1171 'base64_all': False,
1172 'disk_aliases': {'ephemeral0': '/dev/vdb'},
1173@@ -154,59 +170,41 @@
1174
1175
1176 class DataSourceSmartOS(sources.DataSource):
1177+ _unset = "_unset"
1178+ smartos_type = _unset
1179+ md_client = _unset
1180+
1181 def __init__(self, sys_cfg, distro, paths):
1182 sources.DataSource.__init__(self, sys_cfg, distro, paths)
1183- self.is_smartdc = None
1184 self.ds_cfg = util.mergemanydict([
1185 self.ds_cfg,
1186 util.get_cfg_by_path(sys_cfg, DS_CFG_PATH, {}),
1187 BUILTIN_DS_CONFIG])
1188
1189 self.metadata = {}
1190+ self.network_data = None
1191+ self._network_config = None
1192
1193- # SDC LX-Brand Zones lack dmidecode (no /dev/mem) but
1194- # report 'BrandZ virtual linux' as the kernel version
1195- if os.uname()[3].lower() == 'brandz virtual linux':
1196- LOG.debug("Host is SmartOS, guest in Zone")
1197- self.is_smartdc = True
1198- self.smartos_type = 'lx-brand'
1199- self.cfg = {}
1200- self.seed = self.ds_cfg.get("metadata_sockfile")
1201- else:
1202- self.is_smartdc = True
1203- self.smartos_type = 'kvm'
1204- self.seed = self.ds_cfg.get("serial_device")
1205- self.cfg = BUILTIN_CLOUD_CONFIG
1206- self.seed_timeout = self.ds_cfg.get("serial_timeout")
1207- self.smartos_no_base64 = self.ds_cfg.get('no_base64_decode')
1208- self.b64_keys = self.ds_cfg.get('base64_keys')
1209- self.b64_all = self.ds_cfg.get('base64_all')
1210 self.script_base_d = os.path.join(self.paths.get_cpath("scripts"))
1211
1212+ self._init()
1213+
1214 def __str__(self):
1215 root = sources.DataSource.__str__(self)
1216- return "%s [seed=%s]" % (root, self.seed)
1217-
1218- def _get_seed_file_object(self):
1219- if not self.seed:
1220- raise AttributeError("seed device is not set")
1221-
1222- if self.smartos_type == 'lx-brand':
1223- if not stat.S_ISSOCK(os.stat(self.seed).st_mode):
1224- LOG.debug("Seed %s is not a socket", self.seed)
1225- return None
1226- sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
1227- sock.connect(self.seed)
1228- return sock.makefile('rwb')
1229- else:
1230- if not stat.S_ISCHR(os.stat(self.seed).st_mode):
1231- LOG.debug("Seed %s is not a character device")
1232- return None
1233- ser = serial.Serial(self.seed, timeout=self.seed_timeout)
1234- if not ser.isOpen():
1235- raise SystemError("Unable to open %s" % self.seed)
1236- return ser
1237- return None
1238+ return "%s [client=%s]" % (root, self.md_client)
1239+
1240+ def _init(self):
1241+ if self.smartos_type == self._unset:
1242+ self.smartos_type = get_smartos_environ()
1243+ if self.smartos_type is None:
1244+ self.md_client = None
1245+
1246+ if self.md_client == self._unset:
1247+ self.md_client = jmc_client_factory(
1248+ smartos_type=self.smartos_type,
1249+ metadata_sockfile=self.ds_cfg['metadata_sockfile'],
1250+ serial_device=self.ds_cfg['serial_device'],
1251+ serial_timeout=self.ds_cfg['serial_timeout'])
1252
1253 def _set_provisioned(self):
1254 '''Mark the instance provisioning state as successful.
1255@@ -225,50 +223,26 @@
1256 '/'.join([svc_path, 'provision_success']))
1257
1258 def get_data(self):
1259+ self._init()
1260+
1261 md = {}
1262 ud = ""
1263
1264- if not device_exists(self.seed):
1265- LOG.debug("No metadata device '%s' found for SmartOS datasource",
1266- self.seed)
1267- return False
1268-
1269- uname_arch = os.uname()[4]
1270- if uname_arch.startswith("arm") or uname_arch == "aarch64":
1271- # Disabling because dmidcode in dmi_data() crashes kvm process
1272- LOG.debug("Disabling SmartOS datasource on arm (LP: #1243287)")
1273- return False
1274-
1275- # SDC KVM instances will provide dmi data, LX-brand does not
1276- if self.smartos_type == 'kvm':
1277- dmi_info = dmi_data()
1278- if dmi_info is None:
1279- LOG.debug("No dmidata utility found")
1280- return False
1281-
1282- system_type = dmi_info
1283- if 'smartdc' not in system_type.lower():
1284- LOG.debug("Host is not on SmartOS. system_type=%s",
1285- system_type)
1286- return False
1287- LOG.debug("Host is SmartOS, guest in KVM")
1288-
1289- seed_obj = self._get_seed_file_object()
1290- if seed_obj is None:
1291- LOG.debug('Seed file object not found.')
1292- return False
1293- with contextlib.closing(seed_obj) as seed:
1294- b64_keys = self.query('base64_keys', seed, strip=True, b64=False)
1295- if b64_keys is not None:
1296- self.b64_keys = [k.strip() for k in str(b64_keys).split(',')]
1297-
1298- b64_all = self.query('base64_all', seed, strip=True, b64=False)
1299- if b64_all is not None:
1300- self.b64_all = util.is_true(b64_all)
1301-
1302- for ci_noun, attribute in SMARTOS_ATTRIB_MAP.items():
1303- smartos_noun, strip = attribute
1304- md[ci_noun] = self.query(smartos_noun, seed, strip=strip)
1305+ if not self.smartos_type:
1306+ LOG.debug("Not running on smartos")
1307+ return False
1308+
1309+ if not self.md_client.exists():
1310+ LOG.debug("No metadata device '%r' found for SmartOS datasource",
1311+ self.md_client)
1312+ return False
1313+
1314+ for ci_noun, attribute in SMARTOS_ATTRIB_MAP.items():
1315+ smartos_noun, strip = attribute
1316+ md[ci_noun] = self.md_client.get(smartos_noun, strip=strip)
1317+
1318+ for ci_noun, smartos_noun in SMARTOS_ATTRIB_JSON.items():
1319+ md[ci_noun] = self.md_client.get_json(smartos_noun)
1320
1321 # @datadictionary: This key may contain a program that is written
1322 # to a file in the filesystem of the guest on each boot and then
1323@@ -318,6 +292,7 @@
1324 self.metadata = util.mergemanydict([md, self.metadata])
1325 self.userdata_raw = ud
1326 self.vendordata_raw = md['vendor-data']
1327+ self.network_data = md['network-data']
1328
1329 self._set_provisioned()
1330 return True
1331@@ -326,69 +301,20 @@
1332 return self.ds_cfg['disk_aliases'].get(name)
1333
1334 def get_config_obj(self):
1335- return self.cfg
1336+ if self.smartos_type == SMARTOS_ENV_KVM:
1337+ return BUILTIN_CLOUD_CONFIG
1338+ return {}
1339
1340 def get_instance_id(self):
1341 return self.metadata['instance-id']
1342
1343- def query(self, noun, seed_file, strip=False, default=None, b64=None):
1344- if b64 is None:
1345- if noun in self.smartos_no_base64:
1346- b64 = False
1347- elif self.b64_all or noun in self.b64_keys:
1348- b64 = True
1349-
1350- return self._query_data(noun, seed_file, strip=strip,
1351- default=default, b64=b64)
1352-
1353- def _query_data(self, noun, seed_file, strip=False,
1354- default=None, b64=None):
1355- """Makes a request via "GET <NOUN>"
1356-
1357- In the response, the first line is the status, while subsequent
1358- lines are is the value. A blank line with a "." is used to
1359- indicate end of response.
1360-
1361- If the response is expected to be base64 encoded, then set
1362- b64encoded to true. Unfortantely, there is no way to know if
1363- something is 100% encoded, so this method relies on being told
1364- if the data is base64 or not.
1365- """
1366-
1367- if not noun:
1368- return False
1369-
1370- response = JoyentMetadataClient(seed_file).get_metadata(noun)
1371-
1372- if response is None:
1373- return default
1374-
1375- if b64 is None:
1376- b64 = self._query_data('b64-%s' % noun, seed_file, b64=False,
1377- default=False, strip=True)
1378- b64 = util.is_true(b64)
1379-
1380- resp = None
1381- if b64 or strip:
1382- resp = "".join(response).rstrip()
1383- else:
1384- resp = "".join(response)
1385-
1386- if b64:
1387- try:
1388- return util.b64d(resp)
1389- # Bogus input produces different errors in Python 2 and 3;
1390- # catch both.
1391- except (TypeError, binascii.Error):
1392- LOG.warn("Failed base64 decoding key '%s'", noun)
1393- return resp
1394-
1395- return resp
1396-
1397-
1398-def device_exists(device):
1399- """Symplistic method to determine if the device exists or not"""
1400- return os.path.exists(device)
1401+ @property
1402+ def network_config(self):
1403+ if self._network_config is None:
1404+ if self.network_data is not None:
1405+ self._network_config = (
1406+ convert_smartos_network_data(self.network_data))
1407+ return self._network_config
1408
1409
1410 class JoyentMetadataFetchException(Exception):
1411@@ -407,8 +333,11 @@
1412 r' (?P<body>(?P<request_id>[0-9a-f]+) (?P<status>SUCCESS|NOTFOUND)'
1413 r'( (?P<payload>.+))?)')
1414
1415- def __init__(self, metasource):
1416- self.metasource = metasource
1417+ def __init__(self, smartos_type=None, fp=None):
1418+ if smartos_type is None:
1419+ smartos_type = get_smartos_environ()
1420+ self.smartos_type = smartos_type
1421+ self.fp = fp
1422
1423 def _checksum(self, body):
1424 return '{0:08x}'.format(
1425@@ -436,37 +365,229 @@
1426 LOG.debug('Value "%s" found.', value)
1427 return value
1428
1429- def get_metadata(self, metadata_key):
1430- LOG.debug('Fetching metadata key "%s"...', metadata_key)
1431+ def request(self, rtype, param=None):
1432 request_id = '{0:08x}'.format(random.randint(0, 0xffffffff))
1433- message_body = '{0} GET {1}'.format(request_id,
1434- util.b64e(metadata_key))
1435+ message_body = ' '.join((request_id, rtype,))
1436+ if param:
1437+ message_body += ' ' + base64.b64encode(param.encode()).decode()
1438 msg = 'V2 {0} {1} {2}\n'.format(
1439 len(message_body), self._checksum(message_body), message_body)
1440 LOG.debug('Writing "%s" to metadata transport.', msg)
1441- self.metasource.write(msg.encode('ascii'))
1442- self.metasource.flush()
1443+
1444+ need_close = False
1445+ if not self.fp:
1446+ self.open_transport()
1447+ need_close = True
1448+
1449+ self.fp.write(msg.encode('ascii'))
1450+ self.fp.flush()
1451
1452 response = bytearray()
1453- response.extend(self.metasource.read(1))
1454+ response.extend(self.fp.read(1))
1455 while response[-1:] != b'\n':
1456- response.extend(self.metasource.read(1))
1457+ response.extend(self.fp.read(1))
1458+
1459+ if need_close:
1460+ self.close_transport()
1461+
1462 response = response.rstrip().decode('ascii')
1463 LOG.debug('Read "%s" from metadata transport.', response)
1464
1465 if 'SUCCESS' not in response:
1466 return None
1467
1468- return self._get_value_from_frame(request_id, response)
1469-
1470-
1471-def dmi_data():
1472- sys_type = util.read_dmi_data("system-product-name")
1473-
1474- if not sys_type:
1475+ value = self._get_value_from_frame(request_id, response)
1476+ return value
1477+
1478+ def get(self, key, default=None, strip=False):
1479+ result = self.request(rtype='GET', param=key)
1480+ if result is None:
1481+ return default
1482+ if result and strip:
1483+ result = result.strip()
1484+ return result
1485+
1486+ def get_json(self, key, default=None):
1487+ result = self.get(key, default=default)
1488+ if result is None:
1489+ return default
1490+ return json.loads(result)
1491+
1492+ def list(self):
1493+ result = self.request(rtype='KEYS')
1494+ if result:
1495+ result = result.split('\n')
1496+ return result
1497+
1498+ def put(self, key, val):
1499+ param = b' '.join([base64.b64encode(i.encode())
1500+ for i in (key, val)]).decode()
1501+ return self.request(rtype='PUT', param=param)
1502+
1503+ def delete(self, key):
1504+ return self.request(rtype='DELETE', param=key)
1505+
1506+ def close_transport(self):
1507+ if self.fp:
1508+ self.fp.close()
1509+ self.fp = None
1510+
1511+ def __enter__(self):
1512+ if self.fp:
1513+ return self
1514+ self.open_transport()
1515+ return self
1516+
1517+ def __exit__(self, exc_type, exc_value, traceback):
1518+ self.close_transport()
1519+ return
1520+
1521+ def open_transport(self):
1522+ raise NotImplementedError
1523+
1524+
1525+class JoyentMetadataSocketClient(JoyentMetadataClient):
1526+ def __init__(self, socketpath):
1527+ self.socketpath = socketpath
1528+
1529+ def open_transport(self):
1530+ sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
1531+ sock.connect(self.socketpath)
1532+ self.fp = sock.makefile('rwb')
1533+
1534+ def exists(self):
1535+ return os.path.exists(self.socketpath)
1536+
1537+ def __repr__(self):
1538+ return "%s(socketpath=%s)" % (self.__class__.__name__, self.socketpath)
1539+
1540+
1541+class JoyentMetadataSerialClient(JoyentMetadataClient):
1542+ def __init__(self, device, timeout=10, smartos_type=None):
1543+ super(JoyentMetadataSerialClient, self).__init__(smartos_type)
1544+ self.device = device
1545+ self.timeout = timeout
1546+
1547+ def exists(self):
1548+ return os.path.exists(self.device)
1549+
1550+ def open_transport(self):
1551+ ser = serial.Serial(self.device, timeout=self.timeout)
1552+ if not ser.isOpen():
1553+ raise SystemError("Unable to open %s" % self.device)
1554+ self.fp = ser
1555+
1556+ def __repr__(self):
1557+ return "%s(device=%s, timeout=%s)" % (
1558+ self.__class__.__name__, self.device, self.timeout)
1559+
1560+
1561+class JoyentMetadataLegacySerialClient(JoyentMetadataSerialClient):
1562+ """V1 of the protocol was not safe for all values.
1563+ Thus, we allowed the user to pass values in as base64 encoded.
1564+ Users may still reasonably expect to be able to send base64 data
1565+ and have it transparently decoded. So even though the V2 format is
1566+ now used, and is safe (using base64 itself), we keep legacy support.
1567+
1568+ The way for a user to do this was:
1569+ a.) specify 'base64_keys' key whose value is a comma delimited
1570+ list of keys that were base64 encoded.
1571+ b.) base64_all: string interpreted as a boolean that indicates
1572+ if all keys are base64 encoded.
1573+ c.) set a key named b64-<keyname> with a boolean indicating that
1574+ <keyname> is base64 encoded."""
1575+
1576+ def __init__(self, device, timeout=10, smartos_type=None):
1577+ s = super(JoyentMetadataLegacySerialClient, self)
1578+ s.__init__(device, timeout, smartos_type)
1579+ self.base64_keys = None
1580+ self.base64_all = None
1581+
1582+ def _init_base64_keys(self, reset=False):
1583+ if reset:
1584+ self.base64_keys = None
1585+ self.base64_all = None
1586+
1587+ keys = None
1588+ if self.base64_all is None:
1589+ keys = self.list()
1590+ if 'base64_all' in keys:
1591+ self.base64_all = util.is_true(self._get("base64_all"))
1592+ else:
1593+ self.base64_all = False
1594+
1595+ if self.base64_all:
1596+ # short circuit if base64_all is true
1597+ return
1598+
1599+ if self.base64_keys is None:
1600+ if keys is None:
1601+ keys = self.list()
1602+ b64_keys = set()
1603+ if 'base64_keys' in keys:
1604+ b64_keys = set(self._get("base64_keys").split(","))
1605+
1606+ # now add any b64-<keyname> that has a true value
1607+ for key in [k[3:] for k in keys if k.startswith("b64-")]:
1608+ if util.is_true(self._get(key)):
1609+ b64_keys.add(key)
1610+ else:
1611+ if key in b64_keys:
1612+ b64_keys.remove(key)
1613+
1614+ self.base64_keys = b64_keys
1615+
1616+ def _get(self, key, default=None, strip=False):
1617+ return (super(JoyentMetadataLegacySerialClient, self).
1618+ get(key, default=default, strip=strip))
1619+
1620+ def is_b64_encoded(self, key, reset=False):
1621+ if key in NO_BASE64_DECODE:
1622+ return False
1623+
1624+ self._init_base64_keys(reset=reset)
1625+ if self.base64_all:
1626+ return True
1627+
1628+ return key in self.base64_keys
1629+
1630+ def get(self, key, default=None, strip=False):
1631+ mdefault = object()
1632+ val = self._get(key, strip=False, default=mdefault)
1633+ if val is mdefault:
1634+ return default
1635+
1636+ if self.is_b64_encoded(key):
1637+ try:
1638+ val = base64.b64decode(val.encode()).decode()
1639+ # Bogus input produces different errors in Python 2 and 3
1640+ except (TypeError, binascii.Error):
1641+ LOG.warn("Failed base64 decoding key '%s': %s", key, val)
1642+
1643+ if strip:
1644+ val = val.strip()
1645+
1646+ return val
1647+
1648+
1649+def jmc_client_factory(
1650+ smartos_type=None, metadata_sockfile=METADATA_SOCKFILE,
1651+ serial_device=SERIAL_DEVICE, serial_timeout=SERIAL_TIMEOUT,
1652+ uname_version=None):
1653+
1654+ if smartos_type is None:
1655+ smartos_type = get_smartos_environ(uname_version)
1656+
1657+ if smartos_type is None:
1658 return None
1659+ elif smartos_type == SMARTOS_ENV_KVM:
1660+ return JoyentMetadataLegacySerialClient(
1661+ device=serial_device, timeout=serial_timeout,
1662+ smartos_type=smartos_type)
1663+ elif smartos_type == SMARTOS_ENV_LX_BRAND:
1664+ return JoyentMetadataSocketClient(socketpath=metadata_sockfile)
1665
1666- return sys_type
1667+ raise ValueError("Unknown value for smartos_type: %s" % smartos_type)
1668
1669
1670 def write_boot_content(content, content_f, link=None, shebang=False,
1671@@ -522,15 +643,141 @@
1672 util.ensure_dir(os.path.dirname(link))
1673 os.symlink(content_f, link)
1674 except IOError as e:
1675- util.logexc(LOG, "failed establishing content link", e)
1676+ util.logexc(LOG, "failed establishing content link: %s", e)
1677+
1678+
1679+def get_smartos_environ(uname_version=None, product_name=None,
1680+ uname_arch=None):
1681+ uname = os.uname()
1682+ if uname_arch is None:
1683+ uname_arch = uname[4]
1684+
1685+ if uname_arch.startswith("arm") or uname_arch == "aarch64":
1686+ return None
1687+
1688+ # SDC LX-Brand Zones lack dmidecode (no /dev/mem) but
1689+ # report 'BrandZ virtual linux' as the kernel version
1690+ if uname_version is None:
1691+ uname_version = uname[3]
1692+ if uname_version.lower() == 'brandz virtual linux':
1693+ return SMARTOS_ENV_LX_BRAND
1694+
1695+ if product_name is None:
1696+ system_type = util.read_dmi_data("system-product-name")
1697+ else:
1698+ system_type = product_name
1699+
1700+ if system_type and 'smartdc' in system_type.lower():
1701+ return SMARTOS_ENV_KVM
1702+
1703+ return None
1704+
1705+
1706+# Covert SMARTOS 'sdc:nics' data to network_config yaml
1707+def convert_smartos_network_data(network_data=None):
1708+ """Return a dictionary of network_config by parsing provided
1709+ SMARTOS sdc:nics configuration data
1710+
1711+ sdc:nics data is a dictionary of properties of a nic and the ip
1712+ configuration desired. Additional nic dictionaries are appended
1713+ to the list.
1714+
1715+ Converting the format is straightforward though it does include
1716+ duplicate information as well as data which appears to be relevant
1717+ to the hostOS rather than the guest.
1718+
1719+ For each entry in the nics list returned from query sdc:nics, we
1720+ create a type: physical entry, and extract the interface properties:
1721+ 'mac' -> 'mac_address', 'mtu', 'interface' -> 'name'. The remaining
1722+ keys are related to ip configuration. For each ip in the 'ips' list
1723+ we create a subnet entry under 'subnets' pairing the ip to a one in
1724+ the 'gateways' list.
1725+ """
1726+
1727+ valid_keys = {
1728+ 'physical': [
1729+ 'mac_address',
1730+ 'mtu',
1731+ 'name',
1732+ 'params',
1733+ 'subnets',
1734+ 'type',
1735+ ],
1736+ 'subnet': [
1737+ 'address',
1738+ 'broadcast',
1739+ 'dns_nameservers',
1740+ 'dns_search',
1741+ 'gateway',
1742+ 'metric',
1743+ 'netmask',
1744+ 'pointopoint',
1745+ 'routes',
1746+ 'scope',
1747+ 'type',
1748+ ],
1749+ }
1750+
1751+ config = []
1752+ for nic in network_data:
1753+ cfg = {k: v for k, v in nic.items()
1754+ if k in valid_keys['physical']}
1755+ cfg.update({
1756+ 'type': 'physical',
1757+ 'name': nic['interface']})
1758+ if 'mac' in nic:
1759+ cfg.update({'mac_address': nic['mac']})
1760+
1761+ subnets = []
1762+ for ip, gw in zip(nic['ips'], nic['gateways']):
1763+ subnet = {k: v for k, v in nic.items()
1764+ if k in valid_keys['subnet']}
1765+ subnet.update({
1766+ 'type': 'static',
1767+ 'address': ip,
1768+ 'gateway': gw,
1769+ })
1770+ subnets.append(subnet)
1771+ cfg.update({'subnets': subnets})
1772+ config.append(cfg)
1773+
1774+ return {'version': 1, 'config': config}
1775
1776
1777 # Used to match classes to dependencies
1778 datasources = [
1779- (DataSourceSmartOS, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
1780+ (DataSourceSmartOS, (sources.DEP_FILESYSTEM, )),
1781 ]
1782
1783
1784 # Return a list of data sources that match this set of dependencies
1785 def get_datasource_list(depends):
1786 return sources.list_from_depends(depends, datasources)
1787+
1788+
1789+if __name__ == "__main__":
1790+ import sys
1791+ jmc = jmc_client_factory()
1792+ if jmc is None:
1793+ print("Do not appear to be on smartos.")
1794+ sys.exit(1)
1795+ if len(sys.argv) == 1:
1796+ keys = (list(SMARTOS_ATTRIB_JSON.keys()) +
1797+ list(SMARTOS_ATTRIB_MAP.keys()))
1798+ else:
1799+ keys = sys.argv[1:]
1800+
1801+ data = {}
1802+ for key in keys:
1803+ if key in SMARTOS_ATTRIB_JSON:
1804+ keyname = SMARTOS_ATTRIB_JSON[key]
1805+ data[key] = jmc.get_json(keyname)
1806+ else:
1807+ if key in SMARTOS_ATTRIB_MAP:
1808+ keyname, strip = SMARTOS_ATTRIB_MAP[key]
1809+ else:
1810+ keyname, strip = (key, False)
1811+ val = jmc.get(keyname, strip=strip)
1812+ data[key] = jmc.get(keyname, strip=strip)
1813+
1814+ print(json.dumps(data, indent=1))
1815
1816=== modified file 'cloudinit/sources/__init__.py'
1817--- cloudinit/sources/__init__.py 2016-05-12 17:56:26 +0000
1818+++ cloudinit/sources/__init__.py 2016-06-03 19:07:08 +0000
1819@@ -34,6 +34,13 @@
1820 from cloudinit.filters import launch_index
1821 from cloudinit.reporting import events
1822
1823+DSMODE_DISABLED = "disabled"
1824+DSMODE_LOCAL = "local"
1825+DSMODE_NETWORK = "net"
1826+DSMODE_PASS = "pass"
1827+
1828+VALID_DSMODES = [DSMODE_DISABLED, DSMODE_LOCAL, DSMODE_NETWORK]
1829+
1830 DEP_FILESYSTEM = "FILESYSTEM"
1831 DEP_NETWORK = "NETWORK"
1832 DS_PREFIX = 'DataSource'
1833@@ -57,6 +64,7 @@
1834 self.userdata_raw = None
1835 self.vendordata = None
1836 self.vendordata_raw = None
1837+ self.dsmode = DSMODE_NETWORK
1838
1839 # find the datasource config name.
1840 # remove 'DataSource' from classname on front, and remove 'Net' on end.
1841@@ -223,10 +231,35 @@
1842 # quickly (local check only) if self.instance_id is still
1843 return False
1844
1845+ @staticmethod
1846+ def _determine_dsmode(candidates, default=None, valid=None):
1847+ # return the first candidate that is non None, warn if not valid
1848+ if default is None:
1849+ default = DSMODE_NETWORK
1850+
1851+ if valid is None:
1852+ valid = VALID_DSMODES
1853+
1854+ for candidate in candidates:
1855+ if candidate is None:
1856+ continue
1857+ if candidate in valid:
1858+ return candidate
1859+ else:
1860+ LOG.warn("invalid dsmode '%s', using default=%s",
1861+ candidate, default)
1862+ return default
1863+
1864+ return default
1865+
1866 @property
1867 def network_config(self):
1868 return None
1869
1870+ @property
1871+ def first_instance_boot(self):
1872+ return
1873+
1874
1875 def normalize_pubkey_data(pubkey_data):
1876 keys = []
1877
1878=== modified file 'cloudinit/sources/helpers/openstack.py'
1879--- cloudinit/sources/helpers/openstack.py 2016-05-12 17:56:26 +0000
1880+++ cloudinit/sources/helpers/openstack.py 2016-06-03 19:07:08 +0000
1881@@ -190,14 +190,14 @@
1882 versions_available)
1883 return selected_version
1884
1885- def _read_content_path(self, item):
1886+ def _read_content_path(self, item, decode=False):
1887 path = item.get('content_path', '').lstrip("/")
1888 path_pieces = path.split("/")
1889 valid_pieces = [p for p in path_pieces if len(p)]
1890 if not valid_pieces:
1891 raise BrokenMetadata("Item %s has no valid content path" % (item))
1892 path = self._path_join(self.base_path, "openstack", *path_pieces)
1893- return self._path_read(path)
1894+ return self._path_read(path, decode=decode)
1895
1896 def read_v2(self):
1897 """Reads a version 2 formatted location.
1898@@ -298,7 +298,8 @@
1899 net_item = metadata.get("network_config", None)
1900 if net_item:
1901 try:
1902- results['network_config'] = self._read_content_path(net_item)
1903+ content = self._read_content_path(net_item, decode=True)
1904+ results['network_config'] = content
1905 except IOError as e:
1906 raise BrokenMetadata("Failed to read network"
1907 " configuration: %s" % (e))
1908@@ -333,8 +334,8 @@
1909 components = [base] + list(add_ons)
1910 return os.path.join(*components)
1911
1912- def _path_read(self, path):
1913- return util.load_file(path, decode=False)
1914+ def _path_read(self, path, decode=False):
1915+ return util.load_file(path, decode=decode)
1916
1917 def _fetch_available_versions(self):
1918 if self._versions is None:
1919@@ -446,7 +447,7 @@
1920 self._versions = found
1921 return self._versions
1922
1923- def _path_read(self, path):
1924+ def _path_read(self, path, decode=False):
1925
1926 def should_retry_cb(_request_args, cause):
1927 try:
1928@@ -463,7 +464,10 @@
1929 ssl_details=self.ssl_details,
1930 timeout=self.timeout,
1931 exception_cb=should_retry_cb)
1932- return response.contents
1933+ if decode:
1934+ return response.contents.decode()
1935+ else:
1936+ return response.contents
1937
1938 def _path_join(self, base, *add_ons):
1939 return url_helper.combine_url(base, *add_ons)
1940
1941=== modified file 'cloudinit/stages.py'
1942--- cloudinit/stages.py 2016-05-26 13:02:17 +0000
1943+++ cloudinit/stages.py 2016-06-03 19:07:08 +0000
1944@@ -52,6 +52,7 @@
1945 LOG = logging.getLogger(__name__)
1946
1947 NULL_DATA_SOURCE = None
1948+NO_PREVIOUS_INSTANCE_ID = "NO_PREVIOUS_INSTANCE_ID"
1949
1950
1951 class Init(object):
1952@@ -67,6 +68,7 @@
1953 # Changed only when a fetch occurs
1954 self.datasource = NULL_DATA_SOURCE
1955 self.ds_restored = False
1956+ self._previous_iid = None
1957
1958 if reporter is None:
1959 reporter = events.ReportEventStack(
1960@@ -213,6 +215,31 @@
1961 cfg_list = self.cfg.get('datasource_list') or []
1962 return (cfg_list, pkg_list)
1963
1964+ def _restore_from_checked_cache(self, existing):
1965+ if existing not in ("check", "trust"):
1966+ raise ValueError("Unexpected value for existing: %s" % existing)
1967+
1968+ ds = self._restore_from_cache()
1969+ if not ds:
1970+ return (None, "no cache found")
1971+
1972+ run_iid_fn = self.paths.get_runpath('instance_id')
1973+ if os.path.exists(run_iid_fn):
1974+ run_iid = util.load_file(run_iid_fn).strip()
1975+ else:
1976+ run_iid = None
1977+
1978+ if run_iid == ds.get_instance_id():
1979+ return (ds, "restored from cache with run check: %s" % ds)
1980+ elif existing == "trust":
1981+ return (ds, "restored from cache: %s" % ds)
1982+ else:
1983+ if (hasattr(ds, 'check_instance_id') and
1984+ ds.check_instance_id(self.cfg)):
1985+ return (ds, "restored from checked cache: %s" % ds)
1986+ else:
1987+ return (None, "cache invalid in datasource: %s" % ds)
1988+
1989 def _get_data_source(self, existing):
1990 if self.datasource is not NULL_DATA_SOURCE:
1991 return self.datasource
1992@@ -221,19 +248,9 @@
1993 name="check-cache",
1994 description="attempting to read from cache [%s]" % existing,
1995 parent=self.reporter) as myrep:
1996- ds = self._restore_from_cache()
1997- if ds and existing == "trust":
1998- myrep.description = "restored from cache: %s" % ds
1999- elif ds and existing == "check":
2000- if (hasattr(ds, 'check_instance_id') and
2001- ds.check_instance_id(self.cfg)):
2002- myrep.description = "restored from checked cache: %s" % ds
2003- else:
2004- myrep.description = "cache invalid in datasource: %s" % ds
2005- ds = None
2006- else:
2007- myrep.description = "no cache found"
2008
2009+ ds, desc = self._restore_from_checked_cache(existing)
2010+ myrep.description = desc
2011 self.ds_restored = bool(ds)
2012 LOG.debug(myrep.description)
2013
2014@@ -301,23 +318,41 @@
2015
2016 # What the instance id was and is...
2017 iid = self.datasource.get_instance_id()
2018- previous_iid = None
2019 iid_fn = os.path.join(dp, 'instance-id')
2020- try:
2021- previous_iid = util.load_file(iid_fn).strip()
2022- except Exception:
2023- pass
2024- if not previous_iid:
2025- previous_iid = iid
2026+
2027+ previous_iid = self.previous_iid()
2028 util.write_file(iid_fn, "%s\n" % iid)
2029+ util.write_file(self.paths.get_runpath('instance_id'), "%s\n" % iid)
2030 util.write_file(os.path.join(dp, 'previous-instance-id'),
2031 "%s\n" % (previous_iid))
2032+
2033+ self._write_to_cache()
2034 # Ensure needed components are regenerated
2035 # after change of instance which may cause
2036 # change of configuration
2037 self._reset()
2038 return iid
2039
2040+ def previous_iid(self):
2041+ if self._previous_iid is not None:
2042+ return self._previous_iid
2043+
2044+ dp = self.paths.get_cpath('data')
2045+ iid_fn = os.path.join(dp, 'instance-id')
2046+ try:
2047+ self._previous_iid = util.load_file(iid_fn).strip()
2048+ except Exception:
2049+ self._previous_iid = NO_PREVIOUS_INSTANCE_ID
2050+
2051+ LOG.debug("previous iid found to be %s", self._previous_iid)
2052+ return self._previous_iid
2053+
2054+ def is_new_instance(self):
2055+ previous = self.previous_iid()
2056+ ret = (previous == NO_PREVIOUS_INSTANCE_ID or
2057+ previous != self.datasource.get_instance_id())
2058+ return ret
2059+
2060 def fetch(self, existing="check"):
2061 return self._get_data_source(existing=existing)
2062
2063@@ -332,8 +367,6 @@
2064 reporter=self.reporter)
2065
2066 def update(self):
2067- if not self._write_to_cache():
2068- return
2069 self._store_userdata()
2070 self._store_vendordata()
2071
2072@@ -593,15 +626,27 @@
2073 return (ncfg, loc)
2074 return (net.generate_fallback_config(), "fallback")
2075
2076- def apply_network_config(self):
2077+ def apply_network_config(self, bring_up):
2078 netcfg, src = self._find_networking_config()
2079 if netcfg is None:
2080 LOG.info("network config is disabled by %s", src)
2081 return
2082
2083- LOG.info("Applying network configuration from %s: %s", src, netcfg)
2084- try:
2085- return self.distro.apply_network_config(netcfg)
2086+ try:
2087+ LOG.debug("applying net config names for %s" % netcfg)
2088+ self.distro.apply_network_config_names(netcfg)
2089+ except Exception as e:
2090+ LOG.warn("Failed to rename devices: %s", e)
2091+
2092+ if (self.datasource is not NULL_DATA_SOURCE and
2093+ not self.is_new_instance()):
2094+ LOG.debug("not a new instance. network config is not applied.")
2095+ return
2096+
2097+ LOG.info("Applying network configuration from %s bringup=%s: %s",
2098+ src, bring_up, netcfg)
2099+ try:
2100+ return self.distro.apply_network_config(netcfg, bring_up=bring_up)
2101 except NotImplementedError:
2102 LOG.warn("distro '%s' does not implement apply_network_config. "
2103 "networking may not be configured properly." %
2104
2105=== modified file 'setup.py'
2106--- setup.py 2016-05-12 20:49:10 +0000
2107+++ setup.py 2016-06-03 19:07:08 +0000
2108@@ -184,7 +184,6 @@
2109 (USR + '/share/doc/cloud-init/examples/seed',
2110 [f for f in glob('doc/examples/seed/*') if is_f(f)]),
2111 (LIB + '/udev/rules.d', [f for f in glob('udev/*.rules')]),
2112- (LIB + '/udev', ['udev/cloud-init-wait']),
2113 ]
2114 # Use a subclass for install that handles
2115 # adding on the right init system configuration files
2116
2117=== modified file 'systemd/cloud-init-generator'
2118--- systemd/cloud-init-generator 2016-03-19 00:40:54 +0000
2119+++ systemd/cloud-init-generator 2016-06-03 19:07:08 +0000
2120@@ -107,9 +107,6 @@
2121 "ln $CLOUD_SYSTEM_TARGET $link_path"
2122 fi
2123 fi
2124- # this touches /run/cloud-init/enabled, which is read by
2125- # udev/cloud-init-wait. If not present, it will exit quickly.
2126- touch "$LOG_D/$ENABLE"
2127 elif [ "$result" = "$DISABLE" ]; then
2128 if [ -f "$link_path" ]; then
2129 if rm -f "$link_path"; then
2130
2131=== modified file 'tests/unittests/test_datasource/test_configdrive.py'
2132--- tests/unittests/test_datasource/test_configdrive.py 2016-05-12 20:43:11 +0000
2133+++ tests/unittests/test_datasource/test_configdrive.py 2016-06-03 19:07:08 +0000
2134@@ -15,6 +15,7 @@
2135 from contextlib2 import ExitStack
2136
2137 from cloudinit import helpers
2138+from cloudinit import net
2139 from cloudinit import settings
2140 from cloudinit.sources import DataSourceConfigDrive as ds
2141 from cloudinit.sources.helpers import openstack
2142@@ -73,7 +74,7 @@
2143 'type': 'ovs', 'mtu': None, 'id': 'tap2f88d109-5b'},
2144 {'vif_id': '1a5382f8-04c5-4d75-ab98-d666c1ef52cc',
2145 'ethernet_mac_address': 'fa:16:3e:05:30:fe',
2146- 'type': 'ovs', 'mtu': None, 'id': 'tap1a5382f8-04'}
2147+ 'type': 'ovs', 'mtu': None, 'id': 'tap1a5382f8-04', 'name': 'nic0'}
2148 ],
2149 'networks': [
2150 {'link': 'tap2ecc7709-b3', 'type': 'ipv4_dhcp',
2151@@ -88,6 +89,34 @@
2152 ]
2153 }
2154
2155+NETWORK_DATA_2 = {
2156+ "services": [
2157+ {"type": "dns", "address": "1.1.1.191"},
2158+ {"type": "dns", "address": "1.1.1.4"}],
2159+ "networks": [
2160+ {"network_id": "d94bbe94-7abc-48d4-9c82-4628ea26164a", "type": "ipv4",
2161+ "netmask": "255.255.255.248", "link": "eth0",
2162+ "routes": [{"netmask": "0.0.0.0", "network": "0.0.0.0",
2163+ "gateway": "2.2.2.9"}],
2164+ "ip_address": "2.2.2.10", "id": "network0-ipv4"},
2165+ {"network_id": "ca447c83-6409-499b-aaef-6ad1ae995348", "type": "ipv4",
2166+ "netmask": "255.255.255.224", "link": "eth1",
2167+ "routes": [], "ip_address": "3.3.3.24", "id": "network1-ipv4"}],
2168+ "links": [
2169+ {"ethernet_mac_address": "fa:16:3e:dd:50:9a", "mtu": 1500,
2170+ "type": "vif", "id": "eth0", "vif_id": "vif-foo1"},
2171+ {"ethernet_mac_address": "fa:16:3e:a8:14:69", "mtu": 1500,
2172+ "type": "vif", "id": "eth1", "vif_id": "vif-foo2"}]
2173+}
2174+
2175+
2176+KNOWN_MACS = {
2177+ 'fa:16:3e:69:b0:58': 'enp0s1',
2178+ 'fa:16:3e:d4:57:ad': 'enp0s2',
2179+ 'fa:16:3e:dd:50:9a': 'foo1',
2180+ 'fa:16:3e:a8:14:69': 'foo2',
2181+}
2182+
2183 CFG_DRIVE_FILES_V2 = {
2184 'ec2/2009-04-04/meta-data.json': json.dumps(EC2_META),
2185 'ec2/2009-04-04/user-data': USER_DATA,
2186@@ -365,10 +394,54 @@
2187 """Verify that network_data is converted and present on ds object."""
2188 populate_dir(self.tmp, CFG_DRIVE_FILES_V2)
2189 myds = cfg_ds_from_dir(self.tmp)
2190- network_config = ds.convert_network_data(NETWORK_DATA)
2191+ network_config = ds.convert_network_data(NETWORK_DATA,
2192+ known_macs=KNOWN_MACS)
2193 self.assertEqual(myds.network_config, network_config)
2194
2195
2196+class TestConvertNetworkData(TestCase):
2197+ def _getnames_in_config(self, ncfg):
2198+ return set([n['name'] for n in ncfg['config']
2199+ if n['type'] == 'physical'])
2200+
2201+ def test_conversion_fills_names(self):
2202+ ncfg = ds.convert_network_data(NETWORK_DATA, known_macs=KNOWN_MACS)
2203+ expected = set(['nic0', 'enp0s1', 'enp0s2'])
2204+ found = self._getnames_in_config(ncfg)
2205+ self.assertEqual(found, expected)
2206+
2207+ @mock.patch('cloudinit.net.get_interfaces_by_mac')
2208+ def test_convert_reads_system_prefers_name(self, get_interfaces_by_mac):
2209+ macs = KNOWN_MACS.copy()
2210+ macs.update({'fa:16:3e:05:30:fe': 'foonic1',
2211+ 'fa:16:3e:69:b0:58': 'ens1'})
2212+ get_interfaces_by_mac.return_value = macs
2213+
2214+ ncfg = ds.convert_network_data(NETWORK_DATA)
2215+ expected = set(['nic0', 'ens1', 'enp0s2'])
2216+ found = self._getnames_in_config(ncfg)
2217+ self.assertEqual(found, expected)
2218+
2219+ def test_convert_raises_value_error_on_missing_name(self):
2220+ macs = {'aa:aa:aa:aa:aa:00': 'ens1'}
2221+ self.assertRaises(ValueError, ds.convert_network_data,
2222+ NETWORK_DATA, known_macs=macs)
2223+
2224+ def test_conversion_with_route(self):
2225+ ncfg = ds.convert_network_data(NETWORK_DATA_2, known_macs=KNOWN_MACS)
2226+ # not the best test, but see that we get a route in the
2227+ # network config and that it gets rendered to an ENI file
2228+ routes = []
2229+ for n in ncfg['config']:
2230+ for s in n.get('subnets', []):
2231+ routes.extend(s.get('routes', []))
2232+ self.assertIn(
2233+ {'network': '0.0.0.0', 'netmask': '0.0.0.0', 'gateway': '2.2.2.9'},
2234+ routes)
2235+ eni = net.render_interfaces(net.parse_net_config_data(ncfg))
2236+ self.assertIn("route add default gw 2.2.2.9", eni)
2237+
2238+
2239 def cfg_ds_from_dir(seed_d):
2240 found = ds.read_config_drive(seed_d)
2241 cfg_ds = ds.DataSourceConfigDrive(settings.CFG_BUILTIN, None,
2242@@ -387,7 +460,8 @@
2243 cfg_ds.userdata_raw = results.get('userdata')
2244 cfg_ds.version = results.get('version')
2245 cfg_ds.network_json = results.get('networkdata')
2246- cfg_ds._network_config = ds.convert_network_data(cfg_ds.network_json)
2247+ cfg_ds._network_config = ds.convert_network_data(
2248+ cfg_ds.network_json, known_macs=KNOWN_MACS)
2249
2250
2251 def populate_dir(seed_dir, files):
2252
2253=== modified file 'tests/unittests/test_datasource/test_smartos.py'
2254--- tests/unittests/test_datasource/test_smartos.py 2016-05-12 20:43:11 +0000
2255+++ tests/unittests/test_datasource/test_smartos.py 2016-06-03 19:07:08 +0000
2256@@ -25,6 +25,7 @@
2257 from __future__ import print_function
2258
2259 from binascii import crc32
2260+import json
2261 import os
2262 import os.path
2263 import re
2264@@ -40,12 +41,49 @@
2265 from cloudinit.sources import DataSourceSmartOS
2266 from cloudinit.util import b64e
2267
2268-from .. import helpers
2269+from ..helpers import mock, FilesystemMockingTestCase, TestCase
2270
2271-try:
2272- from unittest import mock
2273-except ImportError:
2274- import mock
2275+SDC_NICS = json.loads("""
2276+[
2277+ {
2278+ "nic_tag": "external",
2279+ "primary": true,
2280+ "mtu": 1500,
2281+ "model": "virtio",
2282+ "gateway": "8.12.42.1",
2283+ "netmask": "255.255.255.0",
2284+ "ip": "8.12.42.102",
2285+ "network_uuid": "992fc7ce-6aac-4b74-aed6-7b9d2c6c0bfe",
2286+ "gateways": [
2287+ "8.12.42.1"
2288+ ],
2289+ "vlan_id": 324,
2290+ "mac": "90:b8:d0:f5:e4:f5",
2291+ "interface": "net0",
2292+ "ips": [
2293+ "8.12.42.102/24"
2294+ ]
2295+ },
2296+ {
2297+ "nic_tag": "sdc_overlay/16187209",
2298+ "gateway": "192.168.128.1",
2299+ "model": "virtio",
2300+ "mac": "90:b8:d0:a5:ff:cd",
2301+ "netmask": "255.255.252.0",
2302+ "ip": "192.168.128.93",
2303+ "network_uuid": "4cad71da-09bc-452b-986d-03562a03a0a9",
2304+ "gateways": [
2305+ "192.168.128.1"
2306+ ],
2307+ "vlan_id": 2,
2308+ "mtu": 8500,
2309+ "interface": "net1",
2310+ "ips": [
2311+ "192.168.128.93/22"
2312+ ]
2313+ }
2314+]
2315+""")
2316
2317 MOCK_RETURNS = {
2318 'hostname': 'test-host',
2319@@ -60,79 +98,66 @@
2320 'sdc:vendor-data': '\n'.join(['VENDOR_DATA', '']),
2321 'user-data': '\n'.join(['something', '']),
2322 'user-script': '\n'.join(['/bin/true', '']),
2323+ 'sdc:nics': json.dumps(SDC_NICS),
2324 }
2325
2326 DMI_DATA_RETURN = 'smartdc'
2327
2328
2329-def get_mock_client(mockdata):
2330- class MockMetadataClient(object):
2331-
2332- def __init__(self, serial):
2333- pass
2334-
2335- def get_metadata(self, metadata_key):
2336- return mockdata.get(metadata_key)
2337- return MockMetadataClient
2338-
2339-
2340-class TestSmartOSDataSource(helpers.FilesystemMockingTestCase):
2341+class PsuedoJoyentClient(object):
2342+ def __init__(self, data=None):
2343+ if data is None:
2344+ data = MOCK_RETURNS.copy()
2345+ self.data = data
2346+ return
2347+
2348+ def get(self, key, default=None, strip=False):
2349+ if key in self.data:
2350+ r = self.data[key]
2351+ if strip:
2352+ r = r.strip()
2353+ else:
2354+ r = default
2355+ return r
2356+
2357+ def get_json(self, key, default=None):
2358+ result = self.get(key, default=default)
2359+ if result is None:
2360+ return default
2361+ return json.loads(result)
2362+
2363+ def exists(self):
2364+ return True
2365+
2366+
2367+class TestSmartOSDataSource(FilesystemMockingTestCase):
2368 def setUp(self):
2369 super(TestSmartOSDataSource, self).setUp()
2370
2371+ dsmos = 'cloudinit.sources.DataSourceSmartOS'
2372+ patcher = mock.patch(dsmos + ".jmc_client_factory")
2373+ self.jmc_cfact = patcher.start()
2374+ self.addCleanup(patcher.stop)
2375+ patcher = mock.patch(dsmos + ".get_smartos_environ")
2376+ self.get_smartos_environ = patcher.start()
2377+ self.addCleanup(patcher.stop)
2378+
2379 self.tmp = tempfile.mkdtemp()
2380 self.addCleanup(shutil.rmtree, self.tmp)
2381+ self.paths = c_helpers.Paths({'cloud_dir': self.tmp})
2382+
2383 self.legacy_user_d = tempfile.mkdtemp()
2384- self.addCleanup(shutil.rmtree, self.legacy_user_d)
2385-
2386- # If you should want to watch the logs...
2387- self._log = None
2388- self._log_file = None
2389- self._log_handler = None
2390-
2391- # patch cloud_dir, so our 'seed_dir' is guaranteed empty
2392- self.paths = c_helpers.Paths({'cloud_dir': self.tmp})
2393-
2394- self.unapply = []
2395- super(TestSmartOSDataSource, self).setUp()
2396+ self.orig_lud = DataSourceSmartOS.LEGACY_USER_D
2397+ DataSourceSmartOS.LEGACY_USER_D = self.legacy_user_d
2398
2399 def tearDown(self):
2400- helpers.FilesystemMockingTestCase.tearDown(self)
2401- if self._log_handler and self._log:
2402- self._log.removeHandler(self._log_handler)
2403- apply_patches([i for i in reversed(self.unapply)])
2404+ DataSourceSmartOS.LEGACY_USER_D = self.orig_lud
2405 super(TestSmartOSDataSource, self).tearDown()
2406
2407- def _patchIn(self, root):
2408- self.restore()
2409- self.patchOS(root)
2410- self.patchUtils(root)
2411-
2412- def apply_patches(self, patches):
2413- ret = apply_patches(patches)
2414- self.unapply += ret
2415-
2416- def _get_ds(self, sys_cfg=None, ds_cfg=None, mockdata=None, dmi_data=None,
2417- is_lxbrand=False):
2418- mod = DataSourceSmartOS
2419-
2420- if mockdata is None:
2421- mockdata = MOCK_RETURNS
2422-
2423- if dmi_data is None:
2424- dmi_data = DMI_DATA_RETURN
2425-
2426- def _dmi_data():
2427- return dmi_data
2428-
2429- def _os_uname():
2430- if not is_lxbrand:
2431- # LP: #1243287. tests assume this runs, but running test on
2432- # arm would cause them all to fail.
2433- return ('LINUX', 'NODENAME', 'RELEASE', 'VERSION', 'x86_64')
2434- else:
2435- return ('LINUX', 'NODENAME', 'RELEASE', 'BRANDZ VIRTUAL LINUX',
2436- 'X86_64')
2437+ def _get_ds(self, mockdata=None, mode=DataSourceSmartOS.SMARTOS_ENV_KVM,
2438+ sys_cfg=None, ds_cfg=None):
2439+ self.jmc_cfact.return_value = PsuedoJoyentClient(mockdata)
2440+ self.get_smartos_environ.return_value = mode
2441
2442 if sys_cfg is None:
2443 sys_cfg = {}
2444@@ -141,44 +166,8 @@
2445 sys_cfg['datasource'] = sys_cfg.get('datasource', {})
2446 sys_cfg['datasource']['SmartOS'] = ds_cfg
2447
2448- self.apply_patches([(mod, 'LEGACY_USER_D', self.legacy_user_d)])
2449- self.apply_patches([
2450- (mod, 'JoyentMetadataClient', get_mock_client(mockdata))])
2451- self.apply_patches([(mod, 'dmi_data', _dmi_data)])
2452- self.apply_patches([(os, 'uname', _os_uname)])
2453- self.apply_patches([(mod, 'device_exists', lambda d: True)])
2454- dsrc = mod.DataSourceSmartOS(sys_cfg, distro=None,
2455- paths=self.paths)
2456- self.apply_patches([(dsrc, '_get_seed_file_object', mock.MagicMock())])
2457- return dsrc
2458-
2459- def test_seed(self):
2460- # default seed should be /dev/ttyS1
2461- dsrc = self._get_ds()
2462- ret = dsrc.get_data()
2463- self.assertTrue(ret)
2464- self.assertEqual('kvm', dsrc.smartos_type)
2465- self.assertEqual('/dev/ttyS1', dsrc.seed)
2466-
2467- def test_seed_lxbrand(self):
2468- # default seed should be /dev/ttyS1
2469- dsrc = self._get_ds(is_lxbrand=True)
2470- ret = dsrc.get_data()
2471- self.assertTrue(ret)
2472- self.assertEqual('lx-brand', dsrc.smartos_type)
2473- self.assertEqual('/native/.zonecontrol/metadata.sock', dsrc.seed)
2474-
2475- def test_issmartdc(self):
2476- dsrc = self._get_ds()
2477- ret = dsrc.get_data()
2478- self.assertTrue(ret)
2479- self.assertTrue(dsrc.is_smartdc)
2480-
2481- def test_issmartdc_lxbrand(self):
2482- dsrc = self._get_ds(is_lxbrand=True)
2483- ret = dsrc.get_data()
2484- self.assertTrue(ret)
2485- self.assertTrue(dsrc.is_smartdc)
2486+ return DataSourceSmartOS.DataSourceSmartOS(
2487+ sys_cfg, distro=None, paths=self.paths)
2488
2489 def test_no_base64(self):
2490 ds_cfg = {'no_base64_decode': ['test_var1'], 'all_base': True}
2491@@ -214,58 +203,6 @@
2492 self.assertEqual(MOCK_RETURNS['hostname'],
2493 dsrc.metadata['local-hostname'])
2494
2495- def test_base64_all(self):
2496- # metadata provided base64_all of true
2497- my_returns = MOCK_RETURNS.copy()
2498- my_returns['base64_all'] = "true"
2499- for k in ('hostname', 'cloud-init:user-data'):
2500- my_returns[k] = b64e(my_returns[k])
2501-
2502- dsrc = self._get_ds(mockdata=my_returns)
2503- ret = dsrc.get_data()
2504- self.assertTrue(ret)
2505- self.assertEqual(MOCK_RETURNS['hostname'],
2506- dsrc.metadata['local-hostname'])
2507- self.assertEqual(MOCK_RETURNS['cloud-init:user-data'],
2508- dsrc.userdata_raw)
2509- self.assertEqual(MOCK_RETURNS['root_authorized_keys'],
2510- dsrc.metadata['public-keys'])
2511- self.assertEqual(MOCK_RETURNS['disable_iptables_flag'],
2512- dsrc.metadata['iptables_disable'])
2513- self.assertEqual(MOCK_RETURNS['enable_motd_sys_info'],
2514- dsrc.metadata['motd_sys_info'])
2515-
2516- def test_b64_userdata(self):
2517- my_returns = MOCK_RETURNS.copy()
2518- my_returns['b64-cloud-init:user-data'] = "true"
2519- my_returns['b64-hostname'] = "true"
2520- for k in ('hostname', 'cloud-init:user-data'):
2521- my_returns[k] = b64e(my_returns[k])
2522-
2523- dsrc = self._get_ds(mockdata=my_returns)
2524- ret = dsrc.get_data()
2525- self.assertTrue(ret)
2526- self.assertEqual(MOCK_RETURNS['hostname'],
2527- dsrc.metadata['local-hostname'])
2528- self.assertEqual(MOCK_RETURNS['cloud-init:user-data'],
2529- dsrc.userdata_raw)
2530- self.assertEqual(MOCK_RETURNS['root_authorized_keys'],
2531- dsrc.metadata['public-keys'])
2532-
2533- def test_b64_keys(self):
2534- my_returns = MOCK_RETURNS.copy()
2535- my_returns['base64_keys'] = 'hostname,ignored'
2536- for k in ('hostname',):
2537- my_returns[k] = b64e(my_returns[k])
2538-
2539- dsrc = self._get_ds(mockdata=my_returns)
2540- ret = dsrc.get_data()
2541- self.assertTrue(ret)
2542- self.assertEqual(MOCK_RETURNS['hostname'],
2543- dsrc.metadata['local-hostname'])
2544- self.assertEqual(MOCK_RETURNS['cloud-init:user-data'],
2545- dsrc.userdata_raw)
2546-
2547 def test_userdata(self):
2548 dsrc = self._get_ds(mockdata=MOCK_RETURNS)
2549 ret = dsrc.get_data()
2550@@ -275,6 +212,13 @@
2551 self.assertEqual(MOCK_RETURNS['cloud-init:user-data'],
2552 dsrc.userdata_raw)
2553
2554+ def test_sdc_nics(self):
2555+ dsrc = self._get_ds(mockdata=MOCK_RETURNS)
2556+ ret = dsrc.get_data()
2557+ self.assertTrue(ret)
2558+ self.assertEqual(json.loads(MOCK_RETURNS['sdc:nics']),
2559+ dsrc.metadata['network-data'])
2560+
2561 def test_sdc_scripts(self):
2562 dsrc = self._get_ds(mockdata=MOCK_RETURNS)
2563 ret = dsrc.get_data()
2564@@ -430,18 +374,7 @@
2565 mydscfg['disk_aliases']['FOO'])
2566
2567
2568-def apply_patches(patches):
2569- ret = []
2570- for (ref, name, replace) in patches:
2571- if replace is None:
2572- continue
2573- orig = getattr(ref, name)
2574- setattr(ref, name, replace)
2575- ret.append((ref, name, orig))
2576- return ret
2577-
2578-
2579-class TestJoyentMetadataClient(helpers.FilesystemMockingTestCase):
2580+class TestJoyentMetadataClient(FilesystemMockingTestCase):
2581
2582 def setUp(self):
2583 super(TestJoyentMetadataClient, self).setUp()
2584@@ -481,7 +414,8 @@
2585 mock.Mock(return_value=self.request_id)))
2586
2587 def _get_client(self):
2588- return DataSourceSmartOS.JoyentMetadataClient(self.serial)
2589+ return DataSourceSmartOS.JoyentMetadataClient(
2590+ fp=self.serial, smartos_type=DataSourceSmartOS.SMARTOS_ENV_KVM)
2591
2592 def assertEndsWith(self, haystack, prefix):
2593 self.assertTrue(haystack.endswith(prefix),
2594@@ -495,7 +429,7 @@
2595
2596 def test_get_metadata_writes_a_single_line(self):
2597 client = self._get_client()
2598- client.get_metadata('some_key')
2599+ client.get('some_key')
2600 self.assertEqual(1, self.serial.write.call_count)
2601 written_line = self.serial.write.call_args[0][0]
2602 print(type(written_line))
2603@@ -505,7 +439,7 @@
2604
2605 def _get_written_line(self, key='some_key'):
2606 client = self._get_client()
2607- client.get_metadata(key)
2608+ client.get(key)
2609 return self.serial.write.call_args[0][0]
2610
2611 def test_get_metadata_writes_bytes(self):
2612@@ -549,32 +483,32 @@
2613
2614 def test_get_metadata_reads_a_line(self):
2615 client = self._get_client()
2616- client.get_metadata('some_key')
2617+ client.get('some_key')
2618 self.assertEqual(self.metasource_data_len, self.serial.read.call_count)
2619
2620 def test_get_metadata_returns_valid_value(self):
2621 client = self._get_client()
2622- value = client.get_metadata('some_key')
2623+ value = client.get('some_key')
2624 self.assertEqual(self.metadata_value, value)
2625
2626 def test_get_metadata_throws_exception_for_incorrect_length(self):
2627 self.response_parts['length'] = 0
2628 client = self._get_client()
2629 self.assertRaises(DataSourceSmartOS.JoyentMetadataFetchException,
2630- client.get_metadata, 'some_key')
2631+ client.get, 'some_key')
2632
2633 def test_get_metadata_throws_exception_for_incorrect_crc(self):
2634 self.response_parts['crc'] = 'deadbeef'
2635 client = self._get_client()
2636 self.assertRaises(DataSourceSmartOS.JoyentMetadataFetchException,
2637- client.get_metadata, 'some_key')
2638+ client.get, 'some_key')
2639
2640 def test_get_metadata_throws_exception_for_request_id_mismatch(self):
2641 self.response_parts['request_id'] = 'deadbeef'
2642 client = self._get_client()
2643 client._checksum = lambda _: self.response_parts['crc']
2644 self.assertRaises(DataSourceSmartOS.JoyentMetadataFetchException,
2645- client.get_metadata, 'some_key')
2646+ client.get, 'some_key')
2647
2648 def test_get_metadata_returns_None_if_value_not_found(self):
2649 self.response_parts['payload'] = ''
2650@@ -582,4 +516,24 @@
2651 self.response_parts['length'] = 17
2652 client = self._get_client()
2653 client._checksum = lambda _: self.response_parts['crc']
2654- self.assertIsNone(client.get_metadata('some_key'))
2655+ self.assertIsNone(client.get('some_key'))
2656+
2657+
2658+class TestNetworkConversion(TestCase):
2659+
2660+ def test_convert_simple(self):
2661+ expected = {
2662+ 'version': 1,
2663+ 'config': [
2664+ {'name': 'net0', 'type': 'physical',
2665+ 'subnets': [{'type': 'static', 'gateway': '8.12.42.1',
2666+ 'netmask': '255.255.255.0',
2667+ 'address': '8.12.42.102/24'}],
2668+ 'mtu': 1500, 'mac_address': '90:b8:d0:f5:e4:f5'},
2669+ {'name': 'net1', 'type': 'physical',
2670+ 'subnets': [{'type': 'static', 'gateway': '192.168.128.1',
2671+ 'netmask': '255.255.252.0',
2672+ 'address': '192.168.128.93/22'}],
2673+ 'mtu': 8500, 'mac_address': '90:b8:d0:a5:ff:cd'}]}
2674+ found = DataSourceSmartOS.convert_smartos_network_data(SDC_NICS)
2675+ self.assertEqual(expected, found)
2676
2677=== removed file 'udev/79-cloud-init-net-wait.rules'
2678--- udev/79-cloud-init-net-wait.rules 2016-03-19 00:40:54 +0000
2679+++ udev/79-cloud-init-net-wait.rules 1970-01-01 00:00:00 +0000
2680@@ -1,10 +0,0 @@
2681-# cloud-init cold/hot-plug blocking mechanism
2682-# this file blocks further processing of network events
2683-# until cloud-init local has had a chance to read and apply network
2684-SUBSYSTEM!="net", GOTO="cloudinit_naming_end"
2685-ACTION!="add", GOTO="cloudinit_naming_end"
2686-
2687-IMPORT{program}="/lib/udev/cloud-init-wait"
2688-
2689-LABEL="cloudinit_naming_end"
2690-# vi: ts=4 expandtab syntax=udevrules
2691
2692=== removed file 'udev/cloud-init-wait'
2693--- udev/cloud-init-wait 2016-03-29 13:11:25 +0000
2694+++ udev/cloud-init-wait 1970-01-01 00:00:00 +0000
2695@@ -1,70 +0,0 @@
2696-#!/bin/sh
2697-
2698-CI_NET_READY="/run/cloud-init/network-config-ready"
2699-LOG="/run/cloud-init/${0##*/}.log"
2700-LOG_INIT=0
2701-MAX_WAIT=60
2702-DEBUG=0
2703-
2704-block_until_ready() {
2705- local fname="$1" max="$2"
2706- [ -f "$fname" ] && return 0
2707- # udevadm settle below will exit at the first of 3 conditions
2708- # 1.) timeout 2.) file exists 3.) all in-flight udev events are processed
2709- # since this is being run from a udev event, the 3 wont happen.
2710- # thus, this is essentially a inotify wait or timeout on a file in /run
2711- # that is created by cloud-init-local.
2712- udevadm settle "--timeout=$max" "--exit-if-exists=$fname"
2713-}
2714-
2715-log() {
2716- [ -n "${LOG}" ] || return
2717- [ "${DEBUG:-0}" = "0" ] && return
2718-
2719- if [ $LOG_INIT = 0 ]; then
2720- if [ -d "${LOG%/*}" ] || mkdir -p "${LOG%/*}"; then
2721- LOG_INIT=1
2722- else
2723- echo "${0##*/}: WARN: log init to ${LOG%/*}" 1>&2
2724- return
2725- fi
2726- elif [ "$LOG_INIT" = "-1" ]; then
2727- return
2728- fi
2729- local info="$$ $INTERFACE"
2730- if [ "$DEBUG" -gt 1 ]; then
2731- local up idle
2732- read up idle < /proc/uptime
2733- info="$$ $INTERFACE $up"
2734- fi
2735- echo "[$info]" "$@" >> "$LOG"
2736-}
2737-
2738-main() {
2739- local name="" readyfile="$CI_NET_READY"
2740- local info="INTERFACE=${INTERFACE} ID_NET_NAME=${ID_NET_NAME}"
2741- info="$info ID_NET_NAME_PATH=${ID_NET_NAME_PATH}"
2742- info="$info MAC_ADDRESS=${MAC_ADDRESS}"
2743- log "$info"
2744-
2745- ## Check to see if cloud-init.target is set. If cloud-init is
2746- ## disabled we do not want to do anything.
2747- if [ ! -f "/run/cloud-init/enabled" ]; then
2748- log "cloud-init disabled"
2749- return 0
2750- fi
2751-
2752- if [ "${INTERFACE#lo}" != "$INTERFACE" ]; then
2753- return 0
2754- fi
2755-
2756- block_until_ready "$readyfile" "$MAX_WAIT" ||
2757- { log "failed waiting for ready on $INTERFACE"; return 1; }
2758-
2759- log "net config ready"
2760-}
2761-
2762-main "$@"
2763-exit
2764-
2765-# vi: ts=4 expandtab