cloud-init

Merge lp:~ifeoktistov/cloud-init/remotedisk-setup into lp:~cloud-init-dev/cloud-init/trunk

  1. remotedisk-setup
  2. Merge into trunk
Proposed by ifeoktistov
Status: Rejected
Rejected by: Scott Moser
Proposed branch: lp:~ifeoktistov/cloud-init/remotedisk-setup
Merge into: lp:~cloud-init-dev/cloud-init/trunk
Diff against target: 828 lines (+803/-0)
4 files modified
cloudinit/config/cc_remotedisk_setup.py (+678/-0)
config/cloud.cfg (+1/-0)
doc/examples/cloud-config-remotedisk-setup.txt (+95/-0)
templates/multipath.conf.tmpl (+29/-0)
To merge this branch: bzr merge lp:~ifeoktistov/cloud-init/remotedisk-setup
Reviewer Review Type Date Requested Status
cloud-init Commiters Pending
Review via email: mp+294055@code.launchpad.net

Description of the change

Added remotedisk-setup config module which provides a simple and uniform way to handle remote disks such as:
       - iSCSI LUN's:
           - configures Open iSCSI initiator;
           - configures device multipath;
           - enables necessary services;
           - attaches iSCSI LUN;
           - discovers multipath device;
           - creates logical volume;
           - creates filesystem;
           - mounts filesystem;
           - configures /etc/fstab
       - Hypervisor disks (OpenStack Cinder volumes, AWS EBS, etc):
           - creates logical volume;
           - creates filesystem;
           - mounts filesystem;
           - configures /etc/fstab
       - NFS shares:
           - mounts NFS share;
           - configures /etc/fstab

The module was extensively tested on RHEL/CentOS 6.7, RHEL/CentOS 7.2, and Ubuntu 14.04 in OpenStack/KVM and AWS.
iSCSI is tested against NetApp cDOT storage.

To post a comment you must log in.
Revision history for this message
Scott Moser (smoser) wrote :

Hi, thanks for the merge proposal. sorry for the long lag in looking at it.
This looks interesting, I'll try to take a more deep look at it next week.

Revision history for this message
Scott Moser (smoser) wrote :

Hello,
Thank you for taking the time to contribute to cloud-init. Cloud-init has moved its revision control system to git. As a result, we are marking all bzr merge proposals as 'rejected'. If you would like to re-submit this proposal for review, please do so by following the current HACKING documentation at http://cloudinit.readthedocs.io/en/latest/topics/hacking.html .

This functionality does look interesting.
Long term, we are likely to utilize curtin's storage configuration to accomplish things like this (http://launchpad.net/curtin)

Unmerged revisions

1217. By ifeoktistov

Added remotedisk_setup config module

1216. By ifeoktistov

Added remotedisk_setup config module and updated relevant configurations

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'cloudinit/config/cc_remotedisk_setup.py'
2--- cloudinit/config/cc_remotedisk_setup.py 1970-01-01 00:00:00 +0000
3+++ cloudinit/config/cc_remotedisk_setup.py 2016-05-06 21:33:46 +0000
4@@ -0,0 +1,678 @@
5+# vi: ts=4 expandtab
6+#
7+# Copyright (C) 2009-2010 Canonical Ltd.
8+# Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
9+#
10+# Author: Igor Feoktistov <Igor.Feoktistov@netapp.com>
11+#
12+# This program is free software: you can redistribute it and/or modify
13+# it under the terms of the GNU General Public License version 3, as
14+# published by the Free Software Foundation.
15+#
16+# This program is distributed in the hope that it will be useful,
17+# but WITHOUT ANY WARRANTY; without even the implied warranty of
18+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
19+# GNU General Public License for more details.
20+#
21+# You should have received a copy of the GNU General Public License
22+# along with this program. If not, see <http://www.gnu.org/licenses/>.
23+
24+import logging
25+import os
26+import time
27+import shlex
28+import fnmatch
29+import subprocess
30+import re
31+from string import whitespace
32+
33+from cloudinit.settings import PER_INSTANCE
34+from cloudinit import type_utils
35+from cloudinit import util
36+from cloudinit import templater
37+
38+frequency = PER_INSTANCE
39+
40+WAIT_4_BLOCKDEV_MAPPING_ITER = 60
41+WAIT_4_BLOCKDEV_MAPPING_SLEEP = 5
42+WAIT_4_BLOCKDEV_DEVICE_ITER = 12
43+WAIT_4_BLOCKDEV_DEVICE_SLEEP = 5
44+
45+LVM_CMD = util.which("lvm")
46+ISCSIADM_CMD = util.which("iscsiadm")
47+MULTIPATH_CMD = util.which("multipath")
48+SYSTEMCTL_CMD = util.which("systemctl")
49+CHKCONFIG_CMD = util.which("chkconfig")
50+SERVICE_CMD = util.which("service")
51+FSTAB_PATH = "/etc/fstab"
52+ISCSI_INITIATOR_PATH = "/etc/iscsi/initiatorname.iscsi"
53+
54+
55+def handle(_name, cfg, cloud, log, _args):
56+ if "remotedisk_setup" not in cfg:
57+ log.debug("Skipping module named %s, no configuration found" % _name)
58+ return
59+ remotedisk_setup = cfg.get("remotedisk_setup")
60+ log.debug("setting up remote disk: %s", str(remotedisk_setup))
61+ for definition in remotedisk_setup:
62+ try:
63+ device = definition.get("device")
64+ if device:
65+ if device.startswith("iscsi"):
66+ handle_iscsi(cfg, cloud, log, definition)
67+ elif device.startswith("nfs"):
68+ handle_nfs(cfg, cloud, log, definition)
69+ elif device.startswith("ebs"):
70+ handle_ebs(cfg, cloud, log, definition)
71+ elif device.startswith("ephemeral"):
72+ handle_ebs(cfg, cloud, log, definition)
73+ else:
74+ if "fs_type" in definition:
75+ fs_type = definition.get("fs_type")
76+ if fs_type == "nfs":
77+ handle_nfs(cfg, cloud, log, definition)
78+ else:
79+ handle_ebs(cfg, cloud, log, definition)
80+ else:
81+ util.logexc(log, "Expexted \"fs_type\" parameter")
82+ else:
83+ util.logexc(log, "Expexted \"device\" parameter")
84+ except Exception as e:
85+ util.logexc(log, "Failed during remote disk operation\n"
86+ "Exception: %s" % e)
87+
88+
89+def handle_iscsi(cfg, cloud, log, definition):
90+ # Handle iSCSI LUN
91+ device = definition.get("device")
92+ try:
93+ (iscsi_host,
94+ iscsi_proto,
95+ iscsi_port,
96+ iscsi_lun,
97+ iscsi_target) = device.split(":", 5)[1:]
98+ except Exception as e:
99+ util.logexc(log,
100+ "handle_iscsi: "
101+ "expected \"device\" attribute in the format: "
102+ "\"iscsi:<iSCSI host>:<protocol>:<port>:<LUN>:"
103+ "<iSCSI target name>\": %s" % e)
104+ return
105+ (hostname, fqdn) = util.get_hostname_fqdn(cfg, cloud)
106+ if "initiator_name" in definition:
107+ initiator_name = definition.get("initiator_name")
108+ else:
109+ initiator_name = "iqn.2005-02.com.open-iscsi:%s" % hostname
110+ util.write_file(ISCSI_INITIATOR_PATH, "InitiatorName=%s" % initiator_name)
111+ multipath_tmpl_fn = cloud.get_template_filename("multipath.conf")
112+ if not multipath_tmpl_fn:
113+ util.logexc(log, "handle_iscsi: template multipath.conf not found")
114+ return
115+ templater.render_to_file(multipath_tmpl_fn, "/etc/multipath.conf", {})
116+ if cloud.distro.osfamily == "redhat":
117+ iscsi_services = ["iscsi", "iscsid"]
118+ multipath_services = ["multipathd"]
119+ elif cloud.distro.osfamily == 'debian':
120+ iscsi_services = ["open-iscsi"]
121+ multipath_services = ["multipath-tools"]
122+ else:
123+ util.logexc(log,
124+ "handle_iscsi: "
125+ "unsupported osfamily \"%s\"" % cloud.distro.osfamily)
126+ return
127+ for service in iscsi_services:
128+ _service_wrapper(cloud, log, service, "enable")
129+ _service_wrapper(cloud, log, service, "restart")
130+ for service in multipath_services:
131+ _service_wrapper(cloud, log, service, "enable")
132+ _service_wrapper(cloud, log, service, "restart")
133+ blockdev = _iscsi_lun_discover(log,
134+ iscsi_host,
135+ iscsi_port,
136+ iscsi_lun,
137+ iscsi_target)
138+ if blockdev:
139+ lvm_group = definition.get("lvm_group")
140+ lvm_volume = definition.get("lvm_volume")
141+ fs_type = definition.get("fs_type")
142+ fs_opts = definition.get("fs_opts")
143+ mount_point = definition.get("mount_point")
144+ mount_opts = definition.get("mount_opts")
145+ if not mount_opts:
146+ mount_opts = 'defaults,_netdev'
147+ else:
148+ if mount_opts.find("_netdev") == -1:
149+ mount_opts = "%s,_netdev" % (mount_opts)
150+ fs_freq = definition.get("fs_freq")
151+ if not fs_freq:
152+ fs_freq = "1"
153+ fs_passno = definition.get("fs_passno")
154+ if not fs_passno:
155+ fs_passno = "2"
156+ if lvm_group and lvm_volume:
157+ for vg_name in _list_vg_names():
158+ if vg_name == lvm_group:
159+ util.logexc(log,
160+ "handle_iscsi: "
161+ "logical volume group '%s' exists already"
162+ % lvm_group)
163+ return
164+ for lv_name in _list_lv_names():
165+ if lv_name == lvm_volume:
166+ util.logexc(log,
167+ "handle_iscsi: "
168+ "logical volume '%s' exists already"
169+ % lvm_volume)
170+ return
171+ blockdev = _create_lv(log, blockdev, lvm_group, lvm_volume)
172+ if blockdev:
173+ if mount_point and fs_type:
174+ _create_fs(log, blockdev, fs_type, fs_opts)
175+ _add_fstab_entry(log,
176+ blockdev,
177+ mount_point,
178+ fs_type,
179+ mount_opts,
180+ fs_freq,
181+ fs_passno)
182+ _mount_fs(log, mount_point)
183+ else:
184+ util.logexc(log,
185+ "handle_iscsi: "
186+ "expexted \"mount_point\" "
187+ "and \"fs_type\" parameters")
188+
189+
190+def handle_nfs(cfg, cloud, log, definition):
191+ # Handle NFS share mounts
192+ device = definition.get("device")
193+ if device.startswith("nfs"):
194+ (proto, share_path) = device.split(":", 1)
195+ else:
196+ share_path = device
197+ fs_type = definition.get("fs_type")
198+ mount_point = definition.get("mount_point")
199+ mount_opts = definition.get("mount_opts")
200+ if not mount_opts:
201+ mount_opts = "defaults"
202+ fs_freq = definition.get("fs_freq")
203+ if not fs_freq:
204+ fs_freq = "0"
205+ fs_passno = definition.get("fs_passno")
206+ if not fs_passno:
207+ fs_passno = "0"
208+ if mount_point and fs_type:
209+ _add_fstab_entry(log,
210+ share_path,
211+ mount_point,
212+ fs_type,
213+ mount_opts,
214+ fs_freq,
215+ fs_passno)
216+ _mount_fs(log, mount_point)
217+ else:
218+ util.logexc(log,
219+ "handle_nfs: "
220+ "expexted \"mount_point\" and \"fs_type\" parameters")
221+
222+
223+def handle_ebs(cfg, cloud, log, definition):
224+ # Handle block device either explicitly provided via device path or
225+ # via device name mapping (Amazon/OpenStack)
226+ device = definition.get("device")
227+ blockdev = _cloud_device_2_os_device(cloud, log, device)
228+ if blockdev:
229+ lvm_group = definition.get("lvm_group")
230+ lvm_volume = definition.get("lvm_volume")
231+ fs_type = definition.get("fs_type")
232+ fs_opts = definition.get("fs_opts")
233+ mount_point = definition.get("mount_point")
234+ mount_opts = definition.get("mount_opts")
235+ if not mount_opts:
236+ mount_opts = "defaults"
237+ fs_freq = definition.get("fs_freq")
238+ if not fs_freq:
239+ fs_freq = "1"
240+ fs_passno = definition.get("fs_passno")
241+ if not fs_passno:
242+ fs_passno = "2"
243+ if lvm_group and lvm_volume:
244+ for vg_name in _list_vg_names():
245+ if vg_name == lvm_group:
246+ util.logexc(log,
247+ "handle_ebs: "
248+ "logical volume group '%s' exists already"
249+ % lvm_group)
250+ return
251+ for lv_name in _list_lv_names():
252+ if lv_name == lvm_volume:
253+ util.logexc(log,
254+ "handle_ebs: "
255+ "logical volume '%s' exists already"
256+ % lvm_volume)
257+ return
258+ blockdev = _create_lv(log, blockdev, lvm_group, lvm_volume)
259+ if blockdev:
260+ if mount_point and fs_type:
261+ _create_fs(log, blockdev, fs_type, fs_opts)
262+ _add_fstab_entry(log,
263+ blockdev,
264+ mount_point,
265+ fs_type,
266+ mount_opts,
267+ fs_freq,
268+ fs_passno)
269+ _mount_fs(log, mount_point)
270+ else:
271+ util.logexc(log,
272+ "handle_ebs: "
273+ "expexted \"mount_point\" and "
274+ "\"fs_type\" parameters")
275+
276+
277+def _cloud_device_2_os_device(cloud, log, name):
278+ # Translate cloud device (ebs# and ephemaral#) to OS block device path
279+ blockdev = None
280+ for i in range(WAIT_4_BLOCKDEV_MAPPING_ITER):
281+ if (cloud.datasource.metadata and
282+ "block-device-mapping" in cloud.datasource.metadata):
283+ metadata = cloud.datasource.metadata
284+ else:
285+ if (cloud.datasource.ec2_metadata and
286+ "block-device-mapping" in cloud.datasource.ec2_metadata):
287+ metadata = cloud.datasource.ec2_metadata
288+ else:
289+ util.logexc(log,
290+ "_cloud_device_2_os_device: "
291+ "metadata item block-device-mapping not found")
292+ return None
293+ blockdev_items = metadata["block-device-mapping"].iteritems()
294+ for (map_name, device) in blockdev_items:
295+ if map_name == name:
296+ blockdev = device
297+ break
298+ if blockdev is None:
299+ cloud.datasource.get_data()
300+ time.sleep(WAIT_4_BLOCKDEV_MAPPING_SLEEP)
301+ continue
302+ if blockdev is None:
303+ util.logexc(log,
304+ "_cloud_device_2_os_device: "
305+ "unable to convert %s to a device" % name)
306+ return None
307+ if not blockdev.startswith("/"):
308+ blockdev_path = "/dev/%s" % blockdev
309+ else:
310+ blockdev_path = blockdev
311+ for i in range(WAIT_4_BLOCKDEV_DEVICE_ITER):
312+ if os.path.exists(blockdev_path):
313+ return blockdev_path
314+ time.sleep(WAIT_4_BLOCKDEV_DEVICE_SLEEP)
315+ util.logexc(log,
316+ "_cloud_device_2_os_device: "
317+ "device %s does not exist" % blockdev_path)
318+ return None
319+
320+
321+def _list_vg_names():
322+ # List all LVM volume groups
323+ p = subprocess.Popen([LVM_CMD, "vgs", "-o", "vg_name"],
324+ stdout=subprocess.PIPE, stderr=subprocess.PIPE)
325+ err = p.wait()
326+ if err:
327+ return []
328+ output = p.communicate()[0]
329+ output = output.split("\n")
330+ if not output:
331+ return []
332+ header = output[0].strip()
333+ if header != "VG":
334+ return []
335+ names = []
336+ for name in output[1:]:
337+ if not name:
338+ break
339+ names.append(name.strip())
340+ return names
341+
342+
343+def _list_lv_names():
344+ # List all LVM logical volumes
345+ p = subprocess.Popen([LVM_CMD, "lvs", "-o", "lv_name"],
346+ stdout=subprocess.PIPE, stderr=subprocess.PIPE)
347+ err = p.wait()
348+ if err:
349+ return []
350+ output = p.communicate()[0]
351+ output = output.split("\n")
352+ if not output:
353+ return []
354+ header = output[0].strip()
355+ if header != "LV":
356+ return []
357+ names = []
358+ for name in output[1:]:
359+ if not name:
360+ break
361+ names.append(name.strip())
362+ return names
363+
364+
365+def _create_lv(log, device, vg_name, lv_name):
366+ # Create volume group
367+ pvcreate_cmd = [LVM_CMD, "pvcreate", device]
368+ vgcreate_cmd = [LVM_CMD, "vgcreate", vg_name, device]
369+ lvcreate_cmd = [LVM_CMD,
370+ "lvcreate", "-l", "100%FREE", "--name", lv_name, vg_name]
371+ try:
372+ util.subp(pvcreate_cmd)
373+ util.subp(vgcreate_cmd)
374+ util.subp(lvcreate_cmd)
375+ return "/dev/mapper/%s-%s" % (vg_name, lv_name)
376+ except Exception as e:
377+ util.logexc(log,
378+ "_create_lv: "
379+ "failed to create LVM volume '%s' for device '%s': %s"
380+ % (lv_name, device, e))
381+ return None
382+
383+
384+def _create_fs(log, device, fs_type, fs_opts=None):
385+ # Create filesystem
386+ mkfs_cmd = util.which("mkfs.%s" % fs_type)
387+ if not mkfs_cmd:
388+ mkfs_cmd = util.which("mk%s" % fs_type)
389+ if not mkfs_cmd:
390+ util.logexc(log,
391+ "_create_fs: "
392+ "cannot create filesystem type '%s': "
393+ "failed to find mkfs.%s command" % (fs_type, fs_type))
394+ return
395+ try:
396+ if fs_opts:
397+ util.subp([mkfs_cmd, fs_opts, device])
398+ else:
399+ util.subp([mkfs_cmd, device])
400+ except Exception as e:
401+ util.logexc(log,
402+ "_create_fs: "
403+ "failed to create filesystem type '%s': %s" % (fs_type, e))
404+
405+
406+def _add_fstab_entry(log,
407+ device,
408+ mount_point,
409+ fs_type,
410+ mount_opts,
411+ fs_freq,
412+ fs_passno):
413+ # Create fstab entry
414+ fstab_lines = []
415+ for line in util.load_file(FSTAB_PATH).splitlines():
416+ try:
417+ toks = re.compile("[%s]+" % (whitespace)).split(line)
418+ except:
419+ pass
420+ if len(toks) > 0 and toks[0] == device:
421+ util.logexc(log,
422+ "_add_fstab_entry: "
423+ "file %s has device %s already" % (FSTAB_PATH, device))
424+ return
425+ if len(toks) > 1 and toks[1] == mount_point:
426+ util.logexc(log,
427+ "_add_fstab_entry: "
428+ "file %s has mount point %s already"
429+ % (FSTAB_PATH, mount_point))
430+ return
431+ fstab_lines.append(line)
432+ fstab_lines.extend(["%s\t%s\t%s\t%s\t%s\t%s" %
433+ (device,
434+ mount_point,
435+ fs_type,
436+ mount_opts,
437+ fs_freq,
438+ fs_passno)])
439+ contents = "%s\n" % ('\n'.join(fstab_lines))
440+ util.write_file(FSTAB_PATH, contents)
441+
442+
443+def _mount_fs(log, mount_point):
444+ # Mount filesystem according to fstab entry
445+ try:
446+ util.ensure_dir(mount_point)
447+ except Exception as e:
448+ util.logexc(log,
449+ "_mount_fs: "
450+ "failed to make '%s' mount point directory: %s"
451+ % (mount_point, e))
452+ return
453+ try:
454+ util.subp(["mount", mount_point])
455+ except Exception as e:
456+ util.logexc(log,
457+ "_mount_fs: "
458+ "activating mounts via 'mount %s' failed: %s"
459+ % (mount_point, e))
460+
461+
462+def _service_wrapper(cloud, log, service, command):
463+ # Wrapper for service related commands
464+ if cloud.distro.osfamily == "redhat":
465+ if SYSTEMCTL_CMD:
466+ svc_cmd = [SYSTEMCTL_CMD, command, service]
467+ else:
468+ if command == "enable" or command == "disable":
469+ if CHKCONFIG_CMD:
470+ if command == "enable":
471+ svc_cmd = [CHKCONFIG_CMD, service, "on"]
472+ else:
473+ svc_cmd = [CHKCONFIG_CMD, service, "off"]
474+ else:
475+ util.logexc(log,
476+ "_handle_service: "
477+ "service config command \"chkconfig\" "
478+ "not found")
479+ return
480+ else:
481+ svc_cmd = [SERVICE_CMD, service, command]
482+ elif cloud.distro.osfamily == "debian":
483+ if SYSTEMCTL_CMD:
484+ svc_cmd = [SYSTEMCTL_CMD, command, service]
485+ else:
486+ if command == 'enable' or command == "disable":
487+ if os.path.exists('/usr/sbin/update-rc.d'):
488+ svc_cmd = ['/usr/sbin/update-rc.d', service, "defaults"]
489+ else:
490+ util.logexc(log,
491+ "_handle_service: "
492+ "command \"/usr/sbin/update-rc.d\" not found")
493+ return
494+ else:
495+ svc_cmd = [SERVICE_CMD, service, command]
496+ else:
497+ util.logexc(log,
498+ "_handle_service: "
499+ "unsupported osfamily \"%s\"" % cloud.distro.osfamily)
500+ return
501+ try:
502+ util.subp(svc_cmd, capture=False)
503+ except Exception as e:
504+ util.logexc(log,
505+ "_handle_service: "
506+ "failure to \"%s\" \"%s\": %s" % (command, service, e))
507+
508+
509+def _iscsi_lun_discover(log, iscsi_host, iscsi_port, iscsi_lun, iscsi_target):
510+ # Discover iSCSI target and map LUN ID to multipath device path
511+ blockdev = None
512+ for i in range(WAIT_4_BLOCKDEV_MAPPING_ITER):
513+ try:
514+ util.subp([ISCSIADM_CMD,
515+ "--mode",
516+ "discoverydb",
517+ "--type",
518+ "sendtargets",
519+ "--portal",
520+ "%s:%s" % (iscsi_host, iscsi_port),
521+ "--discover",
522+ "--login",
523+ "all"],
524+ capture=False)
525+ except Exception as e:
526+ util.logexc(log,
527+ "_iscsi_lun_discover: "
528+ "failure in attempt to discover iSCSI LUN for target "
529+ "\"%s\": %s" % (iscsi_target, e))
530+ return None
531+ p = subprocess.Popen([ISCSIADM_CMD, "-m", "node"],
532+ stdout=subprocess.PIPE,
533+ stderr=subprocess.PIPE)
534+ err = p.wait()
535+ if err:
536+ util.logexc(log,
537+ "_iscsi_lun_discover: "
538+ "failure from \"%s -m node\" command" % ISCSIADM_CMD)
539+ return None
540+ output = p.communicate()[0]
541+ output = output.split("\n")
542+ if not output:
543+ util.logexc(log,
544+ "_iscsi_lun_discover: "
545+ "no iSCSI nodes discovered for target \"%s\""
546+ % iscsi_target)
547+ time.sleep(WAIT_4_BLOCKDEV_MAPPING_SLEEP)
548+ continue
549+ for node in output:
550+ iscsi_portal = node.split(",", 1)[0]
551+ if iscsi_portal:
552+ try:
553+ util.subp([ISCSIADM_CMD,
554+ "-m",
555+ "node",
556+ "-T",
557+ iscsi_target,
558+ "-p",
559+ iscsi_portal,
560+ "--op",
561+ "update",
562+ "-n",
563+ "node.startup",
564+ "-v",
565+ "automatic"],
566+ capture=False)
567+ except Exception as e:
568+ util.logexc(log,
569+ "_iscsi_lun_discover: "
570+ "failure in attempt to set automatic binding "
571+ "for target portal \"%s\": %s"
572+ % (iscsi_portal, e))
573+ return None
574+ p = subprocess.Popen([ISCSIADM_CMD, "-m", "session", "-P3"],
575+ stdout=subprocess.PIPE,
576+ stderr=subprocess.PIPE)
577+ err = p.wait()
578+ if err:
579+ util.logexc(log,
580+ "_iscsi_lun_discover: "
581+ "failure from \"%s -m session -P3\" command"
582+ % ISCSIADM_CMD)
583+ return None
584+ output = p.communicate()[0]
585+ output = output.split("\n")
586+ if not output:
587+ util.logexc(log,
588+ "_iscsi_lun_discover: "
589+ "no iSCSI sessions discovered for target \"%s\""
590+ % iscsi_target)
591+ else:
592+ current_iscsi_target = None
593+ current_iscsi_sid = None
594+ current_iscsi_lun = None
595+ for line in output:
596+ m = re.search("^Target: ([a-z0-9\.:-]*)", line)
597+ if m:
598+ current_iscsi_target = m.group(1)
599+ continue
600+ else:
601+ if (current_iscsi_target and
602+ current_iscsi_target == iscsi_target):
603+ m = re.search("SID: ([0-9]*)", line)
604+ if m:
605+ if current_iscsi_sid and not current_iscsi_lun:
606+ try:
607+ util.subp([ISCSIADM_CMD,
608+ "-m",
609+ "session",
610+ "-r",
611+ current_iscsi_sid,
612+ "-u"],
613+ capture=False)
614+ except:
615+ pass
616+ current_iscsi_sid = m.group(1)
617+ current_iscsi_lun = None
618+ continue
619+ m = re.search("scsi[0-9]* Channel [0-9]* "
620+ "Id [0-9]* Lun: ([0-9]*)", line)
621+ if m:
622+ current_iscsi_lun = m.group(1)
623+ continue
624+ if (current_iscsi_lun and
625+ current_iscsi_lun == iscsi_lun):
626+ m = re.search("Attached scsi disk (sd[a-z]*)",
627+ line)
628+ if m:
629+ attached_scsi_disk = m.group(1)
630+ p = subprocess.Popen(["/lib/udev/scsi_id",
631+ "-g", "-u", "-d",
632+ "/dev/%s"
633+ % attached_scsi_disk],
634+ stdout=subprocess.PIPE,
635+ stderr=subprocess.PIPE)
636+ err = p.wait()
637+ if err:
638+ util.logexc(log,
639+ "_iscsi_lun_discover: "
640+ "failure from "
641+ "\"/lib/udev/scsi_id\" "
642+ "command")
643+ return None
644+ output2 = p.communicate()[0]
645+ output2 = output2.split('\n')
646+ if not output2:
647+ util.logexc(log,
648+ "_iscsi_lun_discover: "
649+ "no wwid returned for device "
650+ "\"/dev/%s\""
651+ % attached_scsi_disk)
652+ else:
653+ blockdev = "/dev/mapper/%s" % output2[0]
654+ if current_iscsi_sid and not current_iscsi_lun:
655+ try:
656+ util.subp([ISCSIADM_CMD,
657+ "-m",
658+ "session",
659+ "-r",
660+ current_iscsi_sid,
661+ "-u"],
662+ capture=False)
663+ except:
664+ pass
665+ if blockdev:
666+ break
667+ else:
668+ time.sleep(WAIT_4_BLOCKDEV_MAPPING_SLEEP)
669+ if blockdev:
670+ for i in range(WAIT_4_BLOCKDEV_DEVICE_ITER):
671+ if os.path.exists(blockdev):
672+ return blockdev
673+ try:
674+ util.subp([MULTIPATH_CMD], capture=False)
675+ except Exception as e:
676+ util.logexc(log,
677+ "_iscsi_lun_discover: "
678+ "failure to run \"%s\": %s" % (MULTIPATH_CMD, e))
679+ return None
680+ time.sleep(WAIT_4_BLOCKDEV_DEVICE_SLEEP)
681+ else:
682+ return None
683
684=== modified file 'config/cloud.cfg'
685--- config/cloud.cfg 2016-03-09 22:34:11 +0000
686+++ config/cloud.cfg 2016-05-06 21:33:46 +0000
687@@ -44,6 +44,7 @@
688 # this can be used by upstart jobs for 'start on cloud-config'.
689 - emit_upstart
690 - disk_setup
691+ - remotedisk_setup
692 - mounts
693 - ssh-import-id
694 - locale
695
696=== added file 'doc/examples/cloud-config-remotedisk-setup.txt'
697--- doc/examples/cloud-config-remotedisk-setup.txt 1970-01-01 00:00:00 +0000
698+++ doc/examples/cloud-config-remotedisk-setup.txt 2016-05-06 21:33:46 +0000
699@@ -0,0 +1,95 @@
700+#cloud-config
701+#
702+# The module remotedisk_setup provides a simple and uniform way
703+# to handle remote disks such as:
704+# - iSCSI LUN's:
705+# - configures Open iSCSI initiator;
706+# - configures device multipath;
707+# - enables necessary services;
708+# - attaches iSCSI LUN;
709+# - discovers multipath device;
710+# - creates logical volume;
711+# - creates filesystem;
712+# - mounts filesystem;
713+# - configures /etc/fstab
714+# - Hypervisor disks (OpenStack Cinder volumes, AWS EBS, etc):
715+# - creates logical volume;
716+# - creates filesystem;
717+# - mounts filesystem;
718+# - configures /etc/fstab
719+# - NFS shares:
720+# - mounts NFS share;
721+# - configures /etc/fstab
722+#
723+remotedisk_setup:
724+
725+############################################
726+# Example configuration to handle iSCSI LUN:
727+############################################
728+ - device: 'iscsi:192.168.1.1:6:3260:1:iqn.1992-08.com.netapp:sn.62546b567fbf11e4811590e2ba6cc3b4:vs.10'
729+ lvm_group: 'vg_data1'
730+ lvm_volume: 'lv_data1'
731+ fs_type: 'ext4'
732+ mount_point: '/apps/data1'
733+
734+############################################
735+# Parameters:
736+# mandatory:
737+# device: 'iscsi:<iSCSI target host/LIF>:<transport protocol>:<port>:<LUN ID>:<iSCSI target name>'
738+# fs_type: '<filesystem type>'
739+# mount_point: '<mount point dir path>'
740+# optional:
741+# initiator_name: '<iSCSI initiator name, default is iqn.2005-02.com.open-iscsi:<hostname>>'
742+# mount_opts: '<filesystem mount options, default is "defaults,_netdev">'
743+# lvm_group: '<LVM logical group name>'
744+# lvm_volume: '<LVM logical volume name>'
745+# fs_opts: '<filesystem create options specific to mkfs.fs_type>'
746+# fs_freq: '<fstab fs freq, default is "1">'
747+# fs_passno: '<fstab fs passno, default is "2">'
748+# notes:
749+# missing lvm_group and lvm_volume will cause filesystem creation on top of multipath device
750+#
751+
752+##########################################################
753+# Example configuration to handle OpenStack Cinder volume:
754+##########################################################
755+ - device: 'ebs0'
756+ lvm_group: 'vg_data1'
757+ lvm_volume: 'lv_data1'
758+ fs_type: 'ext4'
759+ mount_point: '/apps/data1'
760+
761+##########################################################
762+# Parameters:
763+# mandatory:
764+# device: 'ebs<0-9> or block device path /dev/vd<b-z>'
765+# fs_type: '<filesystem type>'
766+# mount_point: '<mount point dir path>'
767+# optional:
768+# mount_opts: '<filesystem mount options, default is "defaults">'
769+# lvm_group: '<LVM logical group name>'
770+# lvm_volume: '<LVM logical volume name>'
771+# fs_opts: '<filesystem create options specific to mkfs.fs_type>'
772+# fs_freq: '<fstab fs freq, default is "1">'
773+# fs_passno: '<fstab fs passno, default is "2">'
774+# notes:
775+# missing lvm_group and lvm_volume will cause filesystem creation on top of block device
776+#
777+
778+#############################################
779+# Example configuration to handle NFS shares:
780+#############################################
781+ - device: 'nfs:192.168.1.1:/myshare'
782+ mount_point: '/apps/data1'
783+ mount_opts: 'tcp,rw,rsize=65536,wsize=65536'
784+
785+#############################################
786+# Parameters:
787+# mandatory:
788+# device: 'nfs:<NFS host>:<NFS share path>'
789+# mount_point: '<mount point dir path>'
790+# optional:
791+# mount_opts: '<NFS share mount options, default is "defaults">'
792+# fs_type: 'nfs'
793+# fs_freq: '<fstab fs freq, default is "0">'
794+# fs_passno: '<fstab fs passno, default is "0">'
795
796=== added file 'templates/multipath.conf.tmpl'
797--- templates/multipath.conf.tmpl 1970-01-01 00:00:00 +0000
798+++ templates/multipath.conf.tmpl 2016-05-06 21:33:46 +0000
799@@ -0,0 +1,29 @@
800+defaults {
801+ find_multipaths yes
802+ user_friendly_names no
803+ no_path_retry queue
804+ queue_without_daemon no
805+ flush_on_last_del yes
806+ max_fds max
807+}
808+blacklist {
809+ devnode "^hd[a-z]"
810+ devnode "^vd[a-z]"
811+ devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
812+ devnode "^cciss.*"
813+}
814+devices {
815+ device {
816+ vendor "NETAPP"
817+ product "LUN"
818+ path_grouping_policy group_by_prio
819+ features "3 queue_if_no_path pg_init_retries 50"
820+ prio "alua"
821+ path_checker tur
822+ failback immediate
823+ path_selector "round-robin 0"
824+ hardware_handler "1 alua"
825+ rr_weight uniform
826+ rr_min_io 128
827+ }
828+}