Merge lp:~darkmuggle-deactivatedaccount/cloud-init/lp-1233698 into lp:~cloud-init-dev/cloud-init/trunk

Proposed by Ben Howard
Status: Merged
Merged at revision: 883
Proposed branch: lp:~darkmuggle-deactivatedaccount/cloud-init/lp-1233698
Merge into: lp:~cloud-init-dev/cloud-init/trunk
Diff against target: 642 lines (+314/-61)
7 files modified
cloudinit/config/cc_disk_setup.py (+171/-48)
cloudinit/config/cc_mounts.py (+36/-2)
cloudinit/sources/DataSourceAzure.py (+3/-2)
cloudinit/sources/DataSourceSmartOS.py (+11/-2)
cloudinit/util.py (+77/-0)
doc/examples/cloud-config-disk-setup.txt (+16/-5)
tests/unittests/test_datasource/test_azure.py (+0/-2)
To merge this branch: bzr merge lp:~darkmuggle-deactivatedaccount/cloud-init/lp-1233698
Reviewer Review Type Date Requested Status
Scott Moser Pending
Review via email: mp+188647@code.launchpad.net

Description of the change

Fix for bug #1233698 and Windows Azure's default NTFS ephemeral device.

To post a comment you must log in.
Revision history for this message
Scott Moser (smoser) wrote :

There is really no reason for:
 dd_cmd = util.which("dd")
 subp(dd_cmd)

That is identical to:
 subp(dd_cmd)

purge_disk doesn't seem that it should be something that is partition table specific.
The end result is that you have an unpartitioned disk.

A "wipe the disk" method might look like this:
 def wipe_disk(devname):
   for f in $devname[0-9]*:
      [ -b "$f" ] && wipefs $f || :

   dd if=/dev/zero of=$devname
   out=$(dd if=/dev/zero of="$target" bs=1024 \
          seek=$(($size-1024)) count=1024 2>&1)
   blockdev --rereadpt $devname
   udevadm settle

Interestingly, as you've found, an attempt to "wipe" a disk and then partition it, and then mkfs might end up in there being an existing filesystem at any of the new partitions (ie, if the old disk format had partitions there).
So really what you'd need to do is:
 erase partition tables (gpt is at end and beginning, so zero the first MB and last MB)
 format disk
 rereadpt && udevadm settle
 'wipefs' all partitions
 mkfs

Doing that would actually mean we do not have to pass '-F' to mkfs.extX (note, that -F is not common to all'mkfs' programs so 'mkfs.$FSTYPE -F' wont necessarily work).

Probably could also get away with just calling 'wipefs --all $dev' before formatting.

I'm not sure really what i htink about changing the builtin defaults to overwrite filesystems on azure. that seems dangerous. better if we coudl identify this one scenario and wipe it.

Revision history for this message
Scott Moser (smoser) wrote :

for reference:
 $ sudo lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 20G 0 disk
└─sdb1 8:17 0 20G 0 part

$ sudo blkid /dev/sdb1
/dev/sdb1: LABEL="Temporary Storage" UUID="E4227FB4227F89F6" TYPE="ntfs"

Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

Okay, so there are a couple issues....

First, the use of the device mapping is screwy. The argument against mapping ephemeral0 to /dev/vdb1 (i.e. partition) was that a device should be a device, not a parition. The problem is tha the usage of cloud.get_device_name() in cc_mounts assumes that the device contains a file system and not a partition. That means that the meaning of ephemeral0 and swap are constrained to be on a raw devices only....

But that creates a bigger problem. Previously you stated that you don't want to force the creation of a filesystem. On 12.10 and later, mkfs.ext{2,3,4} refuse to build a file system on a block device (only partitions and special block devices are allowed) with out forcing it.

So this latest merge proposal contains a few work arounds:
1. For SmartOS, the creation is forced because it should be the raw device
2. For Windows Azure, ephemeral0 is defined as /dev/sdb1.
3. For Windows Azure, I added the 'replace_fs' as a means to replace a matching filesystem. This leaves the partition table and the other bits intact, while simply replacing the NTFS filesystem with a ext4 filesystem
4. There is code for the removal of a partition table and all dependant filesystems. It works, but is not used right now.

881. By Scott Moser

examples/cloud-config-user-groups.txt doc fix, fix pep8

fix documentation of mkpasswd usage in
doc/examples/cloud-config-user-groups.txt

Also, Precise's version of pep8 insists on ordering of imports, but saucy's
has not. So we had some incorrect ordering. This fixes './tools/run-pep8'
on precise.

882. By Ben Howard

Added ability to define disks via 'ephemeralX.Y'.
Modified cc_mounts to identify whether ephermalX is partitioned.
Changed datasources for Azure and SmartOS to use 'ephemeralX.Y' format.
Added disk remove functionally

883. By Scott Moser

merge from trunk

884. By Scott Moser

fix pep8

885. By Scott Moser

use native python code for wiping partition table

I'm pretty sure the previous code wasn't seeking correctly
and probably writing near the end, but not to the end.

This is simpler and probably faster.

886. By Scott Moser

remove verbosity of log.info

887. By Ben Howard

Moved ephemeralX.Y handling from Datasource into the cc_disk_setup, which makes it cloud agnostic.

888. By Ben Howard

Configure SmartOS Datasource to be region aware

889. By Ben Howard

Wrapped use of 'lsblk' to a generator function; removed other fcalls to 'lsblk'

890. By Ben Howard

Make {pep8,pylint,test} pass commit.

891. By Ben Howard

Added support for 'ephmeral0.<auto|any|0>' for device mappings in disk
formating support.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'cloudinit/config/cc_disk_setup.py'
--- cloudinit/config/cc_disk_setup.py 2013-10-02 13:25:36 +0000
+++ cloudinit/config/cc_disk_setup.py 2013-10-03 23:32:32 +0000
@@ -19,6 +19,7 @@
19from cloudinit.settings import PER_INSTANCE19from cloudinit.settings import PER_INSTANCE
20from cloudinit import util20from cloudinit import util
21import logging21import logging
22import os
22import shlex23import shlex
2324
24frequency = PER_INSTANCE25frequency = PER_INSTANCE
@@ -29,13 +30,13 @@
29LSBLK_CMD = util.which("lsblk")30LSBLK_CMD = util.which("lsblk")
30BLKID_CMD = util.which("blkid")31BLKID_CMD = util.which("blkid")
31BLKDEV_CMD = util.which("blockdev")32BLKDEV_CMD = util.which("blockdev")
33WIPEFS_CMD = util.which("wipefs")
3234
33LOG = logging.getLogger(__name__)35LOG = logging.getLogger(__name__)
3436
3537
36def handle(_name, cfg, cloud, log, _args):38def handle(_name, cfg, cloud, log, _args):
37 """39 """
38 Call util.prep_disk for disk_setup cloud-config.
39 See doc/examples/cloud-config_disk-setup.txt for documentation on the40 See doc/examples/cloud-config_disk-setup.txt for documentation on the
40 format.41 format.
41 """42 """
@@ -93,6 +94,15 @@
93 LOG.debug("updated disk_setup device entry '%s' to '%s'",94 LOG.debug("updated disk_setup device entry '%s' to '%s'",
94 origname, transformed)95 origname, transformed)
9596
97def reset_part_definition(definition, value):
98 if not value and 'partition' in definition:
99 definition['opartition'] = definition['partition']
100 del definition['partition']
101
102 else:
103 definition['partition'] = value
104
105 return definition
96106
97def update_fs_setup_devices(disk_setup, tformer):107def update_fs_setup_devices(disk_setup, tformer):
98 # update 'fs_setup' dictionary anywhere were a device may occur108 # update 'fs_setup' dictionary anywhere were a device may occur
@@ -103,13 +113,40 @@
103 continue113 continue
104114
105 origname = definition.get('device')115 origname = definition.get('device')
116
106 if origname is None:117 if origname is None:
107 continue118 continue
108119
109 transformed = tformer(origname)120 transformed = None
121 if len(origname.split('.')) == 2:
122 # this maps ephemeralX.Y to a proper disk name. For example,
123 # if the origname is 'ephemeral0.1' and transformed is /dev/sdb
124 # then the returned device will be /dev/sdb1 _if_ /dev/sdb1 exists
125 # otherwise NONE
126 base_name, partition = origname.split('.')
127 tformed = tformer(base_name)
128 LOG.info("base device for %s is %s" % (origname, tformed))
129
130 if partition == "0":
131 transformed = tformed
132 definition = reset_part_definition(definition, None)
133
134 elif partition in ("auto", "any"):
135 definition = reset_part_definition(definition, partition)
136 transformed = tformed
137
138 else:
139 definition = reset_part_definition(definition, None)
140 transformed = util.map_device_alias(tformed, alias=origname)
141 LOG.info("%s is mapped to %s" % (origname, transformed))
142
143 else:
144 transformed = tformer(origname)
145
110 if transformed is None or transformed == origname:146 if transformed is None or transformed == origname:
111 continue147 continue
112148
149 LOG.info("Mapped %s to physical device %s" % (origname, transformed))
113 definition['_origname'] = origname150 definition['_origname'] = origname
114 definition['device'] = transformed151 definition['device'] = transformed
115152
@@ -127,23 +164,56 @@
127 yield key, value164 yield key, value
128165
129166
167def enumerate_disk(device, nodeps=False):
168 """
169 Enumerate the elements of a child device.
170
171 Parameters:
172 device: the kernel device name
173 nodeps <BOOL>: don't enumerate children devices
174
175 Return a dict describing the disk:
176 type: the entry type, i.e disk or part
177 fstype: the filesystem type, if it exists
178 label: file system label, if it exists
179 name: the device name, i.e. sda
180 """
181
182 lsblk_cmd = [LSBLK_CMD, '--pairs', '--out', 'NAME,TYPE,FSTYPE,LABEL',
183 device]
184
185 if nodeps:
186 lsblk_cmd.append('--nodeps')
187
188 info = None
189 try:
190 info, _err = util.subp(lsblk_cmd)
191 except Exception as e:
192 raise Exception("Failed during disk check for %s\n%s" % (device, e))
193
194 parts = [x for x in (info.strip()).splitlines() if len(x.split()) > 0]
195
196 for part in parts:
197 d = {'name': None,
198 'type': None,
199 'fstype': None,
200 'label': None,
201 }
202
203 for key, value in value_splitter(part):
204 d[key.lower()] = value
205
206 yield d
207
208
130def device_type(device):209def device_type(device):
131 """210 """
132 Return the device type of the device by calling lsblk.211 Return the device type of the device by calling lsblk.
133 """212 """
134213
135 lsblk_cmd = [LSBLK_CMD, '--pairs', '--nodeps', '--out', 'NAME,TYPE',214 for d in enumerate_disk(device, nodeps=True):
136 device]215 if "type" in d:
137 info = None216 return d["type"].lower()
138 try:
139 info, _err = util.subp(lsblk_cmd)
140 except Exception as e:
141 raise Exception("Failed during disk check for %s\n%s" % (device, e))
142
143 for key, value in value_splitter(info):
144 if key.lower() == "type":
145 return value.lower()
146
147 return None217 return None
148218
149219
@@ -204,7 +274,7 @@
204274
205275
206def find_device_node(device, fs_type=None, label=None, valid_targets=None,276def find_device_node(device, fs_type=None, label=None, valid_targets=None,
207 label_match=True):277 label_match=True, replace_fs=None):
208 """278 """
209 Find a device that is either matches the spec, or the first279 Find a device that is either matches the spec, or the first
210280
@@ -221,26 +291,12 @@
221 if not valid_targets:291 if not valid_targets:
222 valid_targets = ['disk', 'part']292 valid_targets = ['disk', 'part']
223293
224 lsblk_cmd = [LSBLK_CMD, '--pairs', '--out', 'NAME,TYPE,FSTYPE,LABEL',
225 device]
226 info = None
227 try:
228 info, _err = util.subp(lsblk_cmd)
229 except Exception as e:
230 raise Exception("Failed during disk check for %s\n%s" % (device, e))
231
232 raw_device_used = False294 raw_device_used = False
233 parts = [x for x in (info.strip()).splitlines() if len(x.split()) > 0]295 for d in enumerate_disk(device):
234296
235 for part in parts:297 if d['fstype'] == replace_fs and label_match is False:
236 d = {'name': None,298 # We found a device where we want to replace the FS
237 'type': None,299 return ('/dev/%s' % d['name'], False)
238 'fstype': None,
239 'label': None,
240 }
241
242 for key, value in value_splitter(part):
243 d[key.lower()] = value
244300
245 if (d['fstype'] == fs_type and301 if (d['fstype'] == fs_type and
246 ((label_match and d['label'] == label) or not label_match)):302 ((label_match and d['label'] == label) or not label_match)):
@@ -268,22 +324,20 @@
268324
269def is_disk_used(device):325def is_disk_used(device):
270 """326 """
271 Check if the device is currently used. Returns false if there327 Check if the device is currently used. Returns true if the device
328 has either a file system or a partition entry
272 is no filesystem found on the disk.329 is no filesystem found on the disk.
273 """330 """
274 lsblk_cmd = [LSBLK_CMD, '--pairs', '--out', 'NAME,TYPE',331
275 device]332 # If the child count is higher 1, then there are child nodes
276 info = None333 # such as partition or device mapper nodes
277 try:334 use_count = [x for x in enumerate_disk(device)]
278 info, _err = util.subp(lsblk_cmd)335 if len(use_count.splitlines()) > 1:
279 except Exception as e:
280 # if we error out, we can't use the device
281 util.logexc(LOG,
282 "Error checking for filesystem on %s\n%s" % (device, e))
283 return True336 return True
284337
285 # If there is any output, then the device has something338 # If we see a file system, then its used
286 if len(info.splitlines()) > 1:339 _, check_fstype, _ = check_fs(device)
340 if check_fstype:
287 return True341 return True
288342
289 return False343 return False
@@ -455,6 +509,38 @@
455 return sfdisk_definition509 return sfdisk_definition
456510
457511
512def purge_disk_ptable(device):
513 # wipe the first and last megabyte of a disk (or file)
514 # gpt stores partition table both at front and at end.
515 start_len = 1024 * 1024
516 end_len = 1024 * 1024
517 with open(device, "rb+") as fp:
518 fp.write('\0' * (start_len))
519 fp.seek(-end_len, os.SEEK_END)
520 fp.write('\0' * end_len)
521 fp.flush()
522
523 read_parttbl(device)
524
525
526def purge_disk(device):
527 """
528 Remove parition table entries
529 """
530
531 # wipe any file systems first
532 for d in enumerate_disk(device):
533 if d['type'] not in ["disk", "crypt"]:
534 wipefs_cmd = [WIPEFS_CMD, "--all", "/dev/%s" % d['name']]
535 try:
536 LOG.info("Purging filesystem on /dev/%s" % d['name'])
537 util.subp(wipefs_cmd)
538 except Exception:
539 raise Exception("Failed FS purge of /dev/%s" % d['name'])
540
541 purge_disk_ptable(device)
542
543
458def get_partition_layout(table_type, size, layout):544def get_partition_layout(table_type, size, layout):
459 """545 """
460 Call the appropriate function for creating the table546 Call the appropriate function for creating the table
@@ -542,6 +628,12 @@
542 if not is_device_valid(device):628 if not is_device_valid(device):
543 raise Exception("Device %s is not a disk device!", device)629 raise Exception("Device %s is not a disk device!", device)
544630
631 # Remove the partition table entries
632 if isinstance(layout, str) and layout.lower() == "remove":
633 LOG.debug("Instructed to remove partition table entries")
634 purge_disk(device)
635 return
636
545 LOG.debug("Checking if device layout matches")637 LOG.debug("Checking if device layout matches")
546 if check_partition_layout(table_type, device, layout):638 if check_partition_layout(table_type, device, layout):
547 LOG.debug("Device partitioning layout matches")639 LOG.debug("Device partitioning layout matches")
@@ -565,6 +657,26 @@
565 LOG.debug("Partition table created for %s", device)657 LOG.debug("Partition table created for %s", device)
566658
567659
660def lookup_force_flag(fs):
661 """
662 A force flag might be -F or -F, this look it up
663 """
664 flags = {'ext': '-F',
665 'btrfs': '-f',
666 'xfs': '-f',
667 'reiserfs': '-f',
668 }
669
670 if 'ext' in fs.lower():
671 fs = 'ext'
672
673 if fs.lower() in flags:
674 return flags[fs]
675
676 LOG.warn("Force flag for %s is unknown." % fs)
677 return ''
678
679
568def mkfs(fs_cfg):680def mkfs(fs_cfg):
569 """681 """
570 Create a file system on the device.682 Create a file system on the device.
@@ -592,6 +704,7 @@
592 fs_type = fs_cfg.get('filesystem')704 fs_type = fs_cfg.get('filesystem')
593 fs_cmd = fs_cfg.get('cmd', [])705 fs_cmd = fs_cfg.get('cmd', [])
594 fs_opts = fs_cfg.get('extra_opts', [])706 fs_opts = fs_cfg.get('extra_opts', [])
707 fs_replace = fs_cfg.get('replace_fs', False)
595 overwrite = fs_cfg.get('overwrite', False)708 overwrite = fs_cfg.get('overwrite', False)
596709
597 # This allows you to define the default ephemeral or swap710 # This allows you to define the default ephemeral or swap
@@ -632,17 +745,23 @@
632 label_match = False745 label_match = False
633746
634 device, reuse = find_device_node(device, fs_type=fs_type, label=label,747 device, reuse = find_device_node(device, fs_type=fs_type, label=label,
635 label_match=label_match)748 label_match=label_match,
749 replace_fs=fs_replace)
636 LOG.debug("Automatic device for %s identified as %s", odevice, device)750 LOG.debug("Automatic device for %s identified as %s", odevice, device)
637751
638 if reuse:752 if reuse:
639 LOG.debug("Found filesystem match, skipping formating.")753 LOG.debug("Found filesystem match, skipping formating.")
640 return754 return
641755
756 if not reuse and fs_replace and device:
757 LOG.debug("Replacing file system on %s as instructed." % device)
758
642 if not device:759 if not device:
643 LOG.debug("No device aviable that matches request. "760 LOG.debug("No device aviable that matches request. "
644 "Skipping fs creation for %s", fs_cfg)761 "Skipping fs creation for %s", fs_cfg)
645 return762 return
763 elif not partition or str(partition).lower() == 'none':
764 LOG.debug("Using the raw device to place filesystem %s on" % label)
646765
647 else:766 else:
648 LOG.debug("Error in device identification handling.")767 LOG.debug("Error in device identification handling.")
@@ -682,12 +801,16 @@
682 if label:801 if label:
683 fs_cmd.extend(["-L", label])802 fs_cmd.extend(["-L", label])
684803
804 # File systems that support the -F flag
805 if not fs_cmd and (overwrite or device_type(device) == "disk"):
806 fs_cmd.append(lookup_force_flag(fs_type))
807
685 # Add the extends FS options808 # Add the extends FS options
686 if fs_opts:809 if fs_opts:
687 fs_cmd.extend(fs_opts)810 fs_cmd.extend(fs_opts)
688811
689 LOG.debug("Creating file system %s on %s", label, device)812 LOG.debug("Creating file system %s on %s", label, device)
690 LOG.debug(" Using cmd: %s", "".join(fs_cmd))813 LOG.debug(" Using cmd: %s", " ".join(fs_cmd))
691 try:814 try:
692 util.subp(fs_cmd)815 util.subp(fs_cmd)
693 except Exception as e:816 except Exception as e:
694817
=== modified file 'cloudinit/config/cc_mounts.py'
--- cloudinit/config/cc_mounts.py 2013-03-07 21:27:47 +0000
+++ cloudinit/config/cc_mounts.py 2013-10-03 23:32:32 +0000
@@ -20,6 +20,7 @@
2020
21from string import whitespace # pylint: disable=W040221from string import whitespace # pylint: disable=W0402
2222
23import os.path
23import re24import re
2425
25from cloudinit import type_utils26from cloudinit import type_utils
@@ -75,7 +76,9 @@
75 "name from ephemeral to ephemeral0"), (i + 1))76 "name from ephemeral to ephemeral0"), (i + 1))
7677
77 if is_mdname(startname):78 if is_mdname(startname):
78 newname = cloud.device_name_to_device(startname)79 candidate_name = cloud.device_name_to_device(startname)
80 newname = disk_or_part(candidate_name)
81
79 if not newname:82 if not newname:
80 log.debug("Ignoring nonexistant named mount %s", startname)83 log.debug("Ignoring nonexistant named mount %s", startname)
81 cfgmnt[i][1] = None84 cfgmnt[i][1] = None
@@ -119,7 +122,8 @@
119 # entry has the same device name122 # entry has the same device name
120 for defmnt in defmnts:123 for defmnt in defmnts:
121 startname = defmnt[0]124 startname = defmnt[0]
122 devname = cloud.device_name_to_device(startname)125 candidate_name = cloud.device_name_to_device(startname)
126 devname = disk_or_part(candidate_name)
123 if devname is None:127 if devname is None:
124 log.debug("Ignoring nonexistant named default mount %s", startname)128 log.debug("Ignoring nonexistant named default mount %s", startname)
125 continue129 continue
@@ -198,3 +202,33 @@
198 util.subp(("mount", "-a"))202 util.subp(("mount", "-a"))
199 except:203 except:
200 util.logexc(log, "Activating mounts via 'mount -a' failed")204 util.logexc(log, "Activating mounts via 'mount -a' failed")
205
206
207def disk_or_part(device):
208 """
209 Find where the file system is on the disk, either on
210 the disk itself or on the first partition. We don't go
211 any deeper than partition 1 though.
212 """
213
214 if not device:
215 return None
216
217 short_name = device.split('/')[-1]
218 sys_path = "/sys/block/%s" % short_name
219
220 # if the sys path does not exist but the device exists,
221 # then the device is a partition, no sense looking any further
222 if not os.path.exists(sys_path) and os.path.exists(device):
223 return device
224
225 sys_long_path = sys_path + "/" + short_name + "%s"
226 valid_mappings = [sys_long_path % "1",
227 sys_long_path % "p1",
228 sys_path]
229
230 for cdisk in valid_mappings:
231 if not os.path.exists(cdisk):
232 continue
233 return "/dev/%s" % cdisk.split('/')[-1]
234 return None
201235
=== modified file 'cloudinit/sources/DataSourceAzure.py'
--- cloudinit/sources/DataSourceAzure.py 2013-09-27 17:00:35 +0000
+++ cloudinit/sources/DataSourceAzure.py 2013-10-03 23:32:32 +0000
@@ -54,8 +54,9 @@
54 'layout': True,54 'layout': True,
55 'overwrite': False}55 'overwrite': False}
56 },56 },
57 'fs_setup': [{'filesystem': 'ext4', 'device': 'ephemeral0',57 'fs_setup': [{'filesystem': 'ext4',
58 'partition': 'auto'}],58 'device': 'ephemeral0.1',
59 'replace_fs': 'ntfs'}]
59}60}
6061
61DS_CFG_PATH = ['datasource', DS_NAME]62DS_CFG_PATH = ['datasource', DS_NAME]
6263
=== modified file 'cloudinit/sources/DataSourceSmartOS.py'
--- cloudinit/sources/DataSourceSmartOS.py 2013-09-27 17:35:36 +0000
+++ cloudinit/sources/DataSourceSmartOS.py 2013-10-03 23:32:32 +0000
@@ -46,6 +46,7 @@
46 'user-data': ('user-data', False),46 'user-data': ('user-data', False),
47 'iptables_disable': ('iptables_disable', True),47 'iptables_disable': ('iptables_disable', True),
48 'motd_sys_info': ('motd_sys_info', True),48 'motd_sys_info': ('motd_sys_info', True),
49 'availability_zone': ('region', True),
49}50}
5051
51DS_NAME = 'SmartOS'52DS_NAME = 'SmartOS'
@@ -81,8 +82,9 @@
81 'layout': False,82 'layout': False,
82 'overwrite': False}83 'overwrite': False}
83 },84 },
84 'fs_setup': [{'label': 'ephemeral0', 'filesystem': 'ext3',85 'fs_setup': [{'label': 'ephemeral0',
85 'device': 'ephemeral0', 'partition': 'auto'}],86 'filesystem': 'ext3',
87 'device': 'ephemeral0'}],
86}88}
8789
8890
@@ -174,6 +176,13 @@
174 seed_timeout=self.seed_timeout, default=default,176 seed_timeout=self.seed_timeout, default=default,
175 b64=b64)177 b64=b64)
176178
179 @property
180 def availability_zone(self):
181 try:
182 return self.metadata['availability-zone']
183 except KeyError:
184 return None
185
177186
178def get_serial(seed_device, seed_timeout):187def get_serial(seed_device, seed_timeout):
179 """This is replaced in unit testing, allowing us to replace188 """This is replaced in unit testing, allowing us to replace
180189
=== modified file 'cloudinit/util.py'
--- cloudinit/util.py 2013-09-27 23:42:38 +0000
+++ cloudinit/util.py 2013-10-03 23:32:32 +0000
@@ -32,6 +32,7 @@
32import gzip32import gzip
33import hashlib33import hashlib
34import os34import os
35import os.path
35import platform36import platform
36import pwd37import pwd
37import random38import random
@@ -1826,3 +1827,79 @@
1826 except:1827 except:
1827 pass1828 pass
1828 return ret1829 return ret
1830
1831
1832def map_partition(alias):
1833 """
1834 Return partition number for devices like ephemeral0.0 or ephemeral0.1
1835
1836 Parameters:
1837 alaias: the alias, i.e. ephemeral0 or swap0
1838 device: the actual device to markup
1839
1840 Rules:
1841 - anything after a . is a parittion
1842 - device.0 is the same as device
1843 """
1844
1845 if len(alias.split('.')) == 1:
1846 return None
1847
1848 suffix = alias.split('.')[-1]
1849 try:
1850 if int(suffix) == 0:
1851 return None
1852 return int(suffix)
1853 except ValueError:
1854 pass
1855
1856 return None
1857
1858
1859def map_device_alias(device, partition=None, alias=None):
1860 """
1861 Find the name of the partition. While this might seem rather
1862 straight forward, its not since some devices are '<device><partition>'
1863 while others are '<device>p<partition>'. For example, /dev/xvda3 on EC2
1864 will present as /dev/xvda3p1 for the first partition since /dev/xvda3 is
1865 a block device.
1866
1867 The primary use is to map 'ephemeral0.1' in the datasource to a
1868 real device name
1869 """
1870
1871 if not device:
1872 raise Exception("Device cannot be undefined!")
1873
1874 if not partition and not alias:
1875 raise Exception("partition or alias is required")
1876
1877 if alias:
1878 partition = map_partition(alias)
1879
1880 # if the partition doesn't map, return the device
1881 if not partition:
1882 return device
1883
1884 short_name = device.split('/')[-1]
1885 sys_path = "/sys/block/%s" % short_name
1886
1887 if not os.path.exists(sys_path):
1888 return None
1889
1890 sys_long_path = sys_path + "/" + short_name
1891 valid_mappings = [sys_long_path + "%s" % partition,
1892 sys_long_path + "p%s" % partition]
1893
1894 for cdisk in valid_mappings:
1895 if not os.path.exists(cdisk):
1896 continue
1897
1898 dev_path = "/dev/%s" % cdisk.split('/')[-1]
1899 if os.path.exists(dev_path):
1900 return dev_path
1901 else:
1902 LOG.warn("Specificed parition %s does not exist on %s" % (
1903 partition, device))
1904
1905 return None
18291906
=== modified file 'doc/examples/cloud-config-disk-setup.txt'
--- doc/examples/cloud-config-disk-setup.txt 2013-09-27 23:35:20 +0000
+++ doc/examples/cloud-config-disk-setup.txt 2013-10-03 23:32:32 +0000
@@ -30,8 +30,8 @@
30fs_setup:30fs_setup:
31 - label: ephemeral031 - label: ephemeral0
32 filesystem: ext432 filesystem: ext4
33 device: ephemeral033 device: ephemeral0.1
34 partition: auto34 replace_fs: ntfs
3535
3636
37Default disk definitions for SmartOS37Default disk definitions for SmartOS
@@ -47,8 +47,7 @@
47fs_setup:47fs_setup:
48 - label: ephemeral048 - label: ephemeral0
49 filesystem: ext349 filesystem: ext3
50 device: ephemeral050 device: ephemeral0.0
51 partition: auto
5251
53Cavaut for SmartOS: if ephemeral disk is not defined, then the disk will52Cavaut for SmartOS: if ephemeral disk is not defined, then the disk will
54 not be automatically added to the mounts.53 not be automatically added to the mounts.
@@ -171,6 +170,7 @@
171 device: <DEVICE>170 device: <DEVICE>
172 partition: <PART_VALUE>171 partition: <PART_VALUE>
173 overwrite: <OVERWRITE>172 overwrite: <OVERWRITE>
173 replace_fs: <FS_TYPE>
174174
175Where:175Where:
176 <LABEL>: The file system label to be used. If set to None, no label is176 <LABEL>: The file system label to be used. If set to None, no label is
@@ -187,7 +187,13 @@
187 label as 'ephemeralX' otherwise there may be issues with the mounting187 label as 'ephemeralX' otherwise there may be issues with the mounting
188 of the ephemeral storage layer.188 of the ephemeral storage layer.
189189
190 <PART_VALUE>: The valid options are:190 If you define the device as 'ephemeralX.Y' then Y will be interpetted
191 as a partition value. However, ephermalX.0 is the _same_ as ephemeralX.
192
193 <PART_VALUE>:
194 Partition definitions are overwriten if you use the '<DEVICE>.Y' notation.
195
196 The valid options are:
191 "auto|any": tell cloud-init not to care whether there is a partition197 "auto|any": tell cloud-init not to care whether there is a partition
192 or not. Auto will use the first partition that does not contain a198 or not. Auto will use the first partition that does not contain a
193 file system already. In the absence of a partition table, it will199 file system already. In the absence of a partition table, it will
@@ -236,5 +242,10 @@
236242
237 "false": If an existing file system exists, skip the creation.243 "false": If an existing file system exists, skip the creation.
238244
245 <REPLACE_FS>: This is a special directive, used for Windows Azure that
246 instructs cloud-init to replace a file system of <FS_TYPE>. NOTE:
247 unless you define a label, this requires the use of the 'any' partition
248 directive.
249
239Behavior Caveat: The default behavior is to _check_ if the file system exists.250Behavior Caveat: The default behavior is to _check_ if the file system exists.
240 If a file system matches the specification, then the operation is a no-op.251 If a file system matches the specification, then the operation is a no-op.
241252
=== modified file 'tests/unittests/test_datasource/test_azure.py'
--- tests/unittests/test_datasource/test_azure.py 2013-10-02 13:25:36 +0000
+++ tests/unittests/test_datasource/test_azure.py 2013-10-03 23:32:32 +0000
@@ -328,8 +328,6 @@
328 self.assertTrue(ret)328 self.assertTrue(ret)
329 cfg = dsrc.get_config_obj()329 cfg = dsrc.get_config_obj()
330 self.assertTrue(cfg)330 self.assertTrue(cfg)
331 self.assertEquals(dsrc.device_name_to_device("ephemeral0"),
332 "/dev/sdc")
333331
334 def test_userdata_arrives(self):332 def test_userdata_arrives(self):
335 userdata = "This is my user-data"333 userdata = "This is my user-data"