Merge lp:~darkmuggle-deactivatedaccount/cloud-init/lp-1233698 into lp:~cloud-init-dev/cloud-init/trunk
- lp-1233698
- Merge into trunk
Status: | Merged |
---|---|
Merged at revision: | 883 |
Proposed branch: | lp:~darkmuggle-deactivatedaccount/cloud-init/lp-1233698 |
Merge into: | lp:~cloud-init-dev/cloud-init/trunk |
Diff against target: |
642 lines (+314/-61) 7 files modified
cloudinit/config/cc_disk_setup.py (+171/-48) cloudinit/config/cc_mounts.py (+36/-2) cloudinit/sources/DataSourceAzure.py (+3/-2) cloudinit/sources/DataSourceSmartOS.py (+11/-2) cloudinit/util.py (+77/-0) doc/examples/cloud-config-disk-setup.txt (+16/-5) tests/unittests/test_datasource/test_azure.py (+0/-2) |
To merge this branch: | bzr merge lp:~darkmuggle-deactivatedaccount/cloud-init/lp-1233698 |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Scott Moser | Pending | ||
Review via email: mp+188647@code.launchpad.net |
Commit message
Description of the change
Fix for bug #1233698 and Windows Azure's default NTFS ephemeral device.
Scott Moser (smoser) wrote : | # |
Scott Moser (smoser) wrote : | # |
for reference:
$ sudo lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 20G 0 disk
└─sdb1 8:17 0 20G 0 part
$ sudo blkid /dev/sdb1
/dev/sdb1: LABEL="Temporary Storage" UUID="E4227FB42
Ben Howard (darkmuggle-deactivatedaccount) wrote : | # |
Okay, so there are a couple issues....
First, the use of the device mapping is screwy. The argument against mapping ephemeral0 to /dev/vdb1 (i.e. partition) was that a device should be a device, not a parition. The problem is tha the usage of cloud.get_
But that creates a bigger problem. Previously you stated that you don't want to force the creation of a filesystem. On 12.10 and later, mkfs.ext{2,3,4} refuse to build a file system on a block device (only partitions and special block devices are allowed) with out forcing it.
So this latest merge proposal contains a few work arounds:
1. For SmartOS, the creation is forced because it should be the raw device
2. For Windows Azure, ephemeral0 is defined as /dev/sdb1.
3. For Windows Azure, I added the 'replace_fs' as a means to replace a matching filesystem. This leaves the partition table and the other bits intact, while simply replacing the NTFS filesystem with a ext4 filesystem
4. There is code for the removal of a partition table and all dependant filesystems. It works, but is not used right now.
- 881. By Scott Moser
-
examples/
cloud-config- user-groups. txt doc fix, fix pep8 fix documentation of mkpasswd usage in
doc/examples/cloud-config- user-groups. txt Also, Precise's version of pep8 insists on ordering of imports, but saucy's
has not. So we had some incorrect ordering. This fixes './tools/run-pep8'
on precise. - 882. By Ben Howard
-
Added ability to define disks via 'ephemeralX.Y'.
Modified cc_mounts to identify whether ephermalX is partitioned.
Changed datasources for Azure and SmartOS to use 'ephemeralX.Y' format.
Added disk remove functionally - 883. By Scott Moser
-
merge from trunk
- 884. By Scott Moser
-
fix pep8
- 885. By Scott Moser
-
use native python code for wiping partition table
I'm pretty sure the previous code wasn't seeking correctly
and probably writing near the end, but not to the end.This is simpler and probably faster.
- 886. By Scott Moser
-
remove verbosity of log.info
- 887. By Ben Howard
-
Moved ephemeralX.Y handling from Datasource into the cc_disk_setup, which makes it cloud agnostic.
- 888. By Ben Howard
-
Configure SmartOS Datasource to be region aware
- 889. By Ben Howard
-
Wrapped use of 'lsblk' to a generator function; removed other fcalls to 'lsblk'
- 890. By Ben Howard
-
Make {pep8,pylint,test} pass commit.
- 891. By Ben Howard
-
Added support for 'ephmeral0.
<auto|any| 0>' for device mappings in disk
formating support.
Preview Diff
1 | === modified file 'cloudinit/config/cc_disk_setup.py' |
2 | --- cloudinit/config/cc_disk_setup.py 2013-10-02 13:25:36 +0000 |
3 | +++ cloudinit/config/cc_disk_setup.py 2013-10-03 23:32:32 +0000 |
4 | @@ -19,6 +19,7 @@ |
5 | from cloudinit.settings import PER_INSTANCE |
6 | from cloudinit import util |
7 | import logging |
8 | +import os |
9 | import shlex |
10 | |
11 | frequency = PER_INSTANCE |
12 | @@ -29,13 +30,13 @@ |
13 | LSBLK_CMD = util.which("lsblk") |
14 | BLKID_CMD = util.which("blkid") |
15 | BLKDEV_CMD = util.which("blockdev") |
16 | +WIPEFS_CMD = util.which("wipefs") |
17 | |
18 | LOG = logging.getLogger(__name__) |
19 | |
20 | |
21 | def handle(_name, cfg, cloud, log, _args): |
22 | """ |
23 | - Call util.prep_disk for disk_setup cloud-config. |
24 | See doc/examples/cloud-config_disk-setup.txt for documentation on the |
25 | format. |
26 | """ |
27 | @@ -93,6 +94,15 @@ |
28 | LOG.debug("updated disk_setup device entry '%s' to '%s'", |
29 | origname, transformed) |
30 | |
31 | +def reset_part_definition(definition, value): |
32 | + if not value and 'partition' in definition: |
33 | + definition['opartition'] = definition['partition'] |
34 | + del definition['partition'] |
35 | + |
36 | + else: |
37 | + definition['partition'] = value |
38 | + |
39 | + return definition |
40 | |
41 | def update_fs_setup_devices(disk_setup, tformer): |
42 | # update 'fs_setup' dictionary anywhere were a device may occur |
43 | @@ -103,13 +113,40 @@ |
44 | continue |
45 | |
46 | origname = definition.get('device') |
47 | + |
48 | if origname is None: |
49 | continue |
50 | |
51 | - transformed = tformer(origname) |
52 | + transformed = None |
53 | + if len(origname.split('.')) == 2: |
54 | + # this maps ephemeralX.Y to a proper disk name. For example, |
55 | + # if the origname is 'ephemeral0.1' and transformed is /dev/sdb |
56 | + # then the returned device will be /dev/sdb1 _if_ /dev/sdb1 exists |
57 | + # otherwise NONE |
58 | + base_name, partition = origname.split('.') |
59 | + tformed = tformer(base_name) |
60 | + LOG.info("base device for %s is %s" % (origname, tformed)) |
61 | + |
62 | + if partition == "0": |
63 | + transformed = tformed |
64 | + definition = reset_part_definition(definition, None) |
65 | + |
66 | + elif partition in ("auto", "any"): |
67 | + definition = reset_part_definition(definition, partition) |
68 | + transformed = tformed |
69 | + |
70 | + else: |
71 | + definition = reset_part_definition(definition, None) |
72 | + transformed = util.map_device_alias(tformed, alias=origname) |
73 | + LOG.info("%s is mapped to %s" % (origname, transformed)) |
74 | + |
75 | + else: |
76 | + transformed = tformer(origname) |
77 | + |
78 | if transformed is None or transformed == origname: |
79 | continue |
80 | |
81 | + LOG.info("Mapped %s to physical device %s" % (origname, transformed)) |
82 | definition['_origname'] = origname |
83 | definition['device'] = transformed |
84 | |
85 | @@ -127,23 +164,56 @@ |
86 | yield key, value |
87 | |
88 | |
89 | +def enumerate_disk(device, nodeps=False): |
90 | + """ |
91 | + Enumerate the elements of a child device. |
92 | + |
93 | + Parameters: |
94 | + device: the kernel device name |
95 | + nodeps <BOOL>: don't enumerate children devices |
96 | + |
97 | + Return a dict describing the disk: |
98 | + type: the entry type, i.e disk or part |
99 | + fstype: the filesystem type, if it exists |
100 | + label: file system label, if it exists |
101 | + name: the device name, i.e. sda |
102 | + """ |
103 | + |
104 | + lsblk_cmd = [LSBLK_CMD, '--pairs', '--out', 'NAME,TYPE,FSTYPE,LABEL', |
105 | + device] |
106 | + |
107 | + if nodeps: |
108 | + lsblk_cmd.append('--nodeps') |
109 | + |
110 | + info = None |
111 | + try: |
112 | + info, _err = util.subp(lsblk_cmd) |
113 | + except Exception as e: |
114 | + raise Exception("Failed during disk check for %s\n%s" % (device, e)) |
115 | + |
116 | + parts = [x for x in (info.strip()).splitlines() if len(x.split()) > 0] |
117 | + |
118 | + for part in parts: |
119 | + d = {'name': None, |
120 | + 'type': None, |
121 | + 'fstype': None, |
122 | + 'label': None, |
123 | + } |
124 | + |
125 | + for key, value in value_splitter(part): |
126 | + d[key.lower()] = value |
127 | + |
128 | + yield d |
129 | + |
130 | + |
131 | def device_type(device): |
132 | """ |
133 | Return the device type of the device by calling lsblk. |
134 | """ |
135 | |
136 | - lsblk_cmd = [LSBLK_CMD, '--pairs', '--nodeps', '--out', 'NAME,TYPE', |
137 | - device] |
138 | - info = None |
139 | - try: |
140 | - info, _err = util.subp(lsblk_cmd) |
141 | - except Exception as e: |
142 | - raise Exception("Failed during disk check for %s\n%s" % (device, e)) |
143 | - |
144 | - for key, value in value_splitter(info): |
145 | - if key.lower() == "type": |
146 | - return value.lower() |
147 | - |
148 | + for d in enumerate_disk(device, nodeps=True): |
149 | + if "type" in d: |
150 | + return d["type"].lower() |
151 | return None |
152 | |
153 | |
154 | @@ -204,7 +274,7 @@ |
155 | |
156 | |
157 | def find_device_node(device, fs_type=None, label=None, valid_targets=None, |
158 | - label_match=True): |
159 | + label_match=True, replace_fs=None): |
160 | """ |
161 | Find a device that is either matches the spec, or the first |
162 | |
163 | @@ -221,26 +291,12 @@ |
164 | if not valid_targets: |
165 | valid_targets = ['disk', 'part'] |
166 | |
167 | - lsblk_cmd = [LSBLK_CMD, '--pairs', '--out', 'NAME,TYPE,FSTYPE,LABEL', |
168 | - device] |
169 | - info = None |
170 | - try: |
171 | - info, _err = util.subp(lsblk_cmd) |
172 | - except Exception as e: |
173 | - raise Exception("Failed during disk check for %s\n%s" % (device, e)) |
174 | - |
175 | raw_device_used = False |
176 | - parts = [x for x in (info.strip()).splitlines() if len(x.split()) > 0] |
177 | - |
178 | - for part in parts: |
179 | - d = {'name': None, |
180 | - 'type': None, |
181 | - 'fstype': None, |
182 | - 'label': None, |
183 | - } |
184 | - |
185 | - for key, value in value_splitter(part): |
186 | - d[key.lower()] = value |
187 | + for d in enumerate_disk(device): |
188 | + |
189 | + if d['fstype'] == replace_fs and label_match is False: |
190 | + # We found a device where we want to replace the FS |
191 | + return ('/dev/%s' % d['name'], False) |
192 | |
193 | if (d['fstype'] == fs_type and |
194 | ((label_match and d['label'] == label) or not label_match)): |
195 | @@ -268,22 +324,20 @@ |
196 | |
197 | def is_disk_used(device): |
198 | """ |
199 | - Check if the device is currently used. Returns false if there |
200 | + Check if the device is currently used. Returns true if the device |
201 | + has either a file system or a partition entry |
202 | is no filesystem found on the disk. |
203 | """ |
204 | - lsblk_cmd = [LSBLK_CMD, '--pairs', '--out', 'NAME,TYPE', |
205 | - device] |
206 | - info = None |
207 | - try: |
208 | - info, _err = util.subp(lsblk_cmd) |
209 | - except Exception as e: |
210 | - # if we error out, we can't use the device |
211 | - util.logexc(LOG, |
212 | - "Error checking for filesystem on %s\n%s" % (device, e)) |
213 | + |
214 | + # If the child count is higher 1, then there are child nodes |
215 | + # such as partition or device mapper nodes |
216 | + use_count = [x for x in enumerate_disk(device)] |
217 | + if len(use_count.splitlines()) > 1: |
218 | return True |
219 | |
220 | - # If there is any output, then the device has something |
221 | - if len(info.splitlines()) > 1: |
222 | + # If we see a file system, then its used |
223 | + _, check_fstype, _ = check_fs(device) |
224 | + if check_fstype: |
225 | return True |
226 | |
227 | return False |
228 | @@ -455,6 +509,38 @@ |
229 | return sfdisk_definition |
230 | |
231 | |
232 | +def purge_disk_ptable(device): |
233 | + # wipe the first and last megabyte of a disk (or file) |
234 | + # gpt stores partition table both at front and at end. |
235 | + start_len = 1024 * 1024 |
236 | + end_len = 1024 * 1024 |
237 | + with open(device, "rb+") as fp: |
238 | + fp.write('\0' * (start_len)) |
239 | + fp.seek(-end_len, os.SEEK_END) |
240 | + fp.write('\0' * end_len) |
241 | + fp.flush() |
242 | + |
243 | + read_parttbl(device) |
244 | + |
245 | + |
246 | +def purge_disk(device): |
247 | + """ |
248 | + Remove parition table entries |
249 | + """ |
250 | + |
251 | + # wipe any file systems first |
252 | + for d in enumerate_disk(device): |
253 | + if d['type'] not in ["disk", "crypt"]: |
254 | + wipefs_cmd = [WIPEFS_CMD, "--all", "/dev/%s" % d['name']] |
255 | + try: |
256 | + LOG.info("Purging filesystem on /dev/%s" % d['name']) |
257 | + util.subp(wipefs_cmd) |
258 | + except Exception: |
259 | + raise Exception("Failed FS purge of /dev/%s" % d['name']) |
260 | + |
261 | + purge_disk_ptable(device) |
262 | + |
263 | + |
264 | def get_partition_layout(table_type, size, layout): |
265 | """ |
266 | Call the appropriate function for creating the table |
267 | @@ -542,6 +628,12 @@ |
268 | if not is_device_valid(device): |
269 | raise Exception("Device %s is not a disk device!", device) |
270 | |
271 | + # Remove the partition table entries |
272 | + if isinstance(layout, str) and layout.lower() == "remove": |
273 | + LOG.debug("Instructed to remove partition table entries") |
274 | + purge_disk(device) |
275 | + return |
276 | + |
277 | LOG.debug("Checking if device layout matches") |
278 | if check_partition_layout(table_type, device, layout): |
279 | LOG.debug("Device partitioning layout matches") |
280 | @@ -565,6 +657,26 @@ |
281 | LOG.debug("Partition table created for %s", device) |
282 | |
283 | |
284 | +def lookup_force_flag(fs): |
285 | + """ |
286 | + A force flag might be -F or -F, this look it up |
287 | + """ |
288 | + flags = {'ext': '-F', |
289 | + 'btrfs': '-f', |
290 | + 'xfs': '-f', |
291 | + 'reiserfs': '-f', |
292 | + } |
293 | + |
294 | + if 'ext' in fs.lower(): |
295 | + fs = 'ext' |
296 | + |
297 | + if fs.lower() in flags: |
298 | + return flags[fs] |
299 | + |
300 | + LOG.warn("Force flag for %s is unknown." % fs) |
301 | + return '' |
302 | + |
303 | + |
304 | def mkfs(fs_cfg): |
305 | """ |
306 | Create a file system on the device. |
307 | @@ -592,6 +704,7 @@ |
308 | fs_type = fs_cfg.get('filesystem') |
309 | fs_cmd = fs_cfg.get('cmd', []) |
310 | fs_opts = fs_cfg.get('extra_opts', []) |
311 | + fs_replace = fs_cfg.get('replace_fs', False) |
312 | overwrite = fs_cfg.get('overwrite', False) |
313 | |
314 | # This allows you to define the default ephemeral or swap |
315 | @@ -632,17 +745,23 @@ |
316 | label_match = False |
317 | |
318 | device, reuse = find_device_node(device, fs_type=fs_type, label=label, |
319 | - label_match=label_match) |
320 | + label_match=label_match, |
321 | + replace_fs=fs_replace) |
322 | LOG.debug("Automatic device for %s identified as %s", odevice, device) |
323 | |
324 | if reuse: |
325 | LOG.debug("Found filesystem match, skipping formating.") |
326 | return |
327 | |
328 | + if not reuse and fs_replace and device: |
329 | + LOG.debug("Replacing file system on %s as instructed." % device) |
330 | + |
331 | if not device: |
332 | LOG.debug("No device aviable that matches request. " |
333 | "Skipping fs creation for %s", fs_cfg) |
334 | return |
335 | + elif not partition or str(partition).lower() == 'none': |
336 | + LOG.debug("Using the raw device to place filesystem %s on" % label) |
337 | |
338 | else: |
339 | LOG.debug("Error in device identification handling.") |
340 | @@ -682,12 +801,16 @@ |
341 | if label: |
342 | fs_cmd.extend(["-L", label]) |
343 | |
344 | + # File systems that support the -F flag |
345 | + if not fs_cmd and (overwrite or device_type(device) == "disk"): |
346 | + fs_cmd.append(lookup_force_flag(fs_type)) |
347 | + |
348 | # Add the extends FS options |
349 | if fs_opts: |
350 | fs_cmd.extend(fs_opts) |
351 | |
352 | LOG.debug("Creating file system %s on %s", label, device) |
353 | - LOG.debug(" Using cmd: %s", "".join(fs_cmd)) |
354 | + LOG.debug(" Using cmd: %s", " ".join(fs_cmd)) |
355 | try: |
356 | util.subp(fs_cmd) |
357 | except Exception as e: |
358 | |
359 | === modified file 'cloudinit/config/cc_mounts.py' |
360 | --- cloudinit/config/cc_mounts.py 2013-03-07 21:27:47 +0000 |
361 | +++ cloudinit/config/cc_mounts.py 2013-10-03 23:32:32 +0000 |
362 | @@ -20,6 +20,7 @@ |
363 | |
364 | from string import whitespace # pylint: disable=W0402 |
365 | |
366 | +import os.path |
367 | import re |
368 | |
369 | from cloudinit import type_utils |
370 | @@ -75,7 +76,9 @@ |
371 | "name from ephemeral to ephemeral0"), (i + 1)) |
372 | |
373 | if is_mdname(startname): |
374 | - newname = cloud.device_name_to_device(startname) |
375 | + candidate_name = cloud.device_name_to_device(startname) |
376 | + newname = disk_or_part(candidate_name) |
377 | + |
378 | if not newname: |
379 | log.debug("Ignoring nonexistant named mount %s", startname) |
380 | cfgmnt[i][1] = None |
381 | @@ -119,7 +122,8 @@ |
382 | # entry has the same device name |
383 | for defmnt in defmnts: |
384 | startname = defmnt[0] |
385 | - devname = cloud.device_name_to_device(startname) |
386 | + candidate_name = cloud.device_name_to_device(startname) |
387 | + devname = disk_or_part(candidate_name) |
388 | if devname is None: |
389 | log.debug("Ignoring nonexistant named default mount %s", startname) |
390 | continue |
391 | @@ -198,3 +202,33 @@ |
392 | util.subp(("mount", "-a")) |
393 | except: |
394 | util.logexc(log, "Activating mounts via 'mount -a' failed") |
395 | + |
396 | + |
397 | +def disk_or_part(device): |
398 | + """ |
399 | + Find where the file system is on the disk, either on |
400 | + the disk itself or on the first partition. We don't go |
401 | + any deeper than partition 1 though. |
402 | + """ |
403 | + |
404 | + if not device: |
405 | + return None |
406 | + |
407 | + short_name = device.split('/')[-1] |
408 | + sys_path = "/sys/block/%s" % short_name |
409 | + |
410 | + # if the sys path does not exist but the device exists, |
411 | + # then the device is a partition, no sense looking any further |
412 | + if not os.path.exists(sys_path) and os.path.exists(device): |
413 | + return device |
414 | + |
415 | + sys_long_path = sys_path + "/" + short_name + "%s" |
416 | + valid_mappings = [sys_long_path % "1", |
417 | + sys_long_path % "p1", |
418 | + sys_path] |
419 | + |
420 | + for cdisk in valid_mappings: |
421 | + if not os.path.exists(cdisk): |
422 | + continue |
423 | + return "/dev/%s" % cdisk.split('/')[-1] |
424 | + return None |
425 | |
426 | === modified file 'cloudinit/sources/DataSourceAzure.py' |
427 | --- cloudinit/sources/DataSourceAzure.py 2013-09-27 17:00:35 +0000 |
428 | +++ cloudinit/sources/DataSourceAzure.py 2013-10-03 23:32:32 +0000 |
429 | @@ -54,8 +54,9 @@ |
430 | 'layout': True, |
431 | 'overwrite': False} |
432 | }, |
433 | - 'fs_setup': [{'filesystem': 'ext4', 'device': 'ephemeral0', |
434 | - 'partition': 'auto'}], |
435 | + 'fs_setup': [{'filesystem': 'ext4', |
436 | + 'device': 'ephemeral0.1', |
437 | + 'replace_fs': 'ntfs'}] |
438 | } |
439 | |
440 | DS_CFG_PATH = ['datasource', DS_NAME] |
441 | |
442 | === modified file 'cloudinit/sources/DataSourceSmartOS.py' |
443 | --- cloudinit/sources/DataSourceSmartOS.py 2013-09-27 17:35:36 +0000 |
444 | +++ cloudinit/sources/DataSourceSmartOS.py 2013-10-03 23:32:32 +0000 |
445 | @@ -46,6 +46,7 @@ |
446 | 'user-data': ('user-data', False), |
447 | 'iptables_disable': ('iptables_disable', True), |
448 | 'motd_sys_info': ('motd_sys_info', True), |
449 | + 'availability_zone': ('region', True), |
450 | } |
451 | |
452 | DS_NAME = 'SmartOS' |
453 | @@ -81,8 +82,9 @@ |
454 | 'layout': False, |
455 | 'overwrite': False} |
456 | }, |
457 | - 'fs_setup': [{'label': 'ephemeral0', 'filesystem': 'ext3', |
458 | - 'device': 'ephemeral0', 'partition': 'auto'}], |
459 | + 'fs_setup': [{'label': 'ephemeral0', |
460 | + 'filesystem': 'ext3', |
461 | + 'device': 'ephemeral0'}], |
462 | } |
463 | |
464 | |
465 | @@ -174,6 +176,13 @@ |
466 | seed_timeout=self.seed_timeout, default=default, |
467 | b64=b64) |
468 | |
469 | + @property |
470 | + def availability_zone(self): |
471 | + try: |
472 | + return self.metadata['availability-zone'] |
473 | + except KeyError: |
474 | + return None |
475 | + |
476 | |
477 | def get_serial(seed_device, seed_timeout): |
478 | """This is replaced in unit testing, allowing us to replace |
479 | |
480 | === modified file 'cloudinit/util.py' |
481 | --- cloudinit/util.py 2013-09-27 23:42:38 +0000 |
482 | +++ cloudinit/util.py 2013-10-03 23:32:32 +0000 |
483 | @@ -32,6 +32,7 @@ |
484 | import gzip |
485 | import hashlib |
486 | import os |
487 | +import os.path |
488 | import platform |
489 | import pwd |
490 | import random |
491 | @@ -1826,3 +1827,79 @@ |
492 | except: |
493 | pass |
494 | return ret |
495 | + |
496 | + |
497 | +def map_partition(alias): |
498 | + """ |
499 | + Return partition number for devices like ephemeral0.0 or ephemeral0.1 |
500 | + |
501 | + Parameters: |
502 | + alaias: the alias, i.e. ephemeral0 or swap0 |
503 | + device: the actual device to markup |
504 | + |
505 | + Rules: |
506 | + - anything after a . is a parittion |
507 | + - device.0 is the same as device |
508 | + """ |
509 | + |
510 | + if len(alias.split('.')) == 1: |
511 | + return None |
512 | + |
513 | + suffix = alias.split('.')[-1] |
514 | + try: |
515 | + if int(suffix) == 0: |
516 | + return None |
517 | + return int(suffix) |
518 | + except ValueError: |
519 | + pass |
520 | + |
521 | + return None |
522 | + |
523 | + |
524 | +def map_device_alias(device, partition=None, alias=None): |
525 | + """ |
526 | + Find the name of the partition. While this might seem rather |
527 | + straight forward, its not since some devices are '<device><partition>' |
528 | + while others are '<device>p<partition>'. For example, /dev/xvda3 on EC2 |
529 | + will present as /dev/xvda3p1 for the first partition since /dev/xvda3 is |
530 | + a block device. |
531 | + |
532 | + The primary use is to map 'ephemeral0.1' in the datasource to a |
533 | + real device name |
534 | + """ |
535 | + |
536 | + if not device: |
537 | + raise Exception("Device cannot be undefined!") |
538 | + |
539 | + if not partition and not alias: |
540 | + raise Exception("partition or alias is required") |
541 | + |
542 | + if alias: |
543 | + partition = map_partition(alias) |
544 | + |
545 | + # if the partition doesn't map, return the device |
546 | + if not partition: |
547 | + return device |
548 | + |
549 | + short_name = device.split('/')[-1] |
550 | + sys_path = "/sys/block/%s" % short_name |
551 | + |
552 | + if not os.path.exists(sys_path): |
553 | + return None |
554 | + |
555 | + sys_long_path = sys_path + "/" + short_name |
556 | + valid_mappings = [sys_long_path + "%s" % partition, |
557 | + sys_long_path + "p%s" % partition] |
558 | + |
559 | + for cdisk in valid_mappings: |
560 | + if not os.path.exists(cdisk): |
561 | + continue |
562 | + |
563 | + dev_path = "/dev/%s" % cdisk.split('/')[-1] |
564 | + if os.path.exists(dev_path): |
565 | + return dev_path |
566 | + else: |
567 | + LOG.warn("Specificed parition %s does not exist on %s" % ( |
568 | + partition, device)) |
569 | + |
570 | + return None |
571 | |
572 | === modified file 'doc/examples/cloud-config-disk-setup.txt' |
573 | --- doc/examples/cloud-config-disk-setup.txt 2013-09-27 23:35:20 +0000 |
574 | +++ doc/examples/cloud-config-disk-setup.txt 2013-10-03 23:32:32 +0000 |
575 | @@ -30,8 +30,8 @@ |
576 | fs_setup: |
577 | - label: ephemeral0 |
578 | filesystem: ext4 |
579 | - device: ephemeral0 |
580 | - partition: auto |
581 | + device: ephemeral0.1 |
582 | + replace_fs: ntfs |
583 | |
584 | |
585 | Default disk definitions for SmartOS |
586 | @@ -47,8 +47,7 @@ |
587 | fs_setup: |
588 | - label: ephemeral0 |
589 | filesystem: ext3 |
590 | - device: ephemeral0 |
591 | - partition: auto |
592 | + device: ephemeral0.0 |
593 | |
594 | Cavaut for SmartOS: if ephemeral disk is not defined, then the disk will |
595 | not be automatically added to the mounts. |
596 | @@ -171,6 +170,7 @@ |
597 | device: <DEVICE> |
598 | partition: <PART_VALUE> |
599 | overwrite: <OVERWRITE> |
600 | + replace_fs: <FS_TYPE> |
601 | |
602 | Where: |
603 | <LABEL>: The file system label to be used. If set to None, no label is |
604 | @@ -187,7 +187,13 @@ |
605 | label as 'ephemeralX' otherwise there may be issues with the mounting |
606 | of the ephemeral storage layer. |
607 | |
608 | - <PART_VALUE>: The valid options are: |
609 | + If you define the device as 'ephemeralX.Y' then Y will be interpetted |
610 | + as a partition value. However, ephermalX.0 is the _same_ as ephemeralX. |
611 | + |
612 | + <PART_VALUE>: |
613 | + Partition definitions are overwriten if you use the '<DEVICE>.Y' notation. |
614 | + |
615 | + The valid options are: |
616 | "auto|any": tell cloud-init not to care whether there is a partition |
617 | or not. Auto will use the first partition that does not contain a |
618 | file system already. In the absence of a partition table, it will |
619 | @@ -236,5 +242,10 @@ |
620 | |
621 | "false": If an existing file system exists, skip the creation. |
622 | |
623 | + <REPLACE_FS>: This is a special directive, used for Windows Azure that |
624 | + instructs cloud-init to replace a file system of <FS_TYPE>. NOTE: |
625 | + unless you define a label, this requires the use of the 'any' partition |
626 | + directive. |
627 | + |
628 | Behavior Caveat: The default behavior is to _check_ if the file system exists. |
629 | If a file system matches the specification, then the operation is a no-op. |
630 | |
631 | === modified file 'tests/unittests/test_datasource/test_azure.py' |
632 | --- tests/unittests/test_datasource/test_azure.py 2013-10-02 13:25:36 +0000 |
633 | +++ tests/unittests/test_datasource/test_azure.py 2013-10-03 23:32:32 +0000 |
634 | @@ -328,8 +328,6 @@ |
635 | self.assertTrue(ret) |
636 | cfg = dsrc.get_config_obj() |
637 | self.assertTrue(cfg) |
638 | - self.assertEquals(dsrc.device_name_to_device("ephemeral0"), |
639 | - "/dev/sdc") |
640 | |
641 | def test_userdata_arrives(self): |
642 | userdata = "This is my user-data" |
There is really no reason for:
dd_cmd = util.which("dd")
subp(dd_cmd)
That is identical to:
subp(dd_cmd)
purge_disk doesn't seem that it should be something that is partition table specific.
The end result is that you have an unpartitioned disk.
A "wipe the disk" method might look like this:
def wipe_disk(devname):
for f in $devname[0-9]*:
[ -b "$f" ] && wipefs $f || :
dd if=/dev/zero of=$devname
seek= $(($size- 1024)) count=1024 2>&1)
out=$(dd if=/dev/zero of="$target" bs=1024 \
blockdev --rereadpt $devname
udevadm settle
Interestingly, as you've found, an attempt to "wipe" a disk and then partition it, and then mkfs might end up in there being an existing filesystem at any of the new partitions (ie, if the old disk format had partitions there).
So really what you'd need to do is:
erase partition tables (gpt is at end and beginning, so zero the first MB and last MB)
format disk
rereadpt && udevadm settle
'wipefs' all partitions
mkfs
Doing that would actually mean we do not have to pass '-F' to mkfs.extX (note, that -F is not common to all'mkfs' programs so 'mkfs.$FSTYPE -F' wont necessarily work).
Probably could also get away with just calling 'wipefs --all $dev' before formatting.
I'm not sure really what i htink about changing the builtin defaults to overwrite filesystems on azure. that seems dangerous. better if we coudl identify this one scenario and wipe it.