Merge lp:~raharper/ubuntu/yakkety/curtin/pkg into lp:~smoser/ubuntu/yakkety/curtin/pkg

Proposed by Ryan Harper
Status: Merged
Merged at revision: 59
Proposed branch: lp:~raharper/ubuntu/yakkety/curtin/pkg
Merge into: lp:~smoser/ubuntu/yakkety/curtin/pkg
Diff against target: 9024 lines (+5935/-1556)
69 files modified
Makefile (+3/-1)
curtin/__init__.py (+2/-0)
curtin/block/__init__.py (+176/-47)
curtin/block/clear_holders.py (+387/-0)
curtin/block/lvm.py (+96/-0)
curtin/block/mdadm.py (+18/-5)
curtin/commands/apply_net.py (+156/-1)
curtin/commands/apt_config.py (+11/-9)
curtin/commands/block_info.py (+75/-0)
curtin/commands/block_meta.py (+82/-203)
curtin/commands/block_wipe.py (+0/-1)
curtin/commands/clear_holders.py (+48/-0)
curtin/commands/curthooks.py (+4/-51)
curtin/commands/main.py (+4/-3)
curtin/config.py (+2/-3)
curtin/net/__init__.py (+67/-30)
curtin/net/network_state.py (+45/-1)
curtin/util.py (+45/-8)
debian/changelog (+16/-2)
doc/conf.py (+21/-4)
doc/devel/README-vmtest.txt (+0/-221)
doc/devel/README.txt (+0/-55)
doc/devel/clear_holders_doc.txt (+85/-0)
doc/index.rst (+6/-0)
doc/topics/apt_source.rst (+26/-20)
doc/topics/config.rst (+551/-0)
doc/topics/development.rst (+68/-0)
doc/topics/integration-testing.rst (+245/-0)
doc/topics/networking.rst (+522/-0)
doc/topics/overview.rst (+7/-7)
doc/topics/reporting.rst (+3/-3)
doc/topics/storage.rst (+894/-0)
examples/network-ipv6-bond-vlan.yaml (+56/-0)
examples/tests/basic_network_static_ipv6.yaml (+22/-0)
examples/tests/network_alias.yaml (+125/-0)
examples/tests/network_mtu.yaml (+88/-0)
examples/tests/network_source_ipv6.yaml (+31/-0)
examples/tests/vlan_network_ipv6.yaml (+92/-0)
setup.py (+2/-2)
tests/unittests/helpers.py (+41/-0)
tests/unittests/test_apt_source.py (+17/-3)
tests/unittests/test_block.py (+105/-0)
tests/unittests/test_block_lvm.py (+94/-0)
tests/unittests/test_block_mdadm.py (+28/-23)
tests/unittests/test_clear_holders.py (+329/-0)
tests/unittests/test_net.py (+54/-13)
tests/unittests/test_util.py (+130/-14)
tests/vmtests/helpers.py (+129/-166)
tests/vmtests/test_basic.py (+19/-40)
tests/vmtests/test_bcache_basic.py (+5/-8)
tests/vmtests/test_bonding.py (+0/-204)
tests/vmtests/test_lvm.py (+2/-1)
tests/vmtests/test_mdadm_bcache.py (+21/-17)
tests/vmtests/test_multipath.py (+5/-13)
tests/vmtests/test_network.py (+205/-348)
tests/vmtests/test_network_alias.py (+40/-0)
tests/vmtests/test_network_bonding.py (+63/-0)
tests/vmtests/test_network_enisource.py (+91/-0)
tests/vmtests/test_network_ipv6.py (+53/-0)
tests/vmtests/test_network_ipv6_enisource.py (+26/-0)
tests/vmtests/test_network_ipv6_static.py (+42/-0)
tests/vmtests/test_network_ipv6_vlan.py (+34/-0)
tests/vmtests/test_network_mtu.py (+155/-0)
tests/vmtests/test_network_static.py (+44/-0)
tests/vmtests/test_network_vlan.py (+77/-0)
tests/vmtests/test_nvme.py (+2/-1)
tests/vmtests/test_raid5_bcache.py (+5/-8)
tests/vmtests/test_uefi_basic.py (+15/-17)
tox.ini (+23/-3)
To merge this branch: bzr merge lp:~raharper/ubuntu/yakkety/curtin/pkg
Reviewer Review Type Date Requested Status
curtin developers Pending
Review via email: mp+307217@code.launchpad.net

Commit message

* New upstream snapshot.
  - unittest,tox.ini: catch and fix issue with trusty-level mock of open
  - block/mdadm: add option to ignore mdadm_assemble errors (LP: #1618429)
  - curtin/doc: overhaul curtin documentation for readthedocs.org (LP: #1351085)
  - curtin.util: re-add support for RunInChroot (LP: #1617375)
  - curtin/net: overhaul of eni rendering to handle mixed ipv4/ipv6 configs
  - curtin.block: refactor clear_holders logic into block.clear_holders and cli cmd
  - curtin.apply_net should exit non-zero upon exception. (LP: #1615780)
  - apt: fix bug in disable_suites if sources.list line is blank.
  - vmtests: disable Wily in vmtests
  - Fix the unittests for test_apt_source.

Description of the change

  * New upstream snapshot.
    - unittest,tox.ini: catch and fix issue with trusty-level mock of open
    - block/mdadm: add option to ignore mdadm_assemble errors (LP: #1618429)
    - curtin/doc: overhaul curtin documentation for readthedocs.org (LP: #1351085)
    - curtin.util: re-add support for RunInChroot (LP: #1617375)
    - curtin/net: overhaul of eni rendering to handle mixed ipv4/ipv6 configs
    - curtin.block: refactor clear_holders logic into block.clear_holders and cli cmd
    - curtin.apply_net should exit non-zero upon exception. (LP: #1615780)
    - apt: fix bug in disable_suites if sources.list line is blank.
    - vmtests: disable Wily in vmtests
    - Fix the unittests for test_apt_source.

To post a comment you must log in.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'Makefile'
2--- Makefile 2016-05-05 16:43:40 +0000
3+++ Makefile 2016-09-29 18:49:30 +0000
4@@ -49,5 +49,7 @@
5 sync-images:
6 @$(CWD)/tools/vmtest-sync-images
7
8+clean:
9+ rm -rf doc/_build
10
11-.PHONY: all test pyflakes pyflakes3 pep8 build
12+.PHONY: all clean test pyflakes pyflakes3 pep8 build
13
14=== modified file 'curtin/__init__.py'
15--- curtin/__init__.py 2016-08-05 20:47:14 +0000
16+++ curtin/__init__.py 2016-09-29 18:49:30 +0000
17@@ -37,4 +37,6 @@
18 'APT_CONFIG_V1',
19 ]
20
21+__version__ = "0.1.0"
22+
23 # vi: ts=4 expandtab syntax=python
24
25=== modified file 'curtin/block/__init__.py'
26--- curtin/block/__init__.py 2016-08-05 20:47:14 +0000
27+++ curtin/block/__init__.py 2016-09-29 18:49:30 +0000
28@@ -23,21 +23,31 @@
29 import itertools
30
31 from curtin import util
32+from curtin.block import lvm
33+from curtin.log import LOG
34 from curtin.udev import udevadm_settle
35-from curtin.log import LOG
36
37
38 def get_dev_name_entry(devname):
39+ """
40+ convert device name to path in /dev
41+ """
42 bname = devname.split('/dev/')[-1]
43 return (bname, "/dev/" + bname)
44
45
46 def is_valid_device(devname):
47+ """
48+ check if device is a valid device
49+ """
50 devent = get_dev_name_entry(devname)[1]
51 return is_block_device(devent)
52
53
54 def is_block_device(path):
55+ """
56+ check if path is a block device
57+ """
58 try:
59 return stat.S_ISBLK(os.stat(path).st_mode)
60 except OSError as e:
61@@ -47,12 +57,19 @@
62
63
64 def dev_short(devname):
65+ """
66+ get short form of device name
67+ """
68+ devname = os.path.normpath(devname)
69 if os.path.sep in devname:
70 return os.path.basename(devname)
71 return devname
72
73
74 def dev_path(devname):
75+ """
76+ convert device name to path in /dev
77+ """
78 if devname.startswith('/dev/'):
79 return devname
80 else:
81@@ -100,7 +117,9 @@
82
83
84 def partition_kname(disk_kname, partition_number):
85- """Add number to disk_kname prepending a 'p' if needed"""
86+ """
87+ Add number to disk_kname prepending a 'p' if needed
88+ """
89 for dev_type in ['nvme', 'mmcblk', 'cciss', 'mpath', 'dm']:
90 if disk_kname.startswith(dev_type):
91 partition_number = "p%s" % partition_number
92@@ -108,9 +127,24 @@
93 return "%s%s" % (disk_kname, partition_number)
94
95
96+def sysfs_to_devpath(sysfs_path):
97+ """
98+ convert a path in /sys/class/block to a path in /dev
99+ """
100+ path = kname_to_path(path_to_kname(sysfs_path))
101+ if not is_block_device(path):
102+ raise ValueError('could not find blockdev for sys path: {}'
103+ .format(sysfs_path))
104+ return path
105+
106+
107 def sys_block_path(devname, add=None, strict=True):
108+ """
109+ get path to device in /sys/class/block
110+ """
111 toks = ['/sys/class/block']
112 # insert parent dev if devname is partition
113+ devname = os.path.normpath(devname)
114 (parent, partnum) = get_blockdev_for_partition(devname)
115 if partnum:
116 toks.append(path_to_kname(parent))
117@@ -132,6 +166,9 @@
118
119
120 def _lsblock_pairs_to_dict(lines):
121+ """
122+ parse lsblock output and convert to dict
123+ """
124 ret = {}
125 for line in lines.splitlines():
126 toks = shlex.split(line)
127@@ -147,6 +184,9 @@
128
129
130 def _lsblock(args=None):
131+ """
132+ get lsblock data as dict
133+ """
134 # lsblk --help | sed -n '/Available/,/^$/p' |
135 # sed -e 1d -e '$d' -e 's,^[ ]\+,,' -e 's, .*,,' | sort
136 keys = ['ALIGNMENT', 'DISC-ALN', 'DISC-GRAN', 'DISC-MAX', 'DISC-ZERO',
137@@ -169,8 +209,10 @@
138
139
140 def get_unused_blockdev_info():
141- # return a list of unused block devices. These are devices that
142- # do not have anything mounted on them.
143+ """
144+ return a list of unused block devices.
145+ These are devices that do not have anything mounted on them.
146+ """
147
148 # get a list of top level block devices, then iterate over it to get
149 # devices dependent on those. If the lsblk call for that specific
150@@ -186,7 +228,9 @@
151
152
153 def get_devices_for_mp(mountpoint):
154- # return a list of devices (full paths) used by the provided mountpoint
155+ """
156+ return a list of devices (full paths) used by the provided mountpoint
157+ """
158 bdinfo = _lsblock()
159 found = set()
160 for devname, data in bdinfo.items():
161@@ -207,6 +251,9 @@
162
163
164 def get_installable_blockdevs(include_removable=False, min_size=1024**3):
165+ """
166+ find blockdevs suitable for installation
167+ """
168 good = []
169 unused = get_unused_blockdev_info()
170 for devname, data in unused.items():
171@@ -221,6 +268,11 @@
172
173
174 def get_blockdev_for_partition(devpath):
175+ """
176+ find the parent device for a partition.
177+ returns a tuple of the parent block device and the partition number
178+ if device is not a partition, None will be returned for partition number
179+ """
180 # normalize path
181 rpath = os.path.realpath(devpath)
182
183@@ -234,7 +286,7 @@
184
185 # don't need to try out multiple sysfs paths as path_to_kname handles cciss
186 if not os.path.exists(syspath):
187- raise ValueError("%s had no syspath (%s)" % (devpath, syspath))
188+ raise OSError("%s had no syspath (%s)" % (devpath, syspath))
189
190 ptpath = os.path.join(syspath, "partition")
191 if not os.path.exists(ptpath):
192@@ -255,8 +307,21 @@
193 return (diskdevpath, ptnum)
194
195
196+def get_sysfs_partitions(device):
197+ """
198+ get a list of sysfs paths for partitions under a block device
199+ accepts input as a device kname, sysfs path, or dev path
200+ returns empty list if no partitions available
201+ """
202+ sysfs_path = sys_block_path(device)
203+ return [sys_block_path(kname) for kname in os.listdir(sysfs_path)
204+ if os.path.exists(os.path.join(sysfs_path, kname, 'partition'))]
205+
206+
207 def get_pardevs_on_blockdevs(devs):
208- # return a dict of partitions with their info that are on provided devs
209+ """
210+ return a dict of partitions with their info that are on provided devs
211+ """
212 if devs is None:
213 devs = []
214 devs = [get_dev_name_entry(d)[1] for d in devs]
215@@ -291,7 +356,9 @@
216
217
218 def rescan_block_devices():
219- # run 'blockdev --rereadpt' for all block devices not currently mounted
220+ """
221+ run 'blockdev --rereadpt' for all block devices not currently mounted
222+ """
223 unused = get_unused_blockdev_info()
224 devices = []
225 for devname, data in unused.items():
226@@ -319,6 +386,9 @@
227
228
229 def blkid(devs=None, cache=True):
230+ """
231+ get data about block devices from blkid and convert to dict
232+ """
233 if devs is None:
234 devs = []
235
236@@ -558,50 +628,108 @@
237 def sysfs_partition_data(blockdev=None, sysfs_path=None):
238 # given block device or sysfs_path, return a list of tuples
239 # of (kernel_name, number, offset, size)
240- if blockdev is None and sysfs_path is None:
241- raise ValueError("Blockdev and sysfs_path cannot both be None")
242-
243 if blockdev:
244+ blockdev = os.path.normpath(blockdev)
245 sysfs_path = sys_block_path(blockdev)
246-
247- ptdata = []
248- # /sys/class/block/dev has entries of 'kname' for each partition
249+ elif sysfs_path:
250+ # use normpath to ensure that paths with trailing slash work
251+ sysfs_path = os.path.normpath(sysfs_path)
252+ blockdev = os.path.join('/dev', os.path.basename(sysfs_path))
253+ else:
254+ raise ValueError("Blockdev and sysfs_path cannot both be None")
255
256 # queue property is only on parent devices, ie, we can't read
257 # /sys/class/block/vda/vda1/queue/* as queue is only on the
258 # parent device
259+ sysfs_prefix = sysfs_path
260 (parent, partnum) = get_blockdev_for_partition(blockdev)
261- sysfs_prefix = sysfs_path
262 if partnum:
263 sysfs_prefix = sys_block_path(parent)
264-
265- block_size = int(util.load_file(os.path.join(sysfs_prefix,
266- 'queue/logical_block_size')))
267-
268- block_size = int(
269- util.load_file(os.path.join(sysfs_path, 'queue/logical_block_size')))
270+ partnum = int(partnum)
271+
272+ block_size = int(util.load_file(os.path.join(
273+ sysfs_prefix, 'queue/logical_block_size')))
274 unit = block_size
275- for d in os.listdir(sysfs_path):
276- partd = os.path.join(sysfs_path, d)
277+
278+ ptdata = []
279+ for part_sysfs in get_sysfs_partitions(sysfs_prefix):
280 data = {}
281 for sfile in ('partition', 'start', 'size'):
282- dfile = os.path.join(partd, sfile)
283+ dfile = os.path.join(part_sysfs, sfile)
284 if not os.path.isfile(dfile):
285 continue
286 data[sfile] = int(util.load_file(dfile))
287- if 'partition' not in data:
288- continue
289- ptdata.append((d, data['partition'], data['start'] * unit,
290- data['size'] * unit,))
291+ if partnum is None or data['partition'] == partnum:
292+ ptdata.append((path_to_kname(part_sysfs), data['partition'],
293+ data['start'] * unit, data['size'] * unit,))
294
295 return ptdata
296
297
298+def get_part_table_type(device):
299+ """
300+ check the type of partition table present on the specified device
301+ returns None if no ptable was present or device could not be read
302+ """
303+ # it is neccessary to look for the gpt signature first, then the dos
304+ # signature, because a gpt formatted disk usually has a valid mbr to
305+ # protect the disk from being modified by older partitioning tools
306+ return ('gpt' if check_efi_signature(device) else
307+ 'dos' if check_dos_signature(device) else None)
308+
309+
310+def check_dos_signature(device):
311+ """
312+ check if there is a dos partition table signature present on device
313+ """
314+ # the last 2 bytes of a dos partition table have the signature with the
315+ # value 0xAA55. the dos partition table is always 0x200 bytes long, even if
316+ # the underlying disk uses a larger logical block size, so the start of
317+ # this signature must be at 0x1fe
318+ # https://en.wikipedia.org/wiki/Master_boot_record#Sector_layout
319+ return (is_block_device(device) and util.file_size(device) >= 0x200 and
320+ (util.load_file(device, mode='rb', read_len=2, offset=0x1fe) ==
321+ b'\x55\xAA'))
322+
323+
324+def check_efi_signature(device):
325+ """
326+ check if there is a gpt partition table signature present on device
327+ """
328+ # the gpt partition table header is always on lba 1, regardless of the
329+ # logical block size used by the underlying disk. therefore, a static
330+ # offset cannot be used, the offset to the start of the table header is
331+ # always the sector size of the disk
332+ # the start of the gpt partition table header shoult have the signaure
333+ # 'EFI PART'.
334+ # https://en.wikipedia.org/wiki/GUID_Partition_Table
335+ sector_size = get_blockdev_sector_size(device)[0]
336+ return (is_block_device(device) and
337+ util.file_size(device) >= 2 * sector_size and
338+ (util.load_file(device, mode='rb', read_len=8,
339+ offset=sector_size) == b'EFI PART'))
340+
341+
342+def is_extended_partition(device):
343+ """
344+ check if the specified device path is a dos extended partition
345+ """
346+ # an extended partition must be on a dos disk, must be a partition, must be
347+ # within the first 4 partitions and will have a valid dos signature,
348+ # because the format of the extended partition matches that of a real mbr
349+ (parent_dev, part_number) = get_blockdev_for_partition(device)
350+ return (get_part_table_type(parent_dev) in ['dos', 'msdos'] and
351+ part_number is not None and int(part_number) <= 4 and
352+ check_dos_signature(device))
353+
354+
355 def wipe_file(path, reader=None, buflen=4 * 1024 * 1024):
356- # wipe the existing file at path.
357- # if reader is provided, it will be called as a 'reader(buflen)'
358- # to provide data for each write. Otherwise, zeros are used.
359- # writes will be done in size of buflen.
360+ """
361+ wipe the existing file at path.
362+ if reader is provided, it will be called as a 'reader(buflen)'
363+ to provide data for each write. Otherwise, zeros are used.
364+ writes will be done in size of buflen.
365+ """
366 if reader:
367 readfunc = reader
368 else:
369@@ -610,13 +738,11 @@
370 def readfunc(size):
371 return buf
372
373+ size = util.file_size(path)
374+ LOG.debug("%s is %s bytes. wiping with buflen=%s",
375+ path, size, buflen)
376+
377 with open(path, "rb+") as fp:
378- # get the size by seeking to end.
379- fp.seek(0, 2)
380- size = fp.tell()
381- LOG.debug("%s is %s bytes. wiping with buflen=%s",
382- path, size, buflen)
383- fp.seek(0)
384 while True:
385 pbuf = readfunc(buflen)
386 pos = fp.tell()
387@@ -633,9 +759,11 @@
388
389
390 def quick_zero(path, partitions=True):
391- # zero 1M at front, 1M at end, and 1M at front
392- # if this is a block device and partitions is true, then
393- # zero 1M at front and end of each partition.
394+ """
395+ zero 1M at front, 1M at end, and 1M at front
396+ if this is a block device and partitions is true, then
397+ zero 1M at front and end of each partition.
398+ """
399 buflen = 1024
400 count = 1024
401 zero_size = buflen * count
402@@ -655,6 +783,9 @@
403
404
405 def zero_file_at_offsets(path, offsets, buflen=1024, count=1024, strict=False):
406+ """
407+ write zeros to file at specified offsets
408+ """
409 bmsg = "{path} (size={size}): "
410 m_short = bmsg + "{tot} bytes from {offset} > size."
411 m_badoff = bmsg + "invalid offset {offset}."
412@@ -716,15 +847,13 @@
413 if mode == "pvremove":
414 # We need to use --force --force in case it's already in a volgroup and
415 # pvremove doesn't want to remove it
416- cmds = []
417- cmds.append(["pvremove", "--force", "--force", "--yes", path])
418- cmds.append(["pvscan", "--cache"])
419- cmds.append(["vgscan", "--mknodes", "--cache"])
420+
421 # If pvremove is run and there is no label on the system,
422 # then it exits with 5. That is also okay, because we might be
423 # wiping something that is already blank
424- for cmd in cmds:
425- util.subp(cmd, rcs=[0, 5], capture=True)
426+ util.subp(['pvremove', '--force', '--force', '--yes', path],
427+ rcs=[0, 5], capture=True)
428+ lvm.lvm_scan()
429 elif mode == "zero":
430 wipe_file(path)
431 elif mode == "random":
432
433=== added file 'curtin/block/clear_holders.py'
434--- curtin/block/clear_holders.py 1970-01-01 00:00:00 +0000
435+++ curtin/block/clear_holders.py 2016-09-29 18:49:30 +0000
436@@ -0,0 +1,387 @@
437+# Copyright (C) 2016 Canonical Ltd.
438+#
439+# Author: Wesley Wiedenmeier <wesley.wiedenmeier@canonical.com>
440+#
441+# Curtin is free software: you can redistribute it and/or modify it under
442+# the terms of the GNU Affero General Public License as published by the
443+# Free Software Foundation, either version 3 of the License, or (at your
444+# option) any later version.
445+#
446+# Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
447+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
448+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for
449+# more details.
450+#
451+# You should have received a copy of the GNU Affero General Public License
452+# along with Curtin. If not, see <http://www.gnu.org/licenses/>.
453+
454+"""
455+This module provides a mechanism for shutting down virtual storage layers on
456+top of a block device, making it possible to reuse the block device without
457+having to reboot the system
458+"""
459+
460+import os
461+
462+from curtin import (block, udev, util)
463+from curtin.block import lvm
464+from curtin.log import LOG
465+
466+
467+def _define_handlers_registry():
468+ """
469+ returns instantiated dev_types
470+ """
471+ return {
472+ 'partition': {'shutdown': wipe_superblock,
473+ 'ident': identify_partition},
474+ 'lvm': {'shutdown': shutdown_lvm, 'ident': identify_lvm},
475+ 'crypt': {'shutdown': shutdown_crypt, 'ident': identify_crypt},
476+ 'raid': {'shutdown': shutdown_mdadm, 'ident': identify_mdadm},
477+ 'bcache': {'shutdown': shutdown_bcache, 'ident': identify_bcache},
478+ 'disk': {'ident': lambda x: False, 'shutdown': wipe_superblock},
479+ }
480+
481+
482+def get_dmsetup_uuid(device):
483+ """
484+ get the dm uuid for a specified dmsetup device
485+ """
486+ blockdev = block.sysfs_to_devpath(device)
487+ (out, _) = util.subp(['dmsetup', 'info', blockdev, '-C', '-o', 'uuid',
488+ '--noheadings'], capture=True)
489+ return out.strip()
490+
491+
492+def get_bcache_using_dev(device):
493+ """
494+ Get the /sys/fs/bcache/ path of the bcache volume using specified device
495+ """
496+ # FIXME: when block.bcache is written this should be moved there
497+ sysfs_path = block.sys_block_path(device)
498+ return os.path.realpath(os.path.join(sysfs_path, 'bcache', 'cache'))
499+
500+
501+def shutdown_bcache(device):
502+ """
503+ Shut down bcache for specified bcache device
504+ """
505+ bcache_shutdown_message = ('shutdown_bcache running on {} has determined '
506+ 'that the device has already been shut down '
507+ 'during handling of another bcache dev. '
508+ 'skipping'.format(device))
509+ if not os.path.exists(device):
510+ LOG.info(bcache_shutdown_message)
511+ return
512+
513+ bcache_sysfs = get_bcache_using_dev(device)
514+ if not os.path.exists(bcache_sysfs):
515+ LOG.info(bcache_shutdown_message)
516+ return
517+
518+ LOG.debug('stopping bcache at: %s', bcache_sysfs)
519+ util.write_file(os.path.join(bcache_sysfs, 'stop'), '1', mode=None)
520+
521+
522+def shutdown_lvm(device):
523+ """
524+ Shutdown specified lvm device.
525+ """
526+ device = block.sys_block_path(device)
527+ # lvm devices have a dm directory that containes a file 'name' containing
528+ # '{volume group}-{logical volume}'. The volume can be freed using lvremove
529+ name_file = os.path.join(device, 'dm', 'name')
530+ (vg_name, lv_name) = lvm.split_lvm_name(util.load_file(name_file))
531+ # use two --force flags here in case the volume group that this lv is
532+ # attached two has been damaged
533+ LOG.debug('running lvremove on %s/%s', vg_name, lv_name)
534+ util.subp(['lvremove', '--force', '--force',
535+ '{}/{}'.format(vg_name, lv_name)], rcs=[0, 5])
536+ # if that was the last lvol in the volgroup, get rid of volgroup
537+ if len(lvm.get_lvols_in_volgroup(vg_name)) == 0:
538+ util.subp(['vgremove', '--force', '--force', vg_name], rcs=[0, 5])
539+ # refresh lvmetad
540+ lvm.lvm_scan()
541+
542+
543+def shutdown_crypt(device):
544+ """
545+ Shutdown specified cryptsetup device
546+ """
547+ blockdev = block.sysfs_to_devpath(device)
548+ util.subp(['cryptsetup', 'remove', blockdev], capture=True)
549+
550+
551+def shutdown_mdadm(device):
552+ """
553+ Shutdown specified mdadm device.
554+ """
555+ blockdev = block.sysfs_to_devpath(device)
556+ LOG.debug('using mdadm.mdadm_stop on dev: %s', blockdev)
557+ block.mdadm.mdadm_stop(blockdev)
558+ block.mdadm.mdadm_remove(blockdev)
559+
560+
561+def wipe_superblock(device):
562+ """
563+ Wrapper for block.wipe_volume compatible with shutdown function interface
564+ """
565+ blockdev = block.sysfs_to_devpath(device)
566+ # when operating on a disk that used to have a dos part table with an
567+ # extended partition, attempting to wipe the extended partition will fail
568+ if block.is_extended_partition(blockdev):
569+ LOG.info("extended partitions do not need wiping, so skipping: '%s'",
570+ blockdev)
571+ else:
572+ LOG.info('wiping superblock on %s', blockdev)
573+ block.wipe_volume(blockdev, mode='superblock')
574+
575+
576+def identify_lvm(device):
577+ """
578+ determine if specified device is a lvm device
579+ """
580+ return (block.path_to_kname(device).startswith('dm') and
581+ get_dmsetup_uuid(device).startswith('LVM'))
582+
583+
584+def identify_crypt(device):
585+ """
586+ determine if specified device is dm-crypt device
587+ """
588+ return (block.path_to_kname(device).startswith('dm') and
589+ get_dmsetup_uuid(device).startswith('CRYPT'))
590+
591+
592+def identify_mdadm(device):
593+ """
594+ determine if specified device is a mdadm device
595+ """
596+ return block.path_to_kname(device).startswith('md')
597+
598+
599+def identify_bcache(device):
600+ """
601+ determine if specified device is a bcache device
602+ """
603+ return block.path_to_kname(device).startswith('bcache')
604+
605+
606+def identify_partition(device):
607+ """
608+ determine if specified device is a partition
609+ """
610+ path = os.path.join(block.sys_block_path(device), 'partition')
611+ return os.path.exists(path)
612+
613+
614+def get_holders(device):
615+ """
616+ Look up any block device holders, return list of knames
617+ """
618+ # block.sys_block_path works when given a /sys or /dev path
619+ sysfs_path = block.sys_block_path(device)
620+ # get holders
621+ holders = os.listdir(os.path.join(sysfs_path, 'holders'))
622+ LOG.debug("devname '%s' had holders: %s", device, holders)
623+ return holders
624+
625+
626+def gen_holders_tree(device):
627+ """
628+ generate a tree representing the current storage hirearchy above 'device'
629+ """
630+ device = block.sys_block_path(device)
631+ dev_name = block.path_to_kname(device)
632+ # the holders for a device should consist of the devices in the holders/
633+ # dir in sysfs and any partitions on the device. this ensures that a
634+ # storage tree starting from a disk will include all devices holding the
635+ # disk's partitions
636+ holder_paths = ([block.sys_block_path(h) for h in get_holders(device)] +
637+ block.get_sysfs_partitions(device))
638+ # the DEV_TYPE registry contains a function under the key 'ident' for each
639+ # device type entry that returns true if the device passed to it is of the
640+ # correct type. there should never be a situation in which multiple
641+ # identify functions return true. therefore, it will always work to take
642+ # the device type with the first identify function that returns true as the
643+ # device type for the current device. in the event that no identify
644+ # functions return true, the device will be treated as a disk
645+ # (DEFAULT_DEV_TYPE). the identify function for disk never returns true.
646+ # the next() builtin in python will not raise a StopIteration exception if
647+ # there is a default value defined
648+ dev_type = next((k for k, v in DEV_TYPES.items() if v['ident'](device)),
649+ DEFAULT_DEV_TYPE)
650+ return {
651+ 'device': device, 'dev_type': dev_type, 'name': dev_name,
652+ 'holders': [gen_holders_tree(h) for h in holder_paths],
653+ }
654+
655+
656+def plan_shutdown_holder_trees(holders_trees):
657+ """
658+ plan best order to shut down holders in, taking into account high level
659+ storage layers that may have many devices below them
660+
661+ returns a sorted list of descriptions of storage config entries including
662+ their path in /sys/block and their dev type
663+
664+ can accept either a single storage tree or a list of storage trees assumed
665+ to start at an equal place in storage hirearchy (i.e. a list of trees
666+ starting from disk)
667+ """
668+ # holds a temporary registry of holders to allow cross references
669+ # key = device sysfs path, value = {} of priority level, shutdown function
670+ reg = {}
671+
672+ # normalize to list of trees
673+ if not isinstance(holders_trees, (list, tuple)):
674+ holders_trees = [holders_trees]
675+
676+ def flatten_holders_tree(tree, level=0):
677+ """
678+ add entries from holders tree to registry with level key corresponding
679+ to how many layers from raw disks the current device is at
680+ """
681+ device = tree['device']
682+
683+ # always go with highest level if current device has been
684+ # encountered already. since the device and everything above it is
685+ # re-added to the registry it ensures that any increase of level
686+ # required here will propagate down the tree
687+ # this handles a scenario like mdadm + bcache, where the backing
688+ # device for bcache is a 3nd level item like mdadm, but the cache
689+ # device is 1st level (disk) or second level (partition), ensuring
690+ # that the bcache item is always considered higher level than
691+ # anything else regardless of whether it was added to the tree via
692+ # the cache device or backing device first
693+ if device in reg:
694+ level = max(reg[device]['level'], level)
695+
696+ reg[device] = {'level': level, 'device': device,
697+ 'dev_type': tree['dev_type']}
698+
699+ # handle holders above this level
700+ for holder in tree['holders']:
701+ flatten_holders_tree(holder, level=level + 1)
702+
703+ # flatten the holders tree into the registry
704+ for holders_tree in holders_trees:
705+ flatten_holders_tree(holders_tree)
706+
707+ # return list of entry dicts with highest level first
708+ return [reg[k] for k in sorted(reg, key=lambda x: reg[x]['level'] * -1)]
709+
710+
711+def format_holders_tree(holders_tree):
712+ """
713+ draw a nice dirgram of the holders tree
714+ """
715+ # spacer styles based on output of 'tree --charset=ascii'
716+ spacers = (('`-- ', ' ' * 4), ('|-- ', '|' + ' ' * 3))
717+
718+ def format_tree(tree):
719+ """
720+ format entry and any subentries
721+ """
722+ result = [tree['name']]
723+ holders = tree['holders']
724+ for (holder_no, holder) in enumerate(holders):
725+ spacer_style = spacers[min(len(holders) - (holder_no + 1), 1)]
726+ subtree_lines = format_tree(holder)
727+ for (line_no, line) in enumerate(subtree_lines):
728+ result.append(spacer_style[min(line_no, 1)] + line)
729+ return result
730+
731+ return '\n'.join(format_tree(holders_tree))
732+
733+
734+def get_holder_types(tree):
735+ """
736+ get flattened list of types of holders in holders tree and the devices
737+ they correspond to
738+ """
739+ types = {(tree['dev_type'], tree['device'])}
740+ for holder in tree['holders']:
741+ types.update(get_holder_types(holder))
742+ return types
743+
744+
745+def assert_clear(base_paths):
746+ """
747+ Check if all paths in base_paths are clear to use
748+ """
749+ valid = ('disk', 'partition')
750+ if not isinstance(base_paths, (list, tuple)):
751+ base_paths = [base_paths]
752+ base_paths = [block.sys_block_path(path) for path in base_paths]
753+ for holders_tree in [gen_holders_tree(p) for p in base_paths]:
754+ if any(holder_type not in valid and path not in base_paths
755+ for (holder_type, path) in get_holder_types(holders_tree)):
756+ raise OSError('Storage not clear, remaining:\n{}'
757+ .format(format_holders_tree(holders_tree)))
758+
759+
760+def clear_holders(base_paths, try_preserve=False):
761+ """
762+ Clear all storage layers depending on the devices specified in 'base_paths'
763+ A single device or list of devices can be specified.
764+ Device paths can be specified either as paths in /dev or /sys/block
765+ Will throw OSError if any holders could not be shut down
766+ """
767+ # handle single path
768+ if not isinstance(base_paths, (list, tuple)):
769+ base_paths = [base_paths]
770+
771+ # get current holders and plan how to shut them down
772+ holder_trees = [gen_holders_tree(path) for path in base_paths]
773+ LOG.info('Current device storage tree:\n%s',
774+ '\n'.join(format_holders_tree(tree) for tree in holder_trees))
775+ ordered_devs = plan_shutdown_holder_trees(holder_trees)
776+
777+ # run shutdown functions
778+ for dev_info in ordered_devs:
779+ dev_type = DEV_TYPES.get(dev_info['dev_type'])
780+ shutdown_function = dev_type.get('shutdown')
781+ if not shutdown_function:
782+ continue
783+ if try_preserve and shutdown_function in DATA_DESTROYING_HANDLERS:
784+ LOG.info('shutdown function for holder type: %s is destructive. '
785+ 'attempting to preserve data, so not skipping' %
786+ dev_info['dev_type'])
787+ continue
788+ LOG.info("shutdown running on holder type: '%s' syspath: '%s'",
789+ dev_info['dev_type'], dev_info['device'])
790+ shutdown_function(dev_info['device'])
791+ udev.udevadm_settle()
792+
793+
794+def start_clear_holders_deps():
795+ """
796+ prepare system for clear holders to be able to scan old devices
797+ """
798+ # a mdadm scan has to be started in case there is a md device that needs to
799+ # be detected. if the scan fails, it is either because there are no mdadm
800+ # devices on the system, or because there is a mdadm device in a damaged
801+ # state that could not be started. due to the nature of mdadm tools, it is
802+ # difficult to know which is the case. if any errors did occur, then ignore
803+ # them, since no action needs to be taken if there were no mdadm devices on
804+ # the system, and in the case where there is some mdadm metadata on a disk,
805+ # but there was not enough to start the array, the call to wipe_volume on
806+ # all disks and partitions should be sufficient to remove the mdadm
807+ # metadata
808+ block.mdadm.mdadm_assemble(scan=True, ignore_errors=True)
809+ # the bcache module needs to be present to properly detect bcache devs
810+ # on some systems (precise without hwe kernel) it may not be possible to
811+ # lad the bcache module bcause it is not present in the kernel. if this
812+ # happens then there is no need to halt installation, as the bcache devices
813+ # will never appear and will never prevent the disk from being reformatted
814+ util.subp(['modprobe', 'bcache'], rcs=[0, 1])
815+
816+
817+# anything that is not identified can assumed to be a 'disk' or similar
818+DEFAULT_DEV_TYPE = 'disk'
819+# handlers that should not be run if an attempt is being made to preserve data
820+DATA_DESTROYING_HANDLERS = [wipe_superblock]
821+# types of devices that could be encountered by clear holders and functions to
822+# identify them and shut them down
823+DEV_TYPES = _define_handlers_registry()
824
825=== added file 'curtin/block/lvm.py'
826--- curtin/block/lvm.py 1970-01-01 00:00:00 +0000
827+++ curtin/block/lvm.py 2016-09-29 18:49:30 +0000
828@@ -0,0 +1,96 @@
829+# Copyright (C) 2016 Canonical Ltd.
830+#
831+# Author: Wesley Wiedenmeier <wesley.wiedenmeier@canonical.com>
832+#
833+# Curtin is free software: you can redistribute it and/or modify it under
834+# the terms of the GNU Affero General Public License as published by the
835+# Free Software Foundation, either version 3 of the License, or (at your
836+# option) any later version.
837+#
838+# Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
839+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
840+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for
841+# more details.
842+#
843+# You should have received a copy of the GNU Affero General Public License
844+# along with Curtin. If not, see <http://www.gnu.org/licenses/>.
845+
846+"""
847+This module provides some helper functions for manipulating lvm devices
848+"""
849+
850+from curtin import util
851+from curtin.log import LOG
852+import os
853+
854+# separator to use for lvm/dm tools
855+_SEP = '='
856+
857+
858+def _filter_lvm_info(lvtool, match_field, query_field, match_key):
859+ """
860+ filter output of pv/vg/lvdisplay tools
861+ """
862+ (out, _) = util.subp([lvtool, '-C', '--separator', _SEP, '--noheadings',
863+ '-o', ','.join([match_field, query_field])],
864+ capture=True)
865+ return [qf for (mf, qf) in
866+ [l.strip().split(_SEP) for l in out.strip().splitlines()]
867+ if mf == match_key]
868+
869+
870+def get_pvols_in_volgroup(vg_name):
871+ """
872+ get physical volumes used by volgroup
873+ """
874+ return _filter_lvm_info('pvdisplay', 'vg_name', 'pv_name', vg_name)
875+
876+
877+def get_lvols_in_volgroup(vg_name):
878+ """
879+ get logical volumes in volgroup
880+ """
881+ return _filter_lvm_info('lvdisplay', 'vg_name', 'lv_name', vg_name)
882+
883+
884+def split_lvm_name(full):
885+ """
886+ split full lvm name into tuple of (volgroup, lv_name)
887+ """
888+ # 'dmsetup splitname' is the authoratative source for lvm name parsing
889+ (out, _) = util.subp(['dmsetup', 'splitname', full, '-c', '--noheadings',
890+ '--separator', _SEP, '-o', 'vg_name,lv_name'],
891+ capture=True)
892+ return out.strip().split(_SEP)
893+
894+
895+def lvmetad_running():
896+ """
897+ check if lvmetad is running
898+ """
899+ return os.path.exists(os.environ.get('LVM_LVMETAD_PIDFILE',
900+ '/run/lvmetad.pid'))
901+
902+
903+def lvm_scan():
904+ """
905+ run full scan for volgroups, logical volumes and physical volumes
906+ """
907+ # the lvm tools lvscan, vgscan and pvscan on ubuntu precise do not
908+ # support the flag --cache. the flag is present for the tools in ubuntu
909+ # trusty and later. since lvmetad is used in current releases of
910+ # ubuntu, the --cache flag is needed to ensure that the data cached by
911+ # lvmetad is updated.
912+
913+ # before appending the cache flag though, check if lvmetad is running. this
914+ # ensures that we do the right thing even if lvmetad is supported but is
915+ # not running
916+ release = util.lsb_release().get('codename')
917+ if release in [None, 'UNAVAILABLE']:
918+ LOG.warning('unable to find release number, assuming xenial or later')
919+ release = 'xenial'
920+
921+ for cmd in [['pvscan'], ['vgscan', '--mknodes']]:
922+ if release != 'precise' and lvmetad_running():
923+ cmd.append('--cache')
924+ util.subp(cmd, capture=True)
925
926=== modified file 'curtin/block/mdadm.py'
927--- curtin/block/mdadm.py 2016-05-05 16:43:40 +0000
928+++ curtin/block/mdadm.py 2016-09-29 18:49:30 +0000
929@@ -28,7 +28,7 @@
930 from subprocess import CalledProcessError
931
932 from curtin.block import (dev_short, dev_path, is_valid_device, sys_block_path)
933-from curtin import util
934+from curtin import (util, udev)
935 from curtin.log import LOG
936
937 NOSPARE_RAID_LEVELS = [
938@@ -117,21 +117,34 @@
939 #
940
941
942-def mdadm_assemble(md_devname=None, devices=[], spares=[], scan=False):
943+def mdadm_assemble(md_devname=None, devices=[], spares=[], scan=False,
944+ ignore_errors=False):
945 # md_devname is a /dev/XXXX
946 # devices is non-empty list of /dev/xxx
947 # if spares is non-empt list append of /dev/xxx
948 cmd = ["mdadm", "--assemble"]
949 if scan:
950- cmd += ['--scan']
951+ cmd += ['--scan', '-v']
952 else:
953 valid_mdname(md_devname)
954 cmd += [md_devname, "--run"] + devices
955 if spares:
956 cmd += spares
957
958- util.subp(cmd, capture=True, rcs=[0, 1, 2])
959- util.subp(["udevadm", "settle"])
960+ try:
961+ # mdadm assemble returns 1 when no arrays are found. this might not be
962+ # an error depending on the situation this function was called in, so
963+ # accept a return code of 1
964+ # mdadm assemble returns 2 when called on an array that is already
965+ # assembled. this is not an error, so accept return code of 2
966+ # all other return codes can be accepted with ignore_error set to true
967+ util.subp(cmd, capture=True, rcs=[0, 1, 2])
968+ except util.ProcessExecutionError:
969+ LOG.warning("mdadm_assemble had unexpected return code")
970+ if not ignore_errors:
971+ raise
972+
973+ udev.udevadm_settle()
974
975
976 def mdadm_create(md_devname, raidlevel, devices, spares=None, md_name=""):
977
978=== modified file 'curtin/commands/apply_net.py'
979--- curtin/commands/apply_net.py 2016-05-05 16:43:40 +0000
980+++ curtin/commands/apply_net.py 2016-09-29 18:49:30 +0000
981@@ -26,6 +26,57 @@
982
983 LOG = log.LOG
984
985+IFUPDOWN_IPV6_MTU_PRE_HOOK = """#!/bin/bash -e
986+# injected by curtin installer
987+
988+[ "${IFACE}" != "lo" ] || exit 0
989+
990+# Trigger only if MTU configured
991+[ -n "${IF_MTU}" ] || exit 0
992+
993+read CUR_DEV_MTU </sys/class/net/${IFACE}/mtu ||:
994+read CUR_IPV6_MTU </proc/sys/net/ipv6/conf/${IFACE}/mtu ||:
995+[ -n "${CUR_DEV_MTU}" ] && echo ${CUR_DEV_MTU} > /run/network/${IFACE}_dev.mtu
996+[ -n "${CUR_IPV6_MTU}" ] &&
997+ echo ${CUR_IPV6_MTU} > /run/network/${IFACE}_ipv6.mtu
998+exit 0
999+"""
1000+
1001+IFUPDOWN_IPV6_MTU_POST_HOOK = """#!/bin/bash -e
1002+# injected by curtin installer
1003+
1004+[ "${IFACE}" != "lo" ] || exit 0
1005+
1006+# Trigger only if MTU configured
1007+[ -n "${IF_MTU}" ] || exit 0
1008+
1009+read PRE_DEV_MTU </run/network/${IFACE}_dev.mtu ||:
1010+read CUR_DEV_MTU </sys/class/net/${IFACE}/mtu ||:
1011+read PRE_IPV6_MTU </run/network/${IFACE}_ipv6.mtu ||:
1012+read CUR_IPV6_MTU </proc/sys/net/ipv6/conf/${IFACE}/mtu ||:
1013+
1014+if [ "${ADDRFAM}" = "inet6" ]; then
1015+ # We need to check the underlying interface MTU and
1016+ # raise it if the IPV6 mtu is larger
1017+ if [ ${CUR_DEV_MTU} -lt ${IF_MTU} ]; then
1018+ ip link set ${IFACE} mtu ${IF_MTU}
1019+ fi
1020+ # sysctl -q -e -w net.ipv6.conf.${IFACE}.mtu=${IF_MTU}
1021+ echo ${IF_MTU} >/proc/sys/net/ipv6/conf/${IFACE}/mtu ||:
1022+
1023+elif [ "${ADDRFAM}" = "inet" ]; then
1024+ # handle the clobber case where inet mtu changes v6 mtu.
1025+ # ifupdown will already have set dev mtu, so lower mtu
1026+ # if needed. If v6 mtu was larger, it get's clamped down
1027+ # to the dev MTU value.
1028+ if [ ${PRE_IPV6_MTU} -lt ${CUR_IPV6_MTU} ]; then
1029+ # sysctl -q -e -w net.ipv6.conf.${IFACE}.mtu=${PRE_IPV6_MTU}
1030+ echo ${PRE_IPV6_MTU} >/proc/sys/net/ipv6/conf/${IFACE}/mtu ||:
1031+ fi
1032+fi
1033+exit 0
1034+"""
1035+
1036
1037 def apply_net(target, network_state=None, network_config=None):
1038 if network_state is None and network_config is None:
1039@@ -45,6 +96,108 @@
1040
1041 net.render_network_state(target=target, network_state=ns)
1042
1043+ _maybe_remove_legacy_eth0(target)
1044+ LOG.info('Attempting to remove ipv6 privacy extensions')
1045+ _disable_ipv6_privacy_extensions(target)
1046+ _patch_ifupdown_ipv6_mtu_hook(target)
1047+
1048+
1049+def _patch_ifupdown_ipv6_mtu_hook(target,
1050+ prehookfn="etc/network/if-pre-up.d/mtuipv6",
1051+ posthookfn="etc/network/if-up.d/mtuipv6"):
1052+
1053+ contents = {
1054+ 'prehook': IFUPDOWN_IPV6_MTU_PRE_HOOK,
1055+ 'posthook': IFUPDOWN_IPV6_MTU_POST_HOOK,
1056+ }
1057+
1058+ hookfn = {
1059+ 'prehook': prehookfn,
1060+ 'posthook': posthookfn,
1061+ }
1062+
1063+ for hook in ['prehook', 'posthook']:
1064+ fn = hookfn[hook]
1065+ cfg = util.target_path(target, path=fn)
1066+ LOG.info('Injecting fix for ipv6 mtu settings: %s', cfg)
1067+ util.write_file(cfg, contents[hook], mode=0o755)
1068+
1069+
1070+def _disable_ipv6_privacy_extensions(target,
1071+ path="etc/sysctl.d/10-ipv6-privacy.conf"):
1072+
1073+ """Ubuntu server image sets a preference to use IPv6 privacy extensions
1074+ by default; this races with the cloud-image desire to disable them.
1075+ Resolve this by allowing the cloud-image setting to win. """
1076+
1077+ cfg = util.target_path(target, path=path)
1078+ if not os.path.exists(cfg):
1079+ LOG.warn('Failed to find ipv6 privacy conf file %s', cfg)
1080+ return
1081+
1082+ bmsg = "Disabling IPv6 privacy extensions config may not apply."
1083+ try:
1084+ contents = util.load_file(cfg)
1085+ known_contents = ["net.ipv6.conf.all.use_tempaddr = 2",
1086+ "net.ipv6.conf.default.use_tempaddr = 2"]
1087+ lines = [f.strip() for f in contents.splitlines()
1088+ if not f.startswith("#")]
1089+ if lines == known_contents:
1090+ LOG.info('deleting file: %s', cfg)
1091+ util.del_file(cfg)
1092+ msg = "removed %s with known contents" % cfg
1093+ curtin_contents = '\n'.join(
1094+ ["# IPv6 Privacy Extensions (RFC 4941)",
1095+ "# Disabled by curtin",
1096+ "# net.ipv6.conf.all.use_tempaddr = 2",
1097+ "# net.ipv6.conf.default.use_tempaddr = 2"])
1098+ util.write_file(cfg, curtin_contents)
1099+ else:
1100+ LOG.info('skipping, content didnt match')
1101+ LOG.debug("found content:\n%s", lines)
1102+ LOG.debug("expected contents:\n%s", known_contents)
1103+ msg = (bmsg + " '%s' exists with user configured content." % cfg)
1104+ except:
1105+ msg = bmsg + " %s exists, but could not be read." % cfg
1106+ LOG.exception(msg)
1107+ return
1108+
1109+
1110+def _maybe_remove_legacy_eth0(target,
1111+ path="etc/network/interfaces.d/eth0.cfg"):
1112+ """Ubuntu cloud images previously included a 'eth0.cfg' that had
1113+ hard coded content. That file would interfere with the rendered
1114+ configuration if it was present.
1115+
1116+ if the file does not exist do nothing.
1117+ If the file exists:
1118+ - with known content, remove it and warn
1119+ - with unknown content, leave it and warn
1120+ """
1121+
1122+ cfg = util.target_path(target, path=path)
1123+ if not os.path.exists(cfg):
1124+ LOG.warn('Failed to find legacy network conf file %s', cfg)
1125+ return
1126+
1127+ bmsg = "Dynamic networking config may not apply."
1128+ try:
1129+ contents = util.load_file(cfg)
1130+ known_contents = ["auto eth0", "iface eth0 inet dhcp"]
1131+ lines = [f.strip() for f in contents.splitlines()
1132+ if not f.startswith("#")]
1133+ if lines == known_contents:
1134+ util.del_file(cfg)
1135+ msg = "removed %s with known contents" % cfg
1136+ else:
1137+ msg = (bmsg + " '%s' exists with user configured content." % cfg)
1138+ except:
1139+ msg = bmsg + " %s exists, but could not be read." % cfg
1140+ LOG.exception(msg)
1141+ return
1142+
1143+ LOG.warn(msg)
1144+
1145
1146 def apply_net_main(args):
1147 # curtin apply_net [--net-state=/config/netstate.yml] [--target=/]
1148@@ -76,8 +229,10 @@
1149 apply_net(target=state['target'],
1150 network_state=state['network_state'],
1151 network_config=state['network_config'])
1152+
1153 except Exception:
1154 LOG.exception('failed to apply network config')
1155+ return 1
1156
1157 LOG.info('Applied network configuration successfully')
1158 sys.exit(0)
1159@@ -90,7 +245,7 @@
1160 'metavar': 'NETSTATE', 'action': 'store',
1161 'default': os.environ.get('OUTPUT_NETWORK_STATE')}),
1162 (('-t', '--target'),
1163- {'help': ('target filesystem root to add swap file to. '
1164+ {'help': ('target filesystem root to configure networking to. '
1165 'default is env["TARGET_MOUNT_POINT"]'),
1166 'metavar': 'TARGET', 'action': 'store',
1167 'default': os.environ.get('TARGET_MOUNT_POINT')}),
1168
1169=== modified file 'curtin/commands/apt_config.py'
1170--- curtin/commands/apt_config.py 2016-08-05 20:47:14 +0000
1171+++ curtin/commands/apt_config.py 2016-09-29 18:49:30 +0000
1172@@ -271,17 +271,19 @@
1173 continue
1174
1175 # sources.list allow options in cols[1] which can have spaces
1176- # so the actual suite can be [2] or later
1177+ # so the actual suite can be [2] or later. example:
1178+ # deb [ arch=amd64,armel k=v ] http://example.com/debian
1179 cols = line.split()
1180- pcol = 2
1181- if cols[1].startswith("["):
1182- for col in cols[1:]:
1183- pcol += 1
1184- if col.endswith("]"):
1185- break
1186+ if len(cols) > 1:
1187+ pcol = 2
1188+ if cols[1].startswith("["):
1189+ for col in cols[1:]:
1190+ pcol += 1
1191+ if col.endswith("]"):
1192+ break
1193
1194- if cols[pcol] == releasesuite:
1195- line = '# suite disabled by curtin: %s' % line
1196+ if cols[pcol] == releasesuite:
1197+ line = '# suite disabled by curtin: %s' % line
1198 newsrc += line
1199 retsrc = newsrc
1200
1201
1202=== added file 'curtin/commands/block_info.py'
1203--- curtin/commands/block_info.py 1970-01-01 00:00:00 +0000
1204+++ curtin/commands/block_info.py 2016-09-29 18:49:30 +0000
1205@@ -0,0 +1,75 @@
1206+# Copyright (C) 2016 Canonical Ltd.
1207+#
1208+# Author: Wesley Wiedenmeier <wesley.wiedenmeier@canonical.com>
1209+#
1210+# Curtin is free software: you can redistribute it and/or modify it under
1211+# the terms of the GNU Affero General Public License as published by the
1212+# Free Software Foundation, either version 3 of the License, or (at your
1213+# option) any later version.
1214+#
1215+# Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
1216+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
1217+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for
1218+# more details.
1219+#
1220+# You should have received a copy of the GNU Affero General Public License
1221+# along with Curtin. If not, see <http://www.gnu.org/licenses/>.
1222+
1223+import os
1224+from . import populate_one_subcmd
1225+from curtin import (block, util)
1226+
1227+
1228+def block_info_main(args):
1229+ """get information about block devices, similar to lsblk"""
1230+ if not args.devices:
1231+ raise ValueError('devices to scan must be specified')
1232+ if not all(block.is_block_device(d) for d in args.devices):
1233+ raise ValueError('invalid device(s)')
1234+
1235+ def add_size_to_holders_tree(tree):
1236+ """add size information to generated holders trees"""
1237+ size_file = os.path.join(tree['device'], 'size')
1238+ # size file is always represented in 512 byte sectors even if
1239+ # underlying disk uses a larger logical_block_size
1240+ size = ((512 * int(util.load_file(size_file)))
1241+ if os.path.exists(size_file) else None)
1242+ tree['size'] = util.bytes2human(size) if args.human else str(size)
1243+ for holder in tree['holders']:
1244+ add_size_to_holders_tree(holder)
1245+ return tree
1246+
1247+ def format_name(tree):
1248+ """format information for human readable display"""
1249+ res = {
1250+ 'name': ' - '.join((tree['name'], tree['dev_type'], tree['size'])),
1251+ 'holders': []
1252+ }
1253+ for holder in tree['holders']:
1254+ res['holders'].append(format_name(holder))
1255+ return res
1256+
1257+ trees = [add_size_to_holders_tree(t) for t in
1258+ [block.clear_holders.gen_holders_tree(d) for d in args.devices]]
1259+
1260+ print(util.json_dumps(trees) if args.json else
1261+ '\n'.join(block.clear_holders.format_holders_tree(t) for t in
1262+ [format_name(tree) for tree in trees]))
1263+
1264+ return 0
1265+
1266+
1267+CMD_ARGUMENTS = (
1268+ ('devices',
1269+ {'help': 'devices to get info for', 'default': [], 'nargs': '+'}),
1270+ ('--human',
1271+ {'help': 'output size in human readable format', 'default': False,
1272+ 'action': 'store_true'}),
1273+ (('-j', '--json'),
1274+ {'help': 'output data in json format', 'default': False,
1275+ 'action': 'store_true'}),
1276+)
1277+
1278+
1279+def POPULATE_SUBCMD(parser):
1280+ populate_one_subcmd(parser, CMD_ARGUMENTS, block_info_main)
1281
1282=== modified file 'curtin/commands/block_meta.py'
1283--- curtin/commands/block_meta.py 2016-08-05 20:47:14 +0000
1284+++ curtin/commands/block_meta.py 2016-09-29 18:49:30 +0000
1285@@ -17,9 +17,8 @@
1286
1287 from collections import OrderedDict
1288 from curtin import (block, config, util)
1289-from curtin.block import mdadm
1290+from curtin.block import (mdadm, mkfs, clear_holders, lvm)
1291 from curtin.log import LOG
1292-from curtin.block import mkfs
1293 from curtin.reporter import events
1294
1295 from . import populate_one_subcmd
1296@@ -129,95 +128,6 @@
1297 return "mbr"
1298
1299
1300-def get_holders(devname):
1301- # Look up any block device holders.
1302- # Handle devices and partitions as devnames (vdb, md0, vdb7)
1303- devname_sysfs = block.sys_block_path(devname)
1304- if devname_sysfs:
1305- holders = os.listdir(os.path.join(devname_sysfs, 'holders'))
1306- LOG.debug("devname '%s' had holders: %s", devname, ','.join(holders))
1307- return holders
1308-
1309- LOG.debug('get_holders: did not find sysfs path for %s', devname)
1310- return []
1311-
1312-
1313-def clear_holders(sys_block_path):
1314- holders = os.listdir(os.path.join(sys_block_path, "holders"))
1315- LOG.info("clear_holders running on '%s', with holders '%s'" %
1316- (sys_block_path, holders))
1317- for holder in holders:
1318- # get path to holder in /sys/block, then clear it
1319- try:
1320- holder_realpath = os.path.realpath(
1321- os.path.join(sys_block_path, "holders", holder))
1322- clear_holders(holder_realpath)
1323- except IOError as e:
1324- # something might have already caused the holder to go away
1325- if util.is_file_not_found_exc(e):
1326- pass
1327- pass
1328-
1329- # detect what type of holder is using this volume and shut it down, need to
1330- # find more robust name of doing detection
1331- if "bcache" in sys_block_path:
1332- # bcache device
1333- part_devs = []
1334- for part_dev in glob.glob(os.path.join(sys_block_path,
1335- "slaves", "*", "dev")):
1336- with open(part_dev, "r") as fp:
1337- part_dev_id = fp.read().rstrip()
1338- part_devs.append(
1339- os.path.split(os.path.realpath(os.path.join("/dev/block",
1340- part_dev_id)))[-1])
1341- for cache_dev in glob.glob("/sys/fs/bcache/*/bdev*"):
1342- for part_dev in part_devs:
1343- if part_dev in os.path.realpath(cache_dev):
1344- # This is our bcache device, stop it, wait for udev to
1345- # settle
1346- with open(os.path.join(os.path.split(cache_dev)[0],
1347- "stop"), "w") as fp:
1348- LOG.info("stopping: %s" % fp)
1349- fp.write("1")
1350- udevadm_settle()
1351- break
1352- for part_dev in part_devs:
1353- block.wipe_volume(os.path.join("/dev", part_dev),
1354- mode="superblock")
1355-
1356- if os.path.exists(os.path.join(sys_block_path, "bcache")):
1357- # bcache device that isn't running, if it were, we would have found it
1358- # when we looked for holders
1359- try:
1360- with open(os.path.join(sys_block_path, "bcache", "set", "stop"),
1361- "w") as fp:
1362- LOG.info("stopping: %s" % fp)
1363- fp.write("1")
1364- except IOError as e:
1365- if not util.is_file_not_found_exc(e):
1366- raise e
1367- with open(os.path.join(sys_block_path, "bcache", "stop"),
1368- "w") as fp:
1369- LOG.info("stopping: %s" % fp)
1370- fp.write("1")
1371- udevadm_settle()
1372-
1373- if os.path.exists(os.path.join(sys_block_path, "md")):
1374- # md device
1375- block_dev_kname = block.path_to_kname(sys_block_path)
1376- block_dev = block.kname_to_path(block_dev_kname)
1377- mdadm.mdadm_stop(block_dev)
1378- mdadm.mdadm_remove(block_dev)
1379-
1380- elif os.path.exists(os.path.join(sys_block_path, "dm")):
1381- # Shut down any volgroups
1382- with open(os.path.join(sys_block_path, "dm", "name"), "r") as fp:
1383- name = fp.read().split('-')
1384- util.subp(["lvremove", "--force", name[0].rstrip(), name[1].rstrip()],
1385- rcs=[0, 5])
1386- util.subp(["vgremove", name[0].rstrip()], rcs=[0, 5, 6])
1387-
1388-
1389 def devsync(devpath):
1390 LOG.debug('devsync for %s', devpath)
1391 util.subp(['partprobe', devpath], rcs=[0, 1])
1392@@ -428,72 +338,35 @@
1393
1394
1395 def disk_handler(info, storage_config):
1396+ _dos_names = ['dos', 'msdos']
1397 ptable = info.get('ptable')
1398-
1399 disk = get_path_to_storage_volume(info.get('id'), storage_config)
1400
1401- # Handle preserve flag
1402- if info.get('preserve'):
1403- if not ptable:
1404- # Don't need to check state, return
1405- return
1406-
1407- # Check state of current ptable, try to do this using blkid, but if
1408- # blkid fails then try to fall back to using parted.
1409- _possible_errors = (util.ProcessExecutionError, StopIteration,
1410- IndexError, AttributeError)
1411- try:
1412- (out, _err) = util.subp(["blkid", "-o", "export", disk],
1413- capture=True)
1414- current_ptable = next(l.split('=')[1] for l in out.splitlines()
1415- if 'TYPE' in l)
1416- except _possible_errors:
1417- try:
1418- (out, _err) = util.subp(["parted", disk, "--script", "print"],
1419- capture=True)
1420- current_ptable = next(l.split()[-1] for l in out.splitlines()
1421- if "Partition Table" in l)
1422- except _possible_errors:
1423- raise ValueError("disk '%s' has no readable partition table "
1424- "or cannot be accessed, but preserve is set "
1425- "to true, so cannot continue" % disk)
1426- if not (current_ptable == ptable or
1427- (current_ptable == "dos" and ptable == "msdos")):
1428- raise ValueError("disk '%s' does not have correct "
1429- "partition table, but preserve is "
1430- "set to true, so not creating table."
1431- % info.get('id'))
1432+ if config.value_as_boolean(info.get('preserve')):
1433+ # Handle preserve flag, verifying if ptable specified in config
1434+ if config.value_as_boolean(ptable):
1435+ current_ptable = block.get_part_table_type(disk)
1436+ if not ((ptable in _dos_names and current_ptable in _dos_names) or
1437+ (ptable == 'gpt' and current_ptable == 'gpt')):
1438+ raise ValueError(
1439+ "disk '%s' does not have correct partition table or "
1440+ "cannot be read, but preserve is set to true. "
1441+ "cannot continue installation." % info.get('id'))
1442 LOG.info("disk '%s' marked to be preserved, so keeping partition "
1443 "table" % disk)
1444- return
1445-
1446- # Wipe the disk
1447- if info.get('wipe') and info.get('wipe') != "none":
1448- # The disk has a lable, clear all partitions
1449- mdadm.mdadm_assemble(scan=True)
1450- disk_sysfs_path = block.sys_block_path(disk)
1451- sysfs_partitions = list(
1452- os.path.split(prt)[0] for prt in
1453- glob.glob(os.path.join(disk_sysfs_path, '*', 'partition')))
1454- for partition in sysfs_partitions:
1455- clear_holders(partition)
1456- with open(os.path.join(partition, "dev"), "r") as fp:
1457- block_no = fp.read().rstrip()
1458- partition_path = os.path.realpath(
1459- os.path.join("/dev/block", block_no))
1460- block.wipe_volume(partition_path, mode=info.get('wipe'))
1461-
1462- clear_holders(disk_sysfs_path)
1463- block.wipe_volume(disk, mode=info.get('wipe'))
1464-
1465- # Create partition table on disk
1466- if info.get('ptable'):
1467- LOG.info("labeling device: '%s' with '%s' partition table", disk,
1468- ptable)
1469- if ptable == "gpt":
1470- util.subp(["sgdisk", "--clear", disk])
1471- elif ptable == "msdos":
1472- util.subp(["parted", disk, "--script", "mklabel", "msdos"])
1473+ else:
1474+ # wipe the disk and create the partition table if instructed to do so
1475+ if config.value_as_boolean(info.get('wipe')):
1476+ block.wipe_volume(disk, mode=info.get('wipe'))
1477+ if config.value_as_boolean(ptable):
1478+ LOG.info("labeling device: '%s' with '%s' partition table", disk,
1479+ ptable)
1480+ if ptable == "gpt":
1481+ util.subp(["sgdisk", "--clear", disk])
1482+ elif ptable in _dos_names:
1483+ util.subp(["parted", disk, "--script", "mklabel", "msdos"])
1484+ else:
1485+ raise ValueError('invalid partition table type: %s', ptable)
1486
1487 # Make the name if needed
1488 if info.get('name'):
1489@@ -621,9 +494,9 @@
1490 length_sectors = length_sectors + (logdisks * alignment_offset)
1491
1492 # Handle preserve flag
1493- if info.get('preserve'):
1494+ if config.value_as_boolean(info.get('preserve')):
1495 return
1496- elif storage_config.get(device).get('preserve'):
1497+ elif config.value_as_boolean(storage_config.get(device).get('preserve')):
1498 raise NotImplementedError("Partition '%s' is not marked to be \
1499 preserved, but device '%s' is. At this time, preserving devices \
1500 but not also the partitions on the devices is not supported, \
1501@@ -666,11 +539,16 @@
1502 else:
1503 raise ValueError("parent partition has invalid partition table")
1504
1505- # Wipe the partition if told to do so
1506- if info.get('wipe') and info.get('wipe') != "none":
1507- block.wipe_volume(
1508- get_path_to_storage_volume(info.get('id'), storage_config),
1509- mode=info.get('wipe'))
1510+ # Wipe the partition if told to do so, do not wipe dos extended partitions
1511+ # as this may damage the extended partition table
1512+ if config.value_as_boolean(info.get('wipe')):
1513+ if info.get('flag') == "extended":
1514+ LOG.warn("extended partitions do not need wiping, so skipping: "
1515+ "'%s'" % info.get('id'))
1516+ else:
1517+ block.wipe_volume(
1518+ get_path_to_storage_volume(info.get('id'), storage_config),
1519+ mode=info.get('wipe'))
1520 # Make the name if needed
1521 if storage_config.get(device).get('name') and partition_type != 'extended':
1522 make_dname(info.get('id'), storage_config)
1523@@ -686,7 +564,7 @@
1524 volume_path = get_path_to_storage_volume(volume, storage_config)
1525
1526 # Handle preserve flag
1527- if info.get('preserve'):
1528+ if config.value_as_boolean(info.get('preserve')):
1529 # Volume marked to be preserved, not formatting
1530 return
1531
1532@@ -768,26 +646,21 @@
1533 storage_config))
1534
1535 # Handle preserve flag
1536- if info.get('preserve'):
1537+ if config.value_as_boolean(info.get('preserve')):
1538 # LVM will probably be offline, so start it
1539 util.subp(["vgchange", "-a", "y"])
1540 # Verify that volgroup exists and contains all specified devices
1541- current_paths = []
1542- (out, _err) = util.subp(["pvdisplay", "-C", "--separator", "=", "-o",
1543- "vg_name,pv_name", "--noheadings"],
1544- capture=True)
1545- for line in out.splitlines():
1546- if name in line:
1547- current_paths.append(line.split("=")[-1])
1548- if set(current_paths) != set(device_paths):
1549- raise ValueError("volgroup '%s' marked to be preserved, but does \
1550- not exist or does not contain the right physical \
1551- volumes" % info.get('id'))
1552+ if set(lvm.get_pvols_in_volgroup(name)) != set(device_paths):
1553+ raise ValueError("volgroup '%s' marked to be preserved, but does "
1554+ "not exist or does not contain the right "
1555+ "physical volumes" % info.get('id'))
1556 else:
1557 # Create vgrcreate command and run
1558- cmd = ["vgcreate", name]
1559- cmd.extend(device_paths)
1560- util.subp(cmd)
1561+ # capture output to avoid printing it to log
1562+ util.subp(['vgcreate', name] + device_paths, capture=True)
1563+
1564+ # refresh lvmetad
1565+ lvm.lvm_scan()
1566
1567
1568 def lvm_partition_handler(info, storage_config):
1569@@ -797,28 +670,23 @@
1570 raise ValueError("lvm volgroup for lvm partition must be specified")
1571 if not name:
1572 raise ValueError("lvm partition name must be specified")
1573+ if info.get('ptable'):
1574+ raise ValueError("Partition tables on top of lvm logical volumes is "
1575+ "not supported")
1576
1577 # Handle preserve flag
1578- if info.get('preserve'):
1579- (out, _err) = util.subp(["lvdisplay", "-C", "--separator", "=", "-o",
1580- "lv_name,vg_name", "--noheadings"],
1581- capture=True)
1582- found = False
1583- for line in out.splitlines():
1584- if name in line:
1585- if volgroup == line.split("=")[-1]:
1586- found = True
1587- break
1588- if not found:
1589- raise ValueError("lvm partition '%s' marked to be preserved, but \
1590- does not exist or does not mach storage \
1591- configuration" % info.get('id'))
1592+ if config.value_as_boolean(info.get('preserve')):
1593+ if name not in lvm.get_lvols_in_volgroup(volgroup):
1594+ raise ValueError("lvm partition '%s' marked to be preserved, but "
1595+ "does not exist or does not mach storage "
1596+ "configuration" % info.get('id'))
1597 elif storage_config.get(info.get('volgroup')).get('preserve'):
1598- raise NotImplementedError("Lvm Partition '%s' is not marked to be \
1599- preserved, but volgroup '%s' is. At this time, preserving \
1600- volgroups but not also the lvm partitions on the volgroup is \
1601- not supported, because of the possibility of damaging lvm \
1602- partitions intended to be preserved." % (info.get('id'), volgroup))
1603+ raise NotImplementedError(
1604+ "Lvm Partition '%s' is not marked to be preserved, but volgroup "
1605+ "'%s' is. At this time, preserving volgroups but not also the lvm "
1606+ "partitions on the volgroup is not supported, because of the "
1607+ "possibility of damaging lvm partitions intended to be "
1608+ "preserved." % (info.get('id'), volgroup))
1609 else:
1610 cmd = ["lvcreate", volgroup, "-n", name]
1611 if info.get('size'):
1612@@ -828,9 +696,8 @@
1613
1614 util.subp(cmd)
1615
1616- if info.get('ptable'):
1617- raise ValueError("Partition tables on top of lvm logical volumes is \
1618- not supported")
1619+ # refresh lvmetad
1620+ lvm.lvm_scan()
1621
1622 make_dname(info.get('id'), storage_config)
1623
1624@@ -917,7 +784,7 @@
1625 zip(spare_devices, spare_device_paths)))
1626
1627 # Handle preserve flag
1628- if info.get('preserve'):
1629+ if config.value_as_boolean(info.get('preserve')):
1630 # check if the array is already up, if not try to assemble
1631 if not mdadm.md_check(md_devname, raidlevel,
1632 device_paths, spare_device_paths):
1633@@ -973,9 +840,6 @@
1634 raise ValueError("backing device and cache device for bcache"
1635 " must be specified")
1636
1637- # The bcache module is not loaded when bcache is installed by apt-get, so
1638- # we will load it now
1639- util.subp(["modprobe", "bcache"])
1640 bcache_sysfs = "/sys/fs/bcache"
1641 udevadm_settle(exists=bcache_sysfs)
1642
1643@@ -995,7 +859,7 @@
1644 bcache_device, expected)
1645 return
1646 LOG.debug('bcache device path not found: %s', expected)
1647- local_holders = get_holders(bcache_device)
1648+ local_holders = clear_holders.get_holders(bcache_device)
1649 LOG.debug('got initial holders being "%s"', local_holders)
1650 if len(local_holders) == 0:
1651 raise ValueError("holders == 0 , expected non-zero")
1652@@ -1058,7 +922,7 @@
1653
1654 # via the holders we can identify which bcache device we just created
1655 # for a given backing device
1656- holders = get_holders(backing_device)
1657+ holders = clear_holders.get_holders(backing_device)
1658 if len(holders) != 1:
1659 err = ('Invalid number {} of holding devices:'
1660 ' "{}"'.format(len(holders), holders))
1661@@ -1150,6 +1014,21 @@
1662 # set up reportstack
1663 stack_prefix = state.get('report_stack_prefix', '')
1664
1665+ # shut down any already existing storage layers above any disks used in
1666+ # config that have 'wipe' set
1667+ with events.ReportEventStack(
1668+ name=stack_prefix, reporting_enabled=True, level='INFO',
1669+ description="removing previous storage devices"):
1670+ clear_holders.start_clear_holders_deps()
1671+ disk_paths = [get_path_to_storage_volume(k, storage_config_dict)
1672+ for (k, v) in storage_config_dict.items()
1673+ if v.get('type') == 'disk' and
1674+ config.value_as_boolean(v.get('wipe')) and
1675+ not config.value_as_boolean(v.get('preserve'))]
1676+ clear_holders.clear_holders(disk_paths)
1677+ # if anything was not properly shut down, stop installation
1678+ clear_holders.assert_clear(disk_paths)
1679+
1680 for item_id, command in storage_config_dict.items():
1681 handler = command_handlers.get(command['type'])
1682 if not handler:
1683
1684=== modified file 'curtin/commands/block_wipe.py'
1685--- curtin/commands/block_wipe.py 2016-08-05 20:47:14 +0000
1686+++ curtin/commands/block_wipe.py 2016-09-29 18:49:30 +0000
1687@@ -21,7 +21,6 @@
1688
1689
1690 def wipe_main(args):
1691- # curtin clear-holders device [device2 [device3]]
1692 for blockdev in args.devices:
1693 try:
1694 block.wipe_volume(blockdev, mode=args.mode)
1695
1696=== added file 'curtin/commands/clear_holders.py'
1697--- curtin/commands/clear_holders.py 1970-01-01 00:00:00 +0000
1698+++ curtin/commands/clear_holders.py 2016-09-29 18:49:30 +0000
1699@@ -0,0 +1,48 @@
1700+# Copyright (C) 2016 Canonical Ltd.
1701+#
1702+# Author: Wesley Wiedenmeier <wesley.wiedenmeier@canonical.com>
1703+#
1704+# Curtin is free software: you can redistribute it and/or modify it under
1705+# the terms of the GNU Affero General Public License as published by the
1706+# Free Software Foundation, either version 3 of the License, or (at your
1707+# option) any later version.
1708+#
1709+# Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
1710+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
1711+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for
1712+# more details.
1713+#
1714+# You should have received a copy of the GNU Affero General Public License
1715+# along with Curtin. If not, see <http://www.gnu.org/licenses/>.
1716+
1717+from curtin import block
1718+from . import populate_one_subcmd
1719+
1720+
1721+def clear_holders_main(args):
1722+ """
1723+ wrapper for clear_holders accepting cli args
1724+ """
1725+ if (not all(block.is_block_device(device) for device in args.devices) or
1726+ len(args.devices) == 0):
1727+ raise ValueError('invalid devices specified')
1728+ block.clear_holders.start_clear_holders_deps()
1729+ block.clear_holders.clear_holders(args.devices, try_preserve=args.preserve)
1730+ if args.try_preserve:
1731+ print('ran clear_holders attempting to preserve data. however, '
1732+ 'hotplug support for some devices may cause holders to restart ')
1733+ block.clear_holders.assert_clear(args.devices)
1734+
1735+
1736+CMD_ARGUMENTS = (
1737+ (('devices',
1738+ {'help': 'devices to free', 'default': [], 'nargs': '+'}),
1739+ (('-p', '--preserve'),
1740+ {'help': 'try to shut down holders without erasing anything',
1741+ 'default': False, 'action': 'store_true'}),
1742+ )
1743+)
1744+
1745+
1746+def POPULATE_SUBCMD(parser):
1747+ populate_one_subcmd(parser, CMD_ARGUMENTS, clear_holders_main)
1748
1749=== modified file 'curtin/commands/curthooks.py'
1750--- curtin/commands/curthooks.py 2016-08-05 20:47:14 +0000
1751+++ curtin/commands/curthooks.py 2016-09-29 18:49:30 +0000
1752@@ -28,9 +28,8 @@
1753 from curtin.log import LOG
1754 from curtin import swap
1755 from curtin import util
1756-from curtin import net
1757 from curtin.reporter import events
1758-from curtin.commands import apt_config
1759+from curtin.commands import apply_net, apt_config
1760
1761 from . import populate_one_subcmd
1762
1763@@ -109,42 +108,6 @@
1764 shutil.move(local_conf, local_conf + ".old")
1765
1766
1767-def _maybe_remove_legacy_eth0(target,
1768- path="/etc/network/interfaces.d/eth0.cfg"):
1769- """Ubuntu cloud images previously included a 'eth0.cfg' that had
1770- hard coded content. That file would interfere with the rendered
1771- configuration if it was present.
1772-
1773- if the file does not exist do nothing.
1774- If the file exists:
1775- - with known content, remove it and warn
1776- - with unknown content, leave it and warn
1777- """
1778-
1779- cfg = os.path.sep.join([target, path])
1780- if not os.path.exists(cfg):
1781- LOG.warn('Failed to find legacy conf file %s', cfg)
1782- return
1783-
1784- bmsg = "Dynamic networking config may not apply."
1785- try:
1786- contents = util.load_file(cfg)
1787- known_contents = ["auto eth0", "iface eth0 inet dhcp"]
1788- lines = [f.strip() for f in contents.splitlines()
1789- if not f.startswith("#")]
1790- if lines == known_contents:
1791- util.del_file(cfg)
1792- msg = "removed %s with known contents" % cfg
1793- else:
1794- msg = (bmsg + " '%s' exists with user configured content." % cfg)
1795- except:
1796- msg = bmsg + " %s exists, but could not be read." % cfg
1797- LOG.exception(msg)
1798- return
1799-
1800- LOG.warn(msg)
1801-
1802-
1803 def setup_zipl(cfg, target):
1804 if platform.machine() != 's390x':
1805 return
1806@@ -411,7 +374,6 @@
1807
1808
1809 def apply_networking(target, state):
1810- netstate = state.get('network_state')
1811 netconf = state.get('network_config')
1812 interfaces = state.get('interfaces')
1813
1814@@ -422,22 +384,13 @@
1815 return True
1816 return False
1817
1818- ns = None
1819- if is_valid_src(netstate):
1820- LOG.debug("applying network_state")
1821- ns = net.network_state.from_state_file(netstate)
1822- elif is_valid_src(netconf):
1823- LOG.debug("applying network_config")
1824- ns = net.parse_net_config(netconf)
1825-
1826- if ns is not None:
1827- net.render_network_state(target=target, network_state=ns)
1828+ if is_valid_src(netconf):
1829+ LOG.info("applying network_config")
1830+ apply_net.apply_net(target, network_state=None, network_config=netconf)
1831 else:
1832 LOG.debug("copying interfaces")
1833 copy_interfaces(interfaces, target)
1834
1835- _maybe_remove_legacy_eth0(target)
1836-
1837
1838 def copy_interfaces(interfaces, target):
1839 if not interfaces:
1840
1841=== modified file 'curtin/commands/main.py'
1842--- curtin/commands/main.py 2016-08-05 20:47:14 +0000
1843+++ curtin/commands/main.py 2016-09-29 18:49:30 +0000
1844@@ -26,9 +26,10 @@
1845 from ..deps import install_deps
1846
1847 SUB_COMMAND_MODULES = [
1848- 'apply_net', 'block-meta', 'block-wipe', 'curthooks', 'extract',
1849- 'hook', 'in-target', 'install', 'mkfs', 'net-meta', 'apt-config',
1850- 'pack', 'swap', 'system-install', 'system-upgrade']
1851+ 'apply_net', 'block-info', 'block-meta', 'block-wipe', 'curthooks',
1852+ 'clear-holders', 'extract', 'hook', 'in-target', 'install', 'mkfs',
1853+ 'net-meta', 'apt-config', 'pack', 'swap', 'system-install',
1854+ 'system-upgrade']
1855
1856
1857 def add_subcmd(subparser, subcmd):
1858
1859=== modified file 'curtin/config.py'
1860--- curtin/config.py 2016-03-18 14:16:45 +0000
1861+++ curtin/config.py 2016-09-29 18:49:30 +0000
1862@@ -138,6 +138,5 @@
1863
1864
1865 def value_as_boolean(value):
1866- if value in (False, None, '0', 0, 'False', 'false', ''):
1867- return False
1868- return True
1869+ false_values = (False, None, 0, '0', 'False', 'false', 'None', 'none', '')
1870+ return value not in false_values
1871
1872=== modified file 'curtin/net/__init__.py'
1873--- curtin/net/__init__.py 2016-07-12 16:28:59 +0000
1874+++ curtin/net/__init__.py 2016-09-29 18:49:30 +0000
1875@@ -299,7 +299,7 @@
1876 mac = iface.get('mac_address', '')
1877 # len(macaddr) == 2 * 6 + 5 == 17
1878 if ifname and mac and len(mac) == 17:
1879- content += generate_udev_rule(ifname, mac)
1880+ content += generate_udev_rule(ifname, mac.lower())
1881
1882 return content
1883
1884@@ -349,7 +349,7 @@
1885 'subnets',
1886 'type',
1887 ]
1888- if iface['type'] not in ['bond', 'bridge']:
1889+ if iface['type'] not in ['bond', 'bridge', 'vlan']:
1890 ignore_map.append('mac_address')
1891
1892 for key, value in iface.items():
1893@@ -361,26 +361,52 @@
1894 return content
1895
1896
1897-def render_route(route):
1898- content = "up route add"
1899+def render_route(route, indent=""):
1900+ """When rendering routes for an iface, in some cases applying a route
1901+ may result in the route command returning non-zero which produces
1902+ some confusing output for users manually using ifup/ifdown[1]. To
1903+ that end, we will optionally include an '|| true' postfix to each
1904+ route line allowing users to work with ifup/ifdown without using
1905+ --force option.
1906+
1907+ We may at somepoint not want to emit this additional postfix, and
1908+ add a 'strict' flag to this function. When called with strict=True,
1909+ then we will not append the postfix.
1910+
1911+ 1. http://askubuntu.com/questions/168033/
1912+ how-to-set-static-routes-in-ubuntu-server
1913+ """
1914+ content = []
1915+ up = indent + "post-up route add"
1916+ down = indent + "pre-down route del"
1917+ or_true = " || true"
1918 mapping = {
1919 'network': '-net',
1920 'netmask': 'netmask',
1921 'gateway': 'gw',
1922 'metric': 'metric',
1923 }
1924- for k in ['network', 'netmask', 'gateway', 'metric']:
1925- if k in route:
1926- content += " %s %s" % (mapping[k], route[k])
1927-
1928- content += '\n'
1929- return content
1930-
1931-
1932-def iface_start_entry(iface, index):
1933+ if route['network'] == '0.0.0.0' and route['netmask'] == '0.0.0.0':
1934+ default_gw = " default gw %s" % route['gateway']
1935+ content.append(up + default_gw + or_true)
1936+ content.append(down + default_gw + or_true)
1937+ elif route['network'] == '::' and route['netmask'] == 0:
1938+ # ipv6!
1939+ default_gw = " -A inet6 default gw %s" % route['gateway']
1940+ content.append(up + default_gw + or_true)
1941+ content.append(down + default_gw + or_true)
1942+ else:
1943+ route_line = ""
1944+ for k in ['network', 'netmask', 'gateway', 'metric']:
1945+ if k in route:
1946+ route_line += " %s %s" % (mapping[k], route[k])
1947+ content.append(up + route_line + or_true)
1948+ content.append(down + route_line + or_true)
1949+ return "\n".join(content)
1950+
1951+
1952+def iface_start_entry(iface):
1953 fullname = iface['name']
1954- if index != 0:
1955- fullname += ":%s" % index
1956
1957 control = iface['control']
1958 if control == "auto":
1959@@ -397,6 +423,16 @@
1960 "iface {fullname} {inet} {mode}\n").format(**subst)
1961
1962
1963+def subnet_is_ipv6(subnet):
1964+ # 'static6' or 'dhcp6'
1965+ if subnet['type'].endswith('6'):
1966+ # This is a request for DHCPv6.
1967+ return True
1968+ elif subnet['type'] == 'static' and ":" in subnet['address']:
1969+ return True
1970+ return False
1971+
1972+
1973 def render_interfaces(network_state):
1974 ''' Given state, emit etc/network/interfaces content '''
1975
1976@@ -424,42 +460,43 @@
1977 content += "\n"
1978 subnets = iface.get('subnets', {})
1979 if subnets:
1980- for index, subnet in zip(range(0, len(subnets)), subnets):
1981+ for index, subnet in enumerate(subnets):
1982 if content[-2:] != "\n\n":
1983 content += "\n"
1984 iface['index'] = index
1985 iface['mode'] = subnet['type']
1986 iface['control'] = subnet.get('control', 'auto')
1987 subnet_inet = 'inet'
1988- if iface['mode'].endswith('6'):
1989- # This is a request for DHCPv6.
1990- subnet_inet += '6'
1991- elif iface['mode'] == 'static' and ":" in subnet['address']:
1992- # This is a static IPv6 address.
1993+ if subnet_is_ipv6(subnet):
1994 subnet_inet += '6'
1995 iface['inet'] = subnet_inet
1996- if iface['mode'].startswith('dhcp'):
1997+ if subnet['type'].startswith('dhcp'):
1998 iface['mode'] = 'dhcp'
1999
2000- content += iface_start_entry(iface, index)
2001+ # do not emit multiple 'auto $IFACE' lines as older (precise)
2002+ # ifupdown complains
2003+ if "auto %s\n" % (iface['name']) in content:
2004+ iface['control'] = 'alias'
2005+
2006+ content += iface_start_entry(iface)
2007 content += iface_add_subnet(iface, subnet)
2008 content += iface_add_attrs(iface, index)
2009- if len(subnets) > 1 and index == 0:
2010- for i in range(1, len(subnets)):
2011- content += " post-up ifup %s:%s\n" % (iface['name'],
2012- i)
2013+
2014+ for route in subnet.get('routes', []):
2015+ content += render_route(route, indent=" ") + '\n'
2016+
2017 else:
2018 # ifenslave docs say to auto the slave devices
2019- if 'bond-master' in iface:
2020+ if 'bond-master' in iface or 'bond-slaves' in iface:
2021 content += "auto {name}\n".format(**iface)
2022 content += "iface {name} {inet} {mode}\n".format(**iface)
2023- content += iface_add_attrs(iface, index)
2024+ content += iface_add_attrs(iface, 0)
2025
2026 for route in network_state.get('routes'):
2027 content += render_route(route)
2028
2029 # global replacements until v2 format
2030- content = content.replace('mac_address', 'hwaddress')
2031+ content = content.replace('mac_address', 'hwaddress ether')
2032
2033 # Play nice with others and source eni config files
2034 content += "\nsource /etc/network/interfaces.d/*.cfg\n"
2035
2036=== modified file 'curtin/net/network_state.py'
2037--- curtin/net/network_state.py 2015-10-02 16:19:07 +0000
2038+++ curtin/net/network_state.py 2016-09-29 18:49:30 +0000
2039@@ -121,6 +121,18 @@
2040 iface = interfaces.get(command['name'], {})
2041 for param, val in command.get('params', {}).items():
2042 iface.update({param: val})
2043+
2044+ # convert subnet ipv6 netmask to cidr as needed
2045+ subnets = command.get('subnets')
2046+ if subnets:
2047+ for subnet in subnets:
2048+ if subnet['type'] == 'static':
2049+ if 'netmask' in subnet and ':' in subnet['address']:
2050+ subnet['netmask'] = mask2cidr(subnet['netmask'])
2051+ for route in subnet.get('routes', []):
2052+ if 'netmask' in route:
2053+ route['netmask'] = mask2cidr(route['netmask'])
2054+
2055 iface.update({
2056 'name': command.get('name'),
2057 'type': command.get('type'),
2058@@ -130,7 +142,7 @@
2059 'mtu': command.get('mtu'),
2060 'address': None,
2061 'gateway': None,
2062- 'subnets': command.get('subnets'),
2063+ 'subnets': subnets,
2064 })
2065 self.network_state['interfaces'].update({command.get('name'): iface})
2066 self.dump_network_state()
2067@@ -141,6 +153,7 @@
2068 iface eth0.222 inet static
2069 address 10.10.10.1
2070 netmask 255.255.255.0
2071+ hwaddress ether BC:76:4E:06:96:B3
2072 vlan-raw-device eth0
2073 '''
2074 required_keys = [
2075@@ -332,6 +345,37 @@
2076 return ".".join([str(x) for x in mask])
2077
2078
2079+def ipv4mask2cidr(mask):
2080+ if '.' not in mask:
2081+ return mask
2082+ return sum([bin(int(x)).count('1') for x in mask.split('.')])
2083+
2084+
2085+def ipv6mask2cidr(mask):
2086+ if ':' not in mask:
2087+ return mask
2088+
2089+ bitCount = [0, 0x8000, 0xc000, 0xe000, 0xf000, 0xf800, 0xfc00, 0xfe00,
2090+ 0xff00, 0xff80, 0xffc0, 0xffe0, 0xfff0, 0xfff8, 0xfffc,
2091+ 0xfffe, 0xffff]
2092+ cidr = 0
2093+ for word in mask.split(':'):
2094+ if not word or int(word, 16) == 0:
2095+ break
2096+ cidr += bitCount.index(int(word, 16))
2097+
2098+ return cidr
2099+
2100+
2101+def mask2cidr(mask):
2102+ if ':' in mask:
2103+ return ipv6mask2cidr(mask)
2104+ elif '.' in mask:
2105+ return ipv4mask2cidr(mask)
2106+ else:
2107+ return mask
2108+
2109+
2110 if __name__ == '__main__':
2111 import sys
2112 import random
2113
2114=== modified file 'curtin/util.py'
2115--- curtin/util.py 2016-08-05 20:47:14 +0000
2116+++ curtin/util.py 2016-09-29 18:49:30 +0000
2117@@ -143,6 +143,8 @@
2118 a list of times to sleep in between retries. After each failure
2119 subp will sleep for N seconds and then try again. A value of [1, 3]
2120 means to run, sleep 1, run, sleep 3, run and then return exit code.
2121+ :param target:
2122+ run the command as 'chroot target <args>'
2123 """
2124 retries = []
2125 if "retries" in kwargs:
2126@@ -302,15 +304,29 @@
2127
2128
2129 def write_file(filename, content, mode=0o644, omode="w"):
2130+ """
2131+ write 'content' to file at 'filename' using python open mode 'omode'.
2132+ if mode is not set, then chmod file to mode. mode is 644 by default
2133+ """
2134 ensure_dir(os.path.dirname(filename))
2135 with open(filename, omode) as fp:
2136 fp.write(content)
2137- os.chmod(filename, mode)
2138-
2139-
2140-def load_file(path, mode="r"):
2141+ if mode:
2142+ os.chmod(filename, mode)
2143+
2144+
2145+def load_file(path, mode="r", read_len=None, offset=0):
2146 with open(path, mode) as fp:
2147- return fp.read()
2148+ if offset:
2149+ fp.seek(offset)
2150+ return fp.read(read_len) if read_len else fp.read()
2151+
2152+
2153+def file_size(path):
2154+ """get the size of a file"""
2155+ with open(path, 'rb') as fp:
2156+ fp.seek(0, 2)
2157+ return fp.tell()
2158
2159
2160 def del_file(path):
2161@@ -853,6 +869,18 @@
2162 return val
2163
2164
2165+def bytes2human(size):
2166+ """convert size in bytes to human readable"""
2167+ if not (isinstance(size, (int, float)) and
2168+ int(size) == size and
2169+ int(size) >= 0):
2170+ raise ValueError('size must be a integral value')
2171+ mpliers = {'B': 1, 'K': 2 ** 10, 'M': 2 ** 20, 'G': 2 ** 30, 'T': 2 ** 40}
2172+ unit_order = sorted(mpliers, key=lambda x: -1 * mpliers[x])
2173+ unit = next((u for u in unit_order if (size / mpliers[u]) >= 1), 'B')
2174+ return str(int(size / mpliers[unit])) + unit
2175+
2176+
2177 def import_module(import_str):
2178 """Import a module."""
2179 __import__(import_str)
2180@@ -868,7 +896,9 @@
2181
2182
2183 def is_file_not_found_exc(exc):
2184- return (isinstance(exc, IOError) and exc.errno == errno.ENOENT)
2185+ return (isinstance(exc, (IOError, OSError)) and
2186+ hasattr(exc, 'errno') and
2187+ exc.errno in (errno.ENOENT, errno.EIO, errno.ENXIO))
2188
2189
2190 def _lsb_release(target=None):
2191@@ -916,8 +946,7 @@
2192
2193
2194 def json_dumps(data):
2195- return json.dumps(data, indent=1, sort_keys=True,
2196- separators=(',', ': ')).encode('utf-8')
2197+ return json.dumps(data, indent=1, sort_keys=True, separators=(',', ': '))
2198
2199
2200 def get_platform_arch():
2201@@ -1055,4 +1084,12 @@
2202 return os.path.join(target, path)
2203
2204
2205+class RunInChroot(ChrootableTarget):
2206+ """Backwards compatibility for RunInChroot (LP: #1617375).
2207+ It needs to work like:
2208+ with RunInChroot("/target") as in_chroot:
2209+ in_chroot(["your", "chrooted", "command"])"""
2210+ __call__ = ChrootableTarget.subp
2211+
2212+
2213 # vi: ts=4 expandtab syntax=python
2214
2215=== modified file 'debian/changelog'
2216--- debian/changelog 2016-09-29 18:22:05 +0000
2217+++ debian/changelog 2016-09-29 18:49:30 +0000
2218@@ -1,8 +1,22 @@
2219-curtin (0.1.0~bzr415-0ubuntu2) UNRELEASED; urgency=medium
2220+curtin (0.1.0~bzr425-0ubuntu1) yakkety; urgency=medium
2221
2222+ [ Scott Moser ]
2223 * debian/new-upstream-snapshot: add writing of debian changelog entries.
2224
2225- -- Scott Moser <smoser@ubuntu.com> Thu, 29 Sep 2016 14:20:57 -0400
2226+ [ Ryan Harper ]
2227+ * New upstream snapshot.
2228+ - unittest,tox.ini: catch and fix issue with trusty-level mock of open
2229+ - block/mdadm: add option to ignore mdadm_assemble errors (LP: #1618429)
2230+ - curtin/doc: overhaul curtin documentation for readthedocs.org (LP: #1351085)
2231+ - curtin.util: re-add support for RunInChroot (LP: #1617375)
2232+ - curtin/net: overhaul of eni rendering to handle mixed ipv4/ipv6 configs
2233+ - curtin.block: refactor clear_holders logic into block.clear_holders and cli cmd
2234+ - curtin.apply_net should exit non-zero upon exception. (LP: #1615780)
2235+ - apt: fix bug in disable_suites if sources.list line is blank.
2236+ - vmtests: disable Wily in vmtests
2237+ - Fix the unittests for test_apt_source.
2238+
2239+ -- Ryan Harper <ryan.harper@canonical.com> Thu, 29 Sep 2016 13:31:02 -0500
2240
2241 curtin (0.1.0~bzr415-0ubuntu1) yakkety; urgency=medium
2242
2243
2244=== modified file 'doc/conf.py'
2245--- doc/conf.py 2015-10-02 16:19:07 +0000
2246+++ doc/conf.py 2016-09-29 18:49:30 +0000
2247@@ -13,6 +13,11 @@
2248
2249 import sys, os
2250
2251+# Fix path so we can import curtin.__version__
2252+sys.path.insert(1, os.path.realpath(os.path.join(
2253+ os.path.dirname(__file__), '..')))
2254+import curtin
2255+
2256 # If extensions (or modules to document with autodoc) are in another directory,
2257 # add these directories to sys.path here. If the directory is relative to the
2258 # documentation root, use os.path.abspath to make it absolute, like shown here.
2259@@ -41,16 +46,16 @@
2260
2261 # General information about the project.
2262 project = u'curtin'
2263-copyright = u'2013, Scott Moser'
2264+copyright = u'2016, Scott Moser, Ryan Harper'
2265
2266 # The version info for the project you're documenting, acts as replacement for
2267 # |version| and |release|, also used in various other places throughout the
2268 # built documents.
2269 #
2270 # The short X.Y version.
2271-version = '0.3'
2272+version = curtin.__version__
2273 # The full version, including alpha/beta/rc tags.
2274-release = '0.3'
2275+release = version
2276
2277 # The language for content autogenerated by Sphinx. Refer to documentation
2278 # for a list of supported languages.
2279@@ -93,6 +98,18 @@
2280 # a list of builtin themes.
2281 html_theme = 'classic'
2282
2283+# on_rtd is whether we are on readthedocs.org, this line of code grabbed from
2284+# docs.readthedocs.org
2285+on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
2286+
2287+if not on_rtd: # only import and set the theme if we're building docs locally
2288+ import sphinx_rtd_theme
2289+ html_theme = 'sphinx_rtd_theme'
2290+ html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
2291+
2292+# otherwise, readthedocs.org uses their theme by default, so no need to specify
2293+# it
2294+
2295 # Theme options are theme-specific and customize the look and feel of a theme
2296 # further. For a list of options available for each theme, see the
2297 # documentation.
2298@@ -120,7 +137,7 @@
2299 # Add any paths that contain custom static files (such as style sheets) here,
2300 # relative to this directory. They are copied after the builtin static files,
2301 # so a file named "default.css" will overwrite the builtin "default.css".
2302-html_static_path = ['static']
2303+#html_static_path = ['static']
2304
2305 # If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
2306 # using the given strftime format.
2307
2308=== removed file 'doc/devel/README-vmtest.txt'
2309--- doc/devel/README-vmtest.txt 2016-08-05 20:47:14 +0000
2310+++ doc/devel/README-vmtest.txt 1970-01-01 00:00:00 +0000
2311@@ -1,221 +0,0 @@
2312-== Background ==
2313-Curtin includes a mechanism called 'vmtest' that allows it to actually
2314-do installs and validate a number of configurations.
2315-
2316-The general flow of the vmtests is:
2317- 1. each test has an associated yaml config file for curtin in examples/tests
2318- 2. uses curtin-pack to create the user-data for cloud-init to trigger install
2319- 3. create and install a system using 'tools/launch'.
2320- 3.1 The install environment is booted from a maas ephemeral image.
2321- 3.2 kernel & initrd used are from maas images (not part of the image)
2322- 3.3 network by default is handled via user networking
2323- 3.4 It creates all empty disks required
2324- 3.5 cloud-init datasource is provided by launch
2325- a) like: ds=nocloud-net;seedfrom=http://10.7.0.41:41518/
2326- provided by python webserver start_http
2327- b) via -drive file=/tmp/launch.8VOiOn/seed.img,if=virtio,media=cdrom
2328- as a seed disk (if booted without external kernel)
2329- 3.6 dependencies and other preparations are installed at the beginning by
2330- curtin inside the ephemeral image prior to configuring the target
2331- 4. power off the system.
2332- 5. configure a 'NoCloud' datasource seed image that provides scripts that
2333- will run on first boot.
2334- 5.1 this will contain all our code to gather health data on the install
2335- 5.2 by cloud-init design this runs only once per instance, if you start
2336- the system again this won't be called again
2337- 6. boot the installed system with 'tools/xkvm'.
2338- 6.1 reuses the disks that were installed/configured in the former steps
2339- 6.2 also adds an output disk
2340- 6.3 additionally the seed image for the data gathering is added
2341- 6.4 On this boot it will run the provided scripts, write their output to a
2342- "data" disk and then shut itself down.
2343- 7. extract the data from the output disk
2344- 8. vmtest python code now verifies if the output is as expected.
2345-
2346-== Debugging ==
2347-At 3.1
2348- - one can pull data out of the maas image with
2349- sudo mount-image-callback your.img -- sh -c 'COMMAND'
2350- e.g. sudo mount-image-callback your.img -- sh -c 'cp $MOUNTPOINT/boot/* .'
2351-At step 3.6 -> 4.
2352- - tools/launch can be called in a way to give you console access
2353- to do so just call tools/launch but drop the -serial=x parameter.
2354- One might want to change "'power_state': {'mode': 'poweroff'}" to avoid
2355- the auto reboot before getting control
2356- Replace the directory usually seen in the launch calls with a clean fresh
2357- directory
2358- - In /curtin curtin and its config can be found
2359- - if the system gets that far cloud-init will create a user ubuntu/passw0rd
2360- - otherwise one can use a cloud-image from https://cloud-images.ubuntu.com/
2361- and add a backdoor user via
2362- bzr branch lp:~maas-maintainers/maas/backdoor-image backdoor-image
2363- sudo ./backdoor-image -v --user=<USER> --password-auth --password=<PW> IMG
2364-At step 6 -> 7
2365- - You might want to keep all the temporary images around.
2366- To do so you can set CURTIN_VMTEST_KEEP_DATA_PASS=all:
2367- export CURTIN_VMTEST_KEEP_DATA_PASS=all CURTIN_VMTEST_KEEP_DATA_FAIL=all
2368- That will keep the /tmp/tmpXXXXX directories and all files in there for
2369- further execution.
2370-At step 7
2371- - You might want to take a look at the output disk yourself.
2372- It is a normal qcow image, so one can use mount-image-callback as described
2373- above
2374- - to invoke xkvm on your own take the command you see in the output and
2375- remove the "-serial ..." but add -nographic instead
2376- For graphical console one can add --vnc 127.0.0.1:1
2377-
2378-== Setup ==
2379-In order to run vmtest you'll need some dependencies. To get them, you
2380-can run:
2381- make vmtest-deps
2382-
2383-That will install all necessary dependencies.
2384-
2385-== Running ==
2386-Running tests is done most simply by:
2387-
2388- make vmtest
2389-
2390-If you wish to all tests in test_network.py, do so with:
2391- sudo PATH=$PWD/tools:$PATH nosetests3 tests/vmtests/test_network.py
2392-
2393-Or run a single test with:
2394- sudo PATH=$PWD/tools:$PATH nosetests3 tests/vmtests/test_network.py:WilyTestBasic
2395-
2396-Note:
2397- * currently, the tests have to run as root. The reason for this is that
2398- the kernel and initramfs to boot are extracted from the maas ephemeral
2399- image. This should be fixed at some point, and then 'make vmtest'
2400-
2401- The tests themselves don't actually have to run as root, but the
2402- test setup does.
2403- * the 'tools' directory must be in your path.
2404- * test will set apt: { proxy } in the guests to the value of
2405- 'apt_proxy' environment variable. If that is not set it will
2406- look at the host's apt config and read 'Acquire::HTTP::Proxy'
2407-
2408-== Environment Variables ==
2409-Some environment variables affect the running of vmtest
2410- * apt_proxy:
2411- test will set apt_proxy in the guests to the value of 'apt_proxy'.
2412- If that is not set it will look at the host's apt config and read
2413- 'Acquire::HTTP::Proxy'
2414-
2415- * CURTIN_VMTEST_KEEP_DATA_PASS CURTIN_VMTEST_KEEP_DATA_FAIL:
2416- default:
2417- CURTIN_VMTEST_KEEP_DATA_PASS=none
2418- CURTIN_VMTEST_KEEP_DATA_FAIL=all
2419- These 2 variables determine what portions of the temporary
2420- test data are kept.
2421-
2422- The variables contain a comma ',' delimited list of directories
2423- that should be kept in the case of pass or fail. Additionally,
2424- the values 'all' and 'none' are accepted.
2425-
2426- Each vmtest that runs has its own sub-directory under the top level
2427- CURTIN_VMTEST_TOPDIR. In that directory are directories:
2428- boot: inputs to the system boot (after install)
2429- install: install phase related files
2430- disks: the disks used for installation and boot
2431- logs: install and boot logs
2432- collect: data collected by the boot phase
2433-
2434- * CURTIN_VMTEST_TOPDIR: default $TMPDIR/vmtest-<timestamp>
2435- vmtest puts all test data under this value. By default, it creates
2436- a directory in TMPDIR (/tmp) named with as "vmtest-<timestamp>"
2437-
2438- If you set this value, you must ensure that the directory is either
2439- non-existant or clean.
2440-
2441- * CURTIN_VMTEST_LOG: default $TMPDIR/vmtest-<timestamp>.log
2442- vmtest writes extended log information to this file.
2443- The default puts the log along side the TOPDIR.
2444-
2445- * CURTIN_VMTEST_IMAGE_SYNC: default false (boolean)
2446- if set to true, each run will attempt a sync of images.
2447- If you want to make sure images are always up to date, then set to true.
2448-
2449- * CURTIN_VMTEST_BRIDGE: default 'user'
2450- the network devices will be attached to this bridge. The default is
2451- 'user', which means to use qemu user mode networking. Set it to
2452- 'virbr0' or 'lxcbr0' to use those bridges and then be able to ssh
2453- in directly.
2454-
2455- * CURTIN_VMTEST_BOOT_TIMEOUT: default 300
2456- timeout before giving up on the boot of the installed system.
2457-
2458- * CURTIN_VMTEST_INSTALL_TIMEOUT: default 3000
2459- timeout before giving up on installation.
2460-
2461- * CURTIN_VMTEST_PARALLEL: default ''
2462- only supported through ./tools/jenkins-runner .
2463- -1 : then run one per core.
2464- 0 or '': then run with no parallel
2465- >0 : run with N processes
2466- this modifies the invocation of nosetets to add '--processes' and other
2467- necessary nose arguments (--process-timeout)
2468-
2469- * IMAGE_DIR: default /srv/images
2470- vmtest keeps a mirror of maas ephemeral images in this directory.
2471-
2472- * IMAGES_TO_KEEP: default 1
2473- keep this number of images of each release in the IMAGE_DIR.
2474-
2475-Environment 'boolean' values:
2476- For boolean environment variables the value is considered True
2477- if it is any value other than case insensitive 'false', '' or "0"
2478-
2479-
2480-== Test Class Variables ==
2481-The base VMBaseClass defines several variables that help creating a new test
2482-easily. Among those the common ones are:
2483-
2484-Generic:
2485-- arch_skip
2486- If a test is not supported on an architecture it can list the arch in this
2487- variable to auto-skip the test if executed on that arch.
2488-- conf_file
2489- The configuration that will be processed by this vmtest.
2490-- extra_kern_args
2491- Extra arguments to the guest kernel on boot.
2492-
2493-Data Collection:
2494-- collect_scripts
2495- The commands run when booting into the installed environment to collect the
2496- data for the test to verify a proper execution.
2497-- boot_cloudconf
2498- Extra cloud-init config content for the install phase.
2499- This allows to gather content of the install phase if needed for test
2500- verification.
2501-
2502-
2503-Disk Setup:
2504-- disk_block_size = 512
2505-- disk_driver = 'virtio-blk'
2506- Used to set up the disks for the virtual environment used by the vmtest.
2507- Will set the values used in extra_disks if not specified there.
2508-- extra_disks
2509- A list of extra disks to be created for the testcase. The definition is
2510- dpath:size:driver:block_size
2511-- multipath = False
2512-- multipath_num_paths = 2
2513- Details for the (virtual) multipath setup
2514-- nvme_disks
2515- a shortcut to provide extra disks with the nvme driver
2516-
2517-Timeouts:
2518-- boot_timeout
2519-- install_timeout
2520- Usually set via CURTIN_VMTEST_BOOT_TIMEOUT/CURTIN_VMTEST_INSTALL_TIMEOUT
2521- (see above) environment var, but can be overwritten by a testcase if needed.
2522-
2523-Checks:
2524-- disk_to_check
2525- A disk name that is verified to be existing after the installation.
2526-- fstab_expected
2527-
2528-Release:
2529-- release = None
2530-- krel = None
2531- Those two define the distribution and kernel release to be tested and are
2532- usually set by importing and inheriting from tests/vmtests/releases.py
2533
2534=== removed file 'doc/devel/README.txt'
2535--- doc/devel/README.txt 2015-03-11 13:19:43 +0000
2536+++ doc/devel/README.txt 1970-01-01 00:00:00 +0000
2537@@ -1,55 +0,0 @@
2538-## curtin development ##
2539-
2540-This document describes how to use kvm and ubuntu cloud images
2541-to develop curtin or test install configurations inside kvm.
2542-
2543-## get some dependencies ##
2544-sudo apt-get -qy install kvm libvirt-bin cloud-utils bzr
2545-
2546-## get cloud image to boot (-disk1.img) and one to install (-root.tar.gz)
2547-mkdir -p ~/download
2548-DLDIR=$( cd ~/download && pwd )
2549-rel="trusty"
2550-arch=amd64
2551-burl="http://cloud-images.ubuntu.com/$rel/current/"
2552-for f in $rel-server-cloudimg-${arch}-root.tar.gz $rel-server-cloudimg-${arch}-disk1.img; do
2553- wget "$burl/$f" -O $DLDIR/$f; done
2554-( cd $DLDIR && qemu-img convert -O qcow $rel-server-cloudimg-${arch}-disk1.img $rel-server-cloudimg-${arch}-disk1.qcow2)
2555-
2556-BOOTIMG="$DLDIR/$rel-server-cloudimg-${arch}-disk1.qcow2"
2557-ROOTTGZ="$DLDIR/$rel-server-cloudimg-${arch}-root.tar.gz"
2558-
2559-## get curtin
2560-mkdir -p ~/src
2561-bzr init-repo ~/src/curtin
2562-( cd ~/src/curtin && bzr branch lp:curtin trunk.dist )
2563-( cd ~/src/curtin && bzr branch trunk.dist trunk )
2564-
2565-## work with curtin
2566-cd ~/src/curtin/trunk
2567-# use 'launch' to launch a kvm instance with user data to pack
2568-# up local curtin and run it inside instance.
2569-./tools/launch $BOOTIMG --publish $ROOTTGZ -- curtin install "PUBURL/${ROOTTGZ##*/}"
2570-
2571-## notes about 'launch' ##
2572- * launch has --help so you can see that for some info.
2573- * '--publish' adds a web server at ${HTTP_PORT:-9923}
2574- and puts the files you want available there. You can reference
2575- this url in config or cmdline with 'PUBURL'. For example
2576- '--publish foo.img' will put 'foo.img' at PUBURL/foo.img.
2577- * launch sets 'ubuntu' user password to 'passw0rd'
2578- * launch runs 'kvm -curses'
2579- kvm -curses keyboard info:
2580- 'alt-2' to go to qemu console
2581- * launch puts serial console to 'serial.log' (look there for stuff)
2582- * when logged in
2583- * you can look at /var/log/cloud-init-output.log
2584- * archive should be extracted in /curtin
2585- * shell archive should be in /var/lib/cloud/instance/scripts/part-002
2586- * when logged in, and archive available at
2587-
2588-
2589-## other notes ##
2590- * need to add '--install-deps' or something for curtin
2591- cloud-image in 12.04 has no 'python3'
2592- ideally 'curtin --install-deps install' would get the things it needs
2593
2594=== added file 'doc/devel/clear_holders_doc.txt'
2595--- doc/devel/clear_holders_doc.txt 1970-01-01 00:00:00 +0000
2596+++ doc/devel/clear_holders_doc.txt 2016-09-29 18:49:30 +0000
2597@@ -0,0 +1,85 @@
2598+The new version of clear_holders is based around a data structure called a
2599+holder_tree which represents the current storage hirearchy above a specified
2600+starting device. Each node in a holders tree contains data about the node and a
2601+key 'holders' which contains a list of all nodes that depend on it. The keys in
2602+a holdres_tree node are:
2603+ - device: the path to the device in /sys/class/block
2604+ - dev_type: what type of storage layer the device is. possible values:
2605+ - disk
2606+ - lvm
2607+ - crypt
2608+ - raid
2609+ - bcache
2610+ - disk
2611+ - name: the kname of the device (used for display)
2612+ - holders: holders_trees for devices depending on the current device
2613+
2614+A holders tree can be generated for a device using the function
2615+clear_holders.gen_holders_tree. The device can be specified either as a path in
2616+/sys/class/block or as a path in /dev.
2617+
2618+The new implementation of block.clear_holders shuts down storage devices in a
2619+holders tree starting from the leaves of the tree and ascending towards the
2620+root. The old implementation of clear_holders ascended up each path of the tree
2621+separately, in a pattern similar to depth first search. The problem with the
2622+old implementation is that in some cases either an attempt would be made to
2623+remove one storage device while other devices depended on it or clear_holders
2624+would attempt to shut down the same storage device several times. In order to
2625+cope with this the old version of clear_holders had logic to handle expected
2626+failures and hope for the best moving forward. The new version of clear_holders
2627+is able to run without many anticipated failures.
2628+
2629+The logic to plan what order to shut down storage layers in is in
2630+clear_holders.plan_shutdown_holders_trees. This function accepts either a
2631+single holders tree or a list of holders trees. When run with a list of holders
2632+trees, it assumes that all of these trees start at basically the same layer in
2633+the overall storage hirearcy for the system (i.e. a list of holders trees
2634+starting from all of the target installation disks). This function returns a
2635+list of dictionaries, with each dictionary containing the keys:
2636+ - device: the path to the device in /sys/class/block
2637+ - dev_type: what type of storage layer the device is. possible values:
2638+ - disk
2639+ - lvm
2640+ - crypt
2641+ - raid
2642+ - bcache
2643+ - disk
2644+ - level: the level of the device in the current storage hirearchy
2645+ (starting from 0)
2646+
2647+The items in the list returned by clear_holders.plan_shutdown_holders_trees
2648+should be processed in order to make sure the holders trees are shutdown fully
2649+
2650+The main interface for clear_holders is the function
2651+clear_holders.clear_holders. If the system has just been booted it could be
2652+beneficial to run the function clear_holders.start_clear_holders_deps before
2653+using clear_holders.clear_holders. This ensures clear_holders will be able to
2654+properly storage devices. The function clear_holders.clear_holders can be
2655+passed either a single device or a list of devices and will shut down all
2656+storage devices above the device(s). The devices can be specified either by
2657+path in /dev or by path in /sys/class/block.
2658+
2659+In order to test if a device or devices are free to be partitioned/formatted,
2660+the function clear_holders.assert_clear can be passed either a single device or
2661+a list of devices, with devices specified either by path in /dev or by path in
2662+/sys/class/block. If there are any storage devices that depend on one of the
2663+devices passed to clear_holders.assert_clear, then an OSError will be raised.
2664+If clear_holders.assert_clear does not raise any errors, then the devices
2665+specified should be ready for partitioning.
2666+
2667+It is possible to query further information about storage devices using
2668+clear_holders.
2669+
2670+Holders for a individual device can be queried using clear_holders.get_holders.
2671+Results are returned as a list or knames for holding devices.
2672+
2673+A holders tree can be printed in a human readable format using
2674+clear_holders.format_holders_tree(). Example output:
2675+sda
2676+|-- sda1
2677+|-- sda2
2678+`-- sda5
2679+ `-- dm-0
2680+ |-- dm-1
2681+ `-- dm-2
2682+ `-- dm-3
2683
2684=== modified file 'doc/index.rst'
2685--- doc/index.rst 2015-10-02 16:19:07 +0000
2686+++ doc/index.rst 2016-09-29 18:49:30 +0000
2687@@ -13,7 +13,13 @@
2688 :maxdepth: 2
2689
2690 topics/overview
2691+ topics/config
2692+ topics/apt_source
2693+ topics/networking
2694+ topics/storage
2695 topics/reporting
2696+ topics/development
2697+ topics/integration-testing
2698
2699
2700
2701
2702=== modified file 'doc/topics/apt_source.rst'
2703--- doc/topics/apt_source.rst 2016-08-05 20:47:14 +0000
2704+++ doc/topics/apt_source.rst 2016-09-29 18:49:30 +0000
2705@@ -9,7 +9,7 @@
2706 The feature has an optional target argument which - by default - is used to modify the environment that curtin currently installs (@TARGET_MOUNT_POINT).
2707
2708 Features
2709---------
2710+~~~~~~~~
2711
2712 * Add PGP keys to the APT trusted keyring
2713
2714@@ -39,7 +39,7 @@
2715
2716
2717 Configuration
2718--------------
2719+~~~~~~~~~~~~~
2720
2721 The general configuration of the apt feature is under an element called ``apt``.
2722
2723@@ -53,7 +53,8 @@
2724 The key is the filename and will be prepended by /etc/apt/sources.list.d/ if it doesn't start with a ``/``.
2725 There are certain cases - where no content is written into a source.list file where the filename will be ignored - yet it can still be used as index for merging.
2726
2727-The values inside the entries consist of the following optional entries::
2728+The values inside the entries consist of the following optional entries
2729+
2730 * ``source``: a sources.list entry (some variable replacements apply)
2731
2732 * ``keyid``: providing a key to import via shortid or fingerprint
2733@@ -62,7 +63,8 @@
2734
2735 * ``keyserver``: specify an alternate keyserver to pull keys from that were specified by keyid
2736
2737-The section "sources" is is a dictionary (unlike most block/net configs which are lists). This format allows merging between multiple input files than a list like::
2738+The section "sources" is is a dictionary (unlike most block/net configs which are lists). This format allows merging between multiple input files than a list like ::
2739+
2740 sources:
2741 s1: {'key': 'key1', 'source': 'source1'}
2742
2743@@ -101,8 +103,7 @@
2744
2745 - make an example with a partial mirror that doesn't mirror the backports suite, so backports have to be disabled
2746
2747-That would be specified as
2748-::
2749+That would be specified as ::
2750
2751 apt:
2752 primary:
2753@@ -125,26 +126,31 @@
2754
2755
2756 Common snippets
2757----------------
2758+~~~~~~~~~~~~~~~
2759 This is a collection of additional ideas people can use the feature for customizing their to-be-installed system.
2760
2761 * enable proposed on installing
2762- apt:
2763- sources:
2764- proposed.list: deb $MIRROR $RELEASE-proposed main restricted universe multiverse
2765+
2766+::
2767+
2768+ apt:
2769+ sources:
2770+ proposed.list: deb $MIRROR $RELEASE-proposed main restricted universe multiverse
2771
2772 * Make debug symbols available
2773- apt:
2774-  sources:
2775-   ddebs.list: |
2776-   deb http://ddebs.ubuntu.com $RELEASE main restricted universe multiverse
2777-   deb http://ddebs.ubuntu.com $RELEASE-updates main restricted universe multiverse
2778-  deb http://ddebs.ubuntu.com $RELEASE-security main restricted universe multiverse
2779- deb http://ddebs.ubuntu.com $RELEASE-proposed main restricted universe multiverse
2780-
2781+
2782+::
2783+
2784+ apt:
2785+ sources:
2786+ ddebs.list: |
2787+ deb http://ddebs.ubuntu.com $RELEASE main restricted universe multiverse
2788+  deb http://ddebs.ubuntu.com $RELEASE-updates main restricted universe multiverse
2789+  deb http://ddebs.ubuntu.com $RELEASE-security main restricted universe multiverse
2790+ deb http://ddebs.ubuntu.com $RELEASE-proposed main restricted universe multiverse
2791
2792 Timing
2793-------
2794+~~~~~~
2795 The feature is implemented at the stage of curthooks_commands, which runs just after curtin has extracted the image to the target.
2796 Additionally it can be ran as standalong command "curtin -v --config <yourconfigfile> apt-config".
2797
2798@@ -153,6 +159,6 @@
2799
2800
2801 Dependencies
2802-------------
2803+~~~~~~~~~~~~
2804 Cloud-init might need to resolve dependencies and install packages in the ephemeral environment to run curtin.
2805 Therefore it is recommended to not only provide an apt configuration to curtin for the target, but also one to the install environment via cloud-init.
2806
2807=== added file 'doc/topics/config.rst'
2808--- doc/topics/config.rst 1970-01-01 00:00:00 +0000
2809+++ doc/topics/config.rst 2016-09-29 18:49:30 +0000
2810@@ -0,0 +1,551 @@
2811+====================
2812+Curtin Configuration
2813+====================
2814+
2815+Curtin exposes a number of configuration options for controlling Curtin
2816+behavior during installation.
2817+
2818+
2819+Configuration options
2820+---------------------
2821+Curtin's top level config keys are as follows:
2822+
2823+
2824+- apt_mirrors (``apt_mirrors``)
2825+- apt_proxy (``apt_proxy``)
2826+- block-meta (``block``)
2827+- debconf_selections (``debconf_selections``)
2828+- disable_overlayroot (``disable_overlayroot``)
2829+- grub (``grub``)
2830+- http_proxy (``http_proxy``)
2831+- install (``install``)
2832+- kernel (``kernel``)
2833+- kexec (``kexec``)
2834+- multipath (``multipath``)
2835+- network (``network``)
2836+- power_state (``power_state``)
2837+- reporting (``reporting``)
2838+- restore_dist_interfaces: (``restore_dist_interfaces``)
2839+- sources (``sources``)
2840+- stages (``stages``)
2841+- storage (``storage``)
2842+- swap (``swap``)
2843+- system_upgrade (``system_upgrade``)
2844+- write_files (``write_files``)
2845+
2846+
2847+apt_mirrors
2848+~~~~~~~~~~~
2849+Configure APT mirrors for ``ubuntu_archive`` and ``ubuntu_security``
2850+
2851+**ubuntu_archive**: *<http://local.archive/ubuntu>*
2852+
2853+**ubuntu_security**: *<http://local.archive/ubuntu>*
2854+
2855+If the target OS includes /etc/apt/sources.list, Curtin will replace
2856+the default values for each key set with the supplied mirror URL.
2857+
2858+**Example**::
2859+
2860+ apt_mirrors:
2861+ ubuntu_archive: http://local.archive/ubuntu
2862+ ubuntu_security: http://local.archive/ubuntu
2863+
2864+
2865+apt_proxy
2866+~~~~~~~~~
2867+Curtin will configure an APT HTTP proxy in the target OS
2868+
2869+**apt_proxy**: *<URL to APT proxy>*
2870+
2871+**Example**::
2872+
2873+ apt_proxy: http://squid.mirror:3267/
2874+
2875+
2876+block-meta
2877+~~~~~~~~~~
2878+Configure how Curtin selects and configures disks on the target
2879+system without providing a custom configuration (mode=simple).
2880+
2881+**devices**: *<List of block devices for use>*
2882+
2883+The ``devices`` parameter is a list of block device paths that Curtin may
2884+select from with choosing where to install the OS.
2885+
2886+**boot-partition**: *<dictionary of configuration>*
2887+
2888+The ``boot-partition`` parameter controls how to configure the boot partition
2889+with the following parameters:
2890+
2891+**enabled**: *<boolean>*
2892+
2893+Enabled will forcibly setup a partition on the target device for booting.
2894+
2895+**format**: *<['uefi', 'gpt', 'prep', 'mbr']>*
2896+
2897+Specify the partition format. Some formats, like ``uefi`` and ``prep``
2898+are restricted by platform characteristics.
2899+
2900+**fstype**: *<filesystem type: one of ['ext3', 'ext4'], defaults to 'ext4'>*
2901+
2902+Specify the filesystem format on the boot partition.
2903+
2904+**label**: *<filesystem label: defaults to 'boot'>*
2905+
2906+Specify the filesystem label on the boot partition.
2907+
2908+**Example**::
2909+
2910+ block-meta:
2911+ devices:
2912+ - /dev/sda
2913+ - /dev/sdb
2914+ boot-partition:
2915+ - enabled: True
2916+ format: gpt
2917+ fstype: ext4
2918+ label: my-boot-partition
2919+
2920+
2921+debconf_selections
2922+~~~~~~~~~~~~~~~~~~
2923+Curtin will update the target with debconf set-selection values. Users will
2924+need to be familiar with the package debconf options. Users can probe a
2925+packages' debconf settings by using ``debconf-get-selections``.
2926+
2927+**selection_name**: *<debconf-set-selections input>*
2928+
2929+``debconf-set-selections`` is in the form::
2930+
2931+ <packagename> <packagename/option-name> <type> <value>
2932+
2933+**Example**::
2934+
2935+ debconf_selections:
2936+ set1: |
2937+ cloud-init cloud-init/datasources multiselect MAAS
2938+ lxd lxd/bridge-name string lxdbr0
2939+ set2: lxd lxd/setup-bridge boolean true
2940+
2941+
2942+
2943+disable_overlayroot
2944+~~~~~~~~~~~~~~~~~~~
2945+Curtin disables overlayroot in the target by default.
2946+
2947+**disable_overlayroot**: *<boolean: default True>*
2948+
2949+**Example**::
2950+
2951+ disable_overlayroot: False
2952+
2953+
2954+grub
2955+~~~~
2956+Curtin configures grub as the target machine's boot loader. Users
2957+can control a few options to tailor how the system will boot after
2958+installation.
2959+
2960+**install_devices**: *<list of block device names to install grub>*
2961+
2962+Specify a list of devices onto which grub will attempt to install.
2963+
2964+**replace_linux_default**: *<boolean: default True>*
2965+
2966+Controls whether grub-install will update the Linux Default target
2967+value during installation.
2968+
2969+**update_nvram**: *<boolean: default False>*
2970+
2971+Certain platforms, like ``uefi`` and ``prep`` systems utilize
2972+NVRAM to hold boot configuration settings which control the order in
2973+which devices are booted. Curtin by default will not attempt to
2974+update the NVRAM settings to preserve the system configuration.
2975+Users may want to force NVRAM to be updated such that the next boot
2976+of the system will boot from the installed device.
2977+
2978+**Example**::
2979+
2980+ grub:
2981+ install_devices:
2982+ - /dev/sda1
2983+ replace_linux_default: False
2984+ update_nvram: True
2985+
2986+
2987+http_proxy
2988+~~~~~~~~~~
2989+Curtin will export ``http_proxy`` value into the installer environment.
2990+
2991+**http_proxy**: *<HTTP Proxy URL>*
2992+
2993+**Example**::
2994+
2995+ http_proxy: http://squid.proxy:3728/
2996+
2997+
2998+
2999+install
3000+~~~~~~~
3001+Configure Curtin's install options.
3002+
3003+**log_file**: *<path to write Curtin's install.log data>*
3004+
3005+Curtin logs install progress by default to /var/log/curtin/install.log
3006+
3007+**post_files**: *<List of files to read from host to include in reporting data>*
3008+
3009+Curtin by default will post the ``log_file`` value to any configured reporter.
3010+
3011+**save_install_config**: *<Path to save merged curtin configuration file>*
3012+
3013+Curtin will save the merged configuration data into the target OS at
3014+the path of ``save_install_config``. This defaults to /root/curtin-install-cfg.yaml
3015+
3016+**Example**::
3017+
3018+ install:
3019+ log_file: /tmp/install.log
3020+ post_files:
3021+ - /tmp/install.log
3022+ - /var/log/syslog
3023+ save_install_config: /root/myconf.yaml
3024+
3025+
3026+kernel
3027+~~~~~~
3028+Configure how Curtin selects which kernel to install into the target image.
3029+If ``kernel`` is not configured, Curtin will use the default mapping below
3030+and determine which ``package`` value by looking up the current release
3031+and current kernel version running.
3032+
3033+
3034+**fallback-package**: *<kernel package-name to be used as fallback>*
3035+
3036+Specify a kernel package name to be used if the default package is not
3037+available.
3038+
3039+**mapping**: *<Dictionary mapping Ubuntu release to HWE kernel names>*
3040+
3041+Default mapping for Releases to package names is as follows::
3042+
3043+ precise:
3044+ 3.2.0:
3045+ 3.5.0: -lts-quantal
3046+ 3.8.0: -lts-raring
3047+ 3.11.0: -lts-saucy
3048+ 3.13.0: -lts-trusty
3049+ trusty:
3050+ 3.13.0:
3051+ 3.16.0: -lts-utopic
3052+ 3.19.0: -lts-vivid
3053+ 4.2.0: -lts-wily
3054+ 4.4.0: -lts-xenial
3055+ xenial:
3056+ 4.3.0:
3057+ 4.4.0:
3058+
3059+
3060+**package**: *<Linux kernel package name>*
3061+
3062+Specify the exact package to install in the target OS.
3063+
3064+**Example**::
3065+
3066+ kernel:
3067+ fallback-package: linux-image-generic
3068+ package: linux-image-generic-lts-xenial
3069+ mapping:
3070+ - xenial:
3071+ - 4.4.0: -my-custom-kernel
3072+
3073+
3074+kexec
3075+~~~~~
3076+Curtin can use kexec to "reboot" into the target OS.
3077+
3078+**mode**: *<on>*
3079+
3080+Enable rebooting with kexec.
3081+
3082+**Example**::
3083+
3084+ kexec: on
3085+
3086+
3087+multipath
3088+~~~~~~~~~
3089+Curtin will detect and autoconfigure multipath by default to enable
3090+boot for systems with multipath. Curtin does not apply any advanced
3091+configuration or tuning, rather it uses distro defaults and provides
3092+enough configuration to enable booting.
3093+
3094+**mode**: *<['auto', ['disabled']>*
3095+
3096+Defaults to auto which will configure enough to enable booting on multipath
3097+devices. Disabled will prevent curtin from installing or configuring
3098+multipath.
3099+
3100+**overwrite_bindings**: *<boolean>*
3101+
3102+If ``overwrite_bindings`` is True then Curtin will generate new bindings
3103+file for multipath, overriding any existing binding in the target image.
3104+
3105+**Example**::
3106+
3107+ multipath:
3108+ mode: auto
3109+ overwrite_bindings: True
3110+
3111+
3112+network
3113+~~~~~~~
3114+Configure networking (see Networking section for details).
3115+
3116+**network_option_1**: *<option value>*
3117+
3118+**Example**::
3119+
3120+ network:
3121+ version: 1
3122+ config:
3123+ - type: physical
3124+ name: eth0
3125+ mac_address: "c0:d6:9f:2c:e8:80"
3126+ subnets:
3127+ - type: dhcp4
3128+
3129+
3130+power_state
3131+~~~~~~~~~~~
3132+Curtin can configure the target machine into a specific power state after
3133+completing an installation. Default is to do nothing.
3134+
3135+**delay**: *<Integer seconds to delay change in state>*
3136+
3137+Curtin will wait ``delay`` seconds before changing the power state.
3138+
3139+**mode**: *<New power state is one of: [halt, poweroff, reboot]>*
3140+
3141+Curtin will transition the node into one of the new states listed.
3142+
3143+``halt`` will stop a machine, but may not cut the power to the system.
3144+``poweroff`` will stop a machine and request it shut off the power.
3145+``reboot`` will perform a platform reset.
3146+
3147+**message**: *<message string>*
3148+
3149+The ``message`` string will be broadcast to system consoles prior to
3150+power state change.
3151+
3152+
3153+**Example**::
3154+
3155+ power_state:
3156+ mode: poweroff
3157+ delay: 5
3158+ message: Bye Bye
3159+
3160+
3161+reporting
3162+~~~~~~~~~
3163+Configure installation reporting (see Reporting section for details).
3164+
3165+**Example**::
3166+
3167+ reporting:
3168+ maas:
3169+ level: DEBUG
3170+ type: webhook
3171+ endpoint: http://localhost:8000/
3172+
3173+
3174+restore_dist_interfaces
3175+~~~~~~~~~~~~~~~~~~~~~~~
3176+Curtin can restore a copy of /etc/network/interfaces built in to cloud images.
3177+
3178+**restore_dist_interfaces**: *<boolean>*
3179+
3180+If True, then Curtin will restore the interfaces file into the target.
3181+
3182+
3183+**Example**::
3184+
3185+ restore_dist_interfaces: True
3186+
3187+
3188+sources
3189+~~~~~~~
3190+Specify the root image to install on to the target system. The URI also
3191+configures the method used to copy the image to the target system.
3192+
3193+**sources**: *<List of source URIs>*
3194+
3195+``source URI`` may be one of:
3196+
3197+- **dd-**: Use ``dd`` command to write image to target.
3198+- **cp://**: Use ``rsync`` command to copy source directory to target.
3199+- **file://**: Use ``tar`` command to extract source to target.
3200+- **http[s]://**: Use ``wget | tar`` commands to extract source to target.
3201+
3202+**Example Cloud-image**::
3203+
3204+ sources:
3205+ - https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.gz
3206+
3207+**Example Custom DD image**::
3208+
3209+ sources:
3210+ - dd-img: https://localhost/raw_images/centos-6-3.img
3211+
3212+**Example Copy from booted environment**::
3213+
3214+ sources:
3215+ - cp:///
3216+
3217+
3218+**Example Copy from local tarball**::
3219+
3220+ sources:
3221+ - file:///tmp/root.tar.gz
3222+
3223+
3224+stages
3225+~~~~~~
3226+Curtin installation executes in stages. At each stage, Curtin will look for
3227+a list of commands to run at each stage by reading in from the Curtin config
3228+*<stage_name>_commands* which is a dictionary and each key contains a list
3229+of commands to run. Users may override the stages value to control
3230+what curtin stages execute. During each stage, the commands are executed
3231+in C Locale sort order. Users should name keys in a NN-XXX format where NN
3232+is a two-digit number to exercise control over execution order.
3233+
3234+The following stages are defined in Curtin and
3235+run by default.
3236+
3237+- **early**: *Preparing for Installation*
3238+
3239+This stage runs before any actions are taken for installation. By default
3240+this stage does nothing.
3241+
3242+- **partitioning**: *Select and partition disks for installation*
3243+
3244+This stage runs ``curtin block-meta simple`` by default.
3245+
3246+- **network**: *Probe and configure networking*
3247+
3248+This stage runs ``curtin net-meta auto`` by default.
3249+
3250+- **extract**: *Writing install sources to disk*
3251+
3252+This stage runs ``curtin extract`` by default.
3253+
3254+- **extract**: *Writing install sources to disk*
3255+
3256+This stage runs ``curtin extract`` by default.
3257+
3258+- **curthooks**: *Configuring installed system*
3259+
3260+This stage runs ``curtin curthooks`` by default.
3261+
3262+- **hooks**: *Finalizing installation*
3263+
3264+This stage runs ``curtin hook`` by default.
3265+
3266+- **late**: *Executing late commands*
3267+
3268+This stage runs after Curtin has completed the installation. By default
3269+this stage does nothing.
3270+
3271+**Example Custom Stages**::
3272+
3273+ # Skip the whole install and just run `mystage`
3274+ stages: ['early', 'late', 'mystage']
3275+ mystage_commands:
3276+ 00-cmd: ['/usr/bin/foo']
3277+
3278+**Example Early and Late commands**::
3279+
3280+ early_commands:
3281+ 99-cmd: ['echo', 'I ran last']
3282+ 00-cmd: ['echo', 'I ran first']
3283+ late_commands:
3284+ 50-cmd: ['curtin', 'in-target' '--', 'touch', '/etc/disable_overlayroot']
3285+
3286+
3287+swap
3288+~~~~
3289+Curtin can configure a swapfile on the filesystem in the target system.
3290+Size settings can be integer or string values with suffix. Curtin
3291+supports the following suffixes which multiply the value.
3292+
3293+- **B**: *1*
3294+- **K[B]**: *1 << 10*
3295+- **M[B]**: *1 << 20*
3296+- **G[B]**: *1 << 30*
3297+- **T[B]**: *1 << 40*
3298+
3299+Curtin will use a heuristic to configure the swapfile size if the ``size``
3300+parameter is not set to a specific value. The ``maxsize`` sets the upper
3301+bound of the heuristic calculation.
3302+
3303+**filename**: *<path to swap file>*
3304+
3305+Configure the filename of the swap file. Defaults to /swap.img
3306+
3307+**maxsize**: *<Size string>*
3308+
3309+Configure the max size of the swapfile, defaults to 8GB
3310+
3311+**size**: *<Size string>*
3312+
3313+Configure the exact size of the swapfile. Setting ``size`` to 0 will
3314+disable swap.
3315+
3316+**Example**::
3317+
3318+ swap:
3319+ filename: swap.img
3320+ size: None
3321+ maxsize: 4GB
3322+
3323+
3324+system_upgrade
3325+~~~~~~~~~~~~~~
3326+Control if Curtin runs `dist-upgrade` in target after install. Defaults to
3327+False.
3328+
3329+**enabled**: *<boolean>*
3330+
3331+**Example**::
3332+
3333+ system_upgrade:
3334+ enabled: False
3335+
3336+
3337+write_files
3338+~~~~~~~~~~~
3339+Curtin supports writing out arbitrary data to a file.
3340+``write_files`` accepts a dictionary of entries formatted as follows:
3341+
3342+**path**: *<path and filename to save content>*
3343+
3344+Specify the name and location of where to write the content.
3345+
3346+**permissions**: *<Unix permission string>*
3347+
3348+Specify the permissions mode as an integer or string of numbers.
3349+
3350+**content**: *<data>*
3351+
3352+Specify the content.
3353+
3354+**Example**::
3355+
3356+ write_files:
3357+ f1:
3358+ path: /file1
3359+ content: !!binary |
3360+ f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAwARAAAAAAABAAAAAAAAAAJAVAAAAAAA
3361+ f2: {path: /file2, content: "foobar", permissions: '0666'}
3362
3363=== added file 'doc/topics/development.rst'
3364--- doc/topics/development.rst 1970-01-01 00:00:00 +0000
3365+++ doc/topics/development.rst 2016-09-29 18:49:30 +0000
3366@@ -0,0 +1,68 @@
3367+=================
3368+Developing Curtin
3369+=================
3370+
3371+Curtin developers make use of cloud-images and kvm to help develop and test new
3372+curtin features.
3373+
3374+Install dependencies
3375+====================
3376+
3377+Install some virtualization and cloud packages to get started.::
3378+
3379+ sudo apt-get -qy install kvm libvirt-bin cloud-utils bzr
3380+
3381+
3382+Download cloud images
3383+=====================
3384+Curtin will use two cloud images (-disk1.img) is for booting,
3385+(-root.tar.gz) is used for installing.::
3386+
3387+ mkdir -p ~/download
3388+ DLDIR=$( cd ~/download && pwd )
3389+ rel="trusty"
3390+ arch=amd64
3391+ burl="http://cloud-images.ubuntu.com/$rel/current/"
3392+ for f in $rel-server-cloudimg-${arch}-root.tar.gz $rel-server-cloudimg-${arch}-disk1.img; do
3393+ wget "$burl/$f" -O $DLDIR/$f; done
3394+ ( cd $DLDIR && qemu-img convert -O qcow $rel-server-cloudimg-${arch}-disk1.img $rel-server-cloudimg-${arch}-disk1.qcow2)
3395+
3396+ export BOOTIMG="$DLDIR/$rel-server-cloudimg-${arch}-disk1.qcow2"
3397+ export ROOTTGZ="$DLDIR/$rel-server-cloudimg-${arch}-root.tar.gz"
3398+
3399+
3400+Getting the source
3401+==================
3402+Download curtin source from launchpad via `bzr` command.::
3403+
3404+ mkdir -p ~/src
3405+ bzr init-repo ~/src/curtin
3406+ ( cd ~/src/curtin && bzr branch lp:curtin trunk.dist )
3407+ ( cd ~/src/curtin && bzr branch trunk.dist trunk )
3408+
3409+Using curtin
3410+============
3411+Use `launch` to launch a kvm instance with user data to pack up
3412+local curtin and run it inside an instance.::
3413+
3414+ cd ~/src/curtin/trunk
3415+ ./tools/launch $BOOTIMG --publish $ROOTTGZ -- curtin install "PUBURL/${ROOTTGZ##*/}"
3416+
3417+
3418+Notes about 'launch'
3419+====================
3420+
3421+- launch has --help so you can see that for some info.
3422+- `--publish` adds a web server at ${HTTP_PORT:-$RANDOM_PORT}
3423+ and puts the files you want available there. You can reference
3424+ this url in config or cmdline with 'PUBURL'. For example
3425+ `--publish foo.img` will put `foo.img` at PUBURL/foo.img.
3426+- launch sets 'ubuntu' user password to 'passw0rd'
3427+- launch runs 'kvm -curses'
3428+ kvm -curses keyboard info:
3429+ 'alt-2' to go to qemu console
3430+- launch puts serial console to 'serial.log' (look there for stuff)
3431+- when logged in
3432+ - you can look at /var/log/cloud-init-output.log
3433+ - archive should be extracted in /curtin
3434+ - shell archive should be in /var/lib/cloud/instance/scripts/part-002
3435
3436=== added file 'doc/topics/integration-testing.rst'
3437--- doc/topics/integration-testing.rst 1970-01-01 00:00:00 +0000
3438+++ doc/topics/integration-testing.rst 2016-09-29 18:49:30 +0000
3439@@ -0,0 +1,245 @@
3440+===================
3441+Integration Testing
3442+===================
3443+
3444+Curtin includes an in-tree integration suite that runs hundreds of tests
3445+validating various forms of custom storage and network configurations across
3446+all of the supported Ubuntu LTS releases as well as some of the currently
3447+supported interim releases.
3448+
3449+Background
3450+==========
3451+
3452+Curtin includes a mechanism called 'vmtest' that allows it to actually
3453+do installs and validate a number of configurations.
3454+
3455+The general flow of the vmtests is:
3456+
3457+#. each test has an associated yaml config file for curtin in examples/tests
3458+#. uses curtin-pack to create the user-data for cloud-init to trigger install
3459+#. create and install a system using 'tools/launch'.
3460+
3461+ #. The install environment is booted from a maas ephemeral image.
3462+ #. kernel & initrd used are from maas images (not part of the image)
3463+ #. network by default is handled via user networking
3464+ #. It creates all empty disks required
3465+ #. cloud-init datasource is provided by launch
3466+
3467+ #. like: ds=nocloud-net;seedfrom=http://10.7.0.41:41518/
3468+ provided by python webserver start_http
3469+ #. via -drive file=/tmp/launch.8VOiOn/seed.img,if=virtio,media=cdrom
3470+ as a seed disk (if booted without external kernel)
3471+
3472+ #. dependencies and other preparations are installed at the beginning by
3473+ curtin inside the ephemeral image prior to configuring the target
3474+
3475+#. power off the system.
3476+#. configure a 'NoCloud' datasource seed image that provides scripts that
3477+ will run on first boot.
3478+
3479+ #. this will contain all our code to gather health data on the install
3480+ #. by cloud-init design this runs only once per instance, if you start
3481+ the system again this won't be called again
3482+
3483+#. boot the installed system with 'tools/xkvm'.
3484+
3485+ #. reuses the disks that were installed/configured in the former steps
3486+ #. also adds an output disk
3487+ #. additionally the seed image for the data gathering is added
3488+ #. On this boot it will run the provided scripts, write their output to a
3489+ "data" disk and then shut itself down.
3490+
3491+#. extract the data from the output disk
3492+#. vmtest python code now verifies if the output is as expected.
3493+
3494+Debugging
3495+=========
3496+
3497+At 3.1 one can pull data out of the maas image the command
3498+``mount-image-callback``. For example::
3499+
3500+ sudo mount-image-callback your.img -- sh -c 'cp $MOUNTPOINT/boot/* .'
3501+
3502+At step 3.6 through 4 ``tools/launch`` can be called in a way to give you
3503+console access. To do so just call tools/launch but drop the -serial=x
3504+parameter. One might want to change "'power_state': {'mode': 'poweroff'}" to
3505+avoid the auto reboot before getting control. Replace the directory usually
3506+seen in the launch calls with a clean fresh directory.
3507+
3508+In /curtin curtin and its config can be found. If the system gets that far
3509+cloud-init will create a user creds: ubuntu/passw0rd , otherwise one can use a
3510+cloud-image from https://cloud-images.ubuntu.com/ and add a backdoor user
3511+via::
3512+
3513+ bzr branch lp:~maas-maintainers/maas/backdoor-image backdoor-image
3514+ sudo ./backdoor-image -v --user=<USER> --password-auth --password=<PW> IMG
3515+
3516+At step 6 -> 7 you might want to keep all the temporary images around. To do
3517+so you can set ``CURTIN_VMTEST_KEEP_DATA_PASS=all`` in your environment. ::
3518+
3519+ export CURTIN_VMTEST_KEEP_DATA_PASS=all CURTIN_VMTEST_KEEP_DATA_FAIL=all
3520+
3521+That will keep the /tmp/tmpXXXXX directories and all files in there for further
3522+execution.
3523+
3524+At step 7 you might want to take a look at the output disk yourself. It is a
3525+normal qcow image, so one can use ``mount-image-callback`` as described above.
3526+
3527+To invoke xkvm on your own take the command you see in the output and remove
3528+the "-serial ..." but add ``-nographic`` instead For graphical console one can
3529+add ``--vnc 127.0.0.1:1``
3530+
3531+Setup
3532+=====
3533+
3534+In order to run vmtest you'll need some dependencies. To get them, you
3535+can run::
3536+
3537+ make vmtest-deps
3538+
3539+Running
3540+=======
3541+
3542+Running tests is done most simply by::
3543+
3544+ make vmtest
3545+
3546+If you wish to all tests in test_network.py, do so with::
3547+
3548+ nosetests3 tests/vmtests/test_network.py
3549+
3550+Or run a single test with::
3551+
3552+ nosetests3 tests/vmtests/test_network.py:WilyTestBasic
3553+
3554+
3555+Environment Variables
3556+=====================
3557+
3558+Some environment variables affect the running of vmtest
3559+
3560+- ``apt_proxy``:
3561+
3562+ test will set apt: { proxy } in the guests to the value of ``apt_proxy``
3563+ environment variable. If that is not set it will look at the host's apt
3564+ config and read ``Acquire::HTTP::Proxy``
3565+
3566+- ``CURTIN_VMTEST_KEEP_DATA_PASS``: Defaults to none.
3567+- ``CURTIN_VMTEST_KEEP_DATA_FAIL``: Defaults to all.
3568+
3569+ These 2 variables determine what portions of the temporary
3570+ test data are kept.
3571+
3572+ The variables contain a comma ',' delimited list of directories
3573+ that should be kept in the case of pass or fail. Additionally,
3574+ the values 'all' and 'none' are accepted.
3575+
3576+ Each vmtest that runs has its own sub-directory under the top level
3577+ ``CURTIN_VMTEST_TOPDIR``. In that directory are directories:
3578+
3579+ - ``boot``: inputs to the system boot (after install)
3580+ - ``install``: install phase related files
3581+ - ``disks``: the disks used for installation and boot
3582+ - ``logs``: install and boot logs
3583+ - ``collect``: data collected by the boot phase
3584+
3585+- ``CURTIN_VMTEST_TOPDIR``: default $TMPDIR/vmtest-<timestamp>
3586+
3587+ Vmtest puts all test data under this value. By default, it creates
3588+ a directory in TMPDIR (/tmp) named with as ``vmtest-<timestamp>``
3589+
3590+ If you set this value, you must ensure that the directory is either
3591+ non-existent or clean.
3592+
3593+- ``CURTIN_VMTEST_LOG``: default $TMPDIR/vmtest-<timestamp>.log
3594+
3595+ Vmtest writes extended log information to this file.
3596+ The default puts the log along side the TOPDIR.
3597+
3598+- ``CURTIN_VMTEST_IMAGE_SYNC``: default false (boolean)
3599+
3600+ If set to true, each run will attempt a sync of images.
3601+ If you want to make sure images are always up to date, then set to true.
3602+
3603+- ``CURTIN_VMTEST_BRIDGE``: ``user``
3604+
3605+ The network devices will be attached to this bridge. The default is
3606+ ``user``, which means to use qemu user mode networking. Set it to
3607+ ``virbr0`` or ``lxdbr0`` to use those bridges and then be able to ssh
3608+ in directly.
3609+
3610+- ``CURTIN_VMTEST_BOOT_TIMEOUT``: default 300
3611+
3612+ timeout before giving up on the boot of the installed system.
3613+
3614+- ``CURTIN_VMTEST_INSTALL_TIMEOUT``: default 3000
3615+
3616+ timeout before giving up on installation.
3617+
3618+- ``CURTIN_VMTEST_PARALLEL``: default ''
3619+
3620+ only supported through ./tools/jenkins-runner .
3621+
3622+ - ``-1``: then run one per core.
3623+ - ``0`` or ``''``: run with no parallel
3624+ - ``>0``: run with N processes
3625+
3626+ This modifies the invocation of nosetets to add '--processes' and other
3627+ necessary nose arguments (--process-timeout)
3628+
3629+- ``IMAGE_DIR``: default /srv/images
3630+
3631+ Vmtest keeps a mirror of maas ephemeral images in this directory.
3632+
3633+- ``IMAGES_TO_KEEP``: default 1
3634+
3635+ Controls the number of images of each release retained in the IMAGE_DIR.
3636+
3637+Environment 'boolean' values
3638+============================
3639+
3640+For boolean environment variables the value is considered True
3641+if it is any value other than case insensitive 'false', '' or "0".
3642+
3643+Test Class Variables
3644+====================
3645+
3646+The base VMBaseClass defines several variables that help creating a new test
3647+easily. Among those the common ones are:
3648+
3649+Generic:
3650+
3651+- ``arch_skip``:
3652+
3653+ If a test is not supported on an architecture it can list the arch in this
3654+ variable to auto-skip the test if executed on that arch.
3655+
3656+- ``conf_file``:
3657+
3658+ The configuration that will be processed by this vmtest.
3659+
3660+- ``extra_kern_args``:
3661+
3662+ Extra arguments to the guest kernel on boot.
3663+
3664+Data Collection:
3665+
3666+- ``collect_scripts``:
3667+
3668+ The commands run when booting into the installed environment to collect the
3669+ data for the test to verify a proper execution.
3670+
3671+- ``boot_cloudconf``:
3672+
3673+ Extra cloud-init config content for the install phase. This allows to gather
3674+ content of the install phase if needed for test verification.
3675+
3676+Disk Setup:
3677+
3678+- ``disk_block_size``:
3679+
3680+ Default block size ``512`` bytes.
3681+
3682+- ``disk_driver``:
3683+
3684+ Default block device driver is ``virtio-blk``.
3685
3686=== added file 'doc/topics/networking.rst'
3687--- doc/topics/networking.rst 1970-01-01 00:00:00 +0000
3688+++ doc/topics/networking.rst 2016-09-29 18:49:30 +0000
3689@@ -0,0 +1,522 @@
3690+==========
3691+Networking
3692+==========
3693+
3694+Curtin supports a user-configurable networking configuration format.
3695+This format lets users (including via MAAS) to customize their machines'
3696+networking interfaces by assigning subnet configuration, virtual device
3697+creation (bonds, bridges, vlans) routes and DNS configuration.
3698+
3699+Curtin accepts a YAML input under the top-level ``network`` key
3700+to indicate that a user would like to specify a custom networking
3701+configuration. Required elements of a network configuration are
3702+``config`` and ``version``. Currently curtin only supports
3703+network config version=1. ::
3704+
3705+ network:
3706+ version: 1
3707+ config: []
3708+
3709+Configuration Types
3710+-------------------
3711+Within the network ``config`` portion, users include a list of configuration
3712+types. The current list of support ``type`` values are as follows:
3713+
3714+- Physical (``physical``)
3715+- Bond (``bond``)
3716+- Bridge (``bridge``)
3717+- VLAN (``vlan``)
3718+- Nameserver (``nameserver``)
3719+- Route (``route``)
3720+
3721+Physical, Bond, Bridge and VLAN types may also include IP configuration under
3722+the key ``subnets``.
3723+
3724+- Subnet/IP (``subnets``)
3725+
3726+
3727+Physical
3728+~~~~~~~~
3729+The ``physical`` type configuration represents a "physical" network device,
3730+typically Ethernet-based. At least one of of these entries is required for
3731+external network connectivity. Type ``physical`` requires only one key:
3732+``name``. A ``physical`` device may contain some or all of the following keys:
3733+
3734+**name**: *<desired device name>*
3735+
3736+A devices name must be less than 15 characters. Names exceeding the maximum
3737+will be truncated. This is a limitation of the Linux kernel network-device
3738+structure.
3739+
3740+**mac_address**: *<MAC Address>*
3741+
3742+The MAC Address is a device unique identifier that most Ethernet-based network
3743+devices possess. Specifying a MAC Address is optional.
3744+
3745+
3746+.. note::
3747+
3748+ Curtin will emit a udev rule to provide a persistent mapping between a
3749+ device's ``name`` and the ``mac_address``.
3750+
3751+**mtu**: *<MTU SizeBytes>*
3752+
3753+The MTU key represents a device's Maximum Transmission Unit, the largest size
3754+packet or frame, specified in octets (eight-bit bytes), that can be sent in a
3755+packet- or frame-based network. Specifying ``mtu`` is optional.
3756+
3757+.. note::
3758+
3759+ The possible supported values of a device's MTU is not available at
3760+ configuration time. It's possible to specify a value too large or to
3761+ small for a device and may be ignored by the device.
3762+
3763+
3764+**Physical Example**::
3765+
3766+ network:
3767+ version: 1
3768+ config:
3769+ # Simple network adapter
3770+ - type: physical
3771+ name: interface0
3772+ mac_address: 00:11:22:33:44:55
3773+ # Second nic with Jumbo frames
3774+ - type: physical
3775+ name: jumbo0
3776+ mac_address: aa:11:22:33:44:55
3777+ mtu: 9000
3778+ # 10G pair
3779+ - type: physical
3780+ name: gbe0
3781+ mac_address: cd:11:22:33:44:00
3782+ - type: physical
3783+ name: gbe1
3784+ mac_address: cd:11:22:33:44:02
3785+
3786+Bond
3787+~~~~
3788+A ``bond`` type will configure a Linux software Bond with one or more network
3789+devices. A ``bond`` type requires the following keys:
3790+
3791+**name**: *<desired device name>*
3792+
3793+A devices name must be less than 15 characters. Names exceeding the maximum
3794+will be truncated. This is a limitation of the Linux kernel network-device
3795+structure.
3796+
3797+**mac_address**: *<MAC Address>*
3798+
3799+When specifying MAC Address on a bond this value will be assigned to the bond
3800+device and may be different than the MAC address of any of the underlying
3801+bond interfaces. Specifying a MAC Address is optional. If ``mac_address`` is
3802+not present, then the bond will use one of the MAC Address values from one of
3803+the bond interfaces.
3804+
3805+
3806+**bond_interfaces**: *<List of network device names>*
3807+
3808+The ``bond_interfaces`` key accepts a list of network device ``name`` values
3809+from the configuration. This list may be empty.
3810+
3811+**params**: *<Dictionary of key: value bonding parameter pairs>*
3812+
3813+The ``params`` key in a bond holds a dictionary of bonding parameters.
3814+This dictionary may be empty. For more details on what the various bonding
3815+parameters mean please read the Linux Kernel Bonding.txt.
3816+
3817+Valid ``params`` keys are:
3818+
3819+ - ``active_slave``: Set bond attribute
3820+ - ``ad_actor_key``: Set bond attribute
3821+ - ``ad_actor_sys_prio``: Set bond attribute
3822+ - ``ad_actor_system``: Set bond attribute
3823+ - ``ad_aggregator``: Set bond attribute
3824+ - ``ad_num_ports``: Set bond attribute
3825+ - ``ad_partner_key``: Set bond attribute
3826+ - ``ad_partner_mac``: Set bond attribute
3827+ - ``ad_select``: Set bond attribute
3828+ - ``ad_user_port_key``: Set bond attribute
3829+ - ``all_slaves_active``: Set bond attribute
3830+ - ``arp_all_targets``: Set bond attribute
3831+ - ``arp_interval``: Set bond attribute
3832+ - ``arp_ip_target``: Set bond attribute
3833+ - ``arp_validate``: Set bond attribute
3834+ - ``downdelay``: Set bond attribute
3835+ - ``fail_over_mac``: Set bond attribute
3836+ - ``lacp_rate``: Set bond attribute
3837+ - ``lp_interval``: Set bond attribute
3838+ - ``miimon``: Set bond attribute
3839+ - ``mii_status``: Set bond attribute
3840+ - ``min_links``: Set bond attribute
3841+ - ``mode``: Set bond attribute
3842+ - ``num_grat_arp``: Set bond attribute
3843+ - ``num_unsol_na``: Set bond attribute
3844+ - ``packets_per_slave``: Set bond attribute
3845+ - ``primary``: Set bond attribute
3846+ - ``primary_reselect``: Set bond attribute
3847+ - ``queue_id``: Set bond attribute
3848+ - ``resend_igmp``: Set bond attribute
3849+ - ``slaves``: Set bond attribute
3850+ - ``tlb_dynamic_lb``: Set bond attribute
3851+ - ``updelay``: Set bond attribute
3852+ - ``use_carrier``: Set bond attribute
3853+ - ``xmit_hash_policy``: Set bond attribute
3854+
3855+**Bond Example**::
3856+
3857+ network:
3858+ version: 1
3859+ config:
3860+ # Simple network adapter
3861+ - type: physical
3862+ name: interface0
3863+ mac_address: 00:11:22:33:44:55
3864+ # 10G pair
3865+ - type: physical
3866+ name: gbe0
3867+ mac_address: cd:11:22:33:44:00
3868+ - type: physical
3869+ name: gbe1
3870+ mac_address: cd:11:22:33:44:02
3871+ - type: bond
3872+ name: bond0
3873+ bond_interfaces:
3874+ - gbe0
3875+ - gbe1
3876+ params:
3877+ bond-mode: active-backup
3878+
3879+Bridge
3880+~~~~~~
3881+Type ``bridge`` requires the following keys:
3882+
3883+- ``name``: Set the name of the bridge.
3884+- ``bridge_interfaces``: Specify the ports of a bridge via their ``name``. This list may be empty.
3885+- ``params``: A list of bridge params. For more details, please read the bridge-utils-interfaces manpage.
3886+
3887+Valid keys are:
3888+
3889+ - ``bridge_ageing``: Set the bridge's ageing value.
3890+ - ``bridge_bridgeprio``: Set the bridge device network priority.
3891+ - ``bridge_fd``: Set the bridge's forward delay.
3892+ - ``bridge_hello``: Set the bridge's hello value.
3893+ - ``bridge_hw``: Set the bridge's MAC address.
3894+ - ``bridge_maxage``: Set the bridge's maxage value.
3895+ - ``bridge_maxwait``: Set how long network scripts should wait for the bridge to be up.
3896+ - ``bridge_pathcost``: Set the cost of a specific port on the bridge.
3897+ - ``bridge_portprio``: Set the priority of a specific port on the bridge.
3898+ - ``bridge_ports``: List of devices that are part of the bridge.
3899+ - ``bridge_stp``: Set spanning tree protocol on or off.
3900+ - ``bridge_waitport``: Set amount of time in seconds to wait on specific ports to become available.
3901+
3902+
3903+**Bridge Example**::
3904+
3905+ network:
3906+ version: 1
3907+ config:
3908+ # Simple network adapter
3909+ - type: physical
3910+ name: interface0
3911+ mac_address: 00:11:22:33:44:55
3912+ # Second nic with Jumbo frames
3913+ - type: physical
3914+ name: jumbo0
3915+ mac_address: aa:11:22:33:44:55
3916+ mtu: 9000
3917+ - type: bridge
3918+ name: br0
3919+ bridge_interfaces:
3920+ - jumbo0
3921+ params:
3922+ bridge_ageing: 250
3923+ bridge_bridgeprio: 22
3924+ bridge_fd: 1
3925+ bridge_hello: 1
3926+ bridge_maxage: 10
3927+ bridge_maxwait: 0
3928+ bridge_pathcost:
3929+ - jumbo0 75
3930+ bridge_pathprio:
3931+ - jumbo0 28
3932+ bridge_stp: 'off'
3933+ bridge_maxwait:
3934+ - jumbo0 0
3935+
3936+
3937+VLAN
3938+~~~~
3939+Type ``vlan`` requires the following keys:
3940+
3941+- ``name``: Set the name of the VLAN
3942+- ``vlan_link``: Specify the underlying link via its ``name``.
3943+- ``vlan_id``: Specify the VLAN numeric id.
3944+
3945+**VLAN Example**::
3946+
3947+ network:
3948+ version: 1
3949+ config:
3950+ # Physical interfaces.
3951+ - type: physical
3952+ name: eth0
3953+ mac_address: "c0:d6:9f:2c:e8:80"
3954+ # VLAN interface.
3955+ - type: vlan
3956+ name: eth0.101
3957+ vlan_link: eth0
3958+ vlan_id: 101
3959+ mtu: 1500
3960+
3961+Nameserver
3962+~~~~~~~~~~
3963+
3964+Users can specify a ``nameserver`` type. Nameserver dictionaries include
3965+the following keys:
3966+
3967+- ``address``: List of IPv4 or IPv6 address of nameservers.
3968+- ``search``: List of of hostnames to include in the resolv.conf search path.
3969+
3970+**Nameserver Example**::
3971+
3972+ network:
3973+ version: 1
3974+ config:
3975+ - type: physical
3976+ name: interface0
3977+ mac_address: 00:11:22:33:44:55
3978+ subnets:
3979+ - type: static
3980+ address: 192.168.23.14/27
3981+ gateway: 192.168.23.1
3982+ - type: nameserver:
3983+ address:
3984+ - 192.168.23.2
3985+ - 8.8.8.8
3986+ search:
3987+ - exemplary
3988+
3989+
3990+
3991+Route
3992+~~~~~
3993+
3994+Users can include static routing information as well. A ``route`` dictionary
3995+has the following keys:
3996+
3997+- ``destination``: IPv4 network address with CIDR netmask notation.
3998+- ``gateway``: IPv4 gateway address with CIDR netmask notation.
3999+- ``metric``: Integer which sets the network metric value for this route.
4000+
4001+**Route Example**::
4002+
4003+ network:
4004+ version: 1
4005+ config:
4006+ - type: physical
4007+ name: interface0
4008+ mac_address: 00:11:22:33:44:55
4009+ subnets:
4010+ - type: static
4011+ address: 192.168.23.14/24
4012+ gateway: 192.168.23.1
4013+ - type: route
4014+ destination: 192.168.24.0/24
4015+ gateway: 192.168.24.1
4016+ metric: 3
4017+
4018+Subnet/IP
4019+~~~~~~~~~
4020+
4021+For any network device (one of the Config Types) users can define a list of
4022+``subnets`` which contain ip configuration dictionaries. Multiple subnet
4023+entries will create interface alias allowing a single interface to use different
4024+ip configurations.
4025+
4026+Valid keys for for ``subnets`` include the following:
4027+
4028+- ``type``: Specify the subnet type.
4029+- ``control``: Specify manual, auto or hotplug. Indicates how the interface will be handled during boot.
4030+- ``address``: IPv4 or IPv6 address. It may include CIDR netmask notation.
4031+- ``netmask``: IPv4 subnet mask in dotted format or CIDR notation.
4032+- ``gateway``: IPv4 address of the default gateway for this subnet.
4033+- ``dns_nameserver``: Specify a list of IPv4 dns server IPs to end up in resolv.conf.
4034+- ``dns_search``: Specify a list of search paths to be included in resolv.conf.
4035+
4036+
4037+Subnet types are one of the following:
4038+
4039+- ``dhcp4``: Configure this interface with IPv4 dhcp.
4040+- ``dhcp``: Alias for ``dhcp4``
4041+- ``dhcp6``: Configure this interface with IPv6 dhcp.
4042+- ``static``: Configure this interface with a static IPv4.
4043+- ``static6``: Configure this interface with a static IPv6 .
4044+
4045+When making use of ``dhcp`` types, no additional configuration is needed in the
4046+subnet dictionary.
4047+
4048+
4049+**Subnet DHCP Example**::
4050+
4051+ network:
4052+ version: 1
4053+ config:
4054+ - type: physical
4055+ name: interface0
4056+ mac_address: 00:11:22:33:44:55
4057+ subnets:
4058+ - type: dhcp
4059+
4060+
4061+**Subnet Static Example**::
4062+
4063+ network:
4064+ version: 1
4065+ config:
4066+ - type: physical
4067+ name: interface0
4068+ mac_address: 00:11:22:33:44:55
4069+ subnets:
4070+ - type: static
4071+ address: 192.168.23.14/27
4072+ gateway: 192.168.23.1
4073+ dns_nameservers:
4074+ - 192.168.23.2
4075+ - 8.8.8.8
4076+ dns_search:
4077+ - exemplary.maas
4078+
4079+The following will result in an ``interface0`` using DHCP and ``interface0:1``
4080+using the static subnet configuration.
4081+
4082+**Multiple subnet Example**::
4083+
4084+ network:
4085+ version: 1
4086+ config:
4087+ - type: physical
4088+ name: interface0
4089+ mac_address: 00:11:22:33:44:55
4090+ subnets:
4091+ - type: dhcp
4092+ - type: static
4093+ address: 192.168.23.14/27
4094+ gateway: 192.168.23.1
4095+ dns_nameservers:
4096+ - 192.168.23.2
4097+ - 8.8.8.8
4098+ dns_search:
4099+ - exemplary
4100+
4101+
4102+Multi-layered configurations
4103+----------------------------
4104+
4105+Complex networking sometimes uses layers of configuration. The syntax allows
4106+users to build those layers one at a time. All of the virtual network devices
4107+supported allow specifying an underlying device by their ``name`` value.
4108+
4109+**Bonded VLAN Example**::
4110+
4111+ network:
4112+ version: 1
4113+ config:
4114+ # 10G pair
4115+ - type: physical
4116+ name: gbe0
4117+ mac_address: cd:11:22:33:44:00
4118+ - type: physical
4119+ name: gbe1
4120+ mac_address: cd:11:22:33:44:02
4121+ # Bond.
4122+ - type: bond
4123+ name: bond0
4124+ bond_interfaces:
4125+ - gbe0
4126+ - gbe1
4127+ params:
4128+ bond-mode: 802.3ad
4129+ bond-lacp-rate: fast
4130+ # A Bond VLAN.
4131+ - type: vlan
4132+ name: bond0.200
4133+ vlan_link: bond0
4134+ vlan_id: 200
4135+ subnets:
4136+ - type: dhcp4
4137+
4138+More Examples
4139+-------------
4140+Some more examples to explore the various options available.
4141+
4142+**Multiple VLAN example**::
4143+
4144+ network:
4145+ version: 1
4146+ config:
4147+ - id: eth0
4148+ mac_address: d4:be:d9:a8:49:13
4149+ mtu: 1500
4150+ name: eth0
4151+ subnets:
4152+ - address: 10.245.168.16/21
4153+ dns_nameservers:
4154+ - 10.245.168.2
4155+ gateway: 10.245.168.1
4156+ type: static
4157+ type: physical
4158+ - id: eth1
4159+ mac_address: d4:be:d9:a8:49:15
4160+ mtu: 1500
4161+ name: eth1
4162+ subnets:
4163+ - address: 10.245.188.2/24
4164+ dns_nameservers: []
4165+ type: static
4166+ type: physical
4167+ - id: eth1.2667
4168+ mtu: 1500
4169+ name: eth1.2667
4170+ subnets:
4171+ - address: 10.245.184.2/24
4172+ dns_nameservers: []
4173+ type: static
4174+ type: vlan
4175+ vlan_id: 2667
4176+ vlan_link: eth1
4177+ - id: eth1.2668
4178+ mtu: 1500
4179+ name: eth1.2668
4180+ subnets:
4181+ - address: 10.245.185.1/24
4182+ dns_nameservers: []
4183+ type: static
4184+ type: vlan
4185+ vlan_id: 2668
4186+ vlan_link: eth1
4187+ - id: eth1.2669
4188+ mtu: 1500
4189+ name: eth1.2669
4190+ subnets:
4191+ - address: 10.245.186.1/24
4192+ dns_nameservers: []
4193+ type: static
4194+ type: vlan
4195+ vlan_id: 2669
4196+ vlan_link: eth1
4197+ - id: eth1.2670
4198+ mtu: 1500
4199+ name: eth1.2670
4200+ subnets:
4201+ - address: 10.245.187.2/24
4202+ dns_nameservers: []
4203+ type: static
4204+ type: vlan
4205+ vlan_id: 2670
4206+ vlan_link: eth1
4207+ - address: 10.245.168.2
4208+ search:
4209+ - dellstack
4210+ type: nameserver
4211+
4212
4213=== modified file 'doc/topics/overview.rst'
4214--- doc/topics/overview.rst 2013-07-29 16:12:09 +0000
4215+++ doc/topics/overview.rst 2016-09-29 18:49:30 +0000
4216@@ -20,7 +20,7 @@
4217 ~~~~~~~~~~~~~~~~~~~~~~~~
4218 At the moment, curtin doesn't address how the system that it is running on is booted. It could be booted from a live-cd or from a pxe boot environment. It could even be booted off a disk in the system (although installation to that disk would probably break things).
4219
4220-Curtin's assumption is that a fairly rich linux (Ubuntu) environment is booted.
4221+Curtin's assumption is that a fairly rich Linux (Ubuntu) environment is booted.
4222
4223 Early Commands
4224 ~~~~~~~~~~~~~~
4225@@ -38,13 +38,13 @@
4226
4227 Partitioning
4228 ~~~~~~~~~~~~
4229-Partitioning covers setting up filesystems on the system. A series of commands are run serially in order. At the end, a fstab formated file must be populated in ``OUTPUT_FSTAB`` that contains mount information, and the filesystems are expected to be mounted at the ``TARGET_MOUNT_POINT``.
4230+Partitioning covers setting up filesystems on the system. A series of commands are run serially in order. At the end, a fstab formatted file must be populated in ``OUTPUT_FSTAB`` that contains mount information, and the filesystems are expected to be mounted at the ``TARGET_MOUNT_POINT``.
4231
4232 Any commands can be used to create this filesystem, but curtin contains some tools to facilitate with this process.
4233
4234 **Config Example**::
4235
4236- paritioning_commands:
4237+ partitioning_commands:
4238 10_wipe_filesystems: curtin wipe --quick --all-unused-disks
4239 50_setup_raid: curtin disk-setup --all-disks raid0 /
4240
4241@@ -53,7 +53,7 @@
4242 Partitioning commands have the following environment variables available to them:
4243
4244 - ``WORKING_DIR``: This is simply for some sort of inter-command state. It will be the same directory for each command run and will only be deleted at the end of all partitioning_commands.
4245-- ``OUTPUT_FSTAB``: This is the target path for a fstab file. After all partitioning commands have been run, a file should exist, formated per fstab(5) that describes how the filesystems should be mounted.
4246+- ``OUTPUT_FSTAB``: This is the target path for a fstab file. After all partitioning commands have been run, a file should exist, formatted per fstab(5) that describes how the filesystems should be mounted.
4247 - ``TARGET_MOUNT_POINT``:
4248
4249
4250@@ -61,7 +61,7 @@
4251 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
4252 Networking is done in a similar fashion to partitioning. A series of commands, specified in the config are run. At the end of these commands, a interfaces(5) style file is expected to be written to ``OUTPUT_INTERFACES``.
4253
4254-Note, that as with fstab, this file is not copied verbatum to the target filesystem, but rather made availble to the OS customization stage. That stage may just copy the file verbatum, but may also parse it, and use that as input.
4255+Note, that as with fstab, this file is not copied verbatim to the target filesystem, but rather made available to the OS customization stage. That stage may just copy the file verbatim, but may also parse it, and use that as input.
4256
4257 **Config Example**::
4258
4259@@ -73,7 +73,7 @@
4260 Networking commands have the following environment variables available to them:
4261
4262 - ``WORKING_DIR``: This is simply for some sort of inter-command state. It will be the same directory for each command run and will only be deleted at the end of all network_commands.
4263-- ``OUTPUT_INTERFACES``: This is the target path for an interfaces style file. After all commands have been run, a file should exist, formated per interfaces(5) that describes the systems network setup.
4264+- ``OUTPUT_INTERFACES``: This is the target path for an interfaces style file. After all commands have been run, a file should exist, formatted per interfaces(5) that describes the systems network setup.
4265
4266 Extraction of sources
4267 ~~~~~~~~~~~~~~~~~~~~~
4268@@ -84,7 +84,7 @@
4269 sources:
4270 05_primary: http://cloud-images.ubuntu.com/releases/precise/release/ubuntu-12.04-server-cloudimg-amd64-root.tar.gz
4271
4272-Given the source above, curtin will essentiall do a::
4273+Given the source above, curtin will essentially do a::
4274
4275 wget $URL | tar -Sxvzf
4276
4277
4278=== modified file 'doc/topics/reporting.rst'
4279--- doc/topics/reporting.rst 2016-05-05 16:43:40 +0000
4280+++ doc/topics/reporting.rst 2016-09-29 18:49:30 +0000
4281@@ -7,7 +7,7 @@
4282
4283 Events
4284 ------
4285-Reporting consists of notifcation of a series of 'events. Each event has:
4286+Reporting consists of notification of a series of 'events. Each event has:
4287 - **event_type**: 'start' or 'finish'
4288 - **description**: human readable text
4289 - **name**: and id for this event
4290@@ -48,7 +48,7 @@
4291 Each entry must have a 'type'. The currently supported values are:
4292 - **log**: logs via python logger
4293 - **print**: prints messages to stdout (for debugging)
4294- - **webhook**: posts json formated data to a remote url. Supports Oauth.
4295+ - **webhook**: posts json formatted data to a remote url. Supports Oauth.
4296
4297
4298 Additionally, the webhook reporter will post files on finish of curtin. The user can declare which files should be posted in the ``install`` item via ``post_files`` as shown above. If post_files is not present, it will default to the value of log_file.
4299@@ -154,7 +154,7 @@
4300 Legacy Reporter
4301 ---------------
4302 The legacy 'reporter' config entry is still supported. This was utilized by
4303-MAAS for start/end and posting of the install log at the end of isntallation.
4304+MAAS for start/end and posting of the install log at the end of installation.
4305
4306 Its configuration looks like this:
4307
4308
4309=== added file 'doc/topics/storage.rst'
4310--- doc/topics/storage.rst 1970-01-01 00:00:00 +0000
4311+++ doc/topics/storage.rst 2016-09-29 18:49:30 +0000
4312@@ -0,0 +1,894 @@
4313+=======
4314+Storage
4315+=======
4316+
4317+Curtin supports a user-configurable storage layout. This format lets users
4318+(including via MAAS) to customize their machines' storage configuration by
4319+creating partitions, RAIDs, LVMs, formatting with file systems and setting
4320+mount points.
4321+
4322+Custom storage configuration is handled by the ``block-meta custom`` command
4323+in curtin. Partitioning layout is read as a list of in-order modifications to
4324+make to achieve the desired configuration. The top level configuration key
4325+containing this configuration is ``storage``. This key should contain a
4326+dictionary with at least a version number and the configuration list. The
4327+current config specification is ``version: 1``.
4328+
4329+**Config Example**::
4330+
4331+ storage:
4332+ version: 1
4333+ config:
4334+ - id: sda
4335+ type: disk
4336+ ptable: gpt
4337+ serial: QM00002
4338+ model: QEMU_HARDDISK
4339+
4340+Configuration Types
4341+-------------------
4342+Each entry in the config list is a dictionary with several keys which vary
4343+between commands. The two dictionary keys that every entry in the list needs
4344+to have are ``id: <id>`` and ``type: <type>``.
4345+
4346+An entry's ``id`` allows other entries in the config to refer to a specific
4347+entry. It can be any string other than one with a special meaning in yaml, such
4348+as ``true`` or ``none``.
4349+
4350+An entry's ``type`` tells curtin how to handle a particular entry. Available
4351+commands include:
4352+
4353+- Disk Command (``disk``)
4354+- Partition Command (``partition``)
4355+- Format Command (``format``)
4356+- Mount Command (``mount``)
4357+- LVM_VolGroup Command (``lvm_volgroup``)
4358+- LVM_Partition Command (``lvm_partition``)
4359+- DM_Crypt Command (``dm_crypt``)
4360+- RAID Command (``raid``)
4361+- Bcache Command (``bcache``)
4362+
4363+Disk Command
4364+~~~~~~~~~~~~
4365+The disk command sets up disks for use by curtin. It can wipe the disks, create
4366+partition tables, or just verify that the disks exist with an existing partition
4367+table. A disk command may contain all or some of the following keys:
4368+
4369+**ptable**: *msdos, gpt*
4370+
4371+If the ``ptable`` key is present and a valid type of partition table, curtin
4372+will create an empty partition table of that type on the disk. At the moment,
4373+msdos and gpt partition tables are supported.
4374+
4375+**serial**: *<serial number>*
4376+
4377+In order to uniquely identify a disk on the system its serial number should be
4378+specified. This ensures that even if additional storage devices
4379+are added to the system during installation, or udev rules cause the path to a
4380+disk to change curtin will still be able to correctly identify the disk it
4381+should be operating on using ``/dev/disk/by-id``.
4382+
4383+This is the preferred way to identify a disk and should be used in all
4384+production environments as it is less likely to point to an incorrect device.
4385+
4386+**path**: *<path to device with leading /dev*
4387+
4388+The ``path`` key can be used to identify the disk. If both ``serial`` and
4389+``path`` are specified, curtin will use the serial number and ignore the path
4390+that was specified.
4391+
4392+**model**: *<disk model>*
4393+
4394+This can specify the manufacturer or model of the disk. It is not currently
4395+used by curtin, but can be useful for a human reading a config file. Future
4396+versions of curtin may make use of this information.
4397+
4398+**wipe**: *superblock, superblock-recursive, zero, random*
4399+
4400+If wipe is specified, **the disk contents will be destroyed**. In the case that
4401+a disk is a part of virtual block device, like bcache, RAID array, or LVM, then
4402+curtin will attempt to tear down the virtual device to allow access to the disk
4403+for resetting the disk.
4404+
4405+The most common option for clearing a disk is ``wipe: superblock``. In some
4406+cases use of ``wipe: superblock-recursive`` is useful to ensure that embedded
4407+superblocks on a disk aren't rediscovered during probing. For example, LVM,
4408+bcache and RAID on a partition would have metadata outside of the range of a
4409+superblock wipe of the start and end sections of the disk.
4410+
4411+The ``wipe: zero`` option will write zeros to each sector of the disk.
4412+Depending on the size and speed of the disk; it may take a long time to
4413+complete.
4414+
4415+The ``wipe: random`` option will write pseudo-random data from /dev/urandom
4416+Depending on the size and speed of the disk; it may take a long time to
4417+complete.
4418+
4419+**preserve**: *true, false*
4420+
4421+When the preserve key is present and set to ``true`` curtin will attempt
4422+to use the disk without damaging data present on it. If ``preserve`` is set and
4423+``ptable`` is also set, then curtin will validate that the partition table
4424+specified by ``ptable`` exists on the disk and will raise an error if it does
4425+not. If ``preserve`` is set and ``ptable`` is not, then curtin will be able to
4426+use the disk in later commands, but will not check if the disk has a valid
4427+partition table, and will only verify that the disk exists.
4428+
4429+It can be dangerous to try to move or re-size filesystems and partitions
4430+containing data that needs to be preserved. Therefor curtin does not support
4431+preserving a disk without also preserving the partitions on it. If a disk is
4432+set to be preserved and curtin is told to move a partition on that disk,
4433+installation will stop. It is still possible to reformat partitions that do
4434+not need to be preserved.
4435+
4436+**name**: *<name>*
4437+
4438+If the ``name`` key is present, curtin will create a udev rule that makes a
4439+symbolic link to the disk with the given name value. This makes it easy to find
4440+disks on an installed system. The links are created in
4441+``/dev/disk/by-dname/<name>``.
4442+A link to each partition on the disk will also be created at
4443+``/dev/disk/by-dname/<name>-part<number>``, so if ``name: maindisk`` is set,
4444+the disk will be at ``/dev/disk/by-dname/maindisk`` and the first partition on
4445+it will be at ``/dev/disk/by-dname/maindisk-part1``.
4446+
4447+**grub_device**: *true, false*
4448+
4449+If the ``grub_device`` key is present and set to true, then when post
4450+installation hooks are run grub will be installed onto this disk. In most
4451+situations it is not necessary to specify this value as curtin will detect
4452+and determine which device to use as a boot disk. In cases where the boot
4453+device is on a special volume, such as a RAID array or a LVM Logical Volume,
4454+it may be necessary to specify the device that will hold the grub bootloader.
4455+
4456+**Config Example**::
4457+
4458+ - id: disk0
4459+ type: disk
4460+ ptable: gpt
4461+ serial: QM00002
4462+ model: QEMU_HARDDISK
4463+ name: maindisk
4464+ wipe: superblock
4465+
4466+Partition Command
4467+~~~~~~~~~~~~~~~~~
4468+The partition command creates a single partition on a disk. Curtin only needs
4469+to be told which disk to use and the size of the partition. Additional options
4470+are available.
4471+
4472+**number**: *<number>*
4473+
4474+The partition number can be specified using ``number``. However, numbers must
4475+be in order and some situations, such as extended/logical partitions on msdos
4476+partition tables will require special numbering, so it maybe better to omit
4477+the partition number. If the ``number`` key is not present, curtin will attempt
4478+determine the right number to use.
4479+
4480+**size**: *<size>*
4481+
4482+The partition size can be specified with the ``size`` key. Sizes must be
4483+given with an appropriate SI unit, such as *B, kB, MB, GB, TB*, or using just
4484+the appropriate SI prefix, i.e. *B, k, M, G, T...*
4485+
4486+.. note::
4487+
4488+ Curtin does not adjust size values. If you specific a size that exceeds the
4489+ capacity of a device then installation will fail.
4490+
4491+**device**: *<device id>*
4492+
4493+The ``device`` key refers to the ``id`` of a disk in the storage configuration.
4494+The disk entry must already be defined in the list of commands to ensure that
4495+it has already been processed.
4496+
4497+**wipe**: *superblock, pvremove, zero, random*
4498+
4499+After the partition is added to the disk's partition table, curtin can run a
4500+wipe command on the partition. The wipe command values are the sames as for
4501+disks.
4502+
4503+**flag**: *logical, extended, boot, bios_grub, swap, lvm, raid, home, prep*
4504+
4505+If the ``flag`` key is present, curtin will set the specified flag on the
4506+partition. Note that some flags only apply to msdos partition tables, and some
4507+only apply to gpt partition tables.
4508+
4509+The *logical/extended* partition flags can be used to create logical partitions
4510+on a msdos table. An extended partition should be created containing all of the
4511+empty space on the drive, and logical partitions can be created within it. A
4512+extended partition must already be present to create logical partitions. If the
4513+``number`` flag is set for an extended partition it must be set to 4, and
4514+each logical partition should be numbered starting from 5.
4515+
4516+On msdos partition tables, the *boot* flag sets the boot parameter to that
4517+partition. On gpt partition tables, the boot flag sets the esp flag on the
4518+partition.
4519+
4520+If the host system for curtin has been booted using UEFI then curtin will
4521+install grub to the esp partition. If the system installation media
4522+has been booted using an MBR, grub will be installed onto the disk's MBR.
4523+However, on a disk with a gpt partition table, there is not enough space after
4524+the MBR for grub to store its second stage core.img, so a small un-formatted
4525+partition with the *bios_grub* flag is needed. This partition should be placed
4526+at the beginning of the disk and should be 1MB in size. It should not contain a
4527+filesystem or be mounted anywhere on the system.
4528+
4529+**preserve**: *true, false*
4530+
4531+If the preserve flag is set to true, curtin will verify that the partition
4532+exists and will not modify the partition.
4533+
4534+**Config Example**::
4535+
4536+ - id: disk0-part1
4537+ type: partition
4538+ number: 1
4539+ size: 8GB
4540+ device: disk0
4541+ flag: boot
4542+
4543+Format Command
4544+~~~~~~~~~~~~~~
4545+The format command makes filesystems on a volume. The filesystem type and
4546+target volume can be specified, as well as a few other options.
4547+
4548+**fstype**: ext4, ext3, fat32, fat16, swap, xfs
4549+
4550+The ``fstype`` key specifies what type of filesystem format curtin should use
4551+for this volume. Curtin knows about common Linux filesystems such as ext4/3 and
4552+fat filesystems and makes use of additional parameters and flags to optimize the
4553+filesystem. If the ``fstype`` value is not known to curtin, that is not fatal.
4554+Curtin will check if ``mkfs.<fstype>`` exists and if so, will use that tool to
4555+format the target volume.
4556+
4557+For fat filesystems, the size of the fat table can be specified by entering
4558+*fat64*, *fat32*, *fat16*, or *fat12* instead of just entering *fat*.
4559+If *fat* is used, then ``mkfs.fat`` will automatically determine the best
4560+size fat table to use, probably *fat32*.
4561+
4562+If ``fstype: swap`` is set, curtin will create a swap partition on the target
4563+volume.
4564+
4565+**volume**: *<volume id>*
4566+
4567+The ``volume`` key refers to the ``id`` of the target volume in the storage
4568+config. The target volume must already exist and be accessible. Any type
4569+of target volume can be used as long as it has a block device that curtin
4570+can locate.
4571+
4572+**label**: *<volume name>*
4573+
4574+The ``label`` key tells curtin to create a filesystem LABEL when formatting a
4575+volume. Note that not all filesystem types support names and that there are
4576+length limits for names. For fat filesystems, names are limited to 11
4577+characters. For ext4/3 filesystems, names are limited to 16 characters.
4578+
4579+If curtin does not know about the filesystem type it is using, then the
4580+``label`` key will be ignored, because curtin will not know the correct flags
4581+to set the label value in the filesystem metadata.
4582+
4583+**uuid**: *<uuid>*
4584+
4585+If the ``uuid`` key is set and ``fstype`` is set to *ext4* or *ext3*, then
4586+curtin will set the uuid of the new filesystem to the specified value.
4587+
4588+**preserve**: *true, false*
4589+
4590+If the ``preserve`` key is set to true, curtin will not format the partition.
4591+
4592+**Config Example**::
4593+
4594+ - id: disk0-part1-fs1
4595+ type: format
4596+ fstype: ext4
4597+ label: cloud-image
4598+ volume: disk0-part1
4599+
4600+Mount Command
4601+~~~~~~~~~~~~~
4602+The mount command mounts the target filesystem and creates an entry for it in
4603+the newly installed system's ``/etc/fstab``. The path to the target mountpoint
4604+must be specified as well as the target filesystem.
4605+
4606+**path**: *<path>*
4607+
4608+The ``path`` key tells curtin where the filesystem should be mounted on the
4609+target system. An entry in the target system's ``/etc/fstab`` will be created
4610+for the target device which will mount it in the correct place once the
4611+installed system boots.
4612+
4613+If the device specified is formatted as swap space, then an entry will be added
4614+to the target system's ``/etc/fstab`` to make use of this swap space.
4615+
4616+When entries are created in ``/etc/fstab``, curtin will use the most reliable
4617+method available to identify each device. For regular partitions, curtin will
4618+use the UUID of the filesystem present on the partition. For special devices,
4619+such as RAID arrays, or LVM logical volumes, curtin will use their normal path
4620+in ``/dev``.
4621+
4622+**device**: *<device id>*
4623+
4624+The ``device`` key refers to the ``id`` of the target device in the storage
4625+config. The target device must already contain a valid filesystem and be
4626+accessible.
4627+
4628+**Config Example**::
4629+
4630+ - id: disk0-part1-fs1-mount0
4631+ type: mount
4632+ path: /home
4633+ device: disk0-part1-fs1
4634+
4635+Lvm Volgroup Command
4636+~~~~~~~~~~~~~~~~~~~~
4637+The lvm_volgroup command creates LVM Physical Volumes (PV) and connects them in
4638+a LVM Volume Group (vg). The command requires a name for the volgroup and a
4639+list of the devices that should be used as physical volumes.
4640+
4641+**name**: *<name>*
4642+
4643+The ``name`` key specifies the name of the volume group. It anything can be
4644+used except words with special meanings in YAML, such as *true*, or *none*.
4645+
4646+**devices**: *[]*
4647+
4648+The ``devices`` key gives a list of devices to use as physical volumes. Each
4649+device is specified using the ``id`` of existing devices in the storage config.
4650+Almost anything can be used as a device such as partitions, whole disks, RAID.
4651+
4652+**Config Example**::
4653+
4654+ - id: volgroup1
4655+ type: lvm_volgroup
4656+ name: vg1
4657+ devices:
4658+ - disk0-part2
4659+ - disk1
4660+
4661+Lvm Partition Command
4662+~~~~~~~~~~~~~~~~~~~~~
4663+The lvm_partition command creates a lvm logical volume on the specified
4664+volgroup with the specified size. It also assigns it the specified name.
4665+
4666+**name**: *<name>*
4667+
4668+The ``name`` key specifies the name of the Logical Volume (LV) to be created.
4669+
4670+Curtin creates udev rules for Logical Volumes to give them consistently named
4671+symbolic links in the target system under ``/dev/disk/by-dname/``. The naming
4672+scheme for Logical Volumes follows the pattern
4673+``<volgroup name>-<logical volume name>``. For example a ``lvm_partition``
4674+with ``name`` *lv1* on a ``lvm_volgroup`` named *vg1* would have the path
4675+``/dev/disk/by-dname/vg1-lv1``.
4676+
4677+**volgroup**: *<volgroup id>*
4678+
4679+The ``volgroup`` key specifies the ``id`` of the Volume Group in which to
4680+create the logical volume. The volgroup must already have been created and must
4681+have enough free space on it to create the logical volume. The volgroup should
4682+be specified using the ``id`` key of the volgroup in the storage config, not the
4683+name of the volgroup.
4684+
4685+**size**: *<size>*
4686+
4687+The ``size`` key tells curtin what size to make the logical volume. The size
4688+can be entered in any format that can be processed by the lvm2 tools, so a
4689+number followed by a SI unit should work, i.e. *B, kB, MB, GB, TB*.
4690+
4691+If the ``size`` key is omitted then all remaining space on the volgroup will be
4692+used for the logical volume.
4693+
4694+.. note::
4695+
4696+ Curtin does not adjust size values. If you specific a size that exceeds the
4697+ capacity of a device then installation will fail.
4698+
4699+
4700+**Config Example**::
4701+
4702+ - id: lvm_partition_1
4703+ type: lvm_partition
4704+ name: lv1
4705+ volgroup: volgroup1
4706+ size: 10G
4707+
4708+
4709+**Combined Example**::
4710+
4711+ - id: volgroup1
4712+ type: lvm_volgroup
4713+ name: vg1
4714+ devices:
4715+ - disk0-part2
4716+ - disk1
4717+ - id: lvm_partition_1
4718+ type: lvm_partition
4719+ name: lv1
4720+ volgroup: volgroup1
4721+ size: 10G
4722+
4723+
4724+
4725+Dm-Crypt Command
4726+~~~~~~~~~~~~~~~~
4727+The dm_crypt command creates encrypted volumes using ``cryptsetup``. It
4728+requires a name for the encrypted volume, the volume to be encrypted and a key.
4729+Note that this should not be used for systems where security is a requirement.
4730+The key is stored in plain-text in the storage configuration and it could be
4731+possible for the storage configuration to be intercepted between the utility
4732+that generates it and curtin.
4733+
4734+**volume**: *<volume id>*
4735+
4736+The ``volume`` key gives the volume that is to be encrypted.
4737+
4738+**dm_name**: *<name>*
4739+
4740+The ``name`` key specifies the name of the encrypted volume.
4741+
4742+**key**: *<key>*
4743+
4744+The ``key`` key specifies the password of the encryption key. The target
4745+system will prompt for this password in order to mount the disk.
4746+
4747+.. note::
4748+
4749+ Encrypted disks and partitions are tracked in ``/etc/crypttab`` and will be
4750+ mounted at boot time.
4751+
4752+**Config Example**::
4753+
4754+ - id: lvm_partition_1
4755+ type: dm_crypt
4756+ dm_name: crypto
4757+ volume: sdb1
4758+ key: testkey
4759+
4760+RAID Command
4761+~~~~~~~~~~~~
4762+The RAID command configures Linux Software RAID using mdadm. It needs to be given
4763+a name for the md device, a list of volumes for to compose the md device, an
4764+optional list of devices to be used as spare volumes, and RAID level.
4765+
4766+**name**: *<name>*
4767+
4768+The ``name`` key specifies the name of the md device.
4769+
4770+.. note::
4771+
4772+ Curtin creates a udev rule to create a link to the md device in
4773+ ``/dev/disk/by-dname/<name>`` using the specified name.
4774+
4775+**raidlevel**: *0, 1, 5, 6, 10*
4776+
4777+The ``raidlevel`` key specifies the raid level of the array.
4778+
4779+**devices**: *[]*
4780+
4781+The ``devices`` key specifies a list of the devices that will be used for the
4782+raid array. Each device must be referenced by ``id`` and the device must be
4783+previously defined in the storage configuration. Must not be empty.
4784+
4785+Devices can either be full disks or partition.
4786+
4787+
4788+**spare_devices**: *[]*
4789+
4790+The ``spare_devices`` key specifies a list of the devices that will be used for
4791+spares in the raid array. Each device must be referenced by ``id`` and the
4792+device must be previously defined in the storage configuration. May be empty.
4793+
4794+
4795+**Config Example**::
4796+
4797+ - id: raid_array
4798+ type: raid
4799+ name: md0
4800+ raidlevel: 1
4801+ devices:
4802+ - sdb
4803+ - sdc
4804+ spare_devices:
4805+ - sdd
4806+
4807+Bcache Command
4808+~~~~~~~~~~~~~~
4809+The bcache command will configure a block-cache device using the Linux kernel
4810+bcache module. Bcache allows users to use a typically small, but fast SSD or
4811+NVME device as a cache for larger, slower spinning disks.
4812+
4813+The bcache command needs to be told which device to use hold the data and which
4814+device to use as its cache device. A cache device may be reused with multiple
4815+backing devices.
4816+
4817+
4818+**backing_device**: *<device id>*
4819+
4820+The ``backing_device`` key specifies the item in storage configuration to use
4821+as the backing device. This can be any device that would normally be used with
4822+a filesystem on it, such as a partition or a raid array.
4823+
4824+**cache_device**: *<device id>*
4825+
4826+The ``cache_device`` key specifies the item in the storage configuration to use
4827+as the cache device. This can be a partition or a whole disk. It should be on a
4828+ssd in most cases, as bcache is designed around the performance characteristics
4829+of a ssd.
4830+
4831+**cache_mode**: *writethrough, writeback, writearound, none*
4832+
4833+The ``cache_mode`` key specifies the mode in which bcache operates. The
4834+default mode is writethrough which ensures data hits the backing device
4835+before completing the operation. writeback mode will have higher performance
4836+but exposes dataloss if the cache device fails. writearound will avoid using
4837+the cache for large sequential writes; useful for not evicting smaller
4838+reads/writes from the cache. None effectively disables bcache.
4839+
4840+**name**: *<name>*
4841+
4842+If the ``name`` key is present, curtin will create a link to the device at
4843+``/dev/disk/by-dname/<name>``.
4844+
4845+**Config Example**::
4846+
4847+ - id: bcache0
4848+ type: bcache
4849+ name: cached_raid
4850+ backing_device: raid_array
4851+ cache_device: sdb
4852+
4853+
4854+Additional Examples
4855+-------------------
4856+
4857+Learn by examples.
4858+
4859+- Basic
4860+- LVM
4861+- Bcache
4862+- RAID Boot
4863+- RAID5 + Bcache
4864+
4865+Basic Layout
4866+~~~~~~~~~~~~
4867+
4868+::
4869+
4870+ storage:
4871+ version: 1
4872+ config:
4873+ - id: disk0
4874+ type: disk
4875+ ptable: msdos
4876+ model: QEMU HARDDISK
4877+ path: /dev/vdb
4878+ name: main_disk
4879+ wipe: superblock
4880+ grub_device: true
4881+ - id: disk0-part1
4882+ type: partition
4883+ number: 1
4884+ size: 3GB
4885+ device: disk0
4886+ flag: boot
4887+ - id: disk0-part2
4888+ type: partition
4889+ number: 2
4890+ size: 1GB
4891+ device: disk0
4892+ - id: disk0-part1-format-root
4893+ type: format
4894+ fstype: ext4
4895+ volume: disk0-part1
4896+ - id: disk0-part2-format-home
4897+ type: format
4898+ fstype: ext4
4899+ volume: disk0-part2
4900+ - id: disk0-part1-mount-root
4901+ type: mount
4902+ path: /
4903+ device: disk0-part1-format-root
4904+ - id: disk0-part2-mount-home
4905+ type: mount
4906+ path: /home
4907+ device: disk0-part2-format-home
4908+
4909+LVM
4910+~~~
4911+
4912+::
4913+
4914+ storage:
4915+ version: 1
4916+ config:
4917+ - id: sda
4918+ type: disk
4919+ ptable: msdos
4920+ model: QEMU HARDDISK
4921+ path: /dev/vdb
4922+ name: main_disk
4923+ - id: sda1
4924+ type: partition
4925+ size: 3GB
4926+ device: sda
4927+ flag: boot
4928+ - id: sda_extended
4929+ type: partition
4930+ size: 5G
4931+ flag: extended
4932+ device: sda
4933+ - id: sda2
4934+ type: partition
4935+ size: 2G
4936+ flag: logical
4937+ device: sda
4938+ - id: sda3
4939+ type: partition
4940+ size: 3G
4941+ flag: logical
4942+ device: sda
4943+ - id: volgroup1
4944+ name: vg1
4945+ type: lvm_volgroup
4946+ devices:
4947+ - sda2
4948+ - sda3
4949+ - id: lvmpart1
4950+ name: lv1
4951+ size: 1G
4952+ type: lvm_partition
4953+ volgroup: volgroup1
4954+ - id: lvmpart2
4955+ name: lv2
4956+ type: lvm_partition
4957+ volgroup: volgroup1
4958+ - id: sda1_root
4959+ type: format
4960+ fstype: ext4
4961+ volume: sda1
4962+ - id: lv1_fs
4963+ name: storage
4964+ type: format
4965+ fstype: fat32
4966+ volume: lvmpart1
4967+ - id: lv2_fs
4968+ name: storage
4969+ type: format
4970+ fstype: ext3
4971+ volume: lvmpart2
4972+ - id: sda1_mount
4973+ type: mount
4974+ path: /
4975+ device: sda1_root
4976+ - id: lv1_mount
4977+ type: mount
4978+ path: /srv/data
4979+ device: lv1_fs
4980+ - id: lv2_mount
4981+ type: mount
4982+ path: /srv/backup
4983+ device: lv2_fs
4984+
4985+Bcache
4986+~~~~~~
4987+
4988+::
4989+
4990+ storage:
4991+ version: 1
4992+ config:
4993+ - id: id_rotary0
4994+ type: disk
4995+ name: rotary0
4996+ path: /dev/vdb
4997+ ptable: msdos
4998+ wipe: superblock
4999+ grub_device: true
5000+ - id: id_ssd0
The diff has been truncated for viewing.

Subscribers

People subscribed via source and target branches

to all changes: