clear-holders: detect and remove devices from kernel swap as needed
During testing of subiquity the ephemeral environment may activate
existing swap partitions which then prevent curtin from using a device during
storage configuration. This branch introduces a change to TestBasic to add
a swap partition and a command to the dirty-disk mode to forcibly enable
the swap device to reproduce the error. The fix involves having
clear_holders detect and remove the device (disk or partition) from
the kernel swap via use of swapoff.
Enable simpler use of zfsroot by tagging the root filesystem format with
'zfsroot'. Curtin will then automatically construct zpool and zfs commands
to create a ZFS on Root setup. Some validation of the config included:
* only one 'zfsroot' type is allowed.
* zfsroot entry must be referenced by the / mount.
zfs: only load zfs on xenial+ skip artful/bionic zfs env export
The zfs module is only available on xenial+, so guard against modprobing it on
precise and trusty.
Xenial is the only release which curtin is exporting an environment variable to
aid grub to find zfs devices, therefor we should only check if the environment
has the exported value on xenial, skiptest otherwise.
error logs: automatically collect and tar curtin logs upon error.
If 'curtin install' errors, create a tarfile of curtin configuration, logs
and system info. This tarfile is simplifies triage and/or filing curtin
bugs. By default, the tarfile is written to
/var/log/curtin/curtin-error-logs.tar and can be configured by setting the
install:error_tarfile value.
This branch also:
- uses deepcopy for CONFIG_BUILTIN to avoid manipulating a shared dict
in unit tests
- fixes a workingd variable declaration before raising errors to avoid
undeclared variable errors
vmtests: Support skipping skip_by_date tests based on environment.
Curtin's vmtest code has a function 'skip_by_date' which is used to
skip tests based until a certain date so developers can add a test
for a bug that is not yet fixed in another package. Each skip_by_date
call lists a bug by its bug number.
It serves to allow us to ignore known failures for a limited time but
remind us later to push to get fixes into other packages. The cost is
that someone has to "touch" code at a point in the future to bump the
skip_by_date values or remove them. That works well enough for trunk
where developers can easily make this change.
It causes issues though, in places where the code is not so easily
changed. An example is when we run tests on the version of ubuntu
in the archive. That packaged code is not so easily modified and
any skipby tests are basically time-bombs.
The solution here is to allow the runner of the test to specify
a comma separated list of bugs to skip in the environment variable
CURTIN_VMTEST_SKIP_BY_DATE_BUGS
The special value '*' indicates all skip_by_date bugs should be skipped.
We expect to run tests on packaged code with this environment variable
set to '*'.
Add zpool, zfs storage commands for experimental support of ZFS on root.
curtin/block
- Add get_dev_disk_byid() to return a mapping of devname to disk/by-id
- Add zfs and zpool to install deps dictionary
curtin/block/clear_holders
- Add modprobe zfs
curtin/block/zfs
- implement zpool_create, zfs_create, zfs_list, zfs_export and zfs_mount
commands
curtin/dep
- Add zfs, zfsutils-linux to ephemeral environment as needed
- Introduce a check for required kernel modules (zfs)
curtin/commands/block_meta
- implement handlers for type: zpool and zfs
- add get_poolname resolver
curtin/commands/curthooks
- On xenial, apt-mark hold zfs-dkms in target to prevent installation
to work around bug 1742560.
- Add an injection of a zfs environment variable required for ZFS on
rootfs to work with grub; allows grub to extract the path to the zfs
vdevs full path rather than just the devname (/dev/disk/by-id/foo vs
disk/by-id/foo) to work around bug 1527727.
curtin/commands/install
- Export ZPOOL_VDEV_NAME_PATH=1 into target environment to allow grub to
work with zfs on root
doc/topics/storage.rst
- Add documentation for zpool, zfs commands and experimental ZFS-on-Root
helpers/common
- Update install_grub to skip block-device check if target is zfs.
tests/unittests/test_block
- Add test for disk_byid methods
tests/unittests/test_block_zfs
- Add coverage for block.zfs
tests/unittests/test_clear_holders
- Update test to account for modprobe zfs
tests/vmtests/test_zfsroot
- Add initial zfsroot install and test
