~raharper/curtin:fix/unittest-leak-modprobe

Last commit made on 2020-04-21
Get this branch:
git clone -b fix/unittest-leak-modprobe https://git.launchpad.net/~raharper/curtin
Only Ryan Harper can upload to this branch. If you are Ryan Harper please log in for upload directions.

Branch merges

Branch information

Name:
fix/unittest-leak-modprobe
Repository:
lp:~raharper/curtin

Recent commits

f61faa4... by Ryan Harper on 2020-04-21

Mock out zfs_supported to prevent attempting to load kernel modules

f885a78... by Ryan Harper on 2020-04-21

unittest: do not allow util.subp by default

Bring over our util.subp mock from cloud-init. Rework unittest to
enable subp as needed.

LP: #1873913

22b505c... by Ryan Harper on 2020-04-20

curthooks: support multiple ESP on UEFI bootable systems

On debian systems we generate and set debconf selections values
to tell grub the set of configured devices. Also refactor
setup_grub to extract all UEFI boot partitions and determine
the primary ESP (mounted at /boot/efi). We include all ESPs
in the debconf selection for grub-[multi]-install to use.

Non-UEFI:
  grub-pc grub-pc/install_devices multiselect <devices>

UEFI:
  grub-pc grub-efi/install_devices multiselect <devices>

The grub-efi/install_devices is a new template value used for
handling multiple ESP partitions for boot resilience.

Curtin will check for the grub-multi-install tool and if
present will invoke that tool to install grub to all ESPs.

- Add block.schema update to allow partitions to set
  grub_device: True
- Refactor finding ESPs to prefer partition with grub_device,
  ESPs on disks with grub_device and fallback to grubcfg
  install_devices if we don't find one in storage config.

382af4b... by Ryan Harper on 2020-04-20

block-discover: handle missing multipath 'path' data, use DM_NAME

Probert's multipath data (maps and paths) is not always complete.
In some systems, the paths output may not include the required
mpath friendly name and instead use a serial number. In such
cases we can also check DM_NAME in the multipath device udev data
to extract the multipath_id that we can use to search for a member
device.

LP: #1873728

43db282... by Ryan Harper on 2020-04-20

lvm-over-multipath: handle lookups of multipath members

Subiquity provides a 'path' value for disks in storage config.
This triggered an edge case where curtin attempted to wipe the
underlying scsi disk *while* multipath is enabled resulting in
an error when attempting to get exclusive access to the disk.

This branch resolves this by checking if a disk or partition
is related to multipath and if so returning the device mapper
disk path.

LP: #1869075

2a03535... by Ryan Harper on 2020-04-20

block-meta: don't filter preserve=true devices, select by wipe

Now that wiping and preserving are independent values we consider
any disk or partition with a wipe setting for clearing holders.
An existing scenario where this matters is reusing a partition
which was previously part of a volume-group.

- Add a simplified vmtest scenario for reusing partition that
  was previously part of a volume-group

LP: #1837214

f681092... by Ryan Harper on 2020-04-20

vmtest: basic use dname to lookup disk with multiple partitions

This handles unstable scsi device names (sdc/sdd) which shuffle around
from time to time; instead look up the assigned name via the dname
value. This should fix vmtest failures on BasicScsi on ppc64le/arm64.

e967eeb... by Ryan Harper on 2020-04-16

block-meta: Don't check the ptable type of a disk with no ptable

When re-using a disk without a partition table we do not need to check
the current partition table.

0fed7d6... by Ryan Harper on 2020-04-16

curthooks: always use ChrootableTarget.subp when calling efibootmgr

There was a bug recently introduced when adding the feature to remove
duplicate UEFI boot entries which removed use of ChrootableTarget
subp method. The assumption was that util.subp calls would also get
the target parameter set but this was not the case. The ephemeral
environment does not usually have efibootmgr package installed but
we ensure that it is present in the target system. This branch
replaces calls to util.subp with the ChrootableTarget subp method.

dc215b1... by Ryan Harper on 2020-04-16

storage: enable and use multipath during storage configuration

Curtin has attempted to avoid using multipath enabled during installation
as that requires reworking a number of areas with in curtin that use Linux
tooling which is not multipath friendly. In particular, partitions on
multipath devices appear as disk not as partitions and this makes programs
confused. However, some scenarios just cannot be handled sanely without
multipath devices enabled. One such scenario is creating an LVM root on
top of multipath devices. This branch adds a vmtest scenario to recreate
this deployment configuration. To resolve the issue curtin will now keep
multipath enabled if available and adjust partition handling to make use
of 'kpartx' and 'dmsetup' to create, manipulate and calculate partition
related information. There was quite a bit of fallout with this change
so this branch includes a large amount of changes to handle this all the
way back to Xenial. Xenial and Bionic do not enable multipath by default,
and curtin does not enable multipath in the ephemeral environment, rather
if multipath is presnet in the ephemeral environment then curtin will
make use of it.

Additional changes needed/used to resolve this issue:

- Add --shutdown-plan to curtin clear-holders CLI tool for dumping the
  clear-holders plan of action for a given storage configuration.
- Add --probe-data to curtin block-discover CLI to dump the probert probe
  data out to a file and skip converting to a storage config.
- clear-holders will reload multipath maps to ensure mp is active.
- Add use of --filter to lvm commands to only use mpath devices if mp is
  enabled.
- curthooks now modifies target /etc/lvm.conf with a devices filter if mp
  is enabled.
- Fix dname for mpath devices, using dm-uuid and handling the oddity that
  multipath partitions prefix their dm-uuid with part-X-.
- If multipath enabled, don't hard code root devices to mpath0

LP: #1869075