~raharper/curtin:fix/block-discover-nvme-multipath

Last commit made on 2020-04-09
Get this branch:
git clone -b fix/block-discover-nvme-multipath https://git.launchpad.net/~raharper/curtin
Only Ryan Harper can upload to this branch. If you are Ryan Harper please log in for upload directions.

Branch merges

Branch information

Name:
fix/block-discover-nvme-multipath
Repository:
lp:~raharper/curtin

Recent commits

016b6ca... by Ryan Harper on 2020-04-09

Change unittest to ensure we call the blockdevparser.parse() path

a09ca4c... by Ryan Harper on 2020-04-08

block-discover: detect nvme multipath devices

NVME "multipath" devices are a virtual subsystem inside the kernel
that has nothing to do with traditional multipath. The kernel nvme
driver jumps through hoops to hide a controller node and pretend that
it is a block device when it is emphatically _not_. It's not suprising
that this breaks lots of things.

These phony block devices do present a virtual NVME block device and
curtin block-discover ignores composed virtual devices when
constructing storage-config 'type: disk' devices. We must include such
devices as 'type: disk'.

This patch narrows our filter to exclude any DEVPATH which is
/devices/virtual/block and NVME mp devices use
/devices/virtual/nvme-subsystem.

LP: #1868109

638c9a7... by Ryan Harper on 2020-04-03

format: extra_options should be a list type

Switch to a list for extra_options to allow for specifying whitespace
characters in values passed to mkfs command.

- tox: add block-schema validate env

4264649... by Paride Legovini on 2020-04-01

tox: add pyflakes to the default tox run

The new py3-pyflakes environment tests the code against a specific
version of pyflakes, and it is run by default. This in contrast
with the existing tip-pyflakes environment, which tests against the
latest pyflakes version and it is not part of the default tox run.

8a2fbf0... by Ryan Harper on 2020-03-30

storage_config: Add 'extra_options' parameter to allow custom mkfs

Allow users to provide additional flags to the mkfs command used to
create a filesystem during storage configuration. The extra_options
value is a string that will be appended to the constructed mkfs
command. The value will be split on whitespace to show up as
individual arguments to the mkfs command.

LP: #1869069

7f7b8c4... by Ryan Harper on 2020-03-27

Add support for installing Ubuntu Core 20 images

Update detection methods for Ubuntu Core 20. Use new agreed upon location
for writing out cloud.cfg.d directory with contents provided from MAAS.
Add vmtest coverage for UC20 image as well.

f262b82... by Ryan Harper on 2020-03-25

tox.ini: Fix issues with newer tox on focal

Newer tox on focal is not happy when inhieriting dependencies.
First in py27 testenv, it does not pickup the 'testenv' deps
which includes test-depedndences and dependencies files. Fix
this by explicitly referencing these deps fir testenv:py27.

Second, in testenv:trusty-p27, which recreates specific package
versions to model the trusty build environment we cannot pull in
the default testenv deps, as this breaks with different versions
of mock and other settings. Instead, just bring in the setuptools
dep to ensure we can keep running on py27 for now.

b10aeef... by Ryan Harper on 2020-03-12

vmtest: Fix test_basic.py to run on s390x

The TestBasic currently fails on s390x due to usage of virtio-disks
and an "empty" virtio-scsi device controller. There is an existing
bug in s390x firmware that does not like an "empty" virtio-scsi
controller. Until that bug is fixed, skip TestBasic on s390x and
instead make TestScsiBasic work. This requires capturing output of
/dev/disk/by-path as s390x fstab entries prefer by-path as paths on
s390x are stable[1].

1. https://wiki.ubuntu.com/FSTAB

LP: #1866663

90bc0d7... by Ryan Harper on 2020-03-11

vmtest: use util.load_file for loading collect files

In some cases the content inside collect files may need to be
decoded. Instead of open-coded read and decode, use the common
util.load_file() which already handles decoding data.

- Drop mode= param, no callers used it
- Reuse load_collect_file() for load_collect_file_shell_content
- load_log_file callers specify full path to file

bf03e4f... by Ryan Harper on 2020-03-10

block-meta: refactor storage_config preserve and wipe settings

The subiquity installer has use-cases where users wish to preserve
some devices and then repurpose them into other storage devices.
For example, retain a particular partition and then add it to an
LVM setup. Curtin currently makes users chose to preserve devices
wholesale or chose to wipe them. To support the 'repurpose' use
case we need to be able to preserve a device and then optionally
wipe the data therein independently of preservation.

The block schema has been updated to accept preserve and wipe
settings for each block device type. For the following types,
dasd, disk, partition, format, lvm_volgroup, lvm_partition,
dm_crypt, raid and bcache, Curtin will accept the preserve
boolean, and if set True, will verify various characteristics
of the target device and compare this to the specified config.

For example, preserving a partition which needs a device, size
and some optional values, like flag. Curtin will verify the
device exists, the number is correct, the size of the partition
matches and if the correct flag is set on the device.

The set of verifications performed are storage type specific.
Any verification failure results in Curtin raising a RuntimeError
exception which includes the expected and found values.

Additional work:

- Migrate bcache creation functions from block-meta into
  block.bcache
- Unittest and vmtests added for verifying preserve for each
  storage type.

LP: #1837214