~raharper/curtin:fix/vmtest-use-util-load-file-for-collect

Last commit made on 2020-03-11
Get this branch:
git clone -b fix/vmtest-use-util-load-file-for-collect https://git.launchpad.net/~raharper/curtin
Only Ryan Harper can upload to this branch. If you are Ryan Harper please log in for upload directions.

Branch merges

Branch information

Name:
fix/vmtest-use-util-load-file-for-collect
Repository:
lp:~raharper/curtin

Recent commits

f012843... by Ryan Harper on 2020-03-11

vmtest: use util.load_file for loading collect files

In some cases the content inside collect files may need to be
decoded. Instead of open-coded read and decode, use the common
util.load_file() which already handles decoding data.

bf03e4f... by Ryan Harper on 2020-03-10

block-meta: refactor storage_config preserve and wipe settings

The subiquity installer has use-cases where users wish to preserve
some devices and then repurpose them into other storage devices.
For example, retain a particular partition and then add it to an
LVM setup. Curtin currently makes users chose to preserve devices
wholesale or chose to wipe them. To support the 'repurpose' use
case we need to be able to preserve a device and then optionally
wipe the data therein independently of preservation.

The block schema has been updated to accept preserve and wipe
settings for each block device type. For the following types,
dasd, disk, partition, format, lvm_volgroup, lvm_partition,
dm_crypt, raid and bcache, Curtin will accept the preserve
boolean, and if set True, will verify various characteristics
of the target device and compare this to the specified config.

For example, preserving a partition which needs a device, size
and some optional values, like flag. Curtin will verify the
device exists, the number is correct, the size of the partition
matches and if the correct flag is set on the device.

The set of verifications performed are storage type specific.
Any verification failure results in Curtin raising a RuntimeError
exception which includes the expected and found values.

Additional work:

- Migrate bcache creation functions from block-meta into
  block.bcache
- Unittest and vmtests added for verifying preserve for each
  storage type.

LP: #1837214

1f9c53a... by Ryan Harper on 2020-03-09

block-discover: skip 'multipath' key in blockdevice if mpath name is None

Avoid emitting invalid storage-config yaml by not including a 'multipath'
key in block device output without a valid (not None) value for the key.

c650f6c... by Ryan Harper on 2020-03-05

tox: all py27 environments should use the base py27 deps

We fixed base py27 deps to pin setuptools < 45 to fix an issue
with newer setuptools and py27 deprecation. We need source the
base py27 env deps in all py27 related environments

eacb655... by Ryan Harper on 2020-03-04

uefi: refactor efibootmg handling to support removing duplicate entries

When curtin is reusing an existing ESP, it may already contain a previous
ubuntu entry. efibootmgr complains about duplicate labels, but does not
provide a mechanism to remove the duplicate automatically. In curtin
we use our efibootmgr output parser to find duplicates and remove them.

To enable this we need to ensure that 'efivarfs' is mounted inside the
target. In ChrootableTarget we will add the additional mount automatically
if we're running on a uefi system.

This branch also refactors use of ChrootableTarget context manager to
minimize the number of chroot calls (which include 4 mounts and unmounts).

LP: #1864257

d3cac6f... by Ryan Harper on 2020-03-03

tox: pin setuptools < 45 to allow installing py27 in virtenv

setuptools release 45 broke everyone who installed py27[1]. Fix
this by using the previous release.

1. https://github.com/pypa/setuptools/issues/1963

437caaa... by Ryan Harper on 2020-02-26

block-discover: add dasd parser and emit dasd storage config

Probert now includes a 'dasd' element in the probe data.
If present, block-discover will now parse this data and
emit type:dasd elements in the rendered storage config.

LP: #1862849

82f23e3... by Ryan Harper on 2020-02-14

install-grub: refactor uefi partition/disk searching

Refactor the code used to determine the partition and disk where the
ESP resides. There were a few issues:

 - Disks and partitions are of type 'block' which requires the use of
   [ -b ] or [ -e ]; [ -f ] returns false for block devices.
 - The use of lsblk -on pkname exposed a bug in util-linux where pkname
   did not return a value.

This branch adds additional debug logging and safety checks for any
calculated value and fails on empty values. We replace use of lsblk
with use of sysfs to calculate the parent disk of the partition. Add
a vmtest to verify the path where we use efibootmgr to create an entry.

LP: #1862846

2fa55b9... by Paul Tobias on 2020-02-11

doc: update Canonical contributors URL

The http://www.canonical.com/contributors link shows a 404
error. Replace it with the current Canonical contributor
license agreement page.

4d468ef... by Ryan Harper on 2020-02-07

block-discover: detect additional "extended" partition types in MBR

Surprise! There are additional encoding values for "extended" partitions
in MBRs. Add the 3 additional values (0x5, 0x85, 0xC5) to the mapping
table and add unittest using provided probe data.

LP: #1861251