probe-data-mp.json is attached:
(master)mwhudson@ringil:/opt/opensource/curtin$ python3
Python 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, json; from curtin import log, storage_config
>>> log.basicConfig(verbosity=2, stream=sys.stdout)
>>> c = json.load(open('probe-data-mp.json'))
>>> storage_config.extract_storage_config(c)
Extracting storage config from probe data
Sorting extracted configurations
Validating extracted storage config components
Extracted (unmerged) storage config:
storage:
- id: disk-sda
path: /dev/sda
ptable: gpt
serial: '30000000000000064'
type: disk
wwn: '0x0000000000000064'
- id: disk-sdb
path: /dev/sdb
ptable: gpt
serial: '30000000000000064'
type: disk
wwn: '0x0000000000000064'
- device: disk-sda
flag: bios_grub
id: partition-sda1
number: 1
offset: 1048576
size: 1048576
type: partition
- device: disk-sda
flag: linux
id: partition-sda2
number: 2
offset: 2097152
size: 10734272512
type: partition
- device: disk-sdb
flag: bios_grub
id: partition-sdb1
number: 1
offset: 1048576
size: 1048576
type: partition
- device: disk-sdb
flag: linux
id: partition-sdb2
number: 2
offset: 2097152
size: 10734272512
type: partition
- fstype: ext4
id: format-partition-sda2
type: format
uuid: 80ca57a0-60b7-4328-be41-afedf915effd
volume: partition-sda2
- fstype: ext4
id: format-partition-sdb2
type: format
uuid: 80ca57a0-60b7-4328-be41-afedf915effd
volume: partition-sdb2
- fstype: ext4
id: format-disk-dm-2
type: format
uuid: 80ca57a0-60b7-4328-be41-afedf915effd
volume: disk-dm-2
Generating storage config dependencies
Validate: partition-sda1:SourceType:partition -> (DepId:disk-sda DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: partition-sda2:SourceType:partition -> (DepId:disk-sda DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: partition-sdb1:SourceType:partition -> (DepId:disk-sdb DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: partition-sdb2:SourceType:partition -> (DepId:disk-sdb DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: format-partition-sda2:SourceType:format -> (DepId:partition-sda2 DepType:partition) in SourceDeps:{'dm_crypt', 'disk', 'lvm_partition', 'bcache', 'partition', 'raid'} ? result=True
Validate: partition-sda2:SourceType:partition -> (DepId:disk-sda DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Validate: format-partition-sdb2:SourceType:format -> (DepId:partition-sdb2 DepType:partition) in SourceDeps:{'dm_crypt', 'disk', 'lvm_partition', 'bcache', 'partition', 'raid'} ? result=True
Validate: partition-sdb2:SourceType:partition -> (DepId:disk-sdb DepType:disk) in SourceDeps:{'disk', 'raid', 'bcache', 'partition'} ? result=True
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/opensource/curtin/curtin/storage_config.py", line 1171, in extract_storage_config
tree = get_config_tree(cfg.get('id'), final_config)
File "/opt/opensource/curtin/curtin/storage_config.py", line 274, in get_config_tree
for dep in find_item_dependencies(item, sconfig):
File "/opt/opensource/curtin/curtin/storage_config.py", line 244, in find_item_dependencies
_validate_dep_type(item_id, dep_key, dep, config)
File "/opt/opensource/curtin/curtin/storage_config.py", line 192, in _validate_dep_type
'Invalid dep_id (%s) not in storage config' % dep_id)
ValueError: Invalid dep_id (disk-dm-2) not in storage config
>>>
The immediate problem here I guess is that curtin hasn't figured out that dm-2 is a partition of dm-0. But I'm not really sure in general what should be happening here -- it seems to me that, in general, it would be more useful to emit one disk action with path=/dev/mapper/mpatha, wwn=30000000000000064, serial=serial-a (and have some way in info to see which devices are part of it) and have the partition etc actions refer to that, rather than having disk actions with /dev/sdX paths and duplicate serials etc. Happy to work on this if you agree (and ideally tell me where to get started!)
Thanks for filing the bug. I had some probert data that could reproduce as well.
One of the challenges, as you see, is that multipath "partitions" are really additional dm devices, except, they are actual partitions on the underlying disks.
so, you'll have dm2 "partition" on dm0 (mpath device) which is on top of sda which will have an actual sda2.
So curtin generally wants to ignore the mpath device itself, and deal with _one_ path.
We have a couple of options; MAAS currently dumps *all* of the disks it found in the config, and constructs a layout from just *one* of the disks since MAAS and curtin don't *construct* multipath devices (the construction is how the devices are wired up).
Subiquity can also use this method, including all of the paths, but creating partitions/devices against just a single path. Alternatively, we can exclude any of the additional paths after selecting one of the paths and curtin will detect that the underlying disk is part of a multipath and ensure that the target OS has multipath enabled/configured.