Merge ~raharper/curtin:fix/ptable-label-pmbr into curtin:master

Proposed by Ryan Harper
Status: Merged
Approved by: Ryan Harper
Approved revision: ca77533c21d717e0167043a6ca552e1fa5a28892
Merge reported by: Server Team CI bot
Merged at revision: not available
Proposed branch: ~raharper/curtin:fix/ptable-label-pmbr
Merge into: curtin:master
Diff against target: 109 lines (+27/-6)
4 files modified
curtin/block/schemas.py (+3/-2)
curtin/commands/block_meta.py (+3/-1)
curtin/storage_config.py (+5/-1)
tests/unittests/test_storage_config.py (+16/-2)
Reviewer Review Type Date Requested Status
Chad Smith Approve
Server Team CI bot continuous-integration Approve
Dan Watkins (community) Approve
Dimitri John Ledkov (community) Needs Fixing
Michael Hudson-Doyle Pending
Review via email: mp+374305@code.launchpad.net

Commit message

schema: Add ptable value 'unsupported'

Probert output may include values of partition table 'label'
that curtin does not support. This triggers schema validation
errors for common values, like 'mac' or 'PMBR' and likely
other valid table names but are not something that curtin
can create. Instead of adding more values to the schema, mark
such table types as 'unsupported' allowing the existing table
and partitions to be re-used.

This branch also modifies block_meta disk_handler to not
attempt to validate a preserved partition table if it marked
as 'unsupported'.

LP: #1848535

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

No, that is not the right thing to do from the point of preservation of partition table.

partition tables that curtin doesn't know how to manipulate (ie. create table, create/change partitions) it should be emitted as unknown or frozen.

Kernel can parse the partitions, thus one can reuse those partitions.

A device without a partition table, is different to a device with an unknown partition table we don't know how to modify correctly.

Thus completely omitting partition table, seems, conceptually, to me like a lie.

review: Needs Fixing
Revision history for this message
Dimitri John Ledkov (xnox) :
Revision history for this message
Ryan Harper (raharper) wrote :

We have two paths: one curtin is converting probed data into a storage config; this is where we find "unsupported" partition table types/labels. I generally don't want to encode all of these ever-expanding values into the storage schema. On the other hand, I want subiquity and other users of the schema to be able to verify the config they generated is correct, in which case I do want to encode specific values in the schema that curtin is expected to support.

After discussing this, I'll switch to setting ptable: unsupported for these values that curtin won't create/modify.

8981dde... by Ryan Harper

schema: Add ptable value 'unsupported' for types curtin does not handle

Mark unsupported table types as 'unsupported'. This allows the existing
table and partitions to be re-used. Modify block_meta disk_handler to
not attempt to validate a preserved partition table as we now accept
'unsupported' ptable types; we won't be able to validate those entries.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Dan Watkins (oddbloke) wrote :

One inline question but this otherwise looks good. Thanks!

review: Needs Information
Revision history for this message
Ryan Harper (raharper) :
Revision history for this message
Dan Watkins (oddbloke) wrote :

Thanks!

review: Approve
Revision history for this message
Chad Smith (chad.smith) :
review: Needs Information
Revision history for this message
Ryan Harper (raharper) :
Revision history for this message
Ryan Harper (raharper) wrote :

For the record, simply emitting ptable: unsupported out of the discover command is sufficient to resolve the bug (curtin didn't like the PMBR value). Subiquity will still need a change to replace the ptable value with 'gpt'. And the preserve: true use-case is waiting on further work on the other branch.

be9e2d3... by Ryan Harper

schema: add _ptable_unsupported variable

48fd93a... by Ryan Harper

Restore ptable preserve check, except when 'unsupported'

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

Thanks for this minor reference request for schemas._ptable_unsupported until a followup branch renames public schema '_*' variables.

review: Approve
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
ca77533... by Ryan Harper

Replace use of 'unsupported' with schemas._ptable_unsupported

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

LGTM! Thanks rharper.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/curtin/block/schemas.py b/curtin/block/schemas.py
index aa45d61..fc7e522 100644
--- a/curtin/block/schemas.py
+++ b/curtin/block/schemas.py
@@ -7,6 +7,8 @@ _path_nondev = r'(^/$|^(/[^/]+)+$)'
7_fstypes = ['btrfs', 'ext2', 'ext3', 'ext4', 'fat', 'fat12', 'fat16', 'fat32',7_fstypes = ['btrfs', 'ext2', 'ext3', 'ext4', 'fat', 'fat12', 'fat16', 'fat32',
8 'iso9660', 'vfat', 'jfs', 'ntfs', 'reiserfs', 'swap', 'xfs',8 'iso9660', 'vfat', 'jfs', 'ntfs', 'reiserfs', 'swap', 'xfs',
9 'zfsroot']9 'zfsroot']
10_ptables = ['dos', 'gpt', 'msdos', 'vtoc']
11_ptable_unsupported = 'unsupported'
1012
11definitions = {13definitions = {
12 'id': {'type': 'string'},14 'id': {'type': 'string'},
@@ -14,8 +16,7 @@ definitions = {
14 'devices': {'type': 'array', 'items': {'$ref': '#/definitions/ref_id'}},16 'devices': {'type': 'array', 'items': {'$ref': '#/definitions/ref_id'}},
15 'name': {'type': 'string'},17 'name': {'type': 'string'},
16 'preserve': {'type': 'boolean'},18 'preserve': {'type': 'boolean'},
17 'ptable': {'type': 'string', 'enum': ['dos', 'gpt', 'mac', 'msdos',19 'ptable': {'type': 'string', 'enum': _ptables + [_ptable_unsupported]},
18 'vtoc']},
19 'size': {'type': ['string', 'number'],20 'size': {'type': ['string', 'number'],
20 'minimum': 1,21 'minimum': 1,
21 'pattern': r'^([1-9]\d*(.\d+)?|\d+.\d+)(K|M|G|T)?B?'},22 'pattern': r'^([1-9]\d*(.\d+)?|\d+.\d+)(K|M|G|T)?B?'},
diff --git a/curtin/commands/block_meta.py b/curtin/commands/block_meta.py
index 37b93dd..6afb048 100644
--- a/curtin/commands/block_meta.py
+++ b/curtin/commands/block_meta.py
@@ -2,6 +2,7 @@
22
3from collections import OrderedDict, namedtuple3from collections import OrderedDict, namedtuple
4from curtin import (block, config, paths, util)4from curtin import (block, config, paths, util)
5from curtin.block import schemas
5from curtin.block import (bcache, clear_holders, dasd, iscsi, lvm, mdadm, mkfs,6from curtin.block import (bcache, clear_holders, dasd, iscsi, lvm, mdadm, mkfs,
6 zfs)7 zfs)
7from curtin import distro8from curtin import distro
@@ -32,6 +33,7 @@ SIMPLE = 'simple'
32SIMPLE_BOOT = 'simple-boot'33SIMPLE_BOOT = 'simple-boot'
33CUSTOM = 'custom'34CUSTOM = 'custom'
34BCACHE_REGISTRATION_RETRY = [0.2] * 6035BCACHE_REGISTRATION_RETRY = [0.2] * 60
36PTABLE_UNSUPPORTED = schemas._ptable_unsupported
3537
36CMD_ARGUMENTS = (38CMD_ARGUMENTS = (
37 ((('-D', '--devices'),39 ((('-D', '--devices'),
@@ -536,7 +538,7 @@ def disk_handler(info, storage_config):
536538
537 if config.value_as_boolean(info.get('preserve')):539 if config.value_as_boolean(info.get('preserve')):
538 # Handle preserve flag, verifying if ptable specified in config540 # Handle preserve flag, verifying if ptable specified in config
539 if config.value_as_boolean(ptable):541 if config.value_as_boolean(ptable) and ptable != PTABLE_UNSUPPORTED:
540 current_ptable = block.get_part_table_type(disk)542 current_ptable = block.get_part_table_type(disk)
541 if not ((ptable in _dos_names and current_ptable in _dos_names) or543 if not ((ptable in _dos_names and current_ptable in _dos_names) or
542 (ptable == 'gpt' and current_ptable == 'gpt')):544 (ptable == 'gpt' and current_ptable == 'gpt')):
diff --git a/curtin/storage_config.py b/curtin/storage_config.py
index 3bfa466..03cfb4e 100644
--- a/curtin/storage_config.py
+++ b/curtin/storage_config.py
@@ -743,7 +743,11 @@ class BlockdevParser(ProbertParser):
743 entry.update(uniq_ids)743 entry.update(uniq_ids)
744744
745 if 'ID_PART_TABLE_TYPE' in blockdev_data:745 if 'ID_PART_TABLE_TYPE' in blockdev_data:
746 entry['ptable'] = blockdev_data['ID_PART_TABLE_TYPE']746 ptype = blockdev_data['ID_PART_TABLE_TYPE']
747 if ptype in schemas._ptables:
748 entry['ptable'] = ptype
749 else:
750 entry['ptable'] = schemas._ptable_unsupported
747 return entry751 return entry
748752
749 if entry['type'] == 'partition':753 if entry['type'] == 'partition':
diff --git a/tests/unittests/test_storage_config.py b/tests/unittests/test_storage_config.py
index 154a8b2..d4113ea 100644
--- a/tests/unittests/test_storage_config.py
+++ b/tests/unittests/test_storage_config.py
@@ -43,12 +43,11 @@ class TestStorageConfigSchema(CiTestCase):
43 storage_config.validate_config(config)43 storage_config.validate_config(config)
4444
45 @skipUnlessJsonSchema()45 @skipUnlessJsonSchema()
46 def test_disk_schema_accepts_mac_partition_table(self):46 def test_disk_schema_accepts_missing_ptable(self):
47 disk = {47 disk = {
48 "id": "disk-vdc",48 "id": "disk-vdc",
49 "path": "/dev/vdc",49 "path": "/dev/vdc",
50 "type": "disk",50 "type": "disk",
51 "ptable": "mac",
52 }51 }
53 config = {'config': [disk], 'version': 1}52 config = {'config': [disk], 'version': 1}
54 storage_config.validate_config(config)53 storage_config.validate_config(config)
@@ -385,6 +384,21 @@ class TestBlockdevParser(CiTestCase):
385 self.assertDictEqual(expected_dict,384 self.assertDictEqual(expected_dict,
386 self.bdevp.asdict(blockdev))385 self.bdevp.asdict(blockdev))
387386
387 def test_blockdev_asdict_disk_marks_unknown_ptable_as_unspported(self):
388 blockdev = self.bdevp.blockdev_data['/dev/sda']
389 expected_dict = {
390 'id': 'disk-sda',
391 'type': 'disk',
392 'wwn': '0x3001438034e549a0',
393 'serial': '33001438034e549a0',
394 'ptable': 'unsupported',
395 'path': '/dev/sda',
396 }
397 for invalid in ['mac', 'PMBR']:
398 blockdev['ID_PART_TABLE_TYPE'] = invalid
399 self.assertDictEqual(expected_dict,
400 self.bdevp.asdict(blockdev))
401
388 def test_blockdev_detects_multipath(self):402 def test_blockdev_detects_multipath(self):
389 self.probe_data = _get_data('probert_storage_multipath.json')403 self.probe_data = _get_data('probert_storage_multipath.json')
390 self.bdevp = BlockdevParser(self.probe_data)404 self.bdevp = BlockdevParser(self.probe_data)

Subscribers

People subscribed via source and target branches