Merge ~raharper/curtin:fix/eui-is-not-wwn into curtin:master

Proposed by Ryan Harper
Status: Merged
Approved by: Ryan Harper
Approved revision: 3841a0dca1314b8311456316f7eb30da403d083f
Merge reported by: Server Team CI bot
Merged at revision: not available
Proposed branch: ~raharper/curtin:fix/eui-is-not-wwn
Merge into: curtin:master
Diff against target: 210 lines (+127/-14)
4 files modified
curtin/commands/block_meta.py (+8/-12)
tests/unittests/test_block.py (+26/-0)
tests/unittests/test_commands_block_meta.py (+90/-1)
tests/unittests/test_curthooks.py (+3/-1)
Reviewer Review Type Date Requested Status
Chad Smith Approve
Server Team CI bot continuous-integration Approve
Dan Watkins (community) Approve
Review via email: mp+374566@code.launchpad.net

Commit message

block_meta: use lookup for wwn, fix fallback from wwn, serial, path

NVMe devices may provide a ID_WWN value from udev, but the NVMe rules
do not create a by-id/wwn-$ID_WWN symlink which breaks behavior with
other ID_WWN devices resulting in curtin not finding certain NVMe
devices. Resolve this issue buy using block.lookup_disk on WWN values
as well as serial.

Additional fixes to block_meta.get_path_to_storage_volume:
- get_path_to_storage_volume will try wwn, serial, and ultimately
  path values in order to find the specified disk
- Add os.path.exists check to proposed constructed values, if the
  path is not valid, we continue to use different keys if possible.
- Updated the Exception message to indicate the volume wasn't found
  and emit the config it used.

LP: #1849322

To post a comment you must log in.
~raharper/curtin:fix/eui-is-not-wwn updated
fa3dd45... by Ryan Harper

Drop vmtest wwn; that does not work the way we need

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Dan Watkins (oddbloke) wrote :

A couple of nits (so leaving to you to merge) and some questions for my own edification, but looks good to me. Thanks!

review: Approve
Revision history for this message
Ryan Harper (raharper) :
~raharper/curtin:fix/eui-is-not-wwn updated
493fa9f... by Ryan Harper

Move path check to bottom of loop

92188d8... by Ryan Harper

Use assertEqual with expected list of calls

Revision history for this message
Dan Watkins (oddbloke) wrote :

On Wed, Oct 23, 2019 at 08:22:56PM -0000, Ryan Harper wrote:
> Diff comments:
>
> > diff --git a/curtin/commands/block_meta.py b/curtin/commands/block_meta.py
> > index 37b93dd..74faa63 100644
> > --- a/curtin/commands/block_meta.py
> > +++ b/curtin/commands/block_meta.py
> > @@ -416,21 +419,23 @@ def get_path_to_storage_volume(volume, storage_config):
> > # sys/class/block access is valid. ie, there are no
> > # udev generated values in sysfs
> > volume_path = os.path.realpath(vol_value)
> > - break
> > - elif disk_key == 'wwn':
> > - by_wwn = '/dev/disk/by-id/wwn-%s' % vol.get('wwn')
> > - volume_path = os.path.realpath(by_wwn)
> > + if not os.path.exists(volume_path):
>
> Yes, I can move to bottom of loop.

Thanks!

> > + block_meta.get_path_to_storage_volume(volume, s_cfg)
> > + self.m_lookup.assert_called_with(wwn)
>
> OK, I think I can convert to an assertEqual on the call_args_list.

I think assert_called_with more accurately maps to `assertIn` (though
`assertEqual` may be more correct in the context of a given test, which
would be more like `assert_called_once_with`).

`assert_has_calls` can most easily either be `assertEqual([call(...),
call(...)], m.call_args_list)` for strict ordering, or the same but with
the list wrapped in `set()` for non-strict ordering. Strictly speaking,
though, I think assert_has_calls is really doing something more like
`assertTrue(set([call(...), call(..)]).issubset(set(m.call_args_list)))`
but that probably doesn't matter in most contexts.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

I think we might need to fix the two points I mentioned inline. Will look again when you get a chance to review.

review: Needs Fixing
Revision history for this message
Ryan Harper (raharper) :
Revision history for this message
Chad Smith (chad.smith) wrote :

Approve once we drop the inner try/except and manipulate the outer try/except ValueError to be a 'continue' instead of 'pass'

review: Approve
~raharper/curtin:fix/eui-is-not-wwn updated
3841a0d... by Ryan Harper

Drop inner try/except and continue on Exception

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Ryan Harper (raharper) wrote :

rebuilding, git.lp was unhappy.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

Looks good thanks for the updates.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/curtin/commands/block_meta.py b/curtin/commands/block_meta.py
index 37b93dd..cb2122e 100644
--- a/curtin/commands/block_meta.py
+++ b/curtin/commands/block_meta.py
@@ -404,9 +404,8 @@ def get_path_to_storage_volume(volume, storage_config):
404 try:404 try:
405 if not vol_value:405 if not vol_value:
406 continue406 continue
407 if disk_key == 'serial':407 if disk_key in ['wwn', 'serial']:
408 volume_path = block.lookup_disk(vol_value)408 volume_path = block.lookup_disk(vol_value)
409 break
410 elif disk_key == 'path':409 elif disk_key == 'path':
411 if vol_value.startswith('iscsi:'):410 if vol_value.startswith('iscsi:'):
412 i = iscsi.ensure_disk_connected(vol_value)411 i = iscsi.ensure_disk_connected(vol_value)
@@ -416,21 +415,18 @@ def get_path_to_storage_volume(volume, storage_config):
416 # sys/class/block access is valid. ie, there are no415 # sys/class/block access is valid. ie, there are no
417 # udev generated values in sysfs416 # udev generated values in sysfs
418 volume_path = os.path.realpath(vol_value)417 volume_path = os.path.realpath(vol_value)
419 break
420 elif disk_key == 'wwn':
421 by_wwn = '/dev/disk/by-id/wwn-%s' % vol.get('wwn')
422 volume_path = os.path.realpath(by_wwn)
423 break
424 elif disk_key == 'device_id':418 elif disk_key == 'device_id':
425 dasd_device = dasd.DasdDevice(vol_value)419 dasd_device = dasd.DasdDevice(vol_value)
426 volume_path = dasd_device.devname420 volume_path = dasd_device.devname
427 break
428 except ValueError:421 except ValueError:
429 pass422 continue
423 # verify path exists otherwise try the next key
424 if not os.path.exists(volume_path):
425 volume_path = None
430426
431 if not volume_path:427 if volume_path is None:
432 raise ValueError("serial, wwn or path to block dev must be \428 raise ValueError("Failed to find storage volume id='%s' config: %s"
433 specified to identify disk")429 % (vol['id'], vol))
434430
435 elif vol.get('type') == "lvm_partition":431 elif vol.get('type') == "lvm_partition":
436 # For lvm partitions, a directory in /dev/ should be present with the432 # For lvm partitions, a directory in /dev/ should be present with the
diff --git a/tests/unittests/test_block.py b/tests/unittests/test_block.py
index 167a697..e70503d 100644
--- a/tests/unittests/test_block.py
+++ b/tests/unittests/test_block.py
@@ -103,6 +103,32 @@ class TestBlock(CiTestCase):
103 mock_os_listdir.return_value = ["other"]103 mock_os_listdir.return_value = ["other"]
104 block.lookup_disk(serial)104 block.lookup_disk(serial)
105105
106 @mock.patch("curtin.block.multipath")
107 @mock.patch("curtin.block.os.path.realpath")
108 @mock.patch("curtin.block.os.path.exists")
109 @mock.patch("curtin.block.os.listdir")
110 def test_lookup_disk_find_wwn(self, mock_os_listdir, mock_os_path_exists,
111 mock_os_path_realpath, mock_mpath):
112 wwn = "eui.0025388b710116a1"
113 expected_link = 'nvme-%s' % wwn
114 device = '/wark/nvme0n1'
115 mock_os_listdir.return_value = [
116 "nvme-eui.0025388b710116a1",
117 "nvme-eui.0025388b710116a1-part1",
118 "nvme-eui.0025388b710116a1-part2",
119 ]
120 mock_os_path_exists.return_value = True
121 mock_os_path_realpath.return_value = device
122 mock_mpath.is_mpath_device.return_value = False
123
124 path = block.lookup_disk(wwn)
125
126 mock_os_listdir.assert_called_with("/dev/disk/by-id/")
127 mock_os_path_realpath.assert_called_with("/dev/disk/by-id/" +
128 expected_link)
129 self.assertTrue(mock_os_path_exists.called)
130 self.assertEqual(device, path)
131
106 @mock.patch('curtin.block.udevadm_info')132 @mock.patch('curtin.block.udevadm_info')
107 def test_get_device_mapper_links_returns_first_non_none(self, m_info):133 def test_get_device_mapper_links_returns_first_non_none(self, m_info):
108 """ get_device_mapper_links returns first by sort entry in DEVLINKS."""134 """ get_device_mapper_links returns first by sort entry in DEVLINKS."""
diff --git a/tests/unittests/test_commands_block_meta.py b/tests/unittests/test_commands_block_meta.py
index b2e151e..dab133e 100644
--- a/tests/unittests/test_commands_block_meta.py
+++ b/tests/unittests/test_commands_block_meta.py
@@ -11,6 +11,96 @@ from curtin import paths, util
11from .helpers import CiTestCase11from .helpers import CiTestCase
1212
1313
14class TestGetPathToStorageVolume(CiTestCase):
15
16 def setUp(self):
17 super(TestGetPathToStorageVolume, self).setUp()
18 basepath = 'curtin.commands.block_meta.'
19 self.add_patch(basepath + 'os.path.exists', 'm_exists')
20 self.add_patch(basepath + 'block.lookup_disk', 'm_lookup')
21 self.add_patch(basepath + 'devsync', 'm_devsync')
22
23 def test_block_lookup_called_with_disk_wwn(self):
24 volume = 'mydisk'
25 wwn = self.random_string()
26 cfg = {'id': volume, 'type': 'disk', 'wwn': wwn}
27 s_cfg = OrderedDict({volume: cfg})
28 block_meta.get_path_to_storage_volume(volume, s_cfg)
29 expected_calls = [call(wwn)]
30 self.assertEqual(expected_calls, self.m_lookup.call_args_list)
31
32 def test_block_lookup_called_with_disk_serial(self):
33 volume = 'mydisk'
34 serial = self.random_string()
35 cfg = {'id': volume, 'type': 'disk', 'serial': serial}
36 s_cfg = OrderedDict({volume: cfg})
37 block_meta.get_path_to_storage_volume(volume, s_cfg)
38 expected_calls = [call(serial)]
39 self.assertEqual(expected_calls, self.m_lookup.call_args_list)
40
41 def test_block_lookup_called_with_disk_wwn_fallback_to_serial(self):
42 volume = 'mydisk'
43 wwn = self.random_string()
44 serial = self.random_string()
45 cfg = {'id': volume, 'type': 'disk', 'wwn': wwn, 'serial': serial}
46 s_cfg = OrderedDict({volume: cfg})
47
48 # doesn't find wwn, returns path on serial
49 self.m_lookup.side_effect = iter([ValueError('Error'), 'foo'])
50
51 block_meta.get_path_to_storage_volume(volume, s_cfg)
52 expected_calls = [call(wwn), call(serial)]
53 self.assertEqual(expected_calls, self.m_lookup.call_args_list)
54
55 def test_fallback_to_path_when_lookup_wwn_serial_fail(self):
56 volume = 'mydisk'
57 wwn = self.random_string()
58 serial = self.random_string()
59 path = "/%s/%s" % (self.random_string(), self.random_string())
60 cfg = {'id': volume, 'type': 'disk',
61 'path': path, 'wwn': wwn, 'serial': serial}
62 s_cfg = OrderedDict({volume: cfg})
63
64 # lookups fail
65 self.m_lookup.side_effect = iter([
66 ValueError('Error'), ValueError('Error')])
67
68 result = block_meta.get_path_to_storage_volume(volume, s_cfg)
69 expected_calls = [call(wwn), call(serial)]
70 self.assertEqual(expected_calls, self.m_lookup.call_args_list)
71 self.assertEqual(path, result)
72
73 def test_block_lookup_not_called_with_wwn_or_serial_keys(self):
74 volume = 'mydisk'
75 path = "/%s/%s" % (self.random_string(), self.random_string())
76 cfg = {'id': volume, 'type': 'disk', 'path': path}
77 s_cfg = OrderedDict({volume: cfg})
78 result = block_meta.get_path_to_storage_volume(volume, s_cfg)
79 self.assertEqual(0, self.m_lookup.call_count)
80 self.assertEqual(path, result)
81
82 def test_exception_raise_if_disk_not_found(self):
83 volume = 'mydisk'
84 wwn = self.random_string()
85 serial = self.random_string()
86 path = "/%s/%s" % (self.random_string(), self.random_string())
87 cfg = {'id': volume, 'type': 'disk',
88 'path': path, 'wwn': wwn, 'serial': serial}
89 s_cfg = OrderedDict({volume: cfg})
90
91 # lookups fail
92 self.m_lookup.side_effect = iter([
93 ValueError('Error'), ValueError('Error')])
94 # no path
95 self.m_exists.return_value = False
96
97 with self.assertRaises(ValueError):
98 block_meta.get_path_to_storage_volume(volume, s_cfg)
99 expected_calls = [call(wwn), call(serial)]
100 self.assertEqual(expected_calls, self.m_lookup.call_args_list)
101 self.m_exists.assert_has_calls([call(path)])
102
103
14class TestBlockMetaSimple(CiTestCase):104class TestBlockMetaSimple(CiTestCase):
15 def setUp(self):105 def setUp(self):
16 super(TestBlockMetaSimple, self).setUp()106 super(TestBlockMetaSimple, self).setUp()
@@ -1086,5 +1176,4 @@ class TestDmCryptHandler(CiTestCase):
1086 self.m_subp.assert_has_calls(expected_calls)1176 self.m_subp.assert_has_calls(expected_calls)
1087 self.assertEqual(len(util.load_file(self.crypttab).splitlines()), 1)1177 self.assertEqual(len(util.load_file(self.crypttab).splitlines()), 1)
10881178
1089
1090# vi: ts=4 expandtab syntax=python1179# vi: ts=4 expandtab syntax=python
diff --git a/tests/unittests/test_curthooks.py b/tests/unittests/test_curthooks.py
index 613f068..95276f7 100644
--- a/tests/unittests/test_curthooks.py
+++ b/tests/unittests/test_curthooks.py
@@ -353,7 +353,8 @@ class TestSetupGrub(CiTestCase):
353 self.target, '/dev/vdb'],),353 self.target, '/dev/vdb'],),
354 self.mock_subp.call_args_list[0][0])354 self.mock_subp.call_args_list[0][0])
355355
356 def test_uses_grub_install_on_storage_config(self):356 @patch('curtin.commands.curthooks.os.path.exists')
357 def test_uses_grub_install_on_storage_config(self, m_exists):
357 cfg = {358 cfg = {
358 'storage': {359 'storage': {
359 'version': 1,360 'version': 1,
@@ -368,6 +369,7 @@ class TestSetupGrub(CiTestCase):
368 },369 },
369 }370 }
370 self.subp_output.append(('', ''))371 self.subp_output.append(('', ''))
372 m_exists.return_value = True
371 curthooks.setup_grub(cfg, self.target, osfamily=self.distro_family)373 curthooks.setup_grub(cfg, self.target, osfamily=self.distro_family)
372 self.assertEquals(374 self.assertEquals(
373 ([375 ([

Subscribers

People subscribed via source and target branches