Merge ~raharper/curtin:fix/eui-is-not-wwn into curtin:master
- Git
- lp:~raharper/curtin
- fix/eui-is-not-wwn
- Merge into master
Status: | Merged |
---|---|
Approved by: | Ryan Harper |
Approved revision: | 3841a0dca1314b8311456316f7eb30da403d083f |
Merge reported by: | Server Team CI bot |
Merged at revision: | not available |
Proposed branch: | ~raharper/curtin:fix/eui-is-not-wwn |
Merge into: | curtin:master |
Diff against target: |
210 lines (+127/-14) 4 files modified
curtin/commands/block_meta.py (+8/-12) tests/unittests/test_block.py (+26/-0) tests/unittests/test_commands_block_meta.py (+90/-1) tests/unittests/test_curthooks.py (+3/-1) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Chad Smith | Approve | ||
Server Team CI bot | continuous-integration | Approve | |
Dan Watkins (community) | Approve | ||
Review via email: mp+374566@code.launchpad.net |
Commit message
block_meta: use lookup for wwn, fix fallback from wwn, serial, path
NVMe devices may provide a ID_WWN value from udev, but the NVMe rules
do not create a by-id/wwn-$ID_WWN symlink which breaks behavior with
other ID_WWN devices resulting in curtin not finding certain NVMe
devices. Resolve this issue buy using block.lookup_disk on WWN values
as well as serial.
Additional fixes to block_meta.
- get_path_
path values in order to find the specified disk
- Add os.path.exists check to proposed constructed values, if the
path is not valid, we continue to use different keys if possible.
- Updated the Exception message to indicate the volume wasn't found
and emit the config it used.
LP: #1849322
Description of the change
- fa3dd45... by Ryan Harper
-
Drop vmtest wwn; that does not work the way we need
Server Team CI bot (server-team-bot) wrote : | # |
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:fa3dd457103
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
FAILURE: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:fa3dd457103
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Dan Watkins (oddbloke) wrote : | # |
A couple of nits (so leaving to you to merge) and some questions for my own edification, but looks good to me. Thanks!
Ryan Harper (raharper) : | # |
- 493fa9f... by Ryan Harper
-
Move path check to bottom of loop
- 92188d8... by Ryan Harper
-
Use assertEqual with expected list of calls
Dan Watkins (oddbloke) wrote : | # |
On Wed, Oct 23, 2019 at 08:22:56PM -0000, Ryan Harper wrote:
> Diff comments:
>
> > diff --git a/curtin/
> > index 37b93dd..74faa63 100644
> > --- a/curtin/
> > +++ b/curtin/
> > @@ -416,21 +419,23 @@ def get_path_
> > # sys/class/block access is valid. ie, there are no
> > # udev generated values in sysfs
> > volume_path = os.path.
> > - break
> > - elif disk_key == 'wwn':
> > - by_wwn = '/dev/disk/
> > - volume_path = os.path.
> > + if not os.path.
>
> Yes, I can move to bottom of loop.
Thanks!
> > + block_meta.
> > + self.m_
>
> OK, I think I can convert to an assertEqual on the call_args_list.
I think assert_called_with more accurately maps to `assertIn` (though
`assertEqual` may be more correct in the context of a given test, which
would be more like `assert_
`assert_has_calls` can most easily either be `assertEqual(
call(...)], m.call_args_list)` for strict ordering, or the same but with
the list wrapped in `set()` for non-strict ordering. Strictly speaking,
though, I think assert_has_calls is really doing something more like
`assertTrue(
but that probably doesn't matter in most contexts.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:92188d82e38
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) wrote : | # |
I think we might need to fix the two points I mentioned inline. Will look again when you get a chance to review.
Ryan Harper (raharper) : | # |
Chad Smith (chad.smith) wrote : | # |
Approve once we drop the inner try/except and manipulate the outer try/except ValueError to be a 'continue' instead of 'pass'
- 3841a0d... by Ryan Harper
-
Drop inner try/except and continue on Exception
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:3841a0dca13
https:/
Executed test runs:
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
FAILURE: https:/
Click here to trigger a rebuild:
https:/
Ryan Harper (raharper) wrote : | # |
rebuilding, git.lp was unhappy.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:3841a0dca13
https:/
Executed test runs:
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
SUCCESS: https:/
Click here to trigger a rebuild:
https:/
Chad Smith (chad.smith) wrote : | # |
Looks good thanks for the updates.
Preview Diff
1 | diff --git a/curtin/commands/block_meta.py b/curtin/commands/block_meta.py | |||
2 | index 37b93dd..cb2122e 100644 | |||
3 | --- a/curtin/commands/block_meta.py | |||
4 | +++ b/curtin/commands/block_meta.py | |||
5 | @@ -404,9 +404,8 @@ def get_path_to_storage_volume(volume, storage_config): | |||
6 | 404 | try: | 404 | try: |
7 | 405 | if not vol_value: | 405 | if not vol_value: |
8 | 406 | continue | 406 | continue |
10 | 407 | if disk_key == 'serial': | 407 | if disk_key in ['wwn', 'serial']: |
11 | 408 | volume_path = block.lookup_disk(vol_value) | 408 | volume_path = block.lookup_disk(vol_value) |
12 | 409 | break | ||
13 | 410 | elif disk_key == 'path': | 409 | elif disk_key == 'path': |
14 | 411 | if vol_value.startswith('iscsi:'): | 410 | if vol_value.startswith('iscsi:'): |
15 | 412 | i = iscsi.ensure_disk_connected(vol_value) | 411 | i = iscsi.ensure_disk_connected(vol_value) |
16 | @@ -416,21 +415,18 @@ def get_path_to_storage_volume(volume, storage_config): | |||
17 | 416 | # sys/class/block access is valid. ie, there are no | 415 | # sys/class/block access is valid. ie, there are no |
18 | 417 | # udev generated values in sysfs | 416 | # udev generated values in sysfs |
19 | 418 | volume_path = os.path.realpath(vol_value) | 417 | volume_path = os.path.realpath(vol_value) |
20 | 419 | break | ||
21 | 420 | elif disk_key == 'wwn': | ||
22 | 421 | by_wwn = '/dev/disk/by-id/wwn-%s' % vol.get('wwn') | ||
23 | 422 | volume_path = os.path.realpath(by_wwn) | ||
24 | 423 | break | ||
25 | 424 | elif disk_key == 'device_id': | 418 | elif disk_key == 'device_id': |
26 | 425 | dasd_device = dasd.DasdDevice(vol_value) | 419 | dasd_device = dasd.DasdDevice(vol_value) |
27 | 426 | volume_path = dasd_device.devname | 420 | volume_path = dasd_device.devname |
28 | 427 | break | ||
29 | 428 | except ValueError: | 421 | except ValueError: |
31 | 429 | pass | 422 | continue |
32 | 423 | # verify path exists otherwise try the next key | ||
33 | 424 | if not os.path.exists(volume_path): | ||
34 | 425 | volume_path = None | ||
35 | 430 | 426 | ||
39 | 431 | if not volume_path: | 427 | if volume_path is None: |
40 | 432 | raise ValueError("serial, wwn or path to block dev must be \ | 428 | raise ValueError("Failed to find storage volume id='%s' config: %s" |
41 | 433 | specified to identify disk") | 429 | % (vol['id'], vol)) |
42 | 434 | 430 | ||
43 | 435 | elif vol.get('type') == "lvm_partition": | 431 | elif vol.get('type') == "lvm_partition": |
44 | 436 | # For lvm partitions, a directory in /dev/ should be present with the | 432 | # For lvm partitions, a directory in /dev/ should be present with the |
45 | diff --git a/tests/unittests/test_block.py b/tests/unittests/test_block.py | |||
46 | index 167a697..e70503d 100644 | |||
47 | --- a/tests/unittests/test_block.py | |||
48 | +++ b/tests/unittests/test_block.py | |||
49 | @@ -103,6 +103,32 @@ class TestBlock(CiTestCase): | |||
50 | 103 | mock_os_listdir.return_value = ["other"] | 103 | mock_os_listdir.return_value = ["other"] |
51 | 104 | block.lookup_disk(serial) | 104 | block.lookup_disk(serial) |
52 | 105 | 105 | ||
53 | 106 | @mock.patch("curtin.block.multipath") | ||
54 | 107 | @mock.patch("curtin.block.os.path.realpath") | ||
55 | 108 | @mock.patch("curtin.block.os.path.exists") | ||
56 | 109 | @mock.patch("curtin.block.os.listdir") | ||
57 | 110 | def test_lookup_disk_find_wwn(self, mock_os_listdir, mock_os_path_exists, | ||
58 | 111 | mock_os_path_realpath, mock_mpath): | ||
59 | 112 | wwn = "eui.0025388b710116a1" | ||
60 | 113 | expected_link = 'nvme-%s' % wwn | ||
61 | 114 | device = '/wark/nvme0n1' | ||
62 | 115 | mock_os_listdir.return_value = [ | ||
63 | 116 | "nvme-eui.0025388b710116a1", | ||
64 | 117 | "nvme-eui.0025388b710116a1-part1", | ||
65 | 118 | "nvme-eui.0025388b710116a1-part2", | ||
66 | 119 | ] | ||
67 | 120 | mock_os_path_exists.return_value = True | ||
68 | 121 | mock_os_path_realpath.return_value = device | ||
69 | 122 | mock_mpath.is_mpath_device.return_value = False | ||
70 | 123 | |||
71 | 124 | path = block.lookup_disk(wwn) | ||
72 | 125 | |||
73 | 126 | mock_os_listdir.assert_called_with("/dev/disk/by-id/") | ||
74 | 127 | mock_os_path_realpath.assert_called_with("/dev/disk/by-id/" + | ||
75 | 128 | expected_link) | ||
76 | 129 | self.assertTrue(mock_os_path_exists.called) | ||
77 | 130 | self.assertEqual(device, path) | ||
78 | 131 | |||
79 | 106 | @mock.patch('curtin.block.udevadm_info') | 132 | @mock.patch('curtin.block.udevadm_info') |
80 | 107 | def test_get_device_mapper_links_returns_first_non_none(self, m_info): | 133 | def test_get_device_mapper_links_returns_first_non_none(self, m_info): |
81 | 108 | """ get_device_mapper_links returns first by sort entry in DEVLINKS.""" | 134 | """ get_device_mapper_links returns first by sort entry in DEVLINKS.""" |
82 | diff --git a/tests/unittests/test_commands_block_meta.py b/tests/unittests/test_commands_block_meta.py | |||
83 | index b2e151e..dab133e 100644 | |||
84 | --- a/tests/unittests/test_commands_block_meta.py | |||
85 | +++ b/tests/unittests/test_commands_block_meta.py | |||
86 | @@ -11,6 +11,96 @@ from curtin import paths, util | |||
87 | 11 | from .helpers import CiTestCase | 11 | from .helpers import CiTestCase |
88 | 12 | 12 | ||
89 | 13 | 13 | ||
90 | 14 | class TestGetPathToStorageVolume(CiTestCase): | ||
91 | 15 | |||
92 | 16 | def setUp(self): | ||
93 | 17 | super(TestGetPathToStorageVolume, self).setUp() | ||
94 | 18 | basepath = 'curtin.commands.block_meta.' | ||
95 | 19 | self.add_patch(basepath + 'os.path.exists', 'm_exists') | ||
96 | 20 | self.add_patch(basepath + 'block.lookup_disk', 'm_lookup') | ||
97 | 21 | self.add_patch(basepath + 'devsync', 'm_devsync') | ||
98 | 22 | |||
99 | 23 | def test_block_lookup_called_with_disk_wwn(self): | ||
100 | 24 | volume = 'mydisk' | ||
101 | 25 | wwn = self.random_string() | ||
102 | 26 | cfg = {'id': volume, 'type': 'disk', 'wwn': wwn} | ||
103 | 27 | s_cfg = OrderedDict({volume: cfg}) | ||
104 | 28 | block_meta.get_path_to_storage_volume(volume, s_cfg) | ||
105 | 29 | expected_calls = [call(wwn)] | ||
106 | 30 | self.assertEqual(expected_calls, self.m_lookup.call_args_list) | ||
107 | 31 | |||
108 | 32 | def test_block_lookup_called_with_disk_serial(self): | ||
109 | 33 | volume = 'mydisk' | ||
110 | 34 | serial = self.random_string() | ||
111 | 35 | cfg = {'id': volume, 'type': 'disk', 'serial': serial} | ||
112 | 36 | s_cfg = OrderedDict({volume: cfg}) | ||
113 | 37 | block_meta.get_path_to_storage_volume(volume, s_cfg) | ||
114 | 38 | expected_calls = [call(serial)] | ||
115 | 39 | self.assertEqual(expected_calls, self.m_lookup.call_args_list) | ||
116 | 40 | |||
117 | 41 | def test_block_lookup_called_with_disk_wwn_fallback_to_serial(self): | ||
118 | 42 | volume = 'mydisk' | ||
119 | 43 | wwn = self.random_string() | ||
120 | 44 | serial = self.random_string() | ||
121 | 45 | cfg = {'id': volume, 'type': 'disk', 'wwn': wwn, 'serial': serial} | ||
122 | 46 | s_cfg = OrderedDict({volume: cfg}) | ||
123 | 47 | |||
124 | 48 | # doesn't find wwn, returns path on serial | ||
125 | 49 | self.m_lookup.side_effect = iter([ValueError('Error'), 'foo']) | ||
126 | 50 | |||
127 | 51 | block_meta.get_path_to_storage_volume(volume, s_cfg) | ||
128 | 52 | expected_calls = [call(wwn), call(serial)] | ||
129 | 53 | self.assertEqual(expected_calls, self.m_lookup.call_args_list) | ||
130 | 54 | |||
131 | 55 | def test_fallback_to_path_when_lookup_wwn_serial_fail(self): | ||
132 | 56 | volume = 'mydisk' | ||
133 | 57 | wwn = self.random_string() | ||
134 | 58 | serial = self.random_string() | ||
135 | 59 | path = "/%s/%s" % (self.random_string(), self.random_string()) | ||
136 | 60 | cfg = {'id': volume, 'type': 'disk', | ||
137 | 61 | 'path': path, 'wwn': wwn, 'serial': serial} | ||
138 | 62 | s_cfg = OrderedDict({volume: cfg}) | ||
139 | 63 | |||
140 | 64 | # lookups fail | ||
141 | 65 | self.m_lookup.side_effect = iter([ | ||
142 | 66 | ValueError('Error'), ValueError('Error')]) | ||
143 | 67 | |||
144 | 68 | result = block_meta.get_path_to_storage_volume(volume, s_cfg) | ||
145 | 69 | expected_calls = [call(wwn), call(serial)] | ||
146 | 70 | self.assertEqual(expected_calls, self.m_lookup.call_args_list) | ||
147 | 71 | self.assertEqual(path, result) | ||
148 | 72 | |||
149 | 73 | def test_block_lookup_not_called_with_wwn_or_serial_keys(self): | ||
150 | 74 | volume = 'mydisk' | ||
151 | 75 | path = "/%s/%s" % (self.random_string(), self.random_string()) | ||
152 | 76 | cfg = {'id': volume, 'type': 'disk', 'path': path} | ||
153 | 77 | s_cfg = OrderedDict({volume: cfg}) | ||
154 | 78 | result = block_meta.get_path_to_storage_volume(volume, s_cfg) | ||
155 | 79 | self.assertEqual(0, self.m_lookup.call_count) | ||
156 | 80 | self.assertEqual(path, result) | ||
157 | 81 | |||
158 | 82 | def test_exception_raise_if_disk_not_found(self): | ||
159 | 83 | volume = 'mydisk' | ||
160 | 84 | wwn = self.random_string() | ||
161 | 85 | serial = self.random_string() | ||
162 | 86 | path = "/%s/%s" % (self.random_string(), self.random_string()) | ||
163 | 87 | cfg = {'id': volume, 'type': 'disk', | ||
164 | 88 | 'path': path, 'wwn': wwn, 'serial': serial} | ||
165 | 89 | s_cfg = OrderedDict({volume: cfg}) | ||
166 | 90 | |||
167 | 91 | # lookups fail | ||
168 | 92 | self.m_lookup.side_effect = iter([ | ||
169 | 93 | ValueError('Error'), ValueError('Error')]) | ||
170 | 94 | # no path | ||
171 | 95 | self.m_exists.return_value = False | ||
172 | 96 | |||
173 | 97 | with self.assertRaises(ValueError): | ||
174 | 98 | block_meta.get_path_to_storage_volume(volume, s_cfg) | ||
175 | 99 | expected_calls = [call(wwn), call(serial)] | ||
176 | 100 | self.assertEqual(expected_calls, self.m_lookup.call_args_list) | ||
177 | 101 | self.m_exists.assert_has_calls([call(path)]) | ||
178 | 102 | |||
179 | 103 | |||
180 | 14 | class TestBlockMetaSimple(CiTestCase): | 104 | class TestBlockMetaSimple(CiTestCase): |
181 | 15 | def setUp(self): | 105 | def setUp(self): |
182 | 16 | super(TestBlockMetaSimple, self).setUp() | 106 | super(TestBlockMetaSimple, self).setUp() |
183 | @@ -1086,5 +1176,4 @@ class TestDmCryptHandler(CiTestCase): | |||
184 | 1086 | self.m_subp.assert_has_calls(expected_calls) | 1176 | self.m_subp.assert_has_calls(expected_calls) |
185 | 1087 | self.assertEqual(len(util.load_file(self.crypttab).splitlines()), 1) | 1177 | self.assertEqual(len(util.load_file(self.crypttab).splitlines()), 1) |
186 | 1088 | 1178 | ||
187 | 1089 | |||
188 | 1090 | # vi: ts=4 expandtab syntax=python | 1179 | # vi: ts=4 expandtab syntax=python |
189 | diff --git a/tests/unittests/test_curthooks.py b/tests/unittests/test_curthooks.py | |||
190 | index 613f068..95276f7 100644 | |||
191 | --- a/tests/unittests/test_curthooks.py | |||
192 | +++ b/tests/unittests/test_curthooks.py | |||
193 | @@ -353,7 +353,8 @@ class TestSetupGrub(CiTestCase): | |||
194 | 353 | self.target, '/dev/vdb'],), | 353 | self.target, '/dev/vdb'],), |
195 | 354 | self.mock_subp.call_args_list[0][0]) | 354 | self.mock_subp.call_args_list[0][0]) |
196 | 355 | 355 | ||
198 | 356 | def test_uses_grub_install_on_storage_config(self): | 356 | @patch('curtin.commands.curthooks.os.path.exists') |
199 | 357 | def test_uses_grub_install_on_storage_config(self, m_exists): | ||
200 | 357 | cfg = { | 358 | cfg = { |
201 | 358 | 'storage': { | 359 | 'storage': { |
202 | 359 | 'version': 1, | 360 | 'version': 1, |
203 | @@ -368,6 +369,7 @@ class TestSetupGrub(CiTestCase): | |||
204 | 368 | }, | 369 | }, |
205 | 369 | } | 370 | } |
206 | 370 | self.subp_output.append(('', '')) | 371 | self.subp_output.append(('', '')) |
207 | 372 | m_exists.return_value = True | ||
208 | 371 | curthooks.setup_grub(cfg, self.target, osfamily=self.distro_family) | 373 | curthooks.setup_grub(cfg, self.target, osfamily=self.distro_family) |
209 | 372 | self.assertEquals( | 374 | self.assertEquals( |
210 | 373 | ([ | 375 | ([ |
FAILED: Continuous integration, rev:82f20fee48a 89862e006c15521 bb5078df09610e /jenkins. ubuntu. com/server/ job/curtin- ci/3750/ /jenkins. ubuntu. com/server/ job/curtin- ci/nodes= metal-arm64/ 3750/ /jenkins. ubuntu. com/server/ job/curtin- ci/nodes= metal-ppc64el/ 3750/ /jenkins. ubuntu. com/server/ job/curtin- ci/nodes= metal-s390x/ 3750/ /jenkins. ubuntu. com/server/ job/curtin- ci/nodes= torkoal/ 3750/
https:/
Executed test runs:
FAILURE: https:/
SUCCESS: https:/
FAILURE: https:/
FAILURE: https:/
Click here to trigger a rebuild: /jenkins. ubuntu. com/server/ job/curtin- ci/3750/ /rebuild
https:/