Merge ~mwhudson/curtin:lp-1847771 into curtin:master

[Michael Hudson-Doyle]

storage_config: interpret value, not presence, of DM_MULTIPATH_DEVICE_PATH

Curtin assumes any block device that has the DM_MULTIPATH_DEVICE_PATH
key set in udev is a block device, but that's not correct. Since
multipath-tools 0.7.7 (according to
https://www.spinics.net/lists/dm-devel/msg35965.html), _every_ block
device that multipathd looks at gets that key set, but it is set to "1"
for things that are part of a multipathed device and "0" for things that
are not. So fix the checks in curtin to follow that.

Not tested. But probably right?

[Michael Hudson-Doyle (mwhudson)]

I've tested this now and it prevents the crash on my laptop (whereas the current daily crashes)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:788630651611669da45e79364871e7a4d29f30c8
https://jenkins.ubuntu.com/server/job/curtin-ci/3735/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3735/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3735/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3735/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3735/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3735//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:788630651611669da45e79364871e7a4d29f30c8
https://jenkins.ubuntu.com/server/job/curtin-ci/3736/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3736/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3736/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3736/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3736/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3736//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:788630651611669da45e79364871e7a4d29f30c8
https://jenkins.ubuntu.com/server/job/curtin-ci/3737/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3737/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3737/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3737/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3737/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3737//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:788630651611669da45e79364871e7a4d29f30c8
https://jenkins.ubuntu.com/server/job/curtin-ci/3738/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3738/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3738/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3738/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3738/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3738//rebuild

review: Approve (continuous-integration)

[Paride Legovini (paride)]

This is a bit annoying as I didn't change anything to make the CI run work.

The "torkoal" job runs on amd64 and includes a short vmtest run. The run failed as curtin took more than 3000s to install Bionic. When I tried manually keeping an eye on the logs in worked; when I tried it again from Jenkins it worked (CI run #3). But in the same run the s390x run failed:

======================================================================
ERROR: test_needs_formatting_layout_mismatch (unittests.test_block_dasd.TestNeedsFormatting)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/jenkins/servers/server/workspace/curtin-ci/nodes/metal-s390x/curtin-3737/tests/unittests/test_block_dasd.py", line 311, in setUp
    self.dasd = dasd.DasdDevice(random_device_id())
  File "/var/lib/jenkins/servers/server/workspace/curtin-ci/nodes/metal-s390x/curtin-3737/curtin/block/dasd.py", line 309, in __init__
    super(DasdDevice, self).__init__(device_id)
  File "/var/lib/jenkins/servers/server/workspace/curtin-ci/nodes/metal-s390x/curtin-3737/curtin/block/dasd.py", line 283, in __init__
    _valid_device_id(self.device_id)
  File "/var/lib/jenkins/servers/server/workspace/curtin-ci/nodes/metal-s390x/curtin-3737/curtin/block/dasd.py", line 274, in _valid_device_id
    "device_id invalid: devno not in 0-0x10000: '%s'" % dev)
ValueError: device_id invalid: devno not in 0-0x10000: '10000'

Full output: https://paste.ubuntu.com/p/7MTR8wvxFn/

It worked when I re-run it manually, and then worked when I re-run it from Jenkins.

[Dan Watkins (oddbloke)]

This looks reasonable to me, but I'd like Ryan's review too.

[Dan Watkins (oddbloke)]

OK, as Ryan is out today, I've dug into this more, and I'm happy to land it. Thanks Michael!

review: Approve

[Michael Hudson-Doyle (mwhudson)]

Thanks for the quick review and merge!

diff --git a/curtin/storage_config.py b/curtin/storage_config.py
index 942bc75..3bfa466 100644
--- a/curtin/storage_config.py
+++ b/curtin/storage_config.py
@@ -422,7 +422,7 @@ class ProbertParser(object):
         return None
 
     def is_mpath(self, blockdev):
-        if 'DM_MULTIPATH_DEVICE_PATH' in blockdev:
+        if blockdev.get('DM_MULTIPATH_DEVICE_PATH') == "1":
             return True
 
         return bool('mpath-' in blockdev.get('DM_UUID', ''))
@@ -442,7 +442,7 @@ class ProbertParser(object):
                 return rv
 
     def find_mpath_member(self, blockdev):
-        if 'DM_MULTIPATH_DEVICE_PATH' in blockdev:
+        if blockdev.get('DM_MULTIPATH_DEVICE_PATH') == "1":
             # find all other DM_MULTIPATH_DEVICE_PATH devs with same serial
             serial = blockdev.get('ID_SERIAL')
             members = sorted([os.path.basename(dev['DEVNAME'])
@@ -730,7 +730,7 @@ class BlockdevParser(ProbertParser):
             'type': blockdev_data['DEVTYPE'],
             'id': self.blockdev_to_id(blockdev_data),
         }
-        if 'DM_MULTIPATH_DEVICE_PATH' in blockdev_data:
+        if blockdev_data.get('DM_MULTIPATH_DEVICE_PATH') == "1":
             mpath_name = self.get_mpath_name(blockdev_data)
             entry['multipath'] = mpath_name