ISST-LTE: Multipath disk is not automatically set as 1st boot device after installation

Default Comment by Bridge

Default Comment by Bridge

set package to grub-installer

------- Comment From <email address hidden> 2015-09-30 16:11 EDT-------
I think the problem is grub_util_devname_to_ofpath() doesn't handle multipath devices (mpathX on installer) -- only hd/sd/sr/vdisk/fd devices.

So, it would handle scsi disks (sd) -- the individual paths -- correctly.

Options:
1) Arbitrate an individual path (/dev/sdX) to do grub-install on -- not all simple (e.g., you choose an offline path? it goes offline during grub-install? -- i.e., don't reinvent the multipath wheel)
2) Add logic for grub-install to handle multipath devices (/dev/mapper/mpathX).

Code:
=====

util/grub-install.c
main()
...
if (platform == GRUB_INSTALL_PLATFORM_POWERPC_IEEE1275)
...
switch (platform)
...
case GRUB_INSTALL_PLATFORM_SPARC64_IEEE1275:
case GRUB_INSTALL_PLATFORM_POWERPC_IEEE1275:
case GRUB_INSTALL_PLATFORM_I386_IEEE1275:
{
const char * ofpath = grub_util_devname_to_ofpath (*curdev);
g = xasprintf ("ieee1275/%s", ofpath);
break;
}
...

grub-core/osdep/linux/ofpath.c
grub_util_devname_to_ofpath()
...
if (device[0] == 'h' && device[1] == 'd')
ofpath = of_path_of_ide(name_buf, device, devnode, devicenode);
else if (device[0] == 's'
&& (device[1] == 'd' || device[1] == 'r'))
ofpath = of_path_of_scsi(name_buf, device, devnode, devicenode);
else if (device[0] == 'v' && device[1] == 'd' && device[2] == 'i'
&& device[3] == 's' && device[4] == 'k')
ofpath = of_path_of_vdisk(name_buf, device, devnode, devicenode);
else if (device[0] == 'f' && device[1] == 'd'
&& device[2] == '0' && device[3] == '\0')
/* All the models I've seen have a devalias "floppy".
New models have no floppy at all. */
ofpath = xstrdup ("floppy");
else
{
grub_util_warn (_("unknown device type %s\n"), device);
return NULL;
}
...

------- Comment From <email address hidden> 2015-09-30 16:24 EDT-------
@taco-screen-team

This bug is likely a good one for @mathieu-tl (multipath, grub-installer).

Thanks

------- Comment From <email address hidden> 2015-10-02 19:57 EDT-------
There's a different function path that's relevant here.
It seems the previous one only effects a config or something.
(I couldn't verify at install time w/ gdb yet, but per the call to 'nvram', it's likely to be correct now.)

The analysis points to ofpathname called w/ a /dev/mapper/mpathX path, which it can't work with.
So, the options are still handling this in either grub-installer or grub2.

It should be simpler to handle errors in grub-installer rather than grub2
(even accounting for checks of working/online paths, retries in case of errors, etc).

Function path and comments:
---------------------------------------

The boot device settings are modified here:

main()
@ grub-install.c
...
dev = grub_util_get_os_disk (install_device);
grub_install_register_ieee1275 (0, dev, partno,
"\\\\BootX");
...
grub_install_register_ieee1275 (1, grub_util_get_os_disk (install_device),
0, NULL);
...

There are 2 relevant functions:
- grub_install_register_ieee1275()
- grub_util_get_os_disk()

First,

grub_install_register_ieee1275()
@ grub-core/osdep/unix/platform.c

char *boot_device;
...
ofpath = get_ofpathname (install_device);
...
ptr = grub_stpcpy (boot_device, ofpath);
...
boot_device = get_ofpathname (install_device);
...
if (strcmp (grub_install_get_default_powerpc_machtype (), "chrp_ibm") == 0)
{
char *arg = xasprintf ("boot-device=%s", boot_device);
if (grub_util_exec ((const char * []){ "nvram",
"--update-config", arg, NULL }))
...

which runs 'nvram --update-config boot-device=' with the value printed by 'ofpathname' for that device.

get_ofpathname()
@ grub-core/osdep/unix/platform.c
...
pid = grub_util_exec_pipe ((const char * []){ "ofpathname", dev, NULL }, &fd);
...

Second, 'that device' is obtained by:

grub_util_get_os_disk
@ util/getroot.c
...
return convert_system_partition_to_system_disk (os_dev, &is_part);
...

convert_system_partition_to_system_disk()
@ util/getroot.c
...
if (grub_util_device_is_mapped_stat (&st))
return grub_util_devmapper_part_to_disk (&st, is_part, os_dev);
...

grub_util_devmapper_part_to_disk()
@ grub-core/osdep/devmapper/getroot.c
...
if (grub_util_get_dm_node_linear_info (st->st_rdev,
&major, &minor, 0))
{
*is_part = 1;
return grub_find_device ("/dev",
(major << 8) | minor);
}
*is_part = 0;
return xstrdup (path);

But grub_util_get_dm_node_linear_info() doesn't understand mpath/LVM devmapper targets, only linear.
If it's linear, then it sets major and minor.

grub_util_get_dm_node_linear_info()
@ grub-core/osdep/devmapper/hostdisk.c
...
if (node_uuid && (strncmp (node_uuid, "LVM-", 4) == 0
|| strncmp (node_uuid, "mpath-", 6) == 0))
{
dm_task_destroy (dmt);
break;
}
...
if (grub_strcmp (target, "linear") != 0)
{
dm_task_destroy (dmt);
break;
}
...
major = grub_strtoul (params, &ptr, 10);
...
minor = grub_strtoul (ptr, &ptr, 10);

Otherwise, later on:
return 1;

Which makes grub_util_devmapper_part_to_disk() not to call grub_find_device(),
returning the same path it received.

This path is not understandable by ofpathname, so nvram probably ends up updating something incorrectly.

From the description, I understand this problem is specific to installation in PowerVM mode (PReP boot configuration) with multipath root disk. Is that correct?

grub-installer will configure a PReP partition when not running in PowerVM mode, and will equally fail to configure the path when the root disk is on a multipath device, but the failure would not affect the system's bootability after install when booting in PowerNV mode.

This appears to be broken in ofpathname, provided by powerpc-ibm-utils... Fixing.

@vorlon

> From the description, I understand this problem is specific to installation
> in PowerVM mode (PReP boot configuration) with multipath root disk.
> Is that correct?

That's correct.

> grub-installer will configure a PReP partition when not running in PowerVM mode,
> and will equally fail to configure the path when the root disk is on a multipath device,
> but the failure would not affect the system's bootability after install when booting in PowerNV mode.

I'd suspect that petitboot might do the boot device selection differently,
and might eventually end up choosing a boot device if one is not set in NVRAM,
or might use a different NVRAM key/variable for it.

It's possible, as the pseries (powervm) and powernv (opal) platforms are distinct, thus can do things differently.

This bug was fixed in the package powerpc-ibm-utils - 1.2.26-2ubuntu1

---------------
powerpc-ibm-utils (1.2.26-2ubuntu1) wily; urgency=medium

  * mpath_mappings.patch: Map /dev/mapper/mpath* to the right dm-* devices
    even if they don't come up as symlinks but actual nodes. (LP: #1501015)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 20 Oct 2015 13:59:32 +0100

Does this bug also happen at all on 14.04?

This bug report mentions "The symptom is the same as launchpad bug 1486022 against 14.04.3.", yet it also mentions this is for installs of 15.10.

I've already created the tasks for 14.04 and 15.04, but there is no point in doing stable updates and risking to introduce new issues from a rebuild or a new version if the bug isn't happening on these releases.

Hi @mathieu-tl,

> Does this bug also happen at all on 14.04?

Yes.
The test teams hit this recently, and the patch has been verified to fix the issue in 14.04.x too.
The LTC bugzilla is in the process of being mirrored, once it's done I'll mark it as a duplicate of this one.

I'll attach the debdiff used (just yours modified to apply on trusty).

Can you please apply it for 14.04.4?

Thanks!

> The LTC bugzilla is in the process of being mirrored, once it's done I'll mark it as a duplicate of this one.

The duplicate is LP 1533327.

Of course. I had added a task for trusty just in case because I saw it could theoretically happen on 14.04; although it was unclear whether it actually did. I suppose the new multipath-tools might not help in this regard.

Setting to In Progress, thanks for the debdiff!

Sure thing; thank you.

Hello bugproxy, or anyone else affected,

Accepted powerpc-ibm-utils into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/powerpc-ibm-utils/1.2.26-1~15.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello bugproxy, or anyone else affected,

Accepted powerpc-ibm-utils into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/powerpc-ibm-utils/1.2.26-1~14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Thanks; asked the test team to verify the installation w/ apt-setup/proposed=true, and will update once verified.

The test team verified the package in -proposed works OK.
Setting verification-done.

> Chanh H. Nguyen 2016-01-19 14:43:28 BRST
>> Bug Review: waiting on reporter to verify packages w/ fix from the -proposed
>> repository.
>
> Yes, it works fine when I use the "apt-setup/proposed=true" at the boot command line,
> installation finish and system come back fine to the login prompt.

This bug was fixed in the package powerpc-ibm-utils - 1.2.26-1~14.04.2

---------------
powerpc-ibm-utils (1.2.26-1~14.04.2) trusty; urgency=medium

  * mpath_mappings.patch: Map /dev/mapper/mpath* to the right dm-* devices
    even if they don't come up as symlinks but actual nodes. (LP: #1501015)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 20 Oct 2015 13:59:32 +0100

The verification of the Stable Release Update for powerpc-ibm-utils has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package powerpc-ibm-utils - 1.2.26-1~15.04.2

---------------
powerpc-ibm-utils (1.2.26-1~15.04.2) vivid; urgency=medium

  * mpath_mappings.patch: Map /dev/mapper/mpath* to the right dm-* devices
    even if they don't come up as symlinks but actual nodes. (LP: #1501015)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 20 Oct 2015 13:59:32 +0100