Ubuntu
partman-base package

fix handling of multipathed disks

Bug #1430074 reported by Mauricio Faria de Oliveira
24
This bug affects 1 person
Affects Status Importance Assigned to Milestone
grub-installer (Ubuntu)
Fix Released
Critical
Mathieu Trudel-Lapierre
Trusty
Fix Released
Critical
Mathieu Trudel-Lapierre
partman-base (Ubuntu)
Fix Released
Critical
Mathieu Trudel-Lapierre
Trusty
Fix Released
Critical
Mathieu Trudel-Lapierre
partman-multipath (Ubuntu)
Fix Released
Critical
Mathieu Trudel-Lapierre
Trusty
Fix Released
Critical
Mathieu Trudel-Lapierre

Bug Description

This bug is an umbrella for a number of bugs in the debian-installer with common
root cause: the different disk-partition separator used for multipath partition
block devices by parted_server (libparted) and some components of the installer.

The specific problems will be presented shortly. They are summarized in this bug
rather than reported individually because the solution is common, and involves 3
source packages; so it's simple to glance over the bug (.. big) picture, problem
and solution at once, and likewise for reviewing the patches.

The demonstration and steps to reproduce are contained in an attached text file,
for being an install session with descriptive steps and comments. It's somewhat
long, but very easy to follow. The solution testing is also attached in a text
file (and is shorter).

Overall problem:
---------------

The installer (i.e., its components; e.g., partman-*, grub-installer) makes use
of the 'parted_server' program (which relies on 'libparted') to handle disk and
partition devices. It's used to list disks and their partitions, and to modify
partitions in the disks (e.g., create, resize, flag, delete).

Whenever parted_server (libparted) produces a list of partitions in a multipath
disk (for consumption by installer components), or creates such (i.e., a block
device in /dev/mapper for a partition in a multipath disk), the following disk-
parition separator is used: 'p'. This is hard-coded in libparted (arch/linux.c).

For example: /dev/mapper/mpath0p1

The problems arise on some installer components which have code for handling an
installation on multipath disks/partitions (e.g., partition the multipath disks,
format the partitions in multipath disks, install the bootloader in one of them).

Those components assume the multipath disk-partition separator is '-part'. This
is hard-coded in them.

For example: /dev/mapper/mpath0-part1

Overall solution:
----------------

After the evaluation of a few solution alternatives, this one looked like right.

It's short, simple, and non-invasive -- all changes are delivered in multipath-
specific functions or conditional blocks, except for 2 ppc64el-specific changes,
that actually are required to improve grub-installer for multipath on ppc64el).

Fundamentally, stick to 'p' as multipath disk-partition separator.

Reasons:
- First of all: no impact nor change to non-multipath stuff.
- Second: multipath partitions listed by parted_server exist in /dev/mapper,
          with the listed block device name; consistently:
          - including those created/modified by parted_server,
          - and those existing/non-modified -- untouched by parted_server.

Specific problems:
-----------------

In order of appearance (in the installer). See the attached file with the steps
to reproduce/demonstration.

P.S.: some problems reported are not related to the disk-partition separator,
      but are minor errors related to the multipath support in the installer
      (still contained in multipath-specific functions or conditional blocks)
      and have fixes provided in the best interest of the debian-installer.

Problem #1: The underlying devices of multipath devices are listed in the
---------- partitioning dialogs.

The underlying devices cannot be accessed, because the device-mapper locks them
with exclusive access.

There's already code to do this, but it's broken due to a wrong grep expression,
that checks whether a device is an underlying device of a multipath devices.

Fixed in partman-base:
 - init.d/parted: part_of_multipath():
   Update grep expression for more recent output of 'multipath -l'.

Problem #2: Human-description of a multipath device is not a single device string;
---------- it actually contains multiple strings from the same device name/WWID.

Some multipath devices' WWID have more than only hexadecimal characters; this
breaks a sed substitution expression, thus leaving spaces in its output, which
are not correctly handled.

Fixed in partman-base:
 - lib/base.sh: humandev():
   Accept spaces/non-hex-only characters in multipath WWID.

Problem #3: Non-existing LVM VGs/LVs shown for existing multipath partitions
----------

If multipath partitions are not correctly identified (because of the multipath
disk-partition separator), they end up considered LVM VGs/LVs in the dialog for
listing/choosing partitions.

Fixed in partman-base:
 - lib/base.sh: is_multipath_part():
   Use 'p' (not '-part') as multipath disk-partition separator.

Problem #4: Failure when formatting existing multipath partitions (i.e.,
---------- not created/modified by the partitioner via parted_server)

If multipath partitions are not created/modified with the partitioner, they are
not created in /dev/mapper by parted_server/libparted (note: this would create
them with the 'p' disk-partition separator).

In that case, the multipath partitions end up created in /dev/mapper by kpartx
with the '-part' disk-partition separator in a commit.d/ hook.

Afterwards, in the format.d/ hooks, parted_server lists the partitions in a
multipath disk with the 'p' disk-partition separator, which leads to an error,
because (as described) /dev/mapper/mpathX*p*Y doesn't exist, but mpathX*-part*Y.

Fixed in partman-multipath:
 - commit.d/partition_multipath:
   Use 'p' (not '-part') as multipath disk-partition separator.
   (during d-i.)

Problem #5: Incorrect multipath partition device names in target's /etc/fstab
----------

In the finish.d/ hooks, the device names that are inserted in /target/etc/fstab
come from partition listings by parted_server -- therefore, with the 'p' disk-
partition separator.

This is wrong for the target system, because it has kpartx udev rules shipped
with the multipath-tools package (and carried to initramfs by multipath-tools-boot),
and the kpartx udev rules specify the '-part' disk-partition separator.

So, this is one place where we actually have to change from 'p' to '-part'.

It is done immediately after the hook/script that inserts /dev/mapper devices
in the /target/etc/fstab file, using a sed expression.

Fixed in partman-multipath:
 - (new file) finish.d/fstab_hd_entries_multipath:
   Use '-part' (not 'p') as multipath disk-partition separator.
   (for the target.)

Problem #6: PReP partition-flag is lost on multipath disk every 2 changes to
---------- its partition table

(this should be automatically fixed with problem #1.)

When a multipath disk is first partitioned, and a multipath partition is defined
as PReP boot partition, the partition has the flag 'prep' set.

The flag propagates to the underlying device, obviously (multipath is just multi-
ple paths to the a device).

The next time that disk is partitioned, that occurs normally for the multipath
disk; however, the partitioner notices that the underlying device now has a
partition with the 'prep' flag set, and because it was not requested on the
underlying device (only on the multipath disk), the partitioner resets the flag.

The flag reset in the underlying device also propagates to the multipath device.

Now there's no partition w/ the PReP flag anymore.

This is a problem for grub-installer on ppc64el, because it currently uses the
'prep-bootdev' program, which finds a PReP partition using that flag/libparted.

Fixed in problem #1.

Problem #7: GRUB is installed on an underlying device, not the multipath disk.
----------

The grub-installer script fails to identify the partition with boot filesystem
as a multipath partition because it expects the '-part' disk-partition separator.

Therefore, an underlying disk/partition is selected. Although it is working on
the test scenario, it is certainly not the right approach with multipath, and
should be fixed.

This is fixed differently for non-ppc64el and ppc64el architectures, because
ppc64el currently uses the 'prep-bootdev' program to find a PReP partition.

Fixed in grub-installer for non-ppc64el:
 - grub-installer:
   Use 'p' (not '-part') as multipath disk-partition separator.

Fixed in grub-installer for ppc64el:
 - grub-installer:
   [ppc64el] bootdev/wipe_bootdev: prefer PReP partition in the
   same disk as the boot file system partition ($disc_offered).
 - prep-bootdev:
   add '-l' option to list all PReP partitions.

Steps to Reproduce
------------------

See attachment (reproducing_mpathsep.txt)

Solution
--------

See attachments:
- partman-base_mpathsep.debdiff
- partman-multipath_mpathsep.debdiff
- grub-installer_mpathsep.debdiff

Solution Testing (ppc64el & amd64)
----------------------------------

See attachment (testing_mpathsep.txt)

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
tags: added: patch
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "Patch to partman-base" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Revision history for this message
Adam Conrad (adconrad) wrote :

If the target system uses disk1-part1 style notation, and most of d-i also expects that, wouldn't it make more sense to keep consistent and munge the return from parted to conform to that, rather than changing every other component?

Revision history for this message
Steve Langasek (vorlon) wrote :

Mauricio, what is the relationship between this bug and bug #1429327? Do you expect the patches here to fix the symptoms in the other bug?

Steve Langasek (vorlon)
Changed in debian-installer (Ubuntu):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

@adconrad

> [...] wouldn't it make more sense to keep consistent and munge the return from parted [...]
> rather than changing every other component?

Possibly; that was actually the first option evaluated;
e.g., a sed in partman-base, or to change parted.

However, either way, it affects things that are not related at all to multipath.
(i.e., everything that reads from parted_server/libparted somehow)

I tried to keep the changes contained in multipath stuff, mainly for 2 reasons:
1) they're not related/required to non-multipath stuff at all.
2) it's late to ask for patching general stuff.

And it turned out that 'every other component' were actually just 3,
which, given the 2 reasons above, made me consider this approach.

Otherwise, I'd have suggested something more aligned w/ a simpler change.

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

@vorlon

I'm looking at that bug; will post soon.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

I'm looking into this now.

Changed in debian-installer (Ubuntu):
status: New → In Progress
importance: Undecided → Critical
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

It seems to me like at least some of these changes would also fix other bugs reported for installs on ppc64el, ie. the grub-installer fix will help with another bug with detecting the prep partitions.

Adding tasks for partman-* and grub-installer; I'll close them as appropriate if it turns out it would be better to make the changes elsewhere.

Changed in grub-installer (Ubuntu):
status: New → In Progress
Changed in partman-base (Ubuntu):
status: New → In Progress
Changed in partman-multipath (Ubuntu):
status: New → In Progress
Changed in grub-installer (Ubuntu):
importance: Undecided → Critical
Changed in partman-base (Ubuntu):
importance: Undecided → Critical
Changed in partman-multipath (Ubuntu):
importance: Undecided → Critical
Changed in partman-base (Ubuntu):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Changed in grub-installer (Ubuntu):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Changed in partman-multipath (Ubuntu):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Steve Langasek (vorlon)
Changed in debian-installer (Ubuntu Trusty):
milestone: none → ubuntu-14.04.3
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

@vorlon

> Mauricio, what is the relationship between this bug and bug #1429327?

I took a look at that one today; apparently no relation at all.
That one seems related to hardware or delay somehow.

Please don't care bug #1429327 about for now.
I'll get more time on it tomorrow.

Revision history for this message
Phillip Susi (psusi) wrote :

I agree wholeheartedly with this. Since the dawn of time linux has used 'p' as the separator between base devices that end in a digit and their partition number. multipath-tools broke with this standard and that is the cause of all of these problems, so it is multipath-tools that needs to be brought around to the conventional way of doing things.

bugproxy (bugproxy)
tags: added: architecture-ppc64le bugnameltc-122818 severity-critical targetmilestone-inin1504
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-03-16 17:57 EDT-------
*** Bug 122802 has been marked as a duplicate of this bug. ***

Mathieu Trudel-Lapierre (cyphermox)
summary: - d-i multipath disk-partition separator overhaul
+ fix handling of multipathed disks
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package partman-multipath - 4ubuntu1

---------------
partman-multipath (4ubuntu1) vivid; urgency=medium

  [ Mauricio Faria de Oliveira ]
  * Fix multipath support device naming (LP: #1430074):
    - commit.d/partition_multipath:
      Use 'p' (not '-part') as multipath disk-partition separator.
    - (new file) finish.d/fstab_hd_entries_multipath:
      Use '-part' (not 'p') as multipath disk-partition separator in fstab
      everywhere, as it is how devices will be detected after boot.
 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 17 Mar 2015 00:20:26 -0400

Changed in partman-multipath (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub-installer - 1.78ubuntu27

---------------
grub-installer (1.78ubuntu27) vivid; urgency=medium

  [ Mauricio Faria de Oliveira ]
  * grub-installer: Use 'p' (not '-part') as multipath disk-partition
    separator. This fixes the check of whether the boot file system is on
    multipath (i.e., $disc_offered is a multipath partition). (LP: #1430074)
  * Better handle PReP partitions on ppc64el: (LP: #1415918)
    - prep-bootdev: add '-l' option to list all PReP partitions. With this,
      it's possible to do more things to select a PReP partition.
    - grub-installer [ppc64el]: bootdev/wipe_bootdev: prefer PReP partition in
      the same disk as the boot file system partition ($disc_offered).
 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 17 Mar 2015 00:25:13 -0400

Changed in grub-installer (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package partman-base - 181ubuntu2

---------------
partman-base (181ubuntu2) vivid; urgency=medium

  [ Mauricio Faria de Oliveira ]
  * Fix multipath support: (LP: #1430074)
    - lib/base.sh: is_multipath_part(): Use 'p' (not '-part') as multipath
      disk-partition separator.
    - lib/base.sh: humandev(): Use 'p' (not '-part') as multipath
      disk-partition separator.
    - lib/base.sh: humandev(): Accept spaces in multipath WWID.
    - init.d/parted: part_of_multipath(): Update grep expression for more
      recent output of 'multipath -l'.
 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 17 Mar 2015 00:06:34 -0400

Changed in partman-base (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-03-20 08:19 EDT-------
*** Bug 122802 has been marked as a duplicate of this bug. ***

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-03-23 12:01 EDT-------
*** Bug 122853 has been marked as a duplicate of this bug. ***

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

There as nothing to do in d-i itself for this, all the changes have been made relative to multipath disks in partman* packages and grub-installer.

Changed in debian-installer (Ubuntu Trusty):
status: New → Invalid
Changed in debian-installer (Ubuntu):
status: In Progress → Invalid
Mathieu Trudel-Lapierre (cyphermox)
Changed in partman-base (Ubuntu Trusty):
status: New → In Progress
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Mathieu Trudel-Lapierre (cyphermox)
Changed in partman-multipath (Ubuntu Trusty):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
status: New → In Progress
Changed in grub-installer (Ubuntu Trusty):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
status: New → In Progress
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Mauricio, or anyone else affected,

Accepted partman-multipath into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/partman-multipath/4ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in partman-multipath (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in partman-base (Ubuntu Trusty):
status: In Progress → Fix Committed
Revision history for this message
Adam Conrad (adconrad) wrote :

Hello Mauricio, or anyone else affected,

Accepted partman-base into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/partman-base/172ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in grub-installer (Ubuntu Trusty):
status: In Progress → Fix Committed
Revision history for this message
Adam Conrad (adconrad) wrote :

Hello Mauricio, or anyone else affected,

Accepted grub-installer into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub-installer/1.78ubuntu20.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Mauricio Faria de Oliveira (mfo)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

I verified the changes are OK with trusty-proposed; tags changed.

partman-multipath: the p/-part partition separator is OK w/ the partitioner and fstab.
parman-base: likewise for p/-part, also OK on humandev() and its output in partitioner dialogs, and not listing underlying devices in partitioner dialogs.
grub-installer: likewise for p/-part, selects the boot/PReP partition.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub-installer - 1.78ubuntu20.2

---------------
grub-installer (1.78ubuntu20.2) trusty; urgency=medium

  * grub-installer: Use 'p' (not '-part') as multipath disk-partition
    separator. This fixes the check of whether the boot file system is on
    multipath (i.e., $disc_offered is a multipath partition). (LP: #1430074)
  * Better handle PReP partitions on ppc64el: (LP: #1415918)
    - prep-bootdev: add '-l' option to list all PReP partitions. With this,
      it's possible to do more things to select a PReP partition.
    - grub-installer [ppc64el]: bootdev/wipe_bootdev: prefer PReP partition in
      the same disk as the boot file system partition ($disc_offered).

 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 17 Jul 2015 13:12:57 -0400

Changed in grub-installer (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Adam Conrad (adconrad) wrote : Update Released

The verification of the Stable Release Update for grub-installer has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package partman-multipath - 4ubuntu0.1

---------------
partman-multipath (4ubuntu0.1) trusty; urgency=medium

  * post-base-installer.d/60multipath: copy multipath bindings to target.
    (LP: #1445973)
  * Fix multipath support device naming (LP: #1430074):
    - commit.d/partition_multipath:
      Use 'p' (not '-part') as multipath disk-partition separator.
    - (new file) finish.d/fstab_hd_entries_multipath:
      Use '-part' (not 'p') as multipath disk-partition separator in fstab
      everywhere, as it is how devices will be detected after boot.

 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 17 Jul 2015 12:48:29 -0400

Changed in partman-multipath (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package partman-base - 172ubuntu1.1

---------------
partman-base (172ubuntu1.1) trusty; urgency=medium

  * Fix multipath support: (LP: #1430074)
    - lib/base.sh: is_multipath_part(): Use 'p' (not '-part') as multipath
      disk-partition separator.
    - lib/base.sh: humandev(): Use 'p' (not '-part') as multipath
      disk-partition separator.
    - lib/base.sh: humandev(): Accept spaces in multipath WWID.
    - init.d/parted: part_of_multipath(): Update grep expression for more
      recent output of 'multipath -l'.
  * init.d/parted: part_of_multipath(): Update grep expression for more
    spaces between disk and major:minor (happens when sd[a-z][a-z] exist)
    (LP: #1436020)

 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 17 Jul 2015 12:03:31 -0400

Changed in partman-base (Ubuntu Trusty):
status: Fix Committed → Fix Released
Mathew Hodson (mhodson)
no longer affects: debian-installer (Ubuntu Trusty)
no longer affects: debian-installer (Ubuntu)
Changed in grub-installer (Ubuntu Trusty):
importance: Undecided → Critical
Changed in partman-base (Ubuntu Trusty):
importance: Undecided → Critical
Changed in partman-multipath (Ubuntu Trusty):
importance: Undecided → Critical
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.