multipath shows non-mpath disks as being multipath

Bug #1386637 reported by bugproxy
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
multipath-tools (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Unassigned
Vivid
Won't Fix
High
Unassigned

Bug Description

[Impact]
Users of multipath-tools on systems which exhibit both multipathed and non-multipathed drives or some USB devices (some USB devices may still be picked up by multipath-tools if multipath-capable).

[Test case]
- Boot on a system with multipathed disks with the previous version of multipath-tools
- Upgrade to the new version; verify that /etc/multipath/wwids is created and contains all the right device WWIDs, and that 'sudo multipath -ll' has the same devices.
- Reboot and verify that the system still boots correctly.
- [udev] Using Fibre-Channel hardware: verify that multipath still behaves correctly for path degradation scenarios (see below).

After rebooting, you should verify that /proc/mounts correctly lists the multipath device (/dev/mapper/*), and that /proc/swaps also lists any swap partitions that should be on a multipathed disk as a /dev/dm-* device.

PATH DEGRADATION:
-- This test requires fibre-channel hardware or another method to be able to disconnect paths to a drive.
1) verify that on disconnecting a path, it is properly detected as being in degraded mode by multipath-tools; and shows as such in 'multipath -ll'.
2) verify that on reconnection, the path shows up again as ready or OK in 'multipath -ll'.

[Regression potential]
Systems in complex setups mixing both multipathed and non-multipathed disks, or with particularly slow disk controllers may fail to boot or show delays in booting. Systems requiring a particular set of partitions to be available in early boot (past the typical installs on /, some may require /usr to be available for some daemons to load on boot) could fail to properly start up all the software on boot; these should be considered as regressions only if devices do not properly show up in 'sudo multipath -ll' output.

Users seeing issues should preferrably include the output of 'sudo multipath -v4' to bug reports to help developers in debugging.

Udev rules have been changed in this case, because they interfere with the proper behavior of multipath 0.4.9 with the patches included as backported from 0.5.0; this has the potential to strongly impact the detection of change events on multipath devices; if path degradation or new device detection fails, this should be considered an important regression of multipath-tools.

---

Problem Description
======================
non-multpath disks are classified as multipath after the multipath-tools package is installed.

---uname output---
Linux uu04g1 3.16.0-21-generic #28-Ubuntu SMP Mon Oct 6 15:57:32 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = Tuleta pKVM guest

Steps to Reproduce
=========================
1) Create a guest with an unused, non-mpath disk (not the install disk) (a qemu img file is fine).
2) apt-get install multipath-tools.
3) multipath -ll.
4) Note that your file backed disks are showing up as mpath disks.

Userspace tool common name: multipath
The userspace tool has the following bit modes: 64

System Dump Info:
  The system is not configured to capture a system dump.

Userspace rpm: multipath-tools
Userspace tool obtained from project website: multipath-tools v0.4.9 (05/33, 2016)

== Comment: #1 - Edward R. Cheslek <email address hidden> - ==
root@uu04g1:~# multipath -ll
0QEMU QEMU HARDDISK drive-scsi0-0-1-0 dm-0 QEMU,QEMU HARDDISK
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 0:0:1:0 sdb 8:16 active ready running
0QEMU QEMU HARDDISK drive-scsi0-0-2-0 dm-1 QEMU,QEMU HARDDISK
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 0:0:2:0 sdc 8:32 active ready running
root@uu04g1:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
??sda1 8:1 0 7M 0 part
??sda2 8:2 0 19.1G 0 part /
??sda3 8:3 0 896M 0 part [SWAP]
sdb 8:16 0 20G 0 disk
??0QEMU\x20\x20\x20\x20QEMU\x20HARDDISK\x20\x20\x20drive-scsi0-0-1-0
                                                  252:0 0 20G 0 mpath
sdc 8:32 0 20G 0 disk
??0QEMU\x20\x20\x20\x20QEMU\x20HARDDISK\x20\x20\x20drive-scsi0-0-2-0
                                                  252:1 0 20G 0 mpath
root@uu04g1:~#
root@uu04g1:~# lsscsi
[0:0:0:0] disk QEMU QEMU HARDDISK 2.0. /dev/sda
[0:0:1:0] disk QEMU QEMU HARDDISK 2.0. /dev/sdb
[0:0:2:0] disk QEMU QEMU HARDDISK 2.0. /dev/sdc

== Comment: #2 - Edward R. Cheslek <email address hidden> - 2014-10-08 22:27:09 ==
This is the xml file of the affected guest. Please note that all disks are file backed.

root@uu04g1:~# multipath -l
0QEMU QEMU HARDDISK drive-scsi0-0-1-0 dm-0 QEMU,QEMU HARDDISK
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=-1 status=active
  `- 0:0:1:0 sdb 8:16 active undef running
0QEMU QEMU HARDDISK drive-scsi0-0-2-0 dm-1 QEMU,QEMU HARDDISK
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=-1 status=active
  `- 0:0:2:0 sdc 8:32 active undef running

root@uu04g1:~# multipath -v2
root@uu04g1:~#

Revision history for this message
bugproxy (bugproxy) wrote : xml file of the guest

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-117319 severity-high targetmilestone-inin---
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1386637/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
affects: ubuntu → multipath-tools (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-02-02 04:37 EDT-------
Hello,

Can you please let us know if there are any updates on this bug? Do you need more information from us?

Thanks and regards,

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2015-04-13 19:07 EDT-------
This is working as designed. multipath-tools does what it is told to do via /etc/multipath.conf. You need to blacklist any devices you don't want multipath-tools to create multipath devices for.

description: updated
description: updated
description: updated
description: updated
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted multipath-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.4.9-3ubuntu7.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Mathew Hodson (mhodson)
Changed in multipath-tools (Ubuntu):
importance: Undecided → Medium
Mathew Hodson (mhodson)
Changed in multipath-tools (Ubuntu):
importance: Medium → High
Mathew Hodson (mhodson)
Changed in multipath-tools (Ubuntu):
status: New → Triaged
Steve Langasek (vorlon)
Changed in multipath-tools (Ubuntu Trusty):
status: New → Fix Committed
importance: Undecided → High
Changed in multipath-tools (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

This is already fixed in wily by way of multipath 0.5.0.

Changed in multipath-tools (Ubuntu):
assignee: nobody → brandyseres256@gmail.com (brandyseres256)
Changed in multipath-tools (Ubuntu):
assignee: brandyseres256@gmail.com (brandyseres256) → nobody
Mathew Hodson (mhodson)
Changed in multipath-tools (Ubuntu Trusty):
milestone: none → trusty-updates
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Any news on verifying this?

Edward, looks like the devices aren't set up to have the same serial number, so I'm not sure why they would ever be picked up by multipath as being multipath devices. In any case, have you tested the update in -proposed, which should fix this?

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Download full text (4.2 KiB)

Hi @mathieu-tl,

Marking as verification-failed since the version from -proposed hits a segfault in 'multipath -l'.
Here are some details/stack trace:

It segfaults:

 # multipath -l
 Segmentation fault

Trusty / 14.04.3:

 # lsb_release -d
 Description: Ubuntu 14.04.3 LTS

 # uname -a
 Linux plzp17 3.19.0-29-generic #31~14.04.1-Ubuntu SMP Thu Sep 10 09:45:11 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

Using multipath-tools from trusty-proposed (0.4.9-3ubuntu7.5)

 # apt-cache policy multipath-tools
 multipath-tools:
   Installed: 0.4.9-3ubuntu7.5
   Candidate: 0.4.9-3ubuntu7.5
   Version table:
  *** 0.4.9-3ubuntu7.5 0
  500 http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed/main ppc64el Packages
  100 /var/lib/dpkg/status
      0.4.9-3ubuntu7.4 0
  500 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/main ppc64el Packages
      0.4.9-3ubuntu7 0
  500 http://ports.ubuntu.com/ubuntu-ports/ trusty/main ppc64el Packages

 # dpkg -l | grep 'multipath\|kpartx'
 ii kpartx 0.4.9-3ubuntu7.5 ppc64el create device mappings for partitions
 ii kpartx-boot 0.4.9-3ubuntu7.5 all Provides kpartx during boot
 ii multipath-tools 0.4.9-3ubuntu7.5 ppc64el maintain multipath block device access
 ii multipath-tools-boot 0.4.9-3ubuntu7.5 all Support booting from multipath devices

Problem contained in libmultipath, not any external library/component, in the initialization phase:

 # gdb multipath
 GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
 ...
 (gdb) run -l
 Starting program: /sbin/multipath -l
 ...
 Program received signal SIGSEGV, Segmentation fault.
 0x00003fffb7e3fd70 in vector_alloc_slot () from /lib/libmultipath.so.0
 (gdb) bt
 #0 0x00003fffb7e3fd70 in vector_alloc_slot () from /lib/libmultipath.so.0
 #1 0x00003fffb7e3e200 in keyword_alloc () from /lib/libmultipath.so.0
 #2 0x00003fffb7e3e2c4 in install_keyword_root () from /lib/libmultipath.so.0
 #3 0x00003fffb7e5ac34 in init_keywords () from /lib/libmultipath.so.0
 #4 0x00003fffb7e4a54c in load_config () from /lib/libmultipath.so.0
 #5 0x0000000010002bcc in ?? ()
 #6 0x00003fffb7c94d00 in generic_start_main (main=0x10002b10, argc=<optimized out>,
     argv=0x3ffffffffa28, auxvec=0x3ffffffffb30, init=<optimized out>, rtld_fini=<optimized out>,
     stack_end=<optimized out>, fini=<optimized out>) at ../csu/libc-start.c:287
 #7 0x00003fffb7c94ef8 in __libc_start_main (argc=<optimized out>, argv=<optimized out>,
     ev=<optimized out>, auxvec=<optimized out>, rtld_fini=<optimized out>, stinfo=<optimized out>,
     stack_on_entry=<optimized out>) at ../sysdeps/unix/sysv/linux/powerpc/libc-start.c:93
 #8 0x0000000000000000 in ?? ()
 (gdb)

Downgrading to version in trusty-updates (0.4.9-3ubuntu7.4):

 # apt-get install {multipath-tools,kpartx}{,-boot}=0.4.9-3ubuntu7.4
 ...
 The following packages will be DOWNGRADED:
   kpartx kpartx-boot multipath-tools multipath-tools-boot
 ...
 Get:1 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/main multipath-tools-boot all 0.4.9-3ubuntu7.4 [6,070 B]
 Get:2 http...

Read more...

tags: added: verification-failed
removed: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello bugproxy, or anyone else affected,

Accepted multipath-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.4.9-3ubuntu7.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-failed
tags: added: verification-needed
Revision history for this message
Steve Langasek (vorlon) wrote :

The 0.4.9-3ubuntu7.6 version of multipath-tools is available in trusty-proposed. We have verified that USB disks are no longer being incorrectly tagged as multipath in this version. We have also corrected the regression that caused 'multipath -l' to segfault previously. But we don't have affirmative confirmation that this new version won't cause regressions in the handling of failover paths, which is a concern because of the changes to the udev rules.

Can someone with access to the necessary hardware to test the multipath failover (which according to this bug report requires FC or similar) please verify that this works with the -proposed version of multipath-tools? Edward, Mauricio?

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

@vorlon

Hi Steve,

I've been told you asked for priority in this testing during today's interlock.
It's was already on my list, but now it's moved higher.

I hope sometime next week is not a problem?

Thanks.

Revision history for this message
Steve Langasek (vorlon) wrote :

Mauricio,

If you can test this week, that would be perfect. Thanks!

I see bug #1503305 has been submitted about a display error when injecting a path failure, but that appears to be done with the multipath-tools in 14.04.0 so doesn't bear on this SRU.

Revision history for this message
bugproxy (bugproxy) wrote : xml file of the guest

Default Comment by Bridge

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Mauricio,

Have you managed to test this successfully? I was able to build a vm and test this, but I'm not certain whether the behavior, although it looks correct in detecting the path changes, it indeed correct.

There's one noticeable difference at least on qemu; when the paths are detected as removed they stop being listed at all in multipath -ll output.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

It's been a few days now and I was able to verify the fix in what looked like an appropriate manner; marking verification-done. We should move forward with shipping this fix.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Hi Mathieu,

I apologize on the delay here.
As you may know, I'm a bit away from usual bug work for a few weeks.
I still expect to get that SRU verification done in some time.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.3 KiB)

This bug was fixed in the package multipath-tools - 0.4.9-3ubuntu7.6

---------------
multipath-tools (0.4.9-3ubuntu7.6) trusty; urgency=medium

  * debian/patches/0043-alloc-keywords.patch: fix uninitialized keywords
    vector for configuration parsing for when there is no multipath.conf file
    present.

multipath-tools (0.4.9-3ubuntu7.5) trusty; urgency=medium

  * Cherry-picked patches to ignore local non-mpath devices:
    (LP: #1386637, #1468897)
    - 0001-multipath-add-checker_timeout-default-config-option.patch
    - 0002-Make-params-variable-local.patch
    - 0003-libmultipath-Fix-possible-string-overflow.patch
    - 0004-Update-hwtable-factorization.patch
    - 0005-Fixup-strip-trailing-whitespaces-for-getuid-return-v.patch
    - 0006-Remove-sysfs_attr-cache.patch
    - 0007-Move-setup_thread_attr-to-uevent.c.patch
    - 0008-Use-lists-for-uevent-processing.patch
    - 0009-Start-uevent-service-handler-from-main-thread.patch
    - 0010-libmultipath-rework-sysfs-handling.patch
    - 0011-Rework-sysfs-device-handling-in-multipathd.patch
    - 0012-Only-check-offline-status-for-SCSI-devices.patch
    - 0013-Check-for-offline-path-in-get_prio.patch
    - 0014-libmultipath-Remove-duplicate-calls-to-path_offline.patch
    - 0015-Update-dev_loss_tmo-for-no_path_retry.patch
    - 0016-Reload-map-for-device-read-only-setting-changes.patch
    - 0017-multipath-get-right-sysfs-value-for-checker_timeout.patch
    - 0018-multipath-handle-offlined-paths.patch
    - 0019-multipath-fix-scsi-timeout-code.patch
    - 0020-multipath-make-tgt_node_name-work-for-iscsi-devices.patch
    - 0021-multipath-cleanup-dev_loss_tmo-issues.patch
    - 0022-Fix-for-setting-0-to-fast_io_fail.patch
    - 0023-Fix-fast_io_fail-capping.patch
    - 0024-multipath-enable-getting-uevents-through-libudev.patch
    - 0025-Use-devpath-as-argument-for-sysfs-functions.patch
    - 0026-multipathd-remove-references-to-sysfs_device.patch
    - 0027-multipathd-use-struct-path-as-argument-for-event-pro.patch
    - 0028-Add-global-udev-reference-pointer-to-config.patch
    - 0029-Use-udev-enumeration-during-discovery.patch
    - 0030-use-struct-udev_device-during-discovery.patch
    - 0031-More-debugging-output-when-synchronizing-path-states.patch
    - 0032-Use-struct-udev_device-instead-of-sysdev.patch
    - 0033-discovery-Fixup-cciss-discovery.patch
    - 0035-Use-udev-devices-during-discovery.patch
    - 0036-Remove-all-references-to-hand-craftes-sysfs-code.patch
    - 0037-multipath-libudev-cleanup-and-bugfixes.patch
    - 0038-multipath-check-if-a-device-belongs-to-multipath.patch
    - 0039-multipath-and-wwids_file-multipath.conf-option.patch
    - 0040-multipath-Check-blacklists-as-soon-as-possible.patch
    - 0041-add-wwids-file-cleanup-options.patch
    - 0042-add-find_multipaths-option.patch
  * debian/rules: don't ship 95-multipath.rules udev rules anymore; they are
    not necessary with multipath-tools listening for udev events directly.
  * debian/multipath.udev: removed.
  * debian/multipath-tools.postinst: run multipath -W so that we get a properly
    formed wwids file on upgrade.
  * debian/initramfs/hooks: copy the wwids file in the initramfs so that it is
  ...

Read more...

Changed in multipath-tools (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for multipath-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in multipath-tools (Ubuntu Vivid):
status: New → In Progress
importance: Undecided → High
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted multipath-tools into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.4.9-3ubuntu12.15.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in multipath-tools (Ubuntu Vivid):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote : [multipath-tools/vivid] verification still needed

The fix for this bug has been awaiting testing feedback in the -proposed repository for vivid for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The verification for vivid is no longer relevant as it's EOL by Feb, 2016. Thanks.

Revision history for this message
Martin Pitt (pitti) wrote : Proposed package removed from archive

The version of multipath-tools in the proposed pocket of Vivid that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

Changed in multipath-tools (Ubuntu Vivid):
status: Fix Committed → Won't Fix
tags: removed: verification-needed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.