Trusty: multipathd SIGSEGV on path addition or removal

The attachment "Patch #1 from upstream multipath-tools git" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Thank you so much already for the debugging and identifying the patches!

To continue integrating and testing and finally SRUing the patches one needs to be able to reproduce the test. So far you just said "repeated addition and removal of iSCSI
targets".
Would you have a series of commands at hand from your testing that one can use to recreate this?

Could you check if the recent release for bug 1535898 fixed the issue for you.
It seems so similar.

Thanks for the reply.

As regards 1535989, as I understand that fix is included in 0.4.9-3ubuntu7.14 and I am able to reproduce the issue in that build.

When it comes to reproducing the issue, currently I have something that cannot easily be replicated outside my test environment. Basically my environment has a number (specifically 4) of iSCSI targets serving the same drives available on the network. This test is a loop that randomly either does an iSCSI login to all the paths available for a drive that is available, or does an iSCSI logout to all paths on the device. In turn multipathd processes the path additions or removals triggering the issue.

I think that the issue in commit cb0f7127ba90ab5e8e71fc534a0a16cdbe96a88f is triggered by multipathd processing a path addition to any iSCSI device. I don't think a SIGSEGV is reproducible from this issue alone, but the valgrind/memcheck report of free-after-use is.

I think that the issue in commit 828d2fbaab304d1ec7db2f563a59eaf2c7a516ea would be triggered by mulitpathd processing any path removal from a multipath with multiple paths and is not about iSCSI in particular. In the iSCSI context the path removal is a consequence of an iSCSI logout rather than a physical reconfiguration. I think this is the issue causing the SIGSEGV in practice.

Let me see if I can isolate a test script than can be used standalone and ideally be incorporated into any regression suite you have. It may take a couple of business days.

I've attached two test scripts that configure the local machine to serve as an iSCSI target and use then use those as targets to attach and detach.

The best way to observe the issue is to use the first version of the script with a single target and single iteration of the test loop, while running multipathd under valgrind, i.e.:

    service multipath-tools stop
    valgrind multipathd -d &
    bug1628723 1 1

When I do this, I observe the following in valgrind output:

==31667== Invalid read of size 1
==31667== at 0x4C2E4E1: __GI_strncpy (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31667== by 0x56FD3C9: sysfs_get_tgt_nodename (discovery.c:257)
==31667== by 0x56FDF98: scsi_sysfs_pathinfo (discovery.c:494)
==31667== by 0x56FEA01: sysfs_pathinfo (discovery.c:696)
==31667== by 0x56FF174: pathinfo (discovery.c:830)
==31667== by 0x56FC775: store_pathinfo (discovery.c:57)
==31667== by 0x40504E: uev_add_path (main.c:386)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)
==31667== by 0x5713B47: uevent_dispatch (uevent.c:167)
==31667== by 0x406435: uevqloop (main.c:814)
==31667== by 0x4E3F183: start_thread (pthread_create.c:312)
==31667== Address 0x7176790 is 0 bytes inside a block of size 37 free'd
==31667== at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31667== by 0x54D7A1B: ??? (in /lib/x86_64-linux-gnu/libudev.so.1.3.5)
==31667== by 0x54D7ADD: ??? (in /lib/x86_64-linux-gnu/libudev.so.1.3.5)
==31667== by 0x54D84B6: udev_device_unref (in /lib/x86_64-linux-gnu/libudev.so.1.3.5)
==31667== by 0x56FD3AA: sysfs_get_tgt_nodename (discovery.c:255)
==31667== by 0x56FDF98: scsi_sysfs_pathinfo (discovery.c:494)
==31667== by 0x56FEA01: sysfs_pathinfo (discovery.c:696)
==31667== by 0x56FF174: pathinfo (discovery.c:830)
==31667== by 0x56FC775: store_pathinfo (discovery.c:57)
==31667== by 0x40504E: uev_add_path (main.c:386)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)

Which is due to the issue resolved in commit cb0f7127ba90ab5e8e71fc534a0a16cdbe96a88f,
and later:

==31667== Invalid read of size 8
==31667== at 0x56FC388: find_path_by_dev (structs.c:322)
==31667== by 0x404FBF: uev_add_path (main.c:376)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)
==31667== by 0x5713B47: uevent_dispatch (uevent.c:167)
==31667== by 0x406435: uevqloop (main.c:814)
==31667== by 0x4E3F183: start_thread (pthread_create.c:312)
==31667== by 0x5A2B37C: clone (clone.S:111)
==31667== Address 0x72508a0 is 0 bytes inside a block of size 40 free'd
==31667== at 0x4C2CE8E: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.
so)
==31667== by 0x56F4162: vector_del_slot (vector.c:99)
==31667== by 0x571EF72: verify_paths (structs_vec.c:477)
==31667== by 0x405397: ev_add_path (main.c:446)
==31667== by 0x4050C7: uev_add_path (main.c:398)
==31667== by 0x4062EC: uev_trigger (main.c:777)
==31667== by 0x5713938: service_uevq (uevent.c:118)
==31667== by 0x5713B47: uevent_di...

Both Xenial & Yakkety have 0.5.0-based multipath-tools, and the referred to fixes are already present.

Hello, please test a Trusty build at:

https://launchpad.net/~nacc/+archive/ubuntu/lp1628723

submitted just now, which should include the fixes requested! Please test the PPA and report back, thanks!

So far so good with the PPA: it is clean according to valgrind in my test environment, and has not crashed so far. I will continue to run it overnight to be sure and report again tomorrow.

Thanks!

The PPA has now run for a full day in my test environment without issue. I would regard it as a success.

Hello Christopher, or anyone else affected,

Accepted multipath-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.4.9-3ubuntu7.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Tested multipath-tools 0.4.9-3ubuntu7.15 (arch amd64) from trusty-proposed successfully in my test environment; no crashes and valgrind clean. (kpartx 0.4.9-3ubuntu7.15 also installed.)

Changed tag from verification-needed to verification-done.

This bug was fixed in the package multipath-tools - 0.4.9-3ubuntu7.15

---------------
multipath-tools (0.4.9-3ubuntu7.15) trusty; urgency=medium

  * d/p/0047-Add-existing-multipath-devices-to-wwids-file-on.patch:
    Fix multipathd which does not update /etc/multipath/wwids file
    when reconfigure is invoked. (LP: #1621835)

  [ Dragan Stancevic ]
  * d/p/0048-multipathd-delay-free-pathvec.patch :
    Fix SEGV on multipathd shutdown (LP: #1616213)

  [ Nishanth Aravamudan ]
  * debian/patches/fix_use_after_free.patch: Fix use-after-free bugs.
    Thanks to Christof Schmitt <email address hidden> and
    Benjamin Marzinski <email address hidden>. Closes LP: #1628723.

 -- Louis Bouchard <email address hidden> Mon, 12 Sep 2016 10:43:19 +0200

The verification of the Stable Release Update for multipath-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.