shutdown segv due to race w/ free_waiter threads

Bug #713237 reported by dann frazier
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
multipath-tools (Ubuntu)
Fix Released
Low
Serge Hallyn
Lucid
Won't Fix
Low
Unassigned
Maverick
Won't Fix
Undecided
Unassigned

Bug Description

Binary package hint: multipath-tools

==========================================
SRU Justification:
1. Impact: segv on multipathd shutdown
2. How bug was addressed: a patch was cherrypicked from upstream which fixes this
3. Minimal patch: see patch in attached bzr tree
4. TEST CASE: stop multipathd.
5. Regression potential: none, this patch was taken from upstream.
==========================================

I received a coredump from a customer (running a lucid-based system).
Analyzing the core from multipathd shows that we are crashing in waiter.c::free_waiter().

void free_waiter (void *data)
{
 struct event_thread *wp = (struct event_thread *)data;

 /*
  * indicate in mpp that the wp is already freed storage
  */
 lock(wp->vecs->lock);

 if (wp->mpp)
  /*
   * be careful, mpp may already be freed -- null if so
   */
  wp->mpp->waiter = NULL;
 else
  condlog(3, "free_waiter, mpp freed before wp=%p,", wp);

 unlock(wp->vecs->lock); <<< Crashing here

 if (wp->dmt)
  dm_task_destroy(wp->dmt);

 FREE(wp);
}

The lock is NULL:

(gdb) print wp->vecs->lock
$3 = (pthread_mutex_t *) 0x0

This appears to be the upstream issue fixed here:
http://git.kernel.org/gitweb.cgi?p=linux/storage/multipath-tools/.git;a=commitdiff;h=a403f57b991f3be7c7ea6250d95fad1554c9d6ca

This fix is currently missing from all versions of multipath-tools in lucid->natty.

(The original customer issue was filed in the private bug #712875.)

Tags: testcase

Related branches

Changed in multipath-tools (Ubuntu):
status: New → Confirmed
assignee: nobody → Serge Hallyn (serge-hallyn)
importance: Undecided → Low
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks for the patch, Dann. I would ask you to hit 'propose for merge' from your lp tree, but I think it would be nice to have DEP-3 tags in the patch to help out future maintainers. So I'll add those to my own version and propose that for merge.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Tree proposed for merging. Reminder this bug should be SRU'd to lucid and maverick after the fix is applied to natty.

Changed in multipath-tools (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
dann frazier (dannf) wrote : Re: [Bug 713237] Re: shutdown segv due to race w/ free_waiter threads

On Wed, Feb 09, 2011 at 09:37:17PM -0000, Serge Hallyn wrote:
> Thanks for the patch, Dann. I would ask you to hit 'propose for merge'
> from your lp tree, but I think it would be nice to have DEP-3 tags in
> the patch to help out future maintainers. So I'll add those to my own
> version and propose that for merge.

Sounds good, thanks Serge.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package multipath-tools - 0.4.8-14ubuntu9

---------------
multipath-tools (0.4.8-14ubuntu9) natty; urgency=low

  * Fix segv caused by race condition with free_waiter threads during
    shutdown, using backport from upstream. (LP: #713237)
 -- dann frazier <email address hidden> Thu, 03 Feb 2011 16:37:10 -0700

Changed in multipath-tools (Ubuntu):
status: In Progress → Fix Released
description: updated
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Please test proposed package

Accepted multipath-tools into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in multipath-tools (Ubuntu Lucid):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

The upload to lucid-proposed was deleted, as the version was higher than that of the dev release, oneiric. If you happened to download that version while it was briefly available, you should manually downgrade back to 0.4.8-14ubuntu4, otherwise you run the risk of not updating this package on an upgrade to maverick.

Changed in multipath-tools (Ubuntu Lucid):
importance: Undecided → Low
status: Fix Committed → Triaged
tags: removed: verification-needed
tags: added: testcase
Revision history for this message
Rolf Leggewie (r0lf) wrote :

Maverick has long since stopped to receive any updates. Marking the Maverick task for this ticket as "Won't Fix".

Changed in multipath-tools (Ubuntu Maverick):
status: New → Won't Fix
Revision history for this message
Rolf Leggewie (r0lf) wrote :

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".

Changed in multipath-tools (Ubuntu Lucid):
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.