Mdadm: external metadata arrays and containers are removed incorrectly

Bug #1811795 reported by Mariusz Tkaczyk
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
mdadm (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned
Cosmic
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

 * The response to device fails so far was to remove underlying
   containers. That failed if the device itself was not stopped
   before doing so causing issues.

 * Fix by backporting upstream fix

[Test Case]

 * This needs a Intel(R) Matrix Storage Hardware which limits the
   systems this can be tested.
   1. Create IMSM Container:
       mdadm -CR imsm0 -e imsm -n 2 /dev/sd[bc]
   2. Create IMSM level 0 Array:
       mdadm -CR vol0 -l 0 -n 2 /dev/sd[bc]
   3. Wait for resync to complete
      mdadm --wait /dev/md/vol0
   4. Read /proc/mdstat and check /dev/md/ directory
      cat /proc/mdstat
      ls /dev/md/
   5. Fail one disk incrementaly (simulate device disappearing
      from system)
      mdadm -If sdc
   6. Check mdstat file and /dev/md/
      cat /proc/mdstat
      ls /dev/md/
   Result:
      After disk fail [5] raid 0 array should disappear from mdstat,
      link /dev/md/imsm0 should exist.

 * these should allow someone who is not familiar with the affected
   package to reproduce the bug and verify that the updated package fixes
   the problem.

[Regression Potential]

 * The patch changes the behavior on removing devices. So in that use-case
   lies the potential to regress somebody.
   It adds stopping the device before removing the containers associated
   to it (that is the fix). OTOH if there are cases where the kernel
   signals a device fail it would formerly not have removed it (the bug)
   but there are (very low) chances that somebody relies on that behavior.
   Since this avoids I/O errors on partially removed device elements I'd
   expect taking the change makes it better overall.

[Other Info]

 * n/a

---

Steps to reproduce:

1. Create IMSM container on 2 SATA disks.

2. Create R0 IMSM volume

3. Wait for resync to complete.

4. Fail one of disks

Expected result:
R0 Array should disappear form /proc/mdstat.

Actual result:
R0 array is visible in mdstat, link to container in /dev/md/ is removed.

Related branches

Revision history for this message
Mariusz Tkaczyk (mtkaczyk) wrote :

Fixed in upstream:
https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=cb8f5371352f6c16af5aab8a40861e13aa50fc2b

Fix is included in mdadm 4.1, please update mdadm package.

Thanks,
Mariusz

information type: Public → Private
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in mdadm (Ubuntu):
status: New → Confirmed
Revision history for this message
Pawel Baldysiak (pbaldysi) wrote :

Hi,
Are there any update on this one?

Thanks
Pawel

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The fix is in 4.1 which is released into Ubuntu 19.04 already
There are 4.1~rc1 based versions in 18.04 and 18.10 which do not contain the fix.

Which release are we talking about being affected - Bionic?

Changed in mdadm (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Mariusz Tkaczyk (mtkaczyk) wrote :

Yes, Bionic

Thanks, Mariusz

information type: Private → Public
Revision history for this message
Pawel Baldysiak (pbaldysi) wrote :

Hi,
It is fixed in 19.04 (mdadm there is 4.1).
Any chances to include the fix in 18.04 LTS?

Thanks
Pawel

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi,
I need your help in testing this.
I have created a PPA [1] with the fixes for this issue, but I lack a IMSM HW.

I'd ask the people affected/subscribed to test the PPA.
While doing so please
a) ensure the test steps you have trigger the bug without the PPA
b) ensure the PPA fixes the issue
c) provide the exact steps (commands) that you used to test this (we want to add this to the SRU description)

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1811228-1811795-mdadm-imsm

description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Incomplete waiting for PAP verification and better test steps

Changed in mdadm (Ubuntu Cosmic):
status: New → Incomplete
Changed in mdadm (Ubuntu Bionic):
status: New → Incomplete
Revision history for this message
Mariusz Tkaczyk (mtkaczyk) wrote :

Hi,
Here test steps:
1. Create IMSM Container:
    mdadm -CR imsm0 -e imsm -n 2 /dev/sd[bc]
2. Create IMSM level 0 Array:
    mdadm -CR vol0 -l 0 -n 2 /dev/sd[bc]
3. Wait for resync to complete
   mdadm --wait /dev/md/vol0
4. Read /proc/mdstat and check /dev/md/ directory
   cat /proc/mdstat
   ls /dev/md/
5. Fail one disk incrementaly (simulate device disappearing from system)
   mdadm -If sdc
6. Check mdstat file and /dev/md/
   cat /proc/mdstat
   ls /dev/md/
Result:
   After disk fail [5] raid 0 array should disappear from mdstat, link /dev/md/imsm0 should exist.

The issue is fixed in PPA.

Thanks,
Mariusz

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks for the real test steps and checking them Mariusz.

This will now go to the review by the SRU Team.
Once accepted they will ask for a verification in -proposed (of both affected releases).
Once that is complete this can be released to all users.

description: updated
Changed in mdadm (Ubuntu Bionic):
status: Incomplete → Fix Committed
Changed in mdadm (Ubuntu Cosmic):
status: Incomplete → Fix Committed
Changed in mdadm (Ubuntu Bionic):
status: Fix Committed → In Progress
Changed in mdadm (Ubuntu Cosmic):
status: Fix Committed → In Progress
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Mariusz, or anyone else affected,

Accepted mdadm into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mdadm/4.1~rc1-4ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in mdadm (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Mariusz, or anyone else affected,

Accepted mdadm into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mdadm/4.1~rc1-3~ubuntu18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in mdadm (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Mariusz Tkaczyk (mtkaczyk) wrote :

Hi,
the issue is fixed in bionic-proposed.

Thanks,
Mariusz

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks Mariusz!
Since this verification needs special HW which at least I currently have none of - do you see a chance to also do the verification for cosmic-proposed? Lacking that it can't be really released for Bionic as it would then become an upgrade-regression.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Krzysztof Smoliński (ksmolinski) wrote :

Hi,

I have tested issue in cosmic-proposed. It is fixed.

Regards,
Krzysztof

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mdadm - 4.1~rc1-4ubuntu1.1

---------------
mdadm (4.1~rc1-4ubuntu1.1) cosmic; urgency=medium

  * d/p/lp-1811228-imsm-Block-volume-creation-with-empty-name.patch: avoid
    creating whitespaced or empty names (LP: #1811228)
  * d/p/lp-1811795-remove-external-arrays-and-devices.patch: fix removal of
    failing disks with external metadata (LP: #1811795)

 -- Christian Ehrhardt <email address hidden> Wed, 10 Apr 2019 16:14:17 +0200

Changed in mdadm (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mdadm - 4.1~rc1-3~ubuntu18.04.2

---------------
mdadm (4.1~rc1-3~ubuntu18.04.2) bionic; urgency=medium

  * d/p/lp-1811228-imsm-Block-volume-creation-with-empty-name.patch: avoid
    creating whitespaced or empty names (LP: #1811228)
  * d/p/lp-1811795-remove-external-arrays-and-devices.patch: fix removal of
    failing disks with external metadata (LP: #1811795)

 -- Christian Ehrhardt <email address hidden> Wed, 10 Apr 2019 16:14:17 +0200

Changed in mdadm (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for mdadm has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.