Raid10 array not assembled at boot, all disks are marked as spare

Bug #316670 reported by Jonathan Ernst
62
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Linux
Invalid
Undecided
Unassigned
Release Notes for Ubuntu
Fix Released
Undecided
Unassigned
mdadm (Ubuntu)
Fix Released
High
Dustin Kirkland 
Jaunty
Fix Released
High
Dustin Kirkland 

Bug Description

I have just installed jaunty with a separate boot partition and / on a raid10 array (/dev/md1).

However when I boot, I'm dropped in a initramfs shell and I have to manually assemble the array as all disks are marked as spare. There are also segfaults in dmesg. mdadm[1370]: segfault at 0 ip 000000000040839b sp 00007fffb2af2620 error 4 in mdadm[400000+2a000]

The segfault is easily reproducible on jaunty without needing to do a full install on a raid10 setup; steps to do so:

1) create 4 empty loopback disks:

  for i in $(seq 1 4) ; do
      dd if=/dev/zero of=disk${i} bs=1024 count=512000
      sudo losetup /dev/loop${i} disk${i}
  done

2) create raid10 array

  sudo mdadm -C /dev/md0 -l raid10 -n 4 /dev/loop{1,2,3,4}

3) stop the array:

  sudo mdadm --stop /dev/md0

4) incrementally assemble the first loop device; this is how udev invokes mdadm in /lib/udev/rules.d/85-mdadm.rules.

  sudo mdadm --incremental /dev/loop1

et voila, segv.

Revision history for this message
Jonathan Ernst (jonathan.ernst) wrote :
Revision history for this message
Jack Wasey (jackwasey) wrote :

I have exactly the same problem. The only quirk is that it is a three disk raid 10. The same disks (750, 500 and 250G) also hold a raid 0 array which works absolutely fine.

The /dev/mdadm/mdadm.conf file had an incorrect UUID after I'd had to rebuild the arrays. I have fixed this using mdadm --detail /dev/md0, to get the correct UUID, but others beware.

N.b. re-assembling, and re-creating the array (with assume clean) left all my data intact, and I could mount the ext4 LVM volumes.

here's a snip of my dmesg
[ 16.345255] md: bind<sda1>
[ 16.346671] mdadm[2286]: segfault at 0 ip 000000000040839b sp 00007fff6106a5d0 error 4 in mdadm[400000+2a000]
[ 16.384062] md: bind<sdb1>
[ 16.391313] mdadm[2305]: segfault at 0 ip 000000000040839b sp 00007fffa55e70d0 error 4 in mdadm[400000+2a000]
[ 16.432732] md: bind<sdc1>
[ 16.440382] mdadm[2306]: segfault at 0 ip 000000000040839b sp 00007fff1feb49a0 error 4 in mdadm[400000+2a000]

let me know how to help debug this.

Revision history for this message
Jack Wasey (jackwasey) wrote :

I see an error just before the prompt:

exec of program '/sbin/' failed

I also noticed a /var/lock does not exist error, but it seemed to be there by the time the prompt came up (although var wasn't mounted due to the prior errors).

I have also noticed the following:

Before reboot:
md0 = sd[abc]1

After reboot:
md0 is gone
md_d0 is there with one or two of the original partitions.

To recover, I mdadm --stop the md_d0 bogus array, and recreate the original array, assume-clean, and it seems works fine. I also have to reactive the logical volume with lvchange -ay /dev/raid10/tank

Revision history for this message
Jack Wasey (jackwasey) wrote :

I have just noticed that <code>blkid</code> does not mention the UUID of my dodgy /dev/md3 device. Three other functioning /dev/md devices are shown fine.

<code>vol_id</code> produces normal output, stating the UUID, but this doesn't feature in the blkid output.

I notice also that <code>mdadm --detail /dev/md3</code> shows the the array is local to this host, whereas the other arrays do not have this flag. I have been forced to recreate the array every reboot, so this is perhaps not significant. N.b. I am only booting into one system, not trying to mount the arrays from different OSes.

Revision history for this message
Jack Wasey (jackwasey) wrote :

the same ?kernel bug exists with 2.6.29-rc8 which I got from the mainline ppa repository.

at the moment, this seems to be a raid 10 assembly bug. possibly only for three disk raid10, but raid 0 and raid 10 work for me in this setup.

Revision history for this message
Jonathan Ernst (jonathan.ernst) wrote :

four disk raid10 is affected as well

Revision history for this message
Jack Wasey (jackwasey) wrote :

live Fedora 11 alpha CD: no problems with my raid 10 (although a different array is marked inactive...)

[liveuser@localhost ~]$ uname -a
Linux localhost.localdomain 2.6.29-0.66.rc3.fc11.x86_64 #1 SMP Thu Jan 29 14:44:32 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
[liveuser@localhost ~]$
[liveuser@localhost ~]$ more /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md_d2 : inactive sda8[0](S)
      70043264 blocks

md_d1 : active raid1 sda7[0] sdb7[1]
      68356416 blocks [2/2] [UU]

md_d4 : active raid0 sdc2[0] sde2[1]
      488287424 blocks 64k chunks

md_d3 : active raid10 sdc1[0] sde1[2] sdd1[1]
      366293824 blocks 64K chunks 2 near-copies [3/3] [UUU]

Revision history for this message
Z (zwahri) wrote :

I have this problem as well ( 4 disks + hot spare with one raid10 /lvm) and one raid1 (/) array ).

Workaround:

Setting BOOT_DEGRADED=true in /etc/initramfs-tools/conf.d/mdadm works like a charm.

You need to do an
#sudo update-initramfs -u
afterwards.

Drawback: booting takes longer than normal, since mdadm waits 30s to assemble the (presumably) broken array and I'm not sure what happens if the arrays are *really* broken

Revision history for this message
Jack Wasey (jackwasey) wrote :

with jaunty kernel 2.6.28-10, the raid is no longer assembled at all. The array can be created with mdadm --create --assume-clean, as before, apparently without any problems. this sounds okay if nothing has been written to the array yet, and since it isn't assembled, this should be fine. Unless the problem is in shutting down the array, but no segfault there, so probably not.

No joy with 2.6.28-11, although there are three md raid10 fixes, relating to recovery which I hoped might somehow impinge:
from upstream short changelog:
md: avoid races when stopping resync.
  * md/raid10: Don't call bitmap_cond_end_sync when we are doing recovery.
  * md/raid10: Don't skip more than 1 bitmap-chunk at a time during
    recovery.

this seems like a serious bug to me. if you are suffering, please add your comments; if it is just me and the reporter, please can someone help me work out why? N.b. a fedora kernel with ubuntu starts the array correctly, so this is a kernel or mdadm bug.

Revision history for this message
Jack Wasey (jackwasey) wrote :

exactly the same bug on a completely different set-up. dmesg attached. n.b. SEGFAULTS in the middle.

this is a serious bug! raid10 is an installation option, not an obscure corner case, and it results in a non-booting system. no idea if this bug is also corrupting data silently.

Changed in mdadm (Ubuntu):
status: New → Confirmed
tags: added: regression-potential
Revision history for this message
Steve Beattie (sbeattie) wrote :

There's no indication this is a(n upstream) kernel bug, marking that task invalid.

Changed in mdadm:
assignee: nobody → canonical-server
importance: Undecided → Medium
Changed in linux:
status: New → Invalid
Revision history for this message
Steve Beattie (sbeattie) wrote :

Riasing the priority and nominating for jaunty.

description: updated
description: updated
Changed in mdadm:
importance: Medium → High
Revision history for this message
Leszek 'skolima' Ciesielski (skolima) wrote :

I am experiencing this bug with both two-disk raid10 arrays I use, I have to boot with --boot-degraded ever since I upgraded to Jaunty preview.

Revision history for this message
Steve Beattie (sbeattie) wrote :

When I recreated the bug, I created the duplicate bug report bug 347747 with the apport crash information. The retracer has gone through and provided the stacktrace, which I'm attaching here. The rest of the apport crash information can be seen in my duplicate report.

Revision history for this message
LCID Fire (lcid-fire) wrote :

Same Jaunty problem here.

Revision history for this message
Steve Beattie (sbeattie) wrote :

I do not get the mdadm segfault with any of the other supported types (raid 0, 1, 4, 5, and 6, linear, multipath) so I believe only raid10 is affected by this.

Revision history for this message
LCID Fire (lcid-fire) wrote :

If it's not going to be fixed for jaunty - shouldn't the default be BOOT_DEGRADED=TRUE? Otherwise all people running Raid 10 will get into some (fixable but rather annoying) trouble.

Revision history for this message
Jack Wasey (jackwasey) wrote :

bad idea if there really is a degraded array...

needs fix or disable raid 10 (which is drastic). easier to fix than audit the many possible causes of data loss this might bring about if we left the bug in there. I don't understand why fedora kernels boot Ubuntu without this problem. Maybe someone clever could diff between the kernel i mention above and the ubuntu one: surely a clue there...

Revision history for this message
LCID Fire (lcid-fire) wrote :

Disable raid 10 is not an option. But getting it fixed in a month is IMO questionable, too.
Alternatively perhaps it would be enough to display a warning message when updating to jaunty (and a Raid10 is present) and place a remark in the manpage (let's face it - almost no one is reading release notes).

Revision history for this message
Steve Langasek (vorlon) wrote :

Documented in the Beta technical overview:

  The mdadm package in Ubuntu 9.04 Beta will fail to assemble RAID10 arrays on boot. Other types of RAID are not affected; investigation of this issue is ongoing.

This is a high-priority issue which should be fixed for release, so I don't expect we'll need to document anything in the release notes.

Changed in ubuntu-release-notes:
status: New → Fix Released
Revision history for this message
Steve Beattie (sbeattie) wrote :

With Dustin Kirkland's help, I tested out packages based on debian's 2.6.7.2 and 2.6.8 packages; 2.6.7.2 still causes mdadm to segfault for me when incrementally assembling a raid10 array, but 2.6.8 does not. We'll be digging in to find out what portion of 2.6.8 we need to cherry pick to fix this. Thanks for your patience!

Changed in mdadm:
status: Confirmed → In Progress
Revision history for this message
Steve Langasek (vorlon) wrote :

Looking at the backtrace + the upstream diff, I think the attached patch is the bit that fixes it.

Revision history for this message
Dustin Kirkland  (kirkland) wrote : Re: [Canonical-server] [Bug 316670] Re: Raid10 array not assembled at boot, all disks are marked as spare

Thanks, slangasek.

sbeattie et al.-

I have prepared a new package for you to test, including *only* this
cherry picked patch. Would you please try out:
  * http://people.ubuntu.com/~kirkland/316670/
and report back?

I'll get this committed if you can confirm that this solves the issue...

:-Dustin

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

I'm attaching the debdiff here, for the sake of completeness (and since it only exists in my ramfs)...

:-Dustin

Changed in mdadm (Ubuntu Jaunty):
assignee: canonical-server → kirkland
Revision history for this message
Steve Beattie (sbeattie) wrote :

I can confirm Dustin's package fixes the segfault issue and lets a system with / on raid10 boot. I've also briefly verified that the other raid configurations still work properly. Thanks!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mdadm - 2.6.7.1-1ubuntu8

---------------
mdadm (2.6.7.1-1ubuntu8) jaunty; urgency=low

  Fix for LP: #316670
  * Incremental.c: cherry-pick upstream patch, "Fix NULL pointer oops"
    http://git.debian.org/?p=pkg-mdadm/mdadm.git;a=commit;h=43aaf431f66270080368d4b33378bd3dc0fa1c96

  Fix for LP: #349971 (Thanks, Steve Beattie!)
  * debian/source_mdadm.py: apport hooks for improved bug reporting

 -- Dustin Kirkland <email address hidden> Fri, 27 Mar 2009 18:52:51 -0500

Changed in mdadm:
status: In Progress → Fix Released
Revision history for this message
Jack Wasey (jackwasey) wrote :

great - works for me, BUT i had to update the UUID (mdadm --detail, not vol_id /dev/md...) in /etc/mdadm/mdadm.conf for it to assemble on boot, because I had had to recreate it each time I booted. This won't of course affect people who go straight to the fixed mdadm from intrepid.

Revision history for this message
Jack Wasey (jackwasey) wrote :

i should have said: works for me with three disk raid10.

Why isn't the very simple regression test described done automatically?

Revision history for this message
lunomad (damon-metapaso) wrote :

Just adding my experience here, not sure if it is related.

I upgraded last night to the Jaunty Release Candidate.

Gdebi indicates that my installed mdadm is the same version as the AMD64 version from http://people.ubuntu.com/~kirkland/316670/

My root is on a regular sata disk, and I mount sdb1 and sdc1 as a Raid 1 mdadm+lvm2 (as /dev/md0 and /lvm/lvm-raid/homelv) on /home

The default upgrade did not find my md0, and instead puts sdc1 into a false md_d0.

Every time I boot, I have to drop into a console, execute mdadm --assemble --scan and manually mount the /dev/lvm-raid/homelv . These steps work, except my md0 is degraded to just one disk (sdb1)

I checked /proc/mdstat before the upgrade and everything was fine, so I'm wondering if this is the same bug or a different one?

What's the best route to fix this?

Revision history for this message
Steve Beattie (sbeattie) wrote : Re: [Bug 316670] Re: Raid10 array not assembled at boot, all disks are marked as spare

On Fri, Apr 17, 2009 at 12:59:18PM -0000, lunomad wrote:
> Just adding my experience here, not sure if it is related.
>
> I upgraded last night to the Jaunty Release Candidate.
>
> Gdebi indicates that my installed mdadm is the same version as the AMD64
> version from http://people.ubuntu.com/~kirkland/316670/
>
> My root is on a regular sata disk, and I mount sdb1 and sdc1 as a Raid 1
> mdadm+lvm2 (as /dev/md0 and /lvm/lvm-raid/homelv) on /home
>
> The default upgrade did not find my md0, and instead puts sdc1 into a
> false md_d0.
>
> Every time I boot, I have to drop into a console, execute mdadm
> --assemble --scan and manually mount the /dev/lvm-raid/homelv . These
> steps work, except my md0 is degraded to just one disk (sdb1)

Are you able to hot-add back /dev/sdc1 to the array (e.g. mdadm /dev/md_d0
--add /dev/sdc1)? If so, after successfully resync'ing, does the following
boot still drop you to the busybox shell? Also, before you're dropped
to the shell, are you being asked to boot in degraded mode (the timeout
while waiting for the root filesystem to come up is long, but the boot
degraded question timeout is quite short)?

> I checked /proc/mdstat before the upgrade and everything was fine, so
> I'm wondering if this is the same bug or a different one?

This is likely a separate bug, please open a new one (though feel
free to report the bug number here). Thanks!

--
Steve Beattie
<email address hidden>
http://NxNW.org/~steve/

Revision history for this message
lunomad (damon-metapaso) wrote :

During boot I was prompted with a warning, no valid partitions found on /dev/sdc, hit ctrl-D to continue. Because this isn't a root partition, I was able to continue booting with ctrl-D, which took me all the way to the gdm login. I then dropped to a shell with ctrl-alt-F1. There were no questions about degraded mode booting.

I fixed my problem with the following steps:

1. Stop the false md_d0 array
$ sudo mdadm --stop /dev/md_d0

2. Assemble the degraded md0 (only sdb was found) and mount it
$ sudo mdadm --assemble --scan
$ sudo mount /dev/lvm-raid/homelv /home

3. use mkconf to find the configuration option
$ sudo /usr/share/mdadm/mkconf

4. gedit /etc/mdadm/mdadm.conf to include the output from mkconf AND to use only sdb
DEVICE /dev/sdb*
# definitions of existing MD arrays
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=6a291d6f:27049e9f:b95e1135:52246c3f

5. Reboot --> ubuntu found and mounted my array just fine (mount info for LVM is my fstab)

6. Re-add sdc to md0
$ sudo mdadm /dev/md0 --re-add /dev/sdc1

7. Watch rebuild!
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[0] sdc1[2]
      976759936 blocks [2/1] [U_]
      [==>..................] recovery = 14.2% (139109312/976759936) finish=381.7min speed=36570K/sec

unused devices: <none>

I'm not sure why mdadm wanted to build the (incorrect) md_d0 and not the (correct) md0.

Revision history for this message
Simon Vidmar (vidmar-simon) wrote :

I had this issue on a RAID 10 with f2 layout with 3 disks. After upgrade from intrepid to jaunty (production) RAID(s) 1 were fine, but RAID 10 was inactive with only sdc(4) showing.

Fix was rather simple, only:
$ sudo mdadm --stop /dev/md_d2
$ sudo mdadm --assemble --scan

Result:
   /dev/md2: File exists
   mdadm: /dev/md/2 has been started with 3 drives.

and then:
$ sudo mount -a

and array was back...

Revision history for this message
Simon Vidmar (vidmar-simon) wrote :

In my previous post I forgot to write that previous procedure brought array back, but only until next reboot. To fix this I did the following:
$ sudo mdadm --detail /dev/md2

to display array characteristics
with that information I have updated /etc/mdadm/mdadm.conf
because md2 ARRAY was deleted from there.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.