kdump functionality not working as expected when /boot is a separate partition

The following patch fixes the situation where the kdump data collection fails when the /boot is a separate partition, which is the case when the standard LVM installation is selected.

The patch has been built against the kexec-tools 1:2.0.2-1ubuntu3 (11.10 Oneiric) but also applies to the version found on Lucid. This was done to take into accound the next upcoming version with will be LTS.

The attachment "kdump.initramfs_boot_partition-LP828731.patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

FYI

The patch has been tested on i386 and x86_64 in conjunction with the patch provided for LP#785425 which relies on this patch to work.

Both are confirmed to work and should be adressed together as they touch the same script.

This also affects 10.10 Maverick.

Also, the problem affects other filesystems that are separate partitions, specifically, /var

https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/600055

@James

This affects all distro since Lucid which use a separate /boot partition, which is the default configuration for systems installed with LVM.

Now taking into account separate /var, /usr is a bit different, as this is not a standard way of installing Ubuntu, even though it is totally possible. Maybe adding the logic to take /usr and/or /var into account should be added in the script, but this should be something decided by the foundation team as it goes beyond bug fixing.

@louis sure, but shouldn't these bugs be tagged against various releases so that it's possible to search for a bug and release, and find out whether it's fixed in each release?

Here is the new version of the patch, without any reference to ext2

@louis:

Just wanted to make sure this new patch was verified and tested as well. Let me know if this has been done.

--chris

@chris

Well, no it wasn't apparently :-( Testing with a newly created VM with /boot as XFS fails. I'll work on an update of the patch right away.

@chris

Here is a new version of the patch, which takes into account any type of FS type for /boot. It extracts the type from the /etc/fstab which is then used with the mount command.

I have tested it with /boot as :
  - ext2 (standard)
  - xfs
  - jfs

All those three did generate a vmcore file.

Louis,

Attached is an untested patch for kexec-tools that should ensure that all of /boot, /var, and /usr are mounted correctly in the target system. Could you test that this patch works for you? If so, I'll push this to precise.

It should solve a number of problems related to trying to mount arbitrary filesystems (e.g., /boot) from the initramfs.

Steve,

I'm preparing a test environment and will let you know how the patch behaves as soon as I can test it.

Steve,

I am currently testing a slightly modified version of your patch which seems to work to some extent. Multiple partitions (i.e. /boot, /usr and /var) do work, depending on the type of file system used. A mix of ext2, ext4, xfs work fine but I currently have a failure when /var is jfs : for some reason, but initramfs environment refuses to mount /var as jfs, even if the jfs kernel module is present & loaded.

I will carry on with my test when I return for the xmas vacation in early jan. I'll keep you posted with my findings. Let me know if you want me to post the current version of the patch here.

Hi Louis,

On Thu, Dec 22, 2011 at 10:56:00AM -0000, Louis Bouchard wrote:

> Let me know if you want me to post the current version of the patch here.

As I wasn't expecting modifications to be needed, yes, please do :)

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Hi Steve,

One addition is to check fstab for LVM usage. If used, 'mount --bind' the /dev to gain access to the /dev/mapper entries.

The second thing I found out is that makedumpfile requires /tmp to be writable, so it is not possible to restrict / to be read-only.

Here is the modified patch.

I have done some more tests. I still have the issue when /var is a JFS file system. When switching /var to ext4, the patch works well. For some reason, busybox has problems mounting JFS file systems. Right now, here is what I exhave tested :
/boot XFS
/ ext4
/usr XFS
/var JFS (nok), ext4 (ok)

Merry christmas & a very happy new year.

This bug was fixed in the package kexec-tools - 1:2.0.2-3ubuntu2

---------------
kexec-tools (1:2.0.2-3ubuntu2) precise; urgency=low

  * debian/kdump.initramfs: call /usr/bin/makedumpfile via a chroot command,
    so that if makedumpfile is statically linked, we get proper library
    resolution. Thanks to Louis Bouchard <email address hidden> for
    the patch. LP: #785425.
  * debian/kdump.initramfs: handle the possibility that /usr, /boot, or
    /var is on a separate filesystem and needs to be manually mounted before
    calling makedumpfile. LP: #828731.
  * Depend on makedumpfile, without which the initramfs script doesn't work.
 -- Steve Langasek <email address hidden> Mon, 02 Jan 2012 22:43:33 -0800

@steve, @louis:

I assume this fix needs to be applied to lucid/natty/oneiric as well? What would be the next step in that SRU process?
Thanks,
--chris

On Jan 9, 2012, at 8:15 AM, Chris J Arges wrote:

> I assume this fix needs to be applied to lucid/natty/oneiric as well? What would be the next step in that SRU process?

Maverick, too.

This patch backports the fix to the latest lucid release. This applies cleanly and I built an amd64 version, but did not test.

This patch applies cleanly to maverick.

This patch applies cleanly to natty.

This patch applies cleanly to oneiric.

@everybody

FYI, I have tested kexec-tools from Precise on the following :

 - Lucid
 - Maverick
 - Natty
 - Oneiric

kdump works as expected on all of those with the kexec-tool package from precise.

I loaded this version of kexec-tools, and triggered a crash, but /boot and /var is not mounted..

@james

Most probably, the initramfs did not get rebuilt after updating the package. Most of the time when I test, I issue :

initramfs -u
update-grub

then I reboot. You might want to rebuild the initramfs just in case and test again.

Ahh, that's probably it. I thought I saw the installer rebuild it. I'll report back when I have a chance to test it on some other machines.

Is anyone doing any work to support streaming dumpfiles to a remote location?

The debdiffs generally look good to me, but I don't have sufficient permission to upload this package. Since you're also closing bug 785425 in the debdiffs, I'd also make sure the SRU justification and the ubuntu-sru, ubuntu-sponsors teams are subscribed and the releases properly nominated/targetted.

Ok I believe I've targeted things correctly. Any other actions to progress with the SRU?

I'm going to sponsor these patches into *-proposed.

"Fix Committed"/"verification-needed" are auto-added once the package is accepted into -proposed.

Hello Louis, or anyone else affected,

Accepted kexec-tools into oneiric-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Unsubscribing ubuntu-sponsors as all of the SRUs have been sponsored, and are just awaiting ubuntu-sru review.

Hello Louis, or anyone else affected,

Accepted kexec-tools into lucid-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Hello Martin, Barry,

I have tested the kexec-tools in oneiric-proposed and can confirm that the new package does fix the issue (tested on 11.10 x86_64 only).

Now waiting for the Lucid & Natty packages to test as well.

Louis, it's already in lucid-proposed.

lucid/natty were apparently supposed to be there but weren't. Accepted now.

@steve @martin

I have tested all releases and found an issue with the Lucid in -proposed.

The lucid-proposed-lp828731.debdiff is different from the others and has a couple of extra lines that break the fix which works for other releases. So the version in lucid-proposed should be removed as it _DOES NOT_ fix the problem.

For the other releases (Maverick, Natty, Oneiric and Precise), the patch has been tested and works correctly.

> The lucid-proposed-lp828731.debdiff is different from the others and has
> a couple of extra lines that break the fix which works for other
> releases. So the version in lucid-proposed should be removed as it
> _DOES NOT_ fix the problem.

Right, I've removed this now from lucid-proposed to ensure it doesn't get
promoted by accident.

On Mar 01, 2012, at 06:43 PM, Steve Langasek wrote:

>> The lucid-proposed-lp828731.debdiff is different from the others and has
>> a couple of extra lines that break the fix which works for other
>> releases. So the version in lucid-proposed should be removed as it
>> _DOES NOT_ fix the problem.
>
>Right, I've removed this now from lucid-proposed to ensure it doesn't get
>promoted by accident.

I'll take another look at the Lucid SRU.

Please specify the lines that are incorrect in the Lucid debdiff. Is it these two from kdump.initramfs?

@@ -24,22 +24,50 @@
 grep -q kdump_needed /proc/cmdline || exit 0

 # Make sure makedumpfile assumptions are satisfied.
-test -e $INFO || exit 0
-test -x $MAKEDUMPFILE || exit 0
+test -e $rootmnt/$INFO || exit 0
+test -x $rootmnt/$MAKEDUMPFILE || exit 0

Should the lines be removed or substituted with something else? Better yet, why not attach a diff against the current lucid-proposed version? :) I'd be happy to sponsor it again.

@barry

From what I've seen, they're the only difference between Lucid and the other debdiff. For sure, it is indeed what is causing the failure in Lucid (those lines are not in the other versions). I'll have a closer look at it tomorrow morning and will submit a new debdiff for Lucid. I just want to be sure that we get all ends covered this time. From what I could gather, they should be removed but, moreover, the same patch should apply to all versions.

On Thu, Mar 01, 2012 at 08:14:44PM -0000, Louis Bouchard wrote:
> >From what I've seen, they're the only difference between Lucid and the
> other debdiff. For sure, it is indeed what is causing the failure in
> Lucid (those lines are not in the other versions).

The lines are there in the later patches, but take a different form in later
releases:

 # Make sure makedumpfile assumptions are satisfied.
-while ! test -e "$INFO"; do
+while ! test -e "$rootmnt/$INFO"; do
        panic "kdump: Missing $INFO"
 done
-while ! test -x "$MAKEDUMPFILE"; do
- panic "kdump: Missing $MAKEDUMPFILE"
+
+while ! test -x "$rootmnt/$MAKEDUMPFILE"; do
+ panic "kdump: Missing $rootmnt/$MAKEDUMPFILE"
 done

So it's not clear to me why those lines would cause failures for you on
lucid.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

@steve

The reason why it fails on Lucid is that those line are placed BEFORE the $rootmnt gets mounted as opposed to AFTER the mount for the other distro.

It is like that because Lucid runs an older version of the package (1:2.0.1-1) compared to 1:2.0.1-2 for maverick and natty and 1:2.0.2-1 for Oneiric.

What I have done is to have the kdump.initramfs for Lucid be identical to the ones of the other versions (merely move those lines near the end), recreated the debdiff and rebuilt a test package with it. It now works correctly on Lucid too. But I'm questioning the validity of those steps : shouldn't we just provide 1:2.0.1-2 on Lucid just like for maverick and Natty ?

If not, then I have the debdiff ready for upload so a respin of the lucid-proposed can be done.

Just let me know

FYI,

After further investigation, the difference comes from LP: #600051 as noted in Maverick's kexec-tool changelog :

  * kdump.initramfs: panic if assumptions are not satisfied, instead of
    letting the system continue to boot inside the crash kernel. (LP: #600051)

On Fri, Mar 02, 2012 at 10:36:10AM -0000, Louis Bouchard wrote:
> The reason why it fails on Lucid is that those line are placed BEFORE
> the $rootmnt gets mounted as opposed to AFTER the mount for the other
> distro.

Aha, right - so these lines should be moved, rather than dropped.

> What I have done is to have the kdump.initramfs for Lucid be identical
> to the ones of the other versions (merely move those lines near the
> end), recreated the debdiff and rebuilt a test package with it. It now
> works correctly on Lucid too. But I'm questioning the validity of those
> steps : shouldn't we just provide 1:2.0.1-2 on Lucid just like for
> maverick and Natty ?

That depends on what else has changed in the meantime and whether all the
changes are suitable to include in an SRU.

> If not, then I have the debdiff ready for upload so a respin of the
> lucid-proposed can be done.

Easiest would be to make the one change to move the check below the mount.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

@steve

Moving the lines further down is what we get when running the same patch on the kdump.initramfs scripts of newer versions. So I have created a new debdiff for Lucid which produces the same kdump.initramfs for all versions.

The end result is that all distro will share the same kdump.initramfs script. I have positive tests with a new package built from that debdiff. Will be waiting for the -proposed version.

P.S. I have replaced the former lucid-proposed-lp828731.debdiff by a newer one.

lucid-proposed-lp828731.debdiff

Hello Louis, or anyone else affected,

Accepted kexec-tools into lucid-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

@clint

I have tested the new kexec-tool in lucid-proposed and it does work properly.

kexec-tools is now performing as expected on all supported distros (from Lucid onward). Let's hope that it makes it to -updates soon.

Thanks very much for your invaluable help.

This bug was fixed in the package kexec-tools - 1:2.0.1-1ubuntu5

---------------
kexec-tools (1:2.0.1-1ubuntu5) lucid-proposed; urgency=low

  * Backport changes to fix kdump functionality. LP: #828731.
    - debian/kdump.initramfs: call /usr/bin/makedumpfile via a chroot command,
      so that if makedumpfile is statically linked, we get proper library
      resolution. Thanks to Louis Bouchard <email address hidden> for
      the patch. LP: #785425.
    - debian/kdump.initramfs: handle the possibility that /usr, /boot, or
      /var is on a separate filesystem and needs to be manually mounted before
      calling makedumpfile. LP: #828731.
    - Depend on makedumpfile, without which the initramfs script doesn't work.
    - Fix an unnecessary bashism.
    - Only install the kdump initramfs script and depend on makedumpfile on
      architectures that makedumpfile supports.
 -- Louis Bouchard <email address hidden> Mon, 05 Mar 2012 15:10:26 -0000

This bug was fixed in the package kexec-tools - 1:2.0.1-2ubuntu2.2

---------------
kexec-tools (1:2.0.1-2ubuntu2.2) maverick-proposed; urgency=low

  * Backport changes to fix kdump functionality. LP: #828731.
    - debian/kdump.initramfs: call /usr/bin/makedumpfile via a chroot command,
      so that if makedumpfile is statically linked, we get proper library
      resolution. Thanks to Louis Bouchard <email address hidden> for
      the patch. LP: #785425.
    - debian/kdump.initramfs: handle the possibility that /usr, /boot, or
      /var is on a separate filesystem and needs to be manually mounted before
      calling makedumpfile. LP: #828731.
    - Depend on makedumpfile, without which the initramfs script doesn't work.
    - Fix an unnecessary bashism.
    - Only install the kdump initramfs script and depend on makedumpfile on
      architectures that makedumpfile supports.
 -- Chris J Arges <email address hidden> Wed, 18 Jan 2012 15:32:08 -0600

This bug was fixed in the package kexec-tools - 1:2.0.1-2ubuntu5

---------------
kexec-tools (1:2.0.1-2ubuntu5) natty-proposed; urgency=low

  * Backport changes to fix kdump functionality. LP: #828731.
    - debian/kdump.initramfs: call /usr/bin/makedumpfile via a chroot command,
      so that if makedumpfile is statically linked, we get proper library
      resolution. Thanks to Louis Bouchard <email address hidden> for
      the patch. LP: #785425.
    - debian/kdump.initramfs: handle the possibility that /usr, /boot, or
      /var is on a separate filesystem and needs to be manually mounted before
      calling makedumpfile. LP: #828731.
    - Depend on makedumpfile, without which the initramfs script doesn't work.
    - Fix an unnecessary bashism.
    - Only install the kdump initramfs script and depend on makedumpfile on
      architectures that makedumpfile supports.
 -- Chris J Arges <email address hidden> Wed, 18 Jan 2012 17:05:10 -0600

This bug was fixed in the package kexec-tools - 1:2.0.2-1ubuntu4

---------------
kexec-tools (1:2.0.2-1ubuntu4) oneiric-proposed; urgency=low

  * Backport changes to fix kdump functionality. LP: #828731.
    - debian/kdump.initramfs: call /usr/bin/makedumpfile via a chroot command,
      so that if makedumpfile is statically linked, we get proper library
      resolution. Thanks to Louis Bouchard <email address hidden> for
      the patch. LP: #785425.
    - debian/kdump.initramfs: handle the possibility that /usr, /boot, or
      /var is on a separate filesystem and needs to be manually mounted before
      calling makedumpfile. LP: #828731.
    - Depend on makedumpfile, without which the initramfs script doesn't work.
    - Fix an unnecessary bashism.
    - Only install the kdump initramfs script and depend on makedumpfile on
      architectures that makedumpfile supports.
 -- Chris J Arges <email address hidden> Wed, 18 Jan 2012 14:52:58 -0600

Is this really fixed on Ububuntu 14.04 LTS?

I've just tested triggering a crash dump and it didn't work. A Google search pointed me to this bug. When I see /boot on a separate partition, I just realized it is the damn case! Others like dbgsym kernel from ddeb, kdump-tools, linux-crashdump all installed and properly configured.

The VMs use LVM so they all have a separate /boot partition, that's why kdump was not able to dump the /proc/vmcore.

After using the workaround (umount /boot and use the /boot on /), it worked!

However, there is no 0_dump file under /usr/share/initramfs-tools/scripts/init-bottom/ WTF? So what is causing the problem in Ubuntu 14.04 with a separate /boot partition?

A separate bug needed?

I have a setup with Ubuntu 14.04 LTS server. The /boot is not on a separate partition. I followed the kdump setup as mentioned in https://help.ubuntu.com/14.04/serverguide/kernel-crash-dump.html

After triggering the kdump, system panics but system never comeup/reboot. It get stuck at the call trace of the panic.

Some of the commands output is below it might help to understand the issue:
root@ubuntu:~# mount
/dev/mapper/350002ac01d6314e1-part1 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
root@ubuntu:~#

hbauser@ubuntu:~$ ls /boot/
abi-3.13.0-24-generic grub memtest86+.bin memtest86+_multiboot.bin vmlinuz-3.13.0-24-generic
config-3.13.0-24-generic initrd.img-3.13.0-24-generic memtest86+.elf System.map-3.13.0-24-generic
hbauser@ubuntu:~$

hbauser@ubuntu:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.13.0-24-generic root=UUID=0503c0b3-1acd-4e11-a60a-f5f2618fdb6c ro crashkernel=384M-:128M
hbauser@ubuntu:~$ dmesg | grep -i crash
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.13.0-24-generic root=UUID=0503c0b3-1acd-4e11-a60a-f5f2618fdb6c ro crashkernel=384M-:128M
[ 0.000000] Reserving 128MB of memory at 720MB for crashkernel (System RAM: 32733MB)
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.13.0-24-generic root=UUID=0503c0b3-1acd-4e11-a60a-f5f2618fdb6c ro crashkernel=384M-:128M
hbauser@ubuntu:~$

In another configuration where I have installed OS with LVM and /boot on separate partition. The workaround suggested in this bug also didn't work. Am I missing something?

Nagaraj, I see no reason to think your problem is related to this bug. This bug is specifically about a separate /boot partition, which you do not have, and your symptoms don’t match either. You should file a new report.

First of all, doing ponytail on a 3 year old bug is not going to bring you much attention.

Especially since this bug pertains to a mechanism that is now only used on Precise. It is unfortunate though that the server guide is misleading on how to configure kernel dump. I would suggest to open a bug against the server guide to have this fixed.

This page does a pretty good job at documenting how to generate a kernel crash dump :

 https://wiki.ubuntu.com/Kernel/CrashdumpRecipe

TL;DR : Once the linux-crashdump package has been installed & the server rebooted you need to change the following in /etc/default/kdump-tools :

USE_KDUMP=1

Then :

 $ sudo kdump-config load

For any other visiting this bug, please refrain from adding to it, it is no longer relevant as the kernel dump mechanism has changed

same bug than before in trusty.
/boot on sde5
and / is on sdd2

can you release a fix please. I need to evaluate my kernel panic FAST! thank you

still present in trusty! /boot in sda1 and / is on LVM