initramfs does not get loaded

Status changed to 'Confirmed' because the bug affects multiple users.

This bug may also be affecting focal-server-cloudimg builds at: https://cloud-images.ubuntu.com/focal/.
See: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1874147

Hi, this issue needs attention because it renders the cloud images for Ubuntu 20.04 potentially unusable. Thanks.

I agree with David. IMO this bug should be fixed ASAP. Thanks!

I've traced the issue back to "00_header" and "10_linux" here:

https://git.launchpad.net/ubuntu/+source/grub2/tree/debian/patches/ubuntu-add-initrd-less-boot-fallback.patch?h=ubuntu/focal

diffstat debian/patches/ubuntu-add-initrd-less-boot-fallback.patch
 Makefile.am | 3 ++
 configure.ac | 10 +++++++
 grub-initrd-fallback.service | 12 +++++++++
 util/grub.d/00_header.in | 27 ++++++++++++++++++++
 util/grub.d/10_linux.in | 56 +++++++++++++++++++++++++++++--------------

This is still an issue. I have to manually patch the cloud images with guestmount after downloading them in order to make them to work. Please advise. Thank you!

Also, this bug is not specific to just Azure. I am using libvirt and experiencing this issue.

The git commit which introduced the issue seems to be this:

commit 6a814c759e10feafb40c3669be30aa51eb5ce39b
Author: Mathieu Trudel-Lapierre <email address hidden>
Date: Tue Jul 16 11:31:29 2019 -0400

    Import patches-unapplied version 2.04-1ubuntu1 to ubuntu/eoan-proposed

    Imported using git-ubuntu import.

(at https://git.launchpad.net/ubuntu/+source/grub2/)

When the bug was originally reported on Apr 1, "We installed a Ubuntu 20.04 VM from the .iso file from http://cdimage.ubuntu.com/daily-live/pending/ and don’t see the strange grub issue". It looks the grub version used in the .iso file (on Apr 1) does not have the bug.

I don't think the patch in the link mentioned in comment #6 causes the bug, because that patch was made 2 years ago and we started to see this bug just recently. Of course I can be wrong, since I don't really have a lot of grub knowledge. :-)

I'm not sure if the commit 6a814c759e10 ("Import patches-unapplied version 2.04-1ubuntu1 to ubuntu/eoan-proposed", made on Jul 16 11:31:29 201) causes this bug, either, since it's made almost 10 months ago. Again, I can be wrong. :-) BTW, this commit is huge -- more than 12K lines.

Hmmm, I am not using the ISO, but rather the pre-installed cloudimg, and I am not so sure it is actually fixed. I just tried out the latest one, and can confirm the issue still persists with it:

https://cloud-images.ubuntu.com/focal/20200430.1/focal-server-cloudimg-amd64.img

Here is the output from booting:

[ 0.000000] Linux version 5.4.0-28-generic (buildd@lgw01-amd64-036) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #32-Ubuntu SMP Wed Apr 22 17:40:10 UTC 2020 (Ubuntu 5.4.0-28.32-generic 5.4.30)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-28-generic root=PARTUUID=be1f3b17-864d-49f8-8249-5fb0edc7e606 ro console=tty1 console=ttyS0 panic=-1
...
[ 0.999771] VFS: Cannot open root device "PARTUUID=be1f3b17-864d-49f8-8249-5fb0edc7e606" or unknown-block(0,0): error -6
[ 1.001738] Please append a correct "root=" boot option; here are the available partitions:
[ 1.003295] 0b00 1048575 sr0
[ 1.003295] driver: sr
[ 1.004578] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 1.006119] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-28-generic #32-Ubuntu
[ 1.007522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[ 1.009254] Call Trace:
[ 1.009790] dump_stack+0x6d/0x9a
[ 1.010488] panic+0x101/0x2e3
[ 1.011137] mount_block_root+0x23f/0x2e8
[ 1.011931] mount_root+0x38/0x3a
[ 1.012624] prepare_namespace+0x13f/0x194
[ 1.013443] kernel_init_freeable+0x231/0x255
[ 1.014319] ? rest_init+0xb0/0xb0
[ 1.015018] kernel_init+0xe/0x100
[ 1.015721] ret_from_fork+0x35/0x40
[ 1.016842] Kernel Offset: 0x1ac00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

Upon reboot, initrdfail will now be set, and therefore it will actually load the initrd on the second reboot. But, because I am trying to configure the machine on the first boot using cloud-init NoCloud, it needs to actually work on the first boot. (Anyways, I think it's a moot point that first time booting should work.)

In the above, the "panic=-1" followed by "Cannot open root device" indicates that there is still something amiss, at least with the cloudimg distribution. I can file a different ticket for cloudimg, if it would be more appropriate. Please advise. Thanks!

I'm actually not sure the best place to file a ticket for the cloudimg distribution. Any recommendation appreciated.

Today I installed a Generation-1 Ubuntu 20.04 VM on my local Hyper-V host from the ISO file: https://releases.ubuntu.com/20.04/ubuntu-20.04-live-server-amd64.iso (released on 4/23/2020) and I don't see this bug and the grub version is 2.04-1ubuntu26.

Today I also created a VM on my host from https://cloud-images.ubuntu.com/focal/20200430.1/focal-server-cloudimg-amd64-azure.vhd.zip and can not see the bug either, and the grub version is also 2.04-1ubuntu26.

When the bug was originally reported on Apr 1 against my Azure VM (a Ubuntu 19.10 VM on Azure was created and upgraded to Ubuntu 20.04 by “do-release-upgrade –d"), the grub version was 2.04-1ubuntu22.

So it looks the issue has been fixed in the 26 version?

Sorry, this statement is wrong:
======================================================
Today I also created a VM on my host from https://cloud-images.ubuntu.com/focal/20200430.1/focal-server-cloudimg-amd64-azure.vhd.zip and can not see the bug either, and the grub version is also 2.04-1ubuntu26.
======================================================

Actually I do see the bug as well with the vhd.zip file.

Today I just repeated the test "Create a Gen-1 Ubuntu 19.10 VM on Azure, and upgrade it to Ubuntu 20.04 by “do-release-upgrade –d" and I reproduced this bug again, and the grub version is also 2.04-1ubuntu26!

So I suspect grub itself should be good, but some grub config file (i.e. /etc/grub.d/10_linux?) causes the bug?
I checked my /etc/grub.d/10_linux: after I added line 263, "grub-mkconfig" can generate the needed initrd line correctly:

257 fi
258
259 sed "s/^/$submenu_indentation/" << EOF
260 initrd ${rel_dirname}/${initrd}
261 else
262 linux ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args} panic=-1
263 initrd ${rel_dirname}/${initrd}
264 fi
265 initrdfail
266 EOF

My /etc/grub.d/10_linux is from the grub2-common package (2.04-1ubuntu26). It looks this file in my VM that's upgraded from 19.10 to 20.04 is different from the version of the file in a VM that's created from https://releases.ubuntu.com/20.04/ubuntu-20.04-live-server-amd64.iso

That's why I suspected it is specific to the cloud-image version of Ubuntu 20.04. I don't know how exactly “do-release-upgrade -d" works and where the upgrade procedure pulls the grub2 that lacks the initrd line in the /etc/grub.d/10_linux.

In summary,
1. https://cloud-images.ubuntu.com/focal/20200430.1/focal-server-cloudimg-amd64.img and https://cloud-images.ubuntu.com/focal/20200430.1/focal-server-cloudimg-amd64-azure.vhd.zip have the bug.
2. https://releases.ubuntu.com/20.04/ubuntu-20.04-live-server-amd64.iso does not have the bug.
3. A quick fix is add the needed line 263 (see above), but I think we need to understand how the bug is introduced.

I think commen #6 is correct: it looks the 2018 patch introduced the issue for us, but the patch is originally for "initrd-less boot capabilities." and here we do need the initramfs file.

I guess the patch "ubuntu-add-initrd-less-boot-fallback.patch" is not included into the grub shipped in the 20.04 .iso file ubuntu-20.04-live-server-amd64.iso, but somehow it's included into the grub shipped in the cloud-image? If so, I guess we can fix this bug by removing the patch for cloud-image?

Anyone knows who maintains the grub package shipped in the cloud-images?
Should we report a bug at https://bugs.launchpad.net/ubuntu/+source/grub2 ?

I don't know how to escalate attention to this issue. Is emailing the author of commit 6a814c759e10feafb40c3669be30aa51eb5ce39b, which seems to have introduced the issue, acceptable? I'm surprised it's not causing more issues. I guess people don't care about the cloud images much?

You may have a file /etc/default/grub.d/40-force-partuuid.cfg in your system, that sets the grub var GRUB_FORCE_PARTUUID. If so, you can just comment out that line, or remove the file, and run 'sudo update-grub'. Your /boot/grub/grub.cfg should then no longer have the initrd-less boot with panic backup.

Hey Dan, thanks for getting back. In my case, it's not even my system though: it's Canonical's system; It's a fresh download of a cloudimg from https://cloud-images.ubuntu.com/focal/ which has the issue, so it can't because I didn't build the image right, because I didn't build the image at all. So how would this be able to help? What is the rationale behind the new initrdfail design and 40-force-partuuid.cfg? Thanks.

> In my case, it's not even my system though: it's Canonical's system; It's a fresh download of a cloudimg from https://cloud-images.ubuntu.com/focal/ which has the issue, so it can't because I didn't build the image right, because I didn't build the image at all.

that's correct, you didn't do anything wrong; the cloud-images for focal have the /etc/default/grub.d/40-force-partuuid.cfg file built-in.

> So how would this be able to help? What is the rationale behind the new initrdfail design and 40-force-partuuid.cfg? Thanks.

I'm not sure what the original rationale was, as I wasn't involved in those discussions, although my rough understanding is it's intended to try to speed up boot time. @vorlon may know more about the reason behind it.

Yes, the design here is that, since we provide custom kernels for our cloud partners which have all the drivers needed to boot built in, loading an initrd is a waste of time at boot.

If someone is using a kernel other than the one we provide for the cloud, or in the case of a bug, the system will still boot (slower) after a panic and a reboot to try again with the initrd.

Removing this functionality would be a pessimization. The vast majority of Ubuntu instances in Azure are expected to be booted with the provided linux-azure kernel, which does not require an initrd. Canonical does not support the use of other kernels on Azure. And if someone needs to use a different kernel on an instance without the double-boot process, they can edit the /etc/default/grub.d/40-force-partuuid.cfg config file that is provided with the image at the same time they are installing the different kernel to get the previous behavior.

Thanks for the information @vorlon. I think that makes sense.

What would be the solution for booting focal-server-cloudimg-amd64.img with libvirt (virt-install) (so, not on Azure, as the OP was; so, I think the issue is now fractured into two separate issues with same underlying cause)?

Thanks.

On Tue, May 12, 2020 at 01:50:35AM -0000, David Baird wrote:
> Thanks for the information @vorlon. I think that makes sense.

> What would be the solution for booting focal-server-cloudimg-amd64.img
> with libvirt (virt-install) (so, not on Azure, as the OP was; so, I
> think the issue is now fractured into two separate issues with same
> underlying cause)?

These images should likewise be bootable under libvirt with, at most, a boot
performance penalty because of the second boot.

If you are hitting the panic and reboot, it probably means your libvirt is
configured to use suboptimal storage drivers. The recommended driver is
virtio-scsi rather than virtio-blk.

> If someone is using a kernel other than the one we provide for the cloud, or in the case of a bug, the system will still boot (slower) after a panic and a reboot to try again with the initrd.

Hi Steve, I guess you assume the pattern is "panic/success/success/success/...", but actually the pattern is “panic/success/panic/success/panic/success/...” -- this is pretty confusing. Please refer to the the Bug Description for details.

Ideally grub should be configured to not add the 'initrd' line only for the cloud kernels. Is there a way for grub to tell if the kernel is a cloud kernel or not?

> These images should likewise be bootable under libvirt with, at most, a boot performance penalty because of the second boot.

Well, yes, it does boot on the second boot. Not trying to be contrarian here, but isn't a moot point that booting on the first boot is the de facto way things should work? I.e., a panic/reboot shouldn't be considered the "normal" way to boot on libvirt, or anywhere, for that matter. Furthermore, the panic/reboot breaks the virt-install cloud-init workflow, which expects the first boot to succeed. I mean, that's a solid expectation, right? :)

What is the cloudimg designed to boot with, if not libvirt? I think virtio-scsi is being used. Here's the virt-install line, if it helps,

virt-install \
-n myvm \
--description '' \
--os-type Linux \
--os-variant ubuntu16.04 \
--disk path=/var/lib/libvirt/images/focal-server-cloudimg-amd64.img,bus=scsi,discard=unmap \
--controller type=scsi,model=virtio-scsi \
--cdrom /var/lib/libvirt/images/nocloud.iso \
--virt-type kvm \
--graphics none \
--ram 1024 \
--vcpus 1 \
--network bridge:guestbr,model=virtio,virtualport_type=openvswitch,target=myvm \
--boot hd

Once I do (eventually) get the image to boot through arcane ways, then I can confirm it is loading virtio-scsi:

$ lsmod | grep virt
virtio_net 53248 0
virtio_scsi 24576 2

Virtio-scsi seems like it needs to be compiled into the kernel for this to make sense. Am I missing something here? :)

Thanks.

BTW, the symptom described in the Bug Description also exists in the Ubuntu 20.04 image in Azure Marketplace.

Hi! Has any information surfaced about the cloudimg kernel and virtio-scsi?

After thinking about this for some time, I propose that the "initrdfail" design isn't ultimately helpful and should be reverted, because: (1) It substantially increases the complexity of grub.cfg, (2) it is not clear that any value is being added (but maybe I am just not aware of the scenarios where it is valuable), and (3) it has the effect of masking legitimate problems (like the ones being discussed in this ticket). More on that here:

The actual design intention is to have a kernel that boots correctly on the first try. The cloud images, in fact, should be shipping to boot correctly on the first try. These fail/success/fail/success/etc. boot sequences experienced by myself and @decui are nonsensical and don't really contribute value. A fail/success/fail/success/etc. is not an acceptable workaround, because it breaks things, and it's also just hiding the actual problem: that the kernel hasn't shipped with the right drivers, or the wrong kernel was bundled with an image. These are two definite disadvantages of the "initrdfail" design as it is currently. I'm open to hearing feedback.

Thanks,
David

On Tue, May 12, 2020 at 08:09:12PM -0000, Dexuan Cui wrote:
> Hi Steve, I guess you assume the pattern is
> "panic/success/success/success/...", but actually the pattern is
> “panic/success/panic/success/panic/success/...” -- this is pretty
> confusing. Please refer to the the Bug Description for details.

No, on the contrary, it's intentional that on each fresh reboot it again
attempts an initramfsless boot before falling back to an initramfsfull boot.
Because the expectation is that any image using this feature should boot
without initramfs, and if it doesn't that is a bug or a configuration error
(using the wrong kernel for the target, or not removing the config flag when
testing a different kernel). The intent is that the system remain bootable
even if there is such a bug or configuration error.

On Tue, May 12, 2020 at 08:51:38PM -0000, David Baird wrote:
> What is the cloudimg designed to boot with, if not libvirt? I think
> virtio-scsi is being used. Here's the virt-install line, if it helps,

Ok, it turns out this is a bug in the generic cloud image, which ships with
the linux-generic kernel, a kernel which is not expected to boot without
initramfs.

We should fix this to not set the FORCE_PARTUUID flag on this image, since
there's no expectation that it will work.

I've opened a task on livecd-rootfs for this.

Note that the /minimal/ downloadable cloud image uses the linux-kvm kernel
instead; that is the one that has the virtio drivers built-in, and can be
booted without an initramfs.

Awesome, thanks for categorizing that for me @vorlon! I was trying hard to find the right home for the issue, but didn't even think of livecd-rootfs :)

I still am not too sure about the ultimate value offered by the initramfs fallback. It seems like you would: (a) never want initramfs, or (b) always want initramfs. Whereas the initrdfail mechanism isn't either of those, and it introduces complexity and confusion and potentially masks the actual issue.

So it looks this is considered as a feature rather than a bug for the Ubuntu 20.04 VM image in Azure Marketplacet. To whoever uses such an image on Azure: if you're installing a new kernel that doesn't have the necessary drivers built-in (CONFIG_HYPERV=y, CONFIG_HYPERV_STORAGE=y), you're supposed to comment out the GRUB_FORCE_PARTUUID line in /etc/default/grub.d/40-force-partuuid.cfg and run 'sudo update-grub'.

No change is necessary on the linux-azure.

Speaking of cloud-images generally I propose the following:

* Images in the ubuntu-cpc project by default should boot with an initramfs.
  * Revert the livecd-rootfs change which "Unconditionally set
    GRUB_FORCE_PARTUUID in cloud images"
* Images with custom kernels can boot without an initramfs
  * Set GRUB_FORCE_PARTUUID as part of livefs-rootfs:live-build/functions replace_kernel() as this is used by the ubuntu-cpc project for each image that ships an optimized kernel intended to boot without initramfs.

Steve, I agree with everything you said in comment #21 about removing not this functionality from Azure images. I think there is room to improve the experience for users that change their kernel and encounter this behavior. The panic/reboot behavior is unexpected and the mapping of GRUB_FORCE_PARTUUID to initramfs-less boot is non-obvious to the end user. I don't immediately have suggestions to smooth this rough edge but I feel that we need to take some time on this before completely closing this bug out.

Robert Jennings's proposal from comment #33 landed in groovy v2.672.

A mechanism was also added in groovy v2.678 to detect if we've booted using the panic fallback behavior mentioned by Steve Langasek in comment #21.

Sorry, that was v2.672 and v2.678 of the livecd-rootfs package.

Status changed to 'Confirmed' because the bug affects multiple users.

Hey, is there some transparency into an actual fix here? https://cloud-images.ubuntu.com/focal/20201210/focal-server-cloudimg-amd64.img still has this issue. I still cannot use the cloudimg with libvirt / qemu without modification to workaround this issue.

Hi dhbaird - this was fixed in v2.672, v2.674, and v2.678 of the livecd-rootfs package in groovy; however, those changes have not been backported to focal, yet.

Thanks for the update @davidkrauser, and very happy that groovy is working out. Is the fix due for focal sometime in the coming weeks or months? Any extra info is appreciated. Also happy holidays and end of year, and thank you all for what you do!

Hi dhbaird - sorry, the focal backport has languished in my queue for quite a while now. I have not been able find time to focus on it. Another developer, patviafore, has offered to take the task off my hands, and I believe he is hoping to start looking into it at the start of the new year. Happy holidays to you, too :-)

Is there any update on this?

Hi richmbx,

I am currently working this, and currently doing testing to make sure it has integrated with Focal correctly. I expect it to begin the SRU process in the next day or so.

As an update to this bug:

The work to SRU has stalled, as we have found a regression (https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1902260). Once a fix is SRU'd to Groovy, I'll pick up the fix and SRU it back to Focal.

livecd-rootfs (2.672) groovy; urgency=medium

  [ David Krauser ]
  * Boot with an initramfs by default in cloud images, except when
    using a non-generic kernel.

 -- Robert C Jennings <email address hidden> Fri, 10 Jul 2020 07:47:47 -0500

Hello Joseph, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted livecd-rootfs (2.664.18) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/1.10+20.04ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I have tested the changes in livecd-rootfs 2.664.18. Generic cloud images (qcow2, kvm, etc) cloud images now boot with an initramfs and don't have a double boot (tested with QEMU). I've also tested various cloud images (AWS, Google, Azure, Oracle) that have custom cloud kernels (which make use of the replace_kernel function) and confirmed that they don't boot with initramfs, but they still only have one boot.

The only place I've seen a double boot was when booting the Azure image with qemu, but that was to be expected, since it has the linux-azure kernel and does not include all the things needed to boot in a non-Azure environment.

This bug was fixed in the package livecd-rootfs - 2.664.18

---------------
livecd-rootfs (2.664.18) focal; urgency=medium

  [ Patrick Viafore ]
  * Only try without initrd-less on replaced kernels, not all kernels
  * Provide a mechanism to detect initrd-less fallback (LP: #1870189)

 -- Robert C Jennings <email address hidden> Tue, 23 Feb 2021 14:45:23 -0600

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.