Kernel Panic on Boot After Installation (No initrd in grub.cfg)

Bug #1900917 reported by Pico
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cubic
Fix Released
High
Cubic PPA

Bug Description

Yesterday I ran my usual scripts with the latest version of Cubic (2020.10-34) to create an updated Linux Mint installer.

The custom ISO booted properly into Live Boot as well as straight to Ubiquity installer.

But, when I install the Linux Mint (from either Live Boot or our customized straight-to-Ubiquity setup) in UEFI mode (haven't yet tested BIOS installation), I get a Kernel Panic on boot into the installed Linux Mint.

Upon investigation, I've found that the grub.cfg file does not have any initrd listed in the generated boot option.

I've recreated the custom ISO today to make sure it wasn't a fluke and got the same result.

This did not happen with last months updated build which used the same exact scripts during created. The last working ISO that I created was made on 2020-09-23 using Cubic 2020.09-31.

Is there something in the latest Cubic changes that could have possibly caused this?

I'm doubtful that it is a Linux Mint or Kernel issue because our custom installation process installs all updates via chroot into the installed drive after installation. So, even using my 2020-09-23 ISO, after the installation process, the latest Linux Mint with the latest Kernel is being successfully booted.

Revision history for this message
Pico (picomitchell) wrote :

As always, any thoughts or advice is greatly appreciated. I am not 100% certain this is a Cubic issue, but it seems to be the only variable that has changed between the last working ISO and now (unless I've somehow spaced me doing sometime silly in the meantime).

I believe I've seen in the past that the boot settings applied after the installation are somewhat inherited from the USB installer boot, but I'm not certain how all of that really works.

Revision history for this message
Pico (picomitchell) wrote :

Here is the Kernel Panic if it helps.

Revision history for this message
Pico (picomitchell) wrote :

Here is the BAD grub.cfg that was generated after an installation using the custom ISO built with Cubic 2020.10-34

Revision history for this message
Pico (picomitchell) wrote :

Here is the GOOD grub.cfg that was generated after an installation using the custom ISO built with Cubic 2020-09-23

I see now that the BAD one is also using "vmlinuz-5.4.0-52-generic.efi.signed" while this GOOD one uses "vmlinuz-5.4.0-52-generic" not sure what's up with that.

These installations were both done today in UEFI mode using the same process with the same post installation scripts on the same laptop.

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Pico,

Thanks for the really good details!

You're the second person reporting a vmlinuz file with the extension ".efi.signed" ~after~ installation.

With Bug #1898749, we introduced a new symlink named "vmlinuz.efi" as a work-around for the Pop_OS! installer which has this name hard-coded.

Although this is ONLY on the Customized ISO for the bootstrap process, I wonder if the installer is reading this link on the ISO, and forcing grub.cfg in the installed system to contain "vmlinux*.efi.signed" ?

Changed in cubic:
assignee: nobody → Cubic PPA (cubic-wizard)
Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Unfortunately, I have not been able to reproduce this myself.

Would you be able to help validate if the "vmlinuz.efi" symlink is indeed the culprit?...

Please execute the following to modify Cubic...

$ sudo cp /usr/share/cubic/pages/generate_page.py /usr/share/cubic/pages/generate_page.py.backup
$ sudo sed -i "s|symlink(|# symlink(|g" /usr/share/cubic/pages/generate_page.py

Then run Cubic as you normally would, generate a new *.iso file, install the custom OS, and see if you get the same problem.

To revert the above change to Cubic, execute...

$ sudo mv /usr/share/cubic/pages/generate_page.py.backup /usr/share/cubic/pages/generate_page.py

If you are able to test this, please let me know your results?

Changed in cubic:
status: New → Incomplete
Revision history for this message
Pico (picomitchell) wrote :

Will do!

I'll start that now and let you know how it goes.

Revision history for this message
Pico (picomitchell) wrote :

Yep! Somehow, that little symlink appears to be the culprit.

I deleted my existing ISO, deleted the existing vmlinuz.efi symlink in my project files and then ran your snippet to comment out the vmlinuz.efi symlink creation in Cubic.

After having Cubic re-create a new ISO for that existing project, I put it on a USB and ran an install and the installed OS booted properly.

The grub.cfg file in this latest Linux Mint install show the proper "vmlinuz-5.4.0-52-generic" (NOT ENDING IN ".efi.signed") file and "initrd.img-5.4.0-52-generic" is properly listed on the initrd line right below that.

Thanks so much for your quick reply and quick thinking on a workaround. I know it'll take a bit more work to get Cubic working right for Pop_OS! and other Linux flavors though.

Cubic PPA (cubic-wizard)
Changed in cubic:
status: Incomplete → Confirmed
importance: Undecided → High
Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Pico,

Thanks for validating this; I really appreciate you taking the time to test.

To summarize...

1. The installer auto-generates a /boot/grub/grub.cfg
2. If the installer sees vmlinuz.efi in the /casper directory of the ISO...
   It generates a grub.cfg file that references "vmlinux<kernel-version>.efi.signed"
   But "vmlinux<kernel-version>.efi.signed" does not exist
3. If the installer does NOT see vmlinuz.efi in the /casper directory of the ISO...
   It generates a grub.cfg file that references "vmlinux<kernel-version>"
   And "vmlinux<kernel-version>" does exist

With vmlinuz.efi...
- Linux Mint and Ubuntu fail to boot into the installed OS
- Pop_OS! installs successfully

Without vmlinuz.efi...
- Linux Mint and Ubuntu boot successfully into the installed OS
- Pop_OS! fails to install

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Related Bugs:

Bug #1898749, "Pop!_OS expects vmlinuz on the iso to have the *.efi extension."
Bug #1895770, "Pop!_OS expects the initramfs bootstrap file to be explicitly named "initrd.gz"

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

In order to make sure this works for Ubuntu and other distros, while still working for Pop!_OS, Cubic will only create the vmlinuz.efi link when Pop!_OS is being customized.

Cubic checks the file /etc/os-release to determine which OS being customized. For Pop!_OS, the value for keys "ID" or "ID_LIKE" must contain the text "pop".

In the future, if Pop!_OS fixes the installer to not have "vmlinuz.efi" as a hard coded value, then this stop-gap fix can be removed from Cubic.

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Fix released in Cubic version 2020.10-35-release.

Changed in cubic:
status: Confirmed → Fix Released
Revision history for this message
Pico (picomitchell) wrote :

Today is the first chance I got to actually build a new ISO using the latest Cubic (2020.10-35) and everything is back to working perfectly.

Thanks so much for the quick fix!

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Thank you for testing and getting back to me.
I'm glad it's working now!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.