update-initramfs -k all -c does not create initrd images anymore

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1829805

The screenshot has a kernel error, reassigning

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1829805

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

I am unable to get the system to boot to run apport-collect.

I have tested this using the daily iso image (5/21/2019) on a bare metal Acer laptop (BIOS) and Virtual Box using both BIOS and EFI. All tests yield the same kernel panic result.

I have also tested the daily iso (5/20/2019) for Kubuntu and Xubuntu, both of them worked properly on the same VM setup.

I tested today's (5/23/2019) ISO image in virtualbox and while it still fails, I was able to note that I could mount the installed OS file system from the live environment.

Same problem with me: Version "2019-06-05 16:52" is fine in live mode, but kernel-panics after installation and attempt to boot

It appears from the screenshot the panic is occurring because the initramfs /init script is failing to find and mount the real root file-system.

You can drop to a busybox shell in the initramfs by adding a kernel-command line option manually in the GRUB boot menu.

Tap Esc key to get to the GRUB menu, highlight the default entry, press 'e' to edit it, navigate down to the line beginning "linux ..." and add the option "break=premount" then press Ctrl+X (or F10) to immediately boot with this change.

This should drop you into the busbyox initramfs shell so you can investigate where the root-fs is.

Start with "cat /proc/cmdline" and then verify the root= device actually exists.

There are other points to break= at; On an working system identify them with:

grep maybe_break /usr/share/initramfs-tools/init

The problem here is the installer failed to build/install initrd.img.

After installing into a LVM LV via libvirt and seeing the error I mounted the disk image contained in the LV and explored:

$ sudo losetup -f --show -P /dev/VG02/lubuntu1910
/dev/loop5
$ sudo mount /dev/loop5p1 /mnt/target
$ sudo TERM=screen chroot /mnt/target

# ls -l /boot/
total 13548
-rw-r--r-- 1 root root 224306 Jun 4 16:22 config-5.0.0-17-generic
drwxr-xr-x 5 root root 4096 Jun 18 20:50 grub
lrwxrwxrwx 1 root root 27 Jun 18 17:41 initrd.img -> initrd.img-5.0.0-17-generic
lrwxrwxrwx 1 root root 27 Jun 18 17:41 initrd.img.old -> initrd.img-5.0.0-17-generic
-rw-r--r-- 1 root root 182704 Jan 28 2016 memtest86+.bin
-rw-r--r-- 1 root root 184380 Jan 28 2016 memtest86+.elf
-rw-r--r-- 1 root root 184840 Jan 28 2016 memtest86+_multiboot.bin
-rw------- 1 root root 4378278 Jun 4 16:22 System.map-5.0.0-17-generic
lrwxrwxrwx 1 root root 24 Jun 18 17:41 vmlinuz -> vmlinuz-5.0.0-17-generic
-r--r--r-- 1 root root 8703736 Jun 18 20:50 vmlinuz-5.0.0-17-generic
lrwxrwxrwx 1 root root 24 Jun 18 17:41 vmlinuz.old -> vmlinuz-5.0.0-17-generic

Notice the symlinks from initrd.img{,.old} point to non-existent initrd.img-5.0.0-17-generic,

The problem seems to be in the calamares configuration. The initramfs task is being called before the target's live-* packages have been removed. This is specifically warned about in the calamares initramfs module but we see the timestamps of the initramfs task are before the live-* packages are removed:

# in the live system

root@lubuntu:/usr/lib/x86_64-linux-gnu/calamares/modules/initramfs# cat README.md

## initramfs module

This module is specific to Debian based distros. Post installation on Debian
the initramfs needs to be updated so as to not interrupt the boot process
with a error about fsck.ext4 not being found.

## Debian specific notes

If you're using live-build to build your ISO and setup the runtime env
make sure that you purge the live-\* packages on the target system
before running this module, since live-config dpkg-diverts update-initramfs
and can cause all sorts of fun issues.

#
# chroot to the installed target
#

root@lubuntu:/# grep -n initramfs /var/log/installer/debug
81: "/etc/calamares/modules/initramfscfg.conf"
82: "/usr/share/calamares/modules/initramfscfg.conf"
84: "/etc/calamares/modules/initramfs.conf"
85: "/usr/share/calamares/modules/initramfs.conf"
313:2019-06-18 - 21:25:43 [6]: Starting job "initramfscfg"
316:2019-06-18 - 21:25:43 [6]: Job file "/usr/lib/x86_64-linux-gnu/calamares/modules/initramfscfg/main.py"
317:2019-06-18 - 21:25:43 [6]: Job description from pretty_name "initramfscfg" = "Configuring initramfs."
318:2019-06-18 - 21:25:43 [6]: Starting job "initramfs"
321:2019-06-18 - 21:25:43 [6]: Job file "/usr/lib/x86_64-linux-gnu/calamares/modules/initramfs/main.py"
322:2019-06-18 - 21:25:43 [6]: Job description from pretty_name "initramfs" = "Creating initramfs."
323:2019-06-18 - 21:25:43 [6]: Running "chroot" ("/tmp/calamares-root-kp3scv8s", "update-initramfs", "-k", "all", "-c", "-t")

root@lubuntu:/# cat /var/log/apt/history..log
...
Start-Date: 2019-06-18 21:26:38
Commandline: apt-get --purge -q -y remove ^live-* calamares-settings-lubuntu calamares hunspell-en-us zram-config cifs-utils
Purge: hunspell-en-us:amd64 (1:2018.04.16-1), calamares-settings-lubuntu:amd64 (1:19.10.1), casper:amd64 (1.408), lupin-casper:amd64 (0.57build1), calamares:amd64 (3.2.7-0ubuntu1), cifs-utils:amd64 (2:6.8-2), calamares-settings-ubuntu-common:amd64 (1:19.10.1), zram-config:amd64 (0.5)
End-Date: 2019-06-18 21:26:48

After many hours diving down rabbit holes it turns out this is due to a major change in the behaviour of update-initramfs introduced by Debian in July 2018 which made its way into the Ubuntu archive end of april 2019.

01:02 <TJ-> Right! initramfs-tools had a MAJOR import from Debian with "initramfs-tools (0.133ubuntu1)" April 29th; I've worked back through the changes by Debian to "initramfs-tools (0.132) unstable" of 26th July 2018, which contains "[f39625a] update-initramfs: Make "-k all" take over other initramfs images" and looking at that commit seems to indicate it is the cause

https://salsa.debian.org/kernel-team/initramfs-tools/commit/f39625afd6ba6c1aa2027286dc3ef1c933da14e0

Running the script with shell logging shows that when Calamares initramfs module calls with:

# update-initramfs -k all -c -t

We get:

...
+ break
+ [ 0 -ne 0 ]
+ [ -z c ]
+ [ all = all ]
+ get_sorted_versions
+ linux-version list
+ linux-version sort --reverse
+ read -r version
+ test -e /boot/initrd.img-5.0.0-17-generic
+ read -r version
+ version_list=
+ verbose Available versions:
+ [ 0 = 1 ]
+ [ -z ]
+ verbose Nothing to do, exiting.
+ [ 0 = 1 ]
+ exit 0

Because there is currently no /boot/initrd.img-$VERSION the version_list is empty.

This shouldn't be a concern when we're using the "-c" create option so it looks to be a bug in the logic.

The workaround for Lubuntu calamares installer configuration is to make the file exist before calling it:

 touch /boot/initrd.img-$(uname -r)

Until initramfs-tools is fixed.

We have a second bug that causes the call by update-initramfs::get_sorted_versions() to report nothing when calling (/usr/bin/) "linux-versions list".

This because the target root file-system is copied from the filesystem.squashfs on the ISO which does NOT contain a /boot/vmlinuz-$(uname -r).

calamares has a job to copy the /cdrom/casper/vmlinuz into the target and name it correctly. The job is

contextualprocess@before_bootloader_mkdirs

but it is listed after the "initramfs" job so there is no kernel image to get the version string from when "initramfs" executes. "before_bootloader_mkdirs" needs moving to before "initramfs".

I should have made clear, this 'second bug' is the only bug.

initrmafs-tools is in the clear - it wasn't finding a kernel image because there wasn't one when update-initramfs was executed.

Or is it? Actually I not, because the test in update-initramfs::get_sorted_versions() expects an existing initrd.img-$version:

get_sorted_versions()
{
        version_list="$(
                linux-version list |
                while read -r version; do
                      test -e "${BOOTDIR}/initrd.img-$version" && echo "$version"
                done |
                linux-version sort --reverse
                )"
        verbose "Available versions: ${version_list}"
}

I confused myself because I'd added a workaround for this in calamares by executing a shell

"touch /boot/initrd.img-$(uname -r)"

and forgotten it was there whilst chasing the missing kernel!

We got a potential fix thanks to TJ that seems to be working wonderfully from what I can ascertain:
https://phab.lubuntu.me/D15

Download the raw diff, cd /etc/calamares, sudo patch --strip 2 < /path/to/diff, and install like normal. Done!

I have tested this patch in virtualbox and it works for me.

Just to add my $.02 . . . I also had this problem with perhaps 6/12 daily and then I zsyncd what I think is the 6/15 daily . . . finding the same "kernel panic" report as Dan found. A couple days later I downloaded a fresh 19.04 iso and ran the install . . . and that went well.

Glad to hear that a fix has been engineered, right now I don't have the 19.10 iso installed to flash drive so I can't test out this proposed fix . . . may or may not get to that . . . . I might try to upgrade to 19.10 via the console . . . .

But, this problem was "frustrating" in that, as Dan mentions, the "live" system appeared to be "fine" and the installer did not report "errors" and yet would not boot into the installed system, or tty to try stuff out . . . which possibly would have failed anyway??

The fix is now released. Spinning up a new ISO as we speak.

calamares-settings-ubuntu (1:19.10.2) eoan; urgency=medium

  * Fix initramfs job failing. Thanks to TJ for the fix! (LP: #1829805)
    * Move before_bootloader_mkdirs process before initramfs.
    * Workaround bug in update-initramfs::get_sorted_versions() with
      shellprocess.

 -- Walter Lapchynski <email address hidden> Wed, 19 Jun 2019 19:32:14 -0700

The bug was only fixed in our Calamares configuration and still needs to be fixed in initramfs-tools.

Adjusting bug statuses appropriately and adding the release tag. I'll ping vorlon and xnox on IRC to get their take.

For initramfs-tools... please provide steps to reproduce the issue of when initrds are expected to be created and are not, with steps how to get in such situation (e.g. rm /boot/initrd*; update-initramfs...)

*without* requiring to use calamares, or specific images. ie. from any regular normal working installed system.

Until then, I'm failing to understand what the bug in initramfs-tools is allegedly identified.

There was some discussion regarding this particular bug in #ubuntu-release on 20190620 - https://irclogs.ubuntu.com/2019/06/20/%23ubuntu-release.html. I will not summarize the conversation here but only capture what I understand the bug in initramfs-tools to be.

update-initramfs no longer uses what is in /var/lib/initramfs-tools as a source of knowledge about kernel versions so using "-k all" will not bring a deleted initrd. This behavior does not match the documentation and is a regression. Some choice quotes follow:

"[20:52] <infinity> That appears to completely ignore STATEDIR now.
[20:59] <infinity> Anyhow, the behaviour change above is that it switched from checking STATEDIR to stating the initrd.
[20:59] <infinity> So, instead of "statedir knows what we've done in the past", we have "in you have no initrd, you can't have a new one either".
[20:59] <infinity> s/in you/if you/
[21:02] <infinity> https://salsa.debian.org/kernel-team/initramfs-tools/commit/f39625afd6ba6c1aa2027286dc3ef1c933da14e0
[21:03] <infinity> vorlon: That would be the offending commit. I think the obvious way to get both old and new behaviour combined would just be to add the statedir bit to the list (and sort -u the mess)."

This bug has been fixed in initramfs-tools 0.136 and therefore in Ubuntu 20.04 LTS "Focal Fossa".