flash-kernel failure during kernel upgrade

Bug #1861292 reported by Dave Jones
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
flash-kernel (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Critical
Unassigned

Bug Description

[Impact]

In version 3.98ubuntu10 of flash-kernel (f-k henceforth), when both f-k and the kernel are upgraded in the same cycle, depending on the ordering of dpkg trigger execution, f-k may find the content of /boot "inconsistent" causing it to fail and exit with code 2.

This causes build failures of, for instance, our server-live images.

[Test Case]

* Flash an old image (with an out of date kernel) to an SD card
* Boot the card on a pi with no network access (to ensure unattended-upgrades doesn't do anything until wanted)
* Login and change password
* Disable u-u
* Connect ethernet
* sudo apt update/upgrade
* Observe failure with 3.98ubuntu10 (and check existence of /tmp/flash-kernel-no-kernel-error.log)
* Reflash same old image
* Boot the card without network access
* Disable u-u
* Connect ethernet
* sudo add-apt-repository ppa:waveform/flash-kernel
* sudo apt update/upgrade
* Upgrade should proceed without issue

[Regression Potential]

As with the previous flash-kernel uploads, it is possible that a breakage in the changed code can lead to issues with upgrading kernels (due to f-k being executed via a trigger at the end) or with pi devices in the field not upgrading correctly. We will test all the changes extensively though.

[Original Description]

In version 3.98ubuntu10 of flash-kernel (f-k henceforth), when both f-k and the kernel are upgraded in the same cycle, depending on the ordering of dpkg trigger execution, f-k may find the content of /boot "inconsistent" causing it to fail and exit with code 2:

ubuntu@ubuntu:~$ sudo apt full-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libdns-export1104 libdns1104 libisc-export1100 libisc1100
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  libctf-nobfd0 libctf0 libdns-export1107 libdns1107 libisc-export1104 libisc1104 linux-headers-5.3.0-1017-raspi2 linux-image-5.3.0-1017-raspi2 linux-modules-5.3.0-1017-raspi2 linux-raspi2-headers-5.3.0-1017
  python3-distutils python3-importlib-metadata python3-lib2to3 python3-more-itertools python3-pyrsistent python3-setuptools python3-zipp
The following packages will be upgraded:
  bind9-host binutils binutils-arm-linux-gnueabihf binutils-common dmeventd dmsetup dnsutils e2fsprogs file flash-kernel gcc-9-base geoip-database glib-networking glib-networking-common
  glib-networking-services info install-info isc-dhcp-client isc-dhcp-common landscape-common libappstream4 libassuan0 libbind9-161 libbinutils libcom-err2 libdevmapper-event1.02.1 libdevmapper1.02.1 libelf1
  libext2fs2 libfribidi0 libgcc1 libgeoip1 libirs161 libisccc161 libisccfg163 liblvm2cmd2.03 liblwres161 liblzo2-2 libmagic-mgc libmagic1 libnss-systemd libpam-systemd libpci3 libpcsclite1 libss2 libssh-4
  libssl1.1 libstdc++6 libsystemd0 libudev1 libxml2 linux-headers-raspi2 linux-image-raspi2 linux-raspi2 login logsave lvm2 mdadm openssh-client openssh-server openssh-sftp-server openssl passwd pci.ids
  pciutils python3-blinker python3-cryptography python3-json-pointer python3-jsonschema python3-nacl python3-yaml systemd systemd-sysv u-boot-rpi u-boot-tools ubuntu-minimal ubuntu-server ubuntu-standard udev
79 upgraded, 17 newly installed, 0 to remove and 0 not upgraded.
Need to get 71.3 MB of archives.
After this operation, 227 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ports.ubuntu.com/ubuntu-ports focal/main armhf login armhf 1:4.8-1ubuntu1 [217 kB]
Get:2 http://ppa.launchpad.net/waveform/pi4/ubuntu focal/main armhf u-boot-rpi armhf 2019.07+dfsg-1ubuntu8 [276 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports focal/main armhf libnss-systemd armhf 244.1-0ubuntu2 [114 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports focal/main armhf udev armhf 244.1-0ubuntu2 [1319 kB]
Get:5 http://ppa.launchpad.net/waveform/pi4/ubuntu focal/main armhf u-boot-tools armhf 2019.07+dfsg-1ubuntu8 [164 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports focal/main armhf libudev1 armhf 244.1-0ubuntu2 [69.1 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports focal/main armhf systemd-sysv armhf 244.1-0ubuntu2 [9364 B]
Get:8 http://ports.ubuntu.com/ubuntu-ports focal/main armhf libpam-systemd armhf 244.1-0ubuntu2 [118 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports focal/main armhf systemd armhf 244.1-0ubuntu2 [3533 kB]
...
Setting up python3-more-itertools (4.2.0-1build1) ...
Setting up pci.ids (0.0~2020.01.25-1) ...
Setting up u-boot-rpi:armhf (2019.07+dfsg-1ubuntu8) ...
Setting up linux-raspi2-headers-5.3.0-1017 (5.3.0-1017.19) ...
Setting up libmagic-mgc (1:5.38-4) ...
Setting up libssl1.1:armhf (1.1.1d-2ubuntu3) ...
Setting up binutils-common:armhf (2.33.90.20200122-1ubuntu2) ...
Setting up libmagic1:armhf (1:5.38-4) ...
Setting up libctf-nobfd0:armhf (2.33.90.20200122-1ubuntu2) ...
Setting up python3-yaml (5.3-1ubuntu1) ...
Setting up liblzo2-2:armhf (2.10-2) ...
Setting up systemd (244.1-0ubuntu2) ...
Setting up libcom-err2:armhf (1.45.3-4ubuntu3) ...
Setting up file (1:5.38-4) ...
Setting up libassuan0:armhf (2.5.3-7ubuntu2) ...
Setting up python3-zipp (1.0.0-1) ...
Setting up landscape-common (19.12-0ubuntu3) ...
Setting up libisc-export1104:armhf (1:9.11.14+dfsg-3ubuntu1) ...
Setting up linux-modules-5.3.0-1017-raspi2 (5.3.0-1017.19) ...
Setting up python3-cryptography (2.6.1-4ubuntu1) ...
Setting up liblwres161:armhf (1:9.11.14+dfsg-3ubuntu1) ...
Setting up libfribidi0:armhf (1.0.8-2) ...
Setting up linux-headers-5.3.0-1017-raspi2 (5.3.0-1017.19) ...
Setting up udev (244.1-0ubuntu2) ...
update-initramfs: deferring update (trigger activated)
Setting up libss2:armhf (1.45.3-4ubuntu3) ...
Setting up libpcsclite1:armhf (1.8.26-2ubuntu2) ...
Setting up logsave (1.45.3-4ubuntu3) ...
Setting up libssh-4:armhf (0.9.3-2ubuntu1) ...
Setting up libpci3:armhf (1:3.6.4-1) ...
Setting up libdevmapper1.02.1:armhf (2:1.02.155-2ubuntu8) ...
Setting up python3-pyrsistent:armhf (0.15.5-1) ...
Setting up dmsetup (2:1.02.155-2ubuntu8) ...
update-initramfs: deferring update (trigger activated)
Setting up python3-json-pointer (2.0-0ubuntu1) ...
Setting up libgeoip1:armhf (1.6.12-6) ...
Setting up isc-dhcp-common (4.4.1-2ubuntu6) ...
Setting up libdns-export1107 (1:9.11.14+dfsg-3ubuntu1) ... [60/476]
Setting up glib-networking-common (2.63.3-0build1) ...
Setting up libbinutils:armhf (2.33.90.20200122-1ubuntu2) ...
Setting up geoip-database (20191224-2) ...
Setting up openssl (1.1.1d-2ubuntu3) ...
Setting up python3-lib2to3 (3.8.0-1) ...
Setting up libelf1:armhf (0.176-1.1build1) ...
Setting up libxml2:armhf (2.9.4+dfsg1-8ubuntu3) ...
Setting up info (6.7.0.dfsg.2-5) ...
Setting up python3-nacl (1.3.0-4) ...
Setting up mdadm (4.1-5ubuntu1) ...
update-initramfs: deferring update (trigger activated)
mdcheck_continue.timer is a disabled or a static unit not running, not starting it.
Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ...
Setting up libctf0:armhf (2.33.90.20200122-1ubuntu2) ...
Setting up python3-distutils (3.8.0-1) ...
Setting up glib-networking-services (2.63.3-0build1) ...
Setting up systemd-sysv (244.1-0ubuntu2) ...
Setting up libisc1104:armhf (1:9.11.14+dfsg-3ubuntu1) ...
Setting up flash-kernel (3.98ubuntu10) ...
Using DTB: bcm2710-rpi-3-b.dtb
Installing /lib/firmware/5.3.0-1017-raspi2/device-tree/bcm2710-rpi-3-b.dtb into /boot/dtbs/5.3.0-1017-raspi2/./bcm2710-rpi-3-b.dtb
Installing new bcm2710-rpi-3-b.dtb.
flash-kernel: deferring update (trigger activated)
Setting up python3-importlib-metadata (1.4.0-1ubuntu1) ...
Setting up libdevmapper-event1.02.1:armhf (2:1.02.155-2ubuntu8) ...
Setting up python3-setuptools (44.0.0-1) ...
Setting up openssh-client (1:8.1p1-5) ...
Setting up isc-dhcp-client (4.4.1-2ubuntu6) ...
Setting up libnss-systemd:armhf (244.1-0ubuntu2) ...
Setting up e2fsprogs (1.45.3-4ubuntu3) ...
update-initramfs: deferring update (trigger activated)
e2scrub_all.service is a disabled or a static unit not running, not starting it.
Setting up libappstream4:armhf (0.12.10-1) ...
Setting up linux-image-5.3.0-1017-raspi2 (5.3.0-1017.19) ...
I: /boot/vmlinuz is now a symlink to vmlinuz-5.3.0-1017-raspi2
I: /boot/initrd.img is now a symlink to initrd.img-5.3.0-1017-raspi2
Setting up linux-headers-raspi2 (5.3.0.1017.14) ...
Setting up python3-jsonschema (3.2.0-0ubuntu2) ...
Setting up pciutils (1:3.6.4-1) ...
Setting up linux-image-raspi2 (5.3.0.1017.14) ...
Setting up libpam-systemd:armhf (244.1-0ubuntu2) ...
Setting up libisccc161:armhf (1:9.11.14+dfsg-3ubuntu1) ...
Setting up libdns1107:armhf (1:9.11.14+dfsg-3ubuntu1) ...
Setting up libisccfg163:armhf (1:9.11.14+dfsg-3ubuntu1) ...
Setting up glib-networking:armhf (2.63.3-0build1) ...
Setting up binutils-arm-linux-gnueabihf (2.33.90.20200122-1ubuntu2) ...
Setting up openssh-sftp-server (1:8.1p1-5) ...
Setting up openssh-server (1:8.1p1-5) ...
rescue-ssh.target is a disabled or a static unit, not starting it.
Setting up ubuntu-minimal (1.443) ... [10/476]
Setting up libbind9-161:armhf (1:9.11.14+dfsg-3ubuntu1) ...
Setting up binutils (2.33.90.20200122-1ubuntu2) ...
Setting up libirs161:armhf (1:9.11.14+dfsg-3ubuntu1) ...
Setting up linux-raspi2 (5.3.0.1017.14) ...
Setting up u-boot-tools (2019.07+dfsg-1ubuntu8) ...
Setting up bind9-host (1:9.11.14+dfsg-3ubuntu1) ...
Setting up dnsutils (1:9.11.14+dfsg-3ubuntu1) ...
Setting up ubuntu-standard (1.443) ...
Setting up liblvm2cmd2.03:armhf (2.03.02-2ubuntu8) ...
Setting up dmeventd (2:1.02.155-2ubuntu8) ...
dm-event.service is a disabled or a static unit not running, not starting it.
Setting up lvm2 (2.03.02-2ubuntu8) ...
update-initramfs: deferring update (trigger activated)
Setting up ubuntu-server (1.443) ...
Processing triggers for install-info (6.7.0.dfsg.2-5) ...
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for libglib2.0-0:armhf (2.63.3-1) ...
Processing triggers for libc-bin (2.30-0ubuntu3) ...
Processing triggers for ufw (0.36-1ubuntu6) ...
Processing triggers for man-db (2.9.0-2) ...
Processing triggers for dbus (1.12.16-2ubuntu2) ...
Processing triggers for initramfs-tools (0.133ubuntu12) ...
update-initramfs: Generating /boot/initrd.img-5.3.0-1015-raspi2
Using DTB: bcm2710-rpi-3-b.dtb
Installing /lib/firmware/5.3.0-1015-raspi2/device-tree/bcm2710-rpi-3-b.dtb into /boot/dtbs/5.3.0-1015-raspi2/./bcm2710-rpi-3-b.dtb
Installing new bcm2710-rpi-3-b.dtb.
Ignoring old or unknown version 5.3.0-1015-raspi2 (latest is 5.3.0-1017-raspi2)
Processing triggers for flash-kernel (3.98ubuntu10) ...
Using DTB: bcm2710-rpi-3-b.dtb
Installing /lib/firmware/5.3.0-1017-raspi2/device-tree/bcm2710-rpi-3-b.dtb into /boot/dtbs/5.3.0-1017-raspi2/./bcm2710-rpi-3-b.dtb
Taking backup of bcm2710-rpi-3-b.dtb.
Installing new bcm2710-rpi-3-b.dtb.
dpkg: error processing package flash-kernel (--configure):
 installed flash-kernel package post-installation script subprocess returned error exit status 2
Processing triggers for linux-image-5.3.0-1017-raspi2 (5.3.0-1017.19) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.3.0-1017-raspi2
Using DTB: bcm2710-rpi-3-b.dtb
Installing /lib/firmware/5.3.0-1017-raspi2/device-tree/bcm2710-rpi-3-b.dtb into /boot/dtbs/5.3.0-1017-raspi2/./bcm2710-rpi-3-b.dtb
Taking backup of bcm2710-rpi-3-b.dtb.
Installing new bcm2710-rpi-3-b.dtb.
Installing /lib/firmware/5.3.0-1017-raspi2/device-tree/bcm2710-rpi-3-b.dtb into /boot/dtbs/5.3.0-1017-raspi2/./bcm2710-rpi-3-b.dtb
Taking backup of bcm2710-rpi-3-b.dtb.
Installing new bcm2710-rpi-3-b.dtb.
flash-kernel: deferring update (trigger activated)
/etc/kernel/postinst.d/zz-flash-kernel:
Using DTB: bcm2710-rpi-3-b.dtb
Installing /lib/firmware/5.3.0-1017-raspi2/device-tree/bcm2710-rpi-3-b.dtb into /boot/dtbs/5.3.0-1017-raspi2/./bcm2710-rpi-3-b.dtb
Taking backup of bcm2710-rpi-3-b.dtb.
Installing new bcm2710-rpi-3-b.dtb.
Installing /lib/firmware/5.3.0-1017-raspi2/device-tree/bcm2710-rpi-3-b.dtb into /boot/dtbs/5.3.0-1017-raspi2/./bcm2710-rpi-3-b.dtb
Taking backup of bcm2710-rpi-3-b.dtb.
Installing new bcm2710-rpi-3-b.dtb.
flash-kernel: deferring update (trigger activated)
Errors were encountered while processing:
 flash-kernel
E: Sub-process /usr/bin/dpkg returned an error code (1)

This is caused by code under line 832 of "functions" (https://git.launchpad.net/ubuntu/+source/flash-kernel/tree/functions?id=b2fbda45824cfdd2950349979b1cfda52fa86d9d#n832) checking for an apparent inconsistency in /boot: when one or both of the kernel image or initrd for the expected version don't exist. In the content of /tmp/flash-kernel-no-kernel-error.log was the following output:

+ ls -l /boot/vmlinuz-5.3.0-1017-raspi2
-rw------- 1 root root 7430656 Jan 15 21:09 /boot/vmlinuz-5.3.0-1017-raspi2
+ ls -l /boot/initrd.img-5.3.0-1017-raspi2*
ls: cannot access '/boot/initrd.img-5.3.0-1017-raspi2*': No such file or directory

A subsequent run of "apt install -f" fixed things because, by that point, the kernel's own trigger had executed, ensuring that update-initramfs had been run. In the case that f-k is run "prematurely" and finds itself in this situation (/boot/kernel-$[ver} exists, but /boot/initrd-${ver}) doesn't), it should probably bail out silently under the assumption that whatever is responsible for it will rectify the situation and trigger f-k again (as happens in the kernel postinst hooks).

description: updated
Changed in flash-kernel (Ubuntu Bionic):
importance: Undecided → Critical
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package flash-kernel - 3.98ubuntu11

---------------
flash-kernel (3.98ubuntu11) focal; urgency=medium

  * Permit flash-kernel to workaround an inconsistent /boot directory when
    triggered prior to the kernel finishing its triggers (LP: #1861292)

 -- Dave Jones <email address hidden> Wed, 29 Jan 2020 16:15:05 +0000

Changed in flash-kernel (Ubuntu):
status: New → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Dave, or anyone else affected,

Accepted flash-kernel into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/flash-kernel/3.98ubuntu11~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in flash-kernel (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Dave Jones (waveform)
description: updated
Dave Jones (waveform)
description: updated
Revision history for this message
Paul Larson (pwlars) wrote :

I installed 18.04.3 (both armhf and arm64) on an rpi3b+, enabled proposed, and did a dist-upgrade to try this fix. The flash-kernel version it updated to is 3.98ubuntu11~18.04.1.

Both updated successfully, and there was no log written to /tmp/flash-kernel-no-kernel-error.log

Paul Larson (pwlars)
tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package flash-kernel - 3.98ubuntu11~18.04.1

---------------
flash-kernel (3.98ubuntu11~18.04.1) bionic; urgency=medium

  * Permit flash-kernel to workaround an inconsistent /boot directory when
    triggered prior to the kernel finishing its triggers (LP: #1861292)

flash-kernel (3.98ubuntu10~18.04.1) bionic; urgency=medium

  * SRU of 3.98ubuntu10 to support Raspberry Pi 4B. (LP: #1847587)
  * Removed new entries from db/all.db for machines that were not already
    supported on Bionic
  * Remove "quiet splash" from kernel command line options; Bionic shipped
    without these options and we shouldn't change the boot experience for
    existing users
  * Ensure the "overlays" dir always exists on the boot partition; the Bionic
    image for the Pi 2 shipped without this dir

 -- Dave Jones <email address hidden> Wed, 29 Jan 2020 16:15:05 +0000

Changed in flash-kernel (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for flash-kernel has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.