grub-efi-amd64 from grub2-unsigned has lost kernel/postinst.d script

The bug description here is all wrong. grub-efi-amd64-signed depends on grub-pc | grub-efi-amd64; you never have grub-efi-amd64-signed without one of these. The regression is because grub-efi-amd64 from grub2-unsigned has dropped the conffiles, expecting them to be provided by the grub2-common which it depends on; but we still have the old grub2-common from grub2 source which does not ship these conffiles.

We must either have grub-efi-amd64 reintroduce the conffiles in stable series, or add them to grub2-common.

And it's easier to reintroduce the conffiles to grub-efi-amd64.

How can you introduce conffiles in grub-efi-amd64 & grub-efi-arm64 which is shared across releases? If in later series they have been removed from said package. That will cause a mess in focal+ then, since it will conflict with grub2-common there.

Given that the future is for these conffiles to live in grub2-common, it might be easier to backport the move from grub-{platform} to grub2-common.

This requires uploads of grub2 and grub2-{un,}signed to focal to bump the versioned number in the Replaces (grub2-common) and ensure upgrades from bionic to focal get a newer grub-efi-* with the correct lack of conffiles.

> How can you introduce conffiles in grub-efi-amd64 & grub-efi-arm64 which is shared across releases?

By not sharing it across releases.

> That will cause a mess in focal+ then, since it will conflict with grub2-common there.

Not with separate uploads to bionic and focal.

Which is less of a mess than having grub2-common in < focal having to declare a Replaces: on the versions of the grub binaries from grub2 source shipped in those releases.

technically, the replaces in groovy and hirsute are incomplete; however, we don't support direct upgrades from bionic to any release later than focal, so this is not worth SRUing.

Hello Dimitri, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dimitri, or anyone else affected,

Accepted grub2-unsigned into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.04-1ubuntu44.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dimitri, or anyone else affected,

Accepted grub2-signed into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.167.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dimitri, or anyone else affected,

Accepted grub2-unsigned into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.04-1ubuntu44.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dimitri, or anyone else affected,

Accepted grub2-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.167~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

This SRU is inappropriate. grub2-unsigned must be built in hirsute or impish.

Because the regression potential seems to large to me to suddenly rebuild grub 2.04 with a much older toolchain. Part of the reason for doing this is to build it with one toolchain, have one set of binaries. We've seen arm64 shims not boot when compiled with pre-hirsute toolchain, this is a risky move that would require extreme carefulness and testing effort.

This also makes validation of grub changes substantially harder, as everything now needs to be verified 4 times (don't forget xenial ESM), and triples the number of binaries we have to sign, which is terrible.

There's a reason we went with a binary copy in the first place, instead of building grub2 in each release.

Steve I'm not sure one gets valid grub2 binaries by building with bionic's toolchain on neither amd64 nor arm64.

Hello Dimitri, or anyone else affected,

Accepted grub2-unsigned into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.04-1ubuntu44.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dimitri, or anyone else affected,

Accepted grub2-signed into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.167.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dimitri, or anyone else affected,

Accepted grub2-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.167~18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dimitri, or anyone else affected,

Accepted grub2-signed into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.167~16.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dimitri, or anyone else affected,

Accepted grub2-unsigned into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.04-1ubuntu44.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dimitri, or anyone else affected,

Accepted grub2-signed into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.167.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Bionic: server install from ubuntu-18.04.5-live-server-amd64.iso. Security updates applied by subiquity, including grub-efi-amd64 2.04-1ubuntu44. /etc/kernel/post{inst,rm}.d/zz-update-grub are not present (to my surprise - then I worked out this was because grub wasn't installed in the source squashfs, so this was a new install of the package from -security rather than an upgrade).

grub-pc is not installed.

Enabled -proposed in sources.list.

sudo apt install grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed

/etc/kernel/post{inst,rm}.d/zz-update-grub now exist as expected

sudo apt dist-upgrade to install 4.15.0-144-generic kernel from bionic-proposed (along with other random bits)

Inspecting /boot/grub/grub.cfg shows references to -144-.

Rebooting, and uname shows -144- is booted.

Disabled -proposed again.

sudo do-release-upgrade to focal, to my surprise, does not fail with conffile conflicts when -proposed is disabled. However, it does fail to upgrade grub-efi-amd64* due to focal-updates having an earlier version than bionic-proposed.

$ dpkg -S /etc/kernel/postinst.d/zz-update-grub
grub2-common: /etc/kernel/postinst.d/zz-update-grub
$

I do not understand what's happening here with the silent takeover of the conffiles without an explicit Replaces allowing this; this is not how I understood dpkg to behave.

I think this is verification-done for bionic and verification-failed for grub2 in focal (which should be withdrawn).
I do not understand what's happening with the conffile being

Xenial: server install from ubuntu-16.04.7-server-amd64.iso. Security updates applied by d-i, including grub-efi-amd64 2.04-1ubuntu44. /etc/kernel/post{inst,rm}.d/zz-update-grub are not present, as with bionic.

grub-pc is not installed.

Enabled -proposed in sources.list.

sudo apt install grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed

/etc/kernel/post{inst,rm}.d/zz-update-grub now exist as expected

sudo apt dist-upgrade to install 4.4.0-210-generic kernel from xenial-proposed (along with other random bits)

Inspecting /boot/grub/grub.cfg shows references to -210-.

Rebooting, and uname shows -210- is booted.

I think we discussed this yesterday that we need to properly test this. A smoke test on amd64 is not adequate, we also need to ensure additionally that

- there are no regressions on arm64
  install update in canonistack server and reboot it and make sure it boots.

- there are no regressions in (maas) netbooting.
  I don't know how to verify nicely. I have a script you can point at shim and grub efi binaries to
  boot them similar to MAAS, but there might be more regressions, I clearly can't cover them all :)

Due to the toolchain being different in the rebuilds happening in bionic/focal instead of hirsute, with much older binutils, and missing patches in gcc-8 likely. Because we know that at least shim did not work with old toolchain at all, so are a bit scared :D

FWIW, I'm doing canonistack arm64 testing now, by upgrading the image, then upgrading to proposed.

So xenial: I verified that maas-style netbooting that chainloads to local grub still works.

On arm64 canonistack, both the version in -updates and the version in -proposed fail to boot.

After upgrade, the first boot gives us:

EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

The second boot with updates gave us:

[2J[01;01H[=3h[2J[01;01H[2J[01;01H[=3h[2J[01;01H[2J[01;01H[=3h[2J[01;01H[0m[35m[40m[0m[37m[40merror: symbol `grub_file_filters' not found.
Entering rescue mode...
grub rescue>

The second boot with proposed gave us:

[2J[01;01H[=3h[2J[01;01H[2J[01;01H[=3h[2J[01;01H[2J[01;01H[=3h[2J[01;01H[0m[35m[40m[0m[37m[40merror: relocation 0x113 is not implemented yet.
Entering rescue mode...
grub rescue>

So there is some regression in terms of relocations due to the toolchain change, but it did not boot before either, so I'm unsure if that's a worthwhile regression to block the update.

bionic is good, I could do maas-style chained netbooting, as well as boot arm64 cloud instance without errors.

Since focal and groovy are no-change uploads, it is sufficient to smoke test these by showing that they still boot.

I've done this for both focal and groovy on amd64 and the systems still boot.

We figured out the root causes for the failure on the cloud, and it is not a regression of this update compared to the one in -updates, so I'm marking xenial as verified again.

What is happening on the cloud is that the cloud images ship with the grub "loader" installed into
/boot/efi/EFI/BOOT/BOOTAA64.EFI, which loads the modules and core image from /boot/grub. On upgrade, we install new modules to /boot/grub and the loader to /boot/efi/EFI/ubuntu/grubaa64.efi - we do not replace the removable path one.

We do register the new grub in the boot menu, but the cloud does not seem to persist that - the entry is gone in the next boot (or the 2nd boot?). There are some KVM crashes in the cloud due to
https://github.com/qemu/qemu/commit/089fd80376196adc0274a53eb9729c3ef7ee5ae7 which might be related and corrupt the variable storage, I can't say.

So when we boot the next time, instead of loading the new grubaa64.efi, it will load the BOOTAA64.EFI from 2.02 instead, which then gets confused by the 2.04 modules.

We can fix this by passing --force-extra-removable to grub-install, which is supported in xenial. I believe the maintainer scripts are messed up a bit. I can set force_efi_extra_removable, but the maintainer script checks for no_efi_extra_removable. I reconfigured grub-efi-arm64 to set the latter to true, but the former is still true as well, so we don't pass the argument. Something went wrong in migrating this setting presumably.

# Skip extra installation to the EFI removable media path?
grub-efi-arm64 grub2/no_efi_extra_removable boolean true
# Force extra installation to the EFI removable media path?
grub-efi-arm64 grub2/force_efi_extra_removable boolean true

Although amd64 cloud images don't fail to boot after the upgrade, we've determined that the same fundamental problem - that the contents of the removable path are not updated on grub upgrade - also affects amd64/xenial. This means that all xenial cloud images, whether the update is installed or not, will not be *booting* the binary that has the security fixes.

So we need to fix this for both amd64 and arm64 in xenial by having it --force-extra-removable unconditionally on upgrade.

And: agreed that the current bug is verified for xenial now.

This bug was fixed in the package grub2-unsigned - 2.04-1ubuntu44.1

---------------
grub2-unsigned (2.04-1ubuntu44.1) bionic; urgency=medium

  * debian/install.in: add kernel hooks back to grub-efi-amd64 package since
    grub2-common in older releases does not include it. LP: #1928674.

 -- Steve Langasek <email address hidden> Wed, 19 May 2021 16:31:18 -0700

This bug was fixed in the package grub2-signed - 1.167~16.04.4

---------------
grub2-signed (1.167~16.04.4) xenial; urgency=medium

  * Rebuild against grub2 2.04-1ubuntu44.1. LP: #1928674.
  * key on grub-efi-$(DEB_HOST_ARCH) as the binary package for
    download-signed since grub-efi-* and grub2-common are now built from
    separate sources.

 -- Steve Langasek <email address hidden> Wed, 19 May 2021 23:01:40 -0700

The verification of the Stable Release Update for grub2-unsigned has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package grub2-unsigned - 2.04-1ubuntu44.1

---------------
grub2-unsigned (2.04-1ubuntu44.1) bionic; urgency=medium

  * debian/install.in: add kernel hooks back to grub-efi-amd64 package since
    grub2-common in older releases does not include it. LP: #1928674.

 -- Steve Langasek <email address hidden> Wed, 19 May 2021 16:31:18 -0700

This bug was fixed in the package grub2-signed - 1.167~18.04.3

---------------
grub2-signed (1.167~18.04.3) bionic; urgency=medium

  * Rebuild against grub2 2.04-1ubuntu44.1. LP: #1928674.
  * key on grub-efi-$(DEB_HOST_ARCH) as the binary package for
    download-signed since grub-efi-* and grub2-common are now built from
    separate sources.

 -- Steve Langasek <email address hidden> Thu, 20 May 2021 11:48:10 -0700

This bug was fixed in the package grub2 - 2.04-1ubuntu26.12

---------------
grub2 (2.04-1ubuntu26.12) focal; urgency=medium

  * Bump the version number in the replaces for grub-efi-* to account for
    newer packages in bionic from grub2-unsigned shipping the kernel hook
    conffiles. LP: #1928674.

 -- Steve Langasek <email address hidden> Wed, 19 May 2021 22:50:50 -0700

This bug was fixed in the package grub2-unsigned - 2.04-1ubuntu44.2

---------------
grub2-unsigned (2.04-1ubuntu44.2) focal; urgency=medium

  * No-change rebuild to ensure clean upgrade from bionic. LP: #1928674.

 -- Steve Langasek <email address hidden> Thu, 20 May 2021 00:51:07 +0000

This bug was fixed in the package grub2-signed - 1.167.2

---------------
grub2-signed (1.167.2) focal; urgency=medium

  * Rebuild against grub2 2.04-1ubuntu44.2. LP: #1928674.
  * key on grub-efi-$(DEB_HOST_ARCH) as the binary package for
    download-signed since grub-efi-* and grub2-common are now built from
    separate sources.

 -- Steve Langasek <email address hidden> Wed, 19 May 2021 22:54:30 -0700

This bug was fixed in the package grub2-unsigned - 2.04-1ubuntu44.2

---------------
grub2-unsigned (2.04-1ubuntu44.2) focal; urgency=medium

  * No-change rebuild to ensure clean upgrade from bionic. LP: #1928674.

 -- Steve Langasek <email address hidden> Thu, 20 May 2021 00:51:07 +0000

This bug was fixed in the package grub2-signed - 1.167.2

---------------
grub2-signed (1.167.2) focal; urgency=medium

  * Rebuild against grub2 2.04-1ubuntu44.2. LP: #1928674.
  * key on grub-efi-$(DEB_HOST_ARCH) as the binary package for
    download-signed since grub-efi-* and grub2-common are now built from
    separate sources.

 -- Steve Langasek <email address hidden> Wed, 19 May 2021 22:54:30 -0700

Status changed to 'Confirmed' because the bug affects multiple users.