grub efi doesn't install fs module needed to access root

Status changed to 'Confirmed' because the bug affects multiple users.

In analyzing this over in Bug 1731490; grub-install does not check the filesystem of /boot; even if it did, it still needs to do the follwing:

1) copying the proper modules into /boot/efi/EFI/ubuntu/x86_64-efi/<module>
2) updating /boot/efi/EFI/ubuntu/grub.cfg to insmod the required modules for the the search.fs_uuid to locate root/boot

It's not that simple. The module should have been copied if xfs was detected as the filesystem for /boot... But there's more to it than that. In UEFI, you also need a signed grub binary that already includes the module, otherwise Secure Boot might prohibit you loading it from disk.

This is certainly not a Critical bug as it is only as issue for a fraction of our users who happen to be at the intersection of using XFS and UEFI, but at least I think we have enough information to be able to work on the fix.

Same here. I had to write myself a google drive doc that I could consult every time a the grub-efi package was updated and broke my boot again.

In addition to the steps explained previously, I also copy my working manually edited grub.cfg to emergency.cfg, so that every time I get a failed boot again after a system update, I pull up google drive on my phone to read this emergecy manual boot command:

grub> configfile ${cmdpath}/emergency.cfg

That one command is all you need to enter manually when you are faced with the grub> prompt at boot, and it loads the rest of your normal grub config, and you boot normally.

Then once booted, copy emergency.cfg back over grub.cfg, and update the copy of xfs.mod since it might have been updated by the very update that regnerated grub.cfg.

https://drive.google.com/open?id=1B2FrtIaG75PhFPsjzXnJS0MeDg7rezEP3Ef5fFlrLAo

This definitively affects 18.04 as well. The main culprit is that the /boot/efi/EFI/ubuntu/grubx64.efi binary in 18.04 lacks XFS support.

On 19.04 I still get my grub.cfg broken by normal updates. Not by kernel updates, and not by running update-grub. But there is some package that whenever that package is updated, part of the update process generates a new grub.cfg, and it's a broken one because it makes a bad assumption somewhere.

I eneded up writing myself a little "check_grub_efi_xfs" script that I run any time I think I saw anything about grub fly by when I run "apt upgrade", or, any time grub gets screwed up I run it after I get myself back in.

It's only about 3 lines of code that actually matters, but writes a lot of stuff to the screen just so you don't have ANY worries about it what it did and if your boot files are currently ok before you shut down.

Ideally this should run just before shutdown on every single shutdown. Then you'd never have to worry about it. The only way you'd miss an update again is if you had an ungraceful shutdown after an apt upgrade that included a grub packege update. But despite being a unix and linux admin for over 20 years, I have so far utterly failed to get systemd to do such a simple task. If someone can show me exactly how to write a freaking unit file that does that, I'd be pretty grateful.

Just to refresh, the issue here is, it's possible to install ubuntu with xfs root fs to a machine that already has a factory Win10 install and boot files in a factory EFI partition, without having to make any special /boot partition. All you really need is to place a copy of xfs.mod onto the EFI partition, which is a FAT filesystem that grub already reads files from at boot. And edit the initial grub.cfg that's in the EFI filesystem so that it loads xfs.mod from there too.

Then everything else is fine. The rest of /boot can be in "/", and "/" can be xfs. This allowed me to degfrag and resize the factory Win10 without wiping and reinstalling, install linux in the freed space, with xfs root fs, and no other partition or filesystem changes needed. I can boot back and forth between the original Win10 install or linux with no problem. It didn't break Win10 booting.

It's a perfectly valid and "legal" arrangement, and anything that automatically writes grub config files should have to allow for this possibility without breaking it.

Screen capture from a run of this fixup script, just after today's apt upgrade which broke my grub.cfg once again.
This is today 2019 02 26, so I don't know where the supposed fix is.
It's definitely not yet in 19.04 or in grub-* 2.02+dfsg1-5ubuntu11

So this run shows the actions when it detects that it actually has something to do.
Most times, it will detect that it doesn't have to do anything, and it will take no action those times.

This, and the script itself (attached) should serve as detailed documentation of the admittedly uncommon configuration, and exactly how it gets broken, and exactly how it gets un-broken.

---snip---
root@negre:~# check_grub_efi_xfs
Checking for broken grub + efi + xfs config...

EFI System Partition ESP_DIR=/boot/efi/EFI/ubuntu
total 3488
-rwx------ 1 root root 108 Aug 14 14:47 BOOTX64.CSV
-rwx------ 1 root root 141 Jan 25 11:13 emergency.cfg
drwx------ 2 root root 4096 Dec 14 ...

Correction to that date I write above: 2019 02 06 not 2019 02 26 blargh

... Ah I see, it does look like possibly the debian upstream might have fixed it, and merely the ubuntu package has not yet got it. Which is what this bug declares at the top, so no discrepency there after all.

They mention signed images, where in my own case I have uefi boot security disabled and can boot unsigned things, so I don't know if that fix will still fix me or not, but seems likely.

This bug was fixed in the package grub2 - 2.02+dfsg1-12ubuntu1

---------------
grub2 (2.02+dfsg1-12ubuntu1) disco; urgency=medium

  * Merge against Debian unstable; remaining changes (LP: #564853):
    - debian/control: Update Vcs fields for code location on Ubuntu.
    - debian/control: Breaks shim (<< 13).
    - Secure Boot support: use newer patchset from rhboot repo:
      - many linuxefi_* patches added and modified
      - dropped debian/patches/linuxefi_require_shim.patch
      - renamed: debian/patches/no_insmod_on_sb.patch ->
        debian/patches/linuxefi_no_insmod_on_sb.patch
    - debian/patches/install_signed.patch, grub-install-extra-removable.patch:
      - Make sure if we install shim; it should also be exported as the default
        bootloader to install later to a removable path, if we do.
      - Rework grub-install-extra-removable.patch to reverse its logic: in the
        default case, install the bootloader to /EFI/BOOT, unless we're trying
        to install on a removable device, or explicitly telling grub *not* to
        do it.
      - Install a BOOT.CSV for fallback to use.
      - Make sure postinst and templates know about the replacement of
        --force-extra-removable with --no-extra-removable.
    - debian/patches/add-an-auto-nvram-option-to-grub-install.patch: Add the
      --auto-nvram option to grub-install for auto-detecting NVRAM availability
      before attempting NVRAM updates.
    - debian/build-efi-images: provide a new grub EFI image which enforces that
      loaded kernels are signed for Secure Boot: build gsb$arch.efi; which is
      the same as grub$arch.efi minus the 'linux' module. Without fallback to
      'linux' for unsigned loading, this makes it effectively enforce having a
      signed kernel.
    - Verify that the current and newer kernels are signed when grub is
      updated, to make sure people do not accidentally shutdown without a
      signed kernel.
    - debian/default/grub: replace GRUB_HIDDEN_* variables with the less
      confusing GRUB_TIMEOUT_STYLE=hidden.
    - debian/patches/support_initrd-less_boot.patch: Added knobs to allow
      non-initrd boot config.
    - Disable os-prober for ppc64el on the PowerNV platform, to reduce the
      number of entries/clutter from other OSes in Petitboot
    - debian/patches/shorter_version_info.patch: Only show the upstream version
      in menu and console, and hide the package one in a package_version
      variable.
    - debian/patches/skip_text_gfxpayload_where_not_supported.patch: Skip the
      'text' payload if it's not supported but present in gfxpayload, such as
      on EFI systems.
    - debian/patches/bufio_sensible_block_sizes.patch: Don't use arbitrary file
      fizes as block sizes in bufio: this avoids potentially seeking back in
      the files unnecessarily, which may require re-open files that cannot be
      seeked into, such as via TFTP.
    - debian/patches/ofnet-init-structs-in-bootpath-parser.patch: initialize
      structs in bootpath parser.
    - debian/rules: shuffle files around for now to keep build artefacts
      for signing at the same location as they were expected by Launchpad.
    - debian/ru...

At least for Xenial, this still fails, the xfs module either not build-in, or not copied.

Hello Jeremiah, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I don't fix grub2 often ... And I missed/forgot changing the build-depends version instruction in grub2-signed to no longer point to 8.20 but 8.21.

I just did the modification, re-uploaded, and let sil2100 knows about my oversight

-Build-Depends: debhelper (>= 9), lsb-release, python3, python3-apt, grub-efi-amd64-bin (>= 2.02-2ubuntu8.20) [amd64], grub-efi-arm64-bin (>= 2.02-2ubuntu8.20) [arm64]
+Build-Depends: debhelper (>= 9), lsb-release, python3, python3-apt, grub-efi-amd64-bin (>= 2.02-2ubuntu8.21) [amd64], grub-efi-arm64-bin (>= 2.02-2ubuntu8.21) [arm64]

Sorry for the inconvenient.

- Eric

[VERIFICATION BIONIC]

I have reproduced the problem with grub2 (2.02-2ubuntu8.20) w/ this layout:

/boot/efi
/ xfs

I then boot with a live cd, mount the partitions, mount --bind {dev, proc, run, sys} and chroot to enable proposed and update grub to version 2.02-2ubuntu8.21 and grub2-signed (1.93.24)

And now everything is working fine, and I can no longer reproduce the problem.

- Eric

[PART 2]

Here some output to confirm my above statement:

* release:
$ lsb_release -cs
bionic

* df:
/dev/sda2 24G 8.0G 17G 34% /
/dev/sda1 511M 6.1M 505M 2% /boot/efi

* /etc/fstab:
# / was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/9eb6f661-542a-45e5-adf9-6d93cdd5500d / xfs defaults 0 0
# /boot/efi was on /dev/sda1 during curtin installation
/dev/disk/by-uuid/6966-6E10 /boot/efi vfat defaults 0 0

* proposed pkgs:
dpkg -l | grep -i grub
ii grub-common 2.02-2ubuntu8.21 amd64 GRand Unified Bootloader (common files)
ii grub-efi-amd64 2.02-2ubuntu8.21 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 version)
ii grub-efi-amd64-bin 2.02-2ubuntu8.21 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 binaries)
ii grub-efi-amd64-signed 1.93.24+2.02-2ubuntu8.21 amd64 GRand Unified Bootloader, version 2 (EFI-AMD64 version, signed)
ii grub-legacy-ec2 1:1 all Handles update-grub for ec2 instances
ii grub2-common 2.02-2ubuntu8.21 amd64 GRand Unified Bootloader (common files for version 2

This bug was fixed in the package grub2 - 2.02-2ubuntu8.21

---------------
grub2 (2.02-2ubuntu8.21) bionic; urgency=medium

  * d/build-efi-images:
    - Add xfs module to signed UEFI images
      (Closes: #911147) (LP: #1652822)

 -- Eric Desrochers <email address hidden> Thu, 07 Jan 2021 08:45:31 -0500

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.