recordfail false positive causes headless servers to hang on boot by default

There's an existing grub SRU in flight (2.02~beta2-9ubuntu1.2 in Trusty) so I'm deferring upload for all releases until it clears. These diffs are build tested only.

This bug was fixed in the package grub2 - 2.02~beta2-23

---------------
grub2 (2.02~beta2-23) unstable; urgency=medium

  [ Debconf translations ]
  * [da] Danish (Joe Dalton; closes: #781333).

  [ Felix Zielcke ]
  * Run the tests with LC_MESSAGES=C.UTF-8. Some tests fail with non
    english locale. (Closes: #782580)

  [ Mathieu Trudel-Lapierre ]
  * Backport from upstream:
    - arp, icmp: Fix handling in case of oversized or invalid packets.
      (LP: #1428005)

  [ Robie Basak ]
  * Change the default GRUB_RECORDFAIL_TIMEOUT to 30, so interactive users
    still get the opporunity to intervene after a real boot failure, but
    headless users will not end up stuck after boot failures that were
    really power failures (closes: #782552, LP: #1443735).

 -- Colin Watson <email address hidden> Thu, 14 May 2015 16:18:33 +0100

Uploaded; now awaiting SRU team review.

Hello Robie, or anyone else affected,

Accepted grub2 into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-22ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Robie, or anyone else affected,

Accepted grub2 into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-15ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Robie, or anyone else affected,

Accepted grub2 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-9ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Robie, or anyone else affected,

Accepted grub2 into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/1.99-21ubuntu3.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Robie, or anyone else affected,

Accepted grub2-signed into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.38.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Robie, or anyone else affected,

Accepted grub2-signed into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.46.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Robie, or anyone else affected,

Accepted grub2-signed into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.9~ubuntu12.04.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Robie, or anyone else affected,

Accepted grub2-signed into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.34.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I verified the fix works for vivid (2.02~beta2-22ubuntu1.1), trusty (2.02~beta2-9ubuntu1.3) and precise (1.99-21ubuntu3.18).

This bug was fixed in the package grub2 - 2.02~beta2-9ubuntu1.3

---------------
grub2 (2.02~beta2-9ubuntu1.3) trusty; urgency=medium

  * Do not hang headless servers indefinitely on boot after edge case power
    failure timing (LP: #1443735). Instead, time out after 30 seconds and boot
    anyway, including on non-headless systems.

 -- Robie Basak <email address hidden> Tue, 19 May 2015 13:31:03 +0100

The verification of the Stable Release Update for grub2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package grub2 - 2.02~beta2-22ubuntu1.1

---------------
grub2 (2.02~beta2-22ubuntu1.1) vivid; urgency=medium

  * Do not hang headless servers indefinitely on boot after edge case power
    failure timing (LP: #1443735). Instead, time out after 30 seconds and boot
    anyway, including on non-headless systems.

 -- Robie Basak <email address hidden> Tue, 19 May 2015 13:47:00 +0100

This bug was fixed in the package grub2 - 1.99-21ubuntu3.18

---------------
grub2 (1.99-21ubuntu3.18) precise; urgency=medium

  * Do not hang headless servers indefinitely on boot after edge case power
    failure timing (LP: #1443735). Instead, time out after 30 seconds and boot
    anyway, including on non-headless systems.

 -- Robie Basak <email address hidden> Tue, 19 May 2015 12:22:34 +0100

This bug was fixed in the package grub2-signed - 1.9~ubuntu12.04.9

---------------
grub2-signed (1.9~ubuntu12.04.9) precise; urgency=medium

  * Rebuild against grub-efi-amd64 1.99-21ubuntu3.18 (LP: #1443735).

 -- Robie Basak <email address hidden> Fri, 03 Jul 2015 10:19:22 +0000

This bug was fixed in the package grub2-signed - 1.34.4

---------------
grub2-signed (1.34.4) trusty; urgency=medium

  * Rebuild against grub-efi-amd64 2.02~beta2-9ubuntu1.3 (LP: #1443735).

 -- Robie Basak <email address hidden> Fri, 03 Jul 2015 10:19:28 +0000

This bug was fixed in the package grub2-signed - 1.38.1

---------------
grub2-signed (1.38.1) utopic; urgency=medium

  * Rebuild against grub-efi-amd64 2.02~beta2-15ubuntu0.1 (LP:
    #1443735).

 -- Robie Basak <email address hidden> Fri, 03 Jul 2015 10:19:32 +0000

This bug was fixed in the package grub2-signed - 1.46.1

---------------
grub2-signed (1.46.1) vivid; urgency=medium

  * Rebuild against grub-efi-amd64 2.02~beta2-22ubuntu1.1 (LP:
    #1443735).

 -- Robie Basak <email address hidden> Fri, 03 Jul 2015 10:19:37 +0000

This bug was fixed in the package grub2 - 2.02~beta2-15ubuntu0.1

---------------
grub2 (2.02~beta2-15ubuntu0.1) utopic; urgency=medium

  * Do not hang headless servers indefinitely on boot after edge case power
    failure timing (LP: #1443735). Instead, time out after 30 seconds and boot
    anyway, including on non-headless systems.

 -- Robie Basak <email address hidden> Tue, 19 May 2015 13:44:35 +0100

I just tried grub2 (2.02~beta2-15ubuntu0.1) on Utopic and the fix works as intended.

Something you may wish to consider in relation to this fix:

Proposal 1)
I noticed the bug description already contains the comment "Maybe a future addition to recognizing $recordfail is to have a warning on the boot menu, but that is outside the scope of this report.". I +1 this for some future update.

Proposal 2) - might be deemed to be low enough risk to implement prior to dev/test/release of proposal 1.
Add the line:
GRUB_RECORDFAIL_TIMEOUT=30
to the default /etc/default/grub file

Context:
I arrived here while investigating a hibernate/resume issue with Ubuntu 15.04 on a Acer C710 (parrot) Chromebook.

Although it is possible to make hibernate/resume functional on this device, the hibernate action seems to cause recordfail to be set. As a consequence, I see the grub menu & 30 second countdown on resume. (It's true that prior to this fix I saw the grub menu with no countdown, which was even more perplexing - so thank you :)

There were no settings in /etc/default/grub that had a value of 30, so it wasn't immediately obvious to me what was happening.

Implementing one of these proposals may avert support questions from laptop users, who experience what might seem an unconfigured / unrequested 30 second delay by grub to laptop resume. OTOH: maybe this configuration is such an edge case it is preferable to do nothing at all

In case someone else arrives here on the same journey, I'll link https://gist.github.com/ewann/b0aeedf57bd6879946c6 for related Acer C710 issues.

Ewan,

Thank you for your comments.

> I +1 this for some future update.

I don't think anyone is working on this at the moment, or that there is any plan to update stable releases with it. Please feel free to file a separate bug to track it.

> Add the line:
GRUB_RECORDFAIL_TIMEOUT=30
to the default /etc/default/grub file

We deliberately didn't do this is as will cause a prompt requiring user intervention on update for any user who has modified /etc/default/grub - both for the update during an existing stable release's lifetime and during a release upgrade.

As far as I understand, the recordfail path should only get followed after a boot failure. If you're also getting it at other times, then please file a bug for that separate issue.

Robie,

> We deliberately didn't do this is as will cause a prompt
I suppose another way would be to conditionally append the value via a postinst script, but I further suppose you declined to take this approach because it lacks elegance.

> As far as I understand, the recordfail path should only get followed after a boot failure.
I opened https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475620 - I agree the issue should get corrected at source, if possible.

> Please feel free to file a separate bug
I'll see first the outcome of 1475620.

thanks for your feedback

On Fri, Jul 17, 2015 at 12:02:17PM -0000, Ewan wrote:
> Robie,
>
> > We deliberately didn't do this is as will cause a prompt
> I suppose another way would be to conditionally append the value via a
> postinst script, but I further suppose you declined to take this
> approach because it lacks elegance.

It would cause an unnecessary conffile prompt on the next conffile
update following that, since it would appear to be a user-made change.

That's why the best thing to do is to leave files in /etc/ for only
users to change directly, rather than have maintainer scripts change
them. We do use .d/ directories to try and help with this, but I don't
think it'd help in this case as it won't solve the user confusion issue
you're trying to solve.

> > As far as I understand, the recordfail path should only get followed after a boot failure.
> I opened https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475620
> - I agree the issue should get corrected at source, if possible.

Thank you for filing this. I've moved it to grub2 as I'm pretty sure
it's the grub2 packaging that arranges the recordfail test (as opposed
to kernel code or packaging).