firmware update breaks Ubuntu

Adjusting various tasks for the relevant components. I'm pretty sure this is a regression that is only getting exercised in certain situations of boot entries.

Most notably there was a change was that fwupdate was effectively updated to the "9" release in 0.5-2ubuntu5. This version adds the "Linux Firmware Updater" entry to the boot entry list at the end. If there isn't an "ubuntu" entry earlier on, it's possible that entry is sticking around. The BIOS will only try to build automatic entries if the list was empty, which it's not now.

There are two fixes in upstream for this that may help, but that's predicated on my hypothesis being correct.
https://github.com/rhboot/fwupdate/commit/9bebde2c8b8960d86326f805743edb9293a6d173
https://github.com/rhboot/fwupdate/commit/cd9ea4cbdc93da922ce9abc45c7de63fe252ecc8

My hypothesis can be confirmed by seeing efibootmgr -v output before a failed update. If currently in a failed state, booting a USB key in UEFI mode (without creating a boot entry in BIOS setup) should be able to indicate this too.

Would it make sense to SRU those fixes to 17.10 in any case? The regression might be only in some situations but that's still rather annoying and better to be avoided if we can...

Yeah I agree it's better avoided. It should SRU to 16.04's 05-2ubuntu5 too actually.

fixed in unstable via https://anonscm.debian.org/cgit/uefi/fwupdate.git/commit/?id=7b6d96818db166456bbd9aafb031e7d990e06ef8, should sync to ubuntu for bionic

Adding a fwupdate-signed task since this will require re-spinning all fwup EFI binaries

Hello Merlijn, or anyone else affected,

Accepted fwupdate into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate/9-2ubuntu0.17.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Merlijn, or anyone else affected,

Accepted fwupdate-signed into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate-signed/1.14ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

@Lukasz,
It looks like it's in unapproved for both artful and bionic.

This bug was fixed in the package fwupdate - 9-3

---------------
fwupdate (9-3) unstable; urgency=medium

  * Disable "Linux Firmware Updater" boot entry after installation
    (LP: #1730343)
  * update standards version
  * Turn on hardening flags

 -- Mario Limonciello <email address hidden> Mon, 06 Nov 2017 09:17:16 -0600

Hello Merlijn, or anyone else affected,

Accepted fwupdate into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate/9-1ubuntu0.17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Merlijn, or anyone else affected,

Accepted fwupdate-signed into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate-signed/1.13ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

An upload of fwupdate-signed to xenial-proposed has been rejected from the upload queue for the following reason: "Rejected, ubuntu6 is not available anywhere yet. Please re-upload once ubuntu6 is made available for review.".

Xenial I was running into problems. thanks for rejecting the fwupdate-signed package there until it's ready.

@Lukasz, it seems that fwupdate-signed isn't uploading from the feedback on the upstream issue.

DEBUG Considering changefile 1/ubuntu/fwupdate-signed_1.14ubuntu0.2_amd64.changes
DEBUG Finding fresh policy
INFO Processing upload fwupdate-signed_1.14ubuntu0.2_amd64.changes
INFO Upload was rejected:
INFO fwupdate-amd64-signed_1.14ubuntu0.2+9-2ubuntu0.17.10.1_amd64.deb: Version older than that in the archive. 1.14ubuntu0.2+9-2ubuntu0.17.10.1 <= 1.14+9-2
INFO fwupdate-signed_1.14ubuntu0.2+9-2ubuntu0.17.10.1_amd64.deb: Version older than that in the archive. 1.14ubuntu0.2+9-2ubuntu0.17.10.1 <= 1.14+9-2
INFO Committing the transaction and any mails associated with this upload.

I'm not sure the proper version to give this, considering it being an SRU.
I think you'll need to reject the fwupdate-signed packages from the -proposed in artful and zesty though until we come up with the right version to use.

Any advice?

Maybe 1.14.1 and 1.13.1 (like done in xenial for 1.11)

I've done uploads for 1.14.1 and 1.13.1 for artful and zesty respectively.

I tested fwupdate in artful-proposed and it works and fixes the issue.

(Without secure boot that comment means)

On Tue, Nov 21, 2017, 02:11 Merlijn Sebrechts <email address hidden>
wrote:

> I tested fwupdate in artful-proposed and it works and fixes the issue.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1730343
>
> Title:
> firmware update breaks Ubuntu
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fwupd/+bug/1730343/+subscriptions
>

Can we please get this moving forward. Even more users have reported to have killed their system with this bug, and we can expect many many more who haven't reported it. Why is this still "triaged" for Xenial? Xenial is the current LTS so the bug there affects 20x the amount of users as artful and that's what Dell is currently shipping on their fwupd-enabled laptops.

Just imagine what will happen if Dell releases a new firmware update for one of their laptops... We've been lucky this hasn't happened yet, so the only people running into this bug are people who have upgraded from a version where fwupd wasn't enabled or new users.

It failed for artful.. we're waiting for the signed package to be accepted
again.

On Sun, Nov 26, 2017, 05:52 Merlijn Sebrechts <email address hidden>
wrote:

> Just imagine what will happen if Dell releases a new firmware update for
> one of their laptops... We've been lucky this hasn't happened yet, so
> the only people running into this bug are people who have upgraded from
> a version where fwupd wasn't enabled or new users.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1730343
>
> Title:
> firmware update breaks Ubuntu
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fwupd/+bug/1730343/+subscriptions
>

Ok, I was scratching my head what happened, but it seems the fwupdate-signed upload failed to upload because we seemed to have had fwupdate-signed version 1.14+9-2 published in artful at some point. See the following link to the binary:

https://launchpad.net/ubuntu/artful/amd64/fwupdate-signed/1.14+9-2

Need to figure out why that was there. We now need to have a fwupdate-signed version that's higher than 1.14+9-2 (which is why the previous one got rejected).

So that should be what I uploaded now. Can you look at what's in unapproved
queue?

On Thu, Nov 30, 2017, 08:35 Łukasz Zemczak <email address hidden>
wrote:

> Ok, I was scratching my head what happened, but it seems the fwupdate-
> signed upload failed to upload because we seemed to have had fwupdate-
> signed version 1.14+9-2 published in artful at some point. See the
> following link to the binary:
>
> https://launchpad.net/ubuntu/artful/amd64/fwupdate-signed/1.14+9-2
>
> Need to figure out why that was there. We now need to have a fwupdate-
> signed version that's higher than 1.14+9-2 (which is why the previous
> one got rejected).
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1730343
>
> Title:
> firmware update breaks Ubuntu
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fwupd/+bug/1730343/+subscriptions
>

Ah, I see it now. Let me review/approve that.

Hello Merlijn, or anyone else affected,

Accepted fwupdate-signed into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate-signed/1.14.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This package fixes the bug.

Package: fwupdate-signed
Version: 1.14.1+9-2ubuntu0.17.10.1

And what about Xenial?

There's also a fwupdate-signed waiting in the zesty queue; that should fix the current failed-to-upload.

Xenial indeed needs to be updated; perhaps the simplest in this case is to bring it to the same version as in zesty?

Mario, what do you think?

I was having some trouble with patching xenial and got busy with something
else, let me try one more time.

On Fri, Dec 1, 2017, 13:40 Mathieu Trudel-Lapierre <email address hidden>
wrote:

> There's also a fwupdate-signed waiting in the zesty queue; that should
> fix the current failed-to-upload.
>
> Xenial indeed needs to be updated; perhaps the simplest in this case is
> to bring it to the same version as in zesty?
>
> Mario, what do you think?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1730343
>
> Title:
> firmware update breaks Ubuntu
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fwupd/+bug/1730343/+subscriptions
>

OK so I got xenial compiling by a few extra backported patches. I've uploaded fwupdate and fwupdate-signed for xenial to proposed.

This bug was fixed in the package fwupdate - 9-2ubuntu0.17.10.1

---------------
fwupdate (9-2ubuntu0.17.10.1) artful; urgency=medium

  * Disable "Linux Firmware Updater" boot entry after installation
    (LP: #1730343)

 -- Mario Limonciello <email address hidden> Mon, 06 Nov 2017 09:52:16 -0600

The verification of the Stable Release Update for fwupdate has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package fwupdate-signed - 1.14.1

---------------
fwupdate-signed (1.14.1) artful; urgency=medium

  * Build-depend on fwupdate 9-2ubuntu0.17.10.1 (LP: #1730343).

 -- Mario Limonciello <email address hidden> Mon, 20 Nov 2017 17:08:04 -0600

Hello, it was requested I also leave a comment here. Yesterday I experienced this issue on Ubuntu 16.04 LTS.

I then found this github ticket: https://github.com/rhboot/fwupdate/issues/86

Which had a workaround that got my machine working again.

I was told that "The Ubuntu archive admin needs to release the uploaded package for 16.04."

Many people have commented on the bug reports, and more people have given a "thumbs up" to my comment explaining the workaround to get their system booting again.

This is an issue that is actively bricking many people's machines. The only reason why we haven't yet seen thousands of people having this issue is because Dell hasn't yet released a firmware update since the bug was introduced. As a result, only people with new laptops or those who just upgraded from a version that doesn't have fwupdate experience this issue.

Please, fix this issue as soon as possible.

Hello Merlijn, or anyone else affected,

Accepted fwupdate-signed into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate-signed/1.13.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Can someone from the SRU team please tell me what needs to happen in order to get this release out to Xenial?

Hello Merlijn, or anyone else affected,

Accepted fwupdate into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate/0.5-2ubuntu6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Merlijn, or anyone else affected,

Accepted fwupdate-signed into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate-signed/1.11.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello all,

I just tried the proposed package on 16.04 and it failed.

I took this picture with my phone: https://www.dropbox.com/s/6r6zeadqakbtjg5/2018-01-04%2023.35.59-1.jpg?dl=0

After that screen it just continued with a regular boot.

Thank for the feedback Koen. Considering the failure there are other intermediary patches causing problems in less obvious ways.

Since the "9" release is stable in all these other Ubuntu releases with no other major complaints, I'm instead pulling in all changes to efi/fwupdate.c from 0.5 to 9 release.

I've redone the upload for both packages, so it will be awaiting an Ubuntu archive admin to accept it.

Hello Merlijn, or anyone else affected,

Accepted fwupdate into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate/0.5-2ubuntu7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Merlijn, or anyone else affected,

Accepted fwupdate-signed into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupdate-signed/1.11.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The latest upload consists of a very high number of patches which might quickly get hard to maintain. For the future maybe it would make sense to backport the whole 9 version to xenial instead of patching just one component with the code from the 9-codebase.

Unfortunately moving libfwup to 9 also means backporting newer efivar, so
that's why it makes more sense to only update EFI application.

Thanks for reviewing and accepting.

On Thu, Jan 18, 2018, 02:36 Łukasz Zemczak <email address hidden>
wrote:

> The latest upload consists of a very high number of patches which might
> quickly get hard to maintain. For the future maybe it would make sense
> to backport the whole 9 version to xenial instead of patching just one
> component with the code from the 9-codebase.
>
> ** Changed in: fwupdate-signed (Ubuntu Zesty)
> Status: Fix Committed => Won't Fix
>
> ** Changed in: fwupdate (Ubuntu Zesty)
> Status: Fix Committed => Won't Fix
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1730343
>
> Title:
> firmware update breaks Ubuntu
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fwupd/+bug/1730343/+subscriptions
>

I've downloaded the fwupdate 0.5-2ubuntu7/fwupdate-signed 1.11.3 proposed from an XPS 9350 running BIOS 1.5.0 on Ubuntu 16.04.

I successfully upgraded to 1.6.1 via UEFI capsule and booted back into Ubuntu.
I also confirmed that there is no remaining Linux Firmware Updater \fwupx64.efi NVRAM EFI entry.

For me the update was also successful. Went from bios version 1.7.5 to 1.8.1

This bug was fixed in the package fwupdate - 0.5-2ubuntu7

---------------
fwupdate (0.5-2ubuntu7) xenial; urgency=medium

  * Redo patch series. (LP: #1730343)
    - Backport ALL patches to EFI application up through the "9" release.
    - The last two attempts have introduced unobvious bugs due to nature
      of cherry picking.
    - The "9" release of the EFI application however has been stable in
      future releases of Ubuntu.

 -- Mario Limonciello <email address hidden> Mon, 15 Jan 2018 22:59:48 -0600

This bug was fixed in the package fwupdate-signed - 1.11.3

---------------
fwupdate-signed (1.11.3) xenial; urgency=medium

  * Build-depend on fwupdate 0.5-2ubuntu7 (LP: #1730343)

 -- Mario Limonciello <email address hidden> Mon, 15 Jan 2018 23:06:45 -0600