98-reboot-required and Interaction with livepatch

The position of the Security Team has been consistent that kernel live patching allows users to defer reboots, it does not allow users to avoid them. Because not all security fixes are included in live patches, and because correlating the live patch CVEs to the kernel deb CVEs requires knowledge that's external to the packages themselves, hiding the 'reboot required' message will give users a false sense of security about their system.

Cc:ing Tyler for any further comment.

Whatever our policy is going to be here, it should be consistent across the board for both desktop and server (which may fall out naturally from changes to update-notifier, but maybe not).

I think we can agree that the MOTD messaging gets confusing. The best way to fix it is the question.

For example, we go from this:
(...)
*** System restart required ***
 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

Last login: Tue Feb 6 11:37:27 2018 from 192.168.122.1

ubuntu@ubuntu:~$ cat /var/run/reboot-required.pkgs
linux-image-4.4.0-112-generic
linux-base
linux-base
ubuntu@ubuntu:~$ uname -r
4.4.0-87-generic

To this, after I install livepatch and login again:
(...)
*** System restart required ***
 * Canonical Livepatch is enabled.
   - All available patches applied.

Last login: Tue Feb 6 11:41:02 2018 from 192.168.122.1
ubuntu@ubuntu:~$

And I do have a lot of CVEs shown in the livepatch status output. "Do I need to reboot or not?" is a question I can imagine the user asking.

Status changed to 'Confirmed' because the bug affects multiple users.

Actually, I think there's more to it. Here are some thoughts

1. The compelling use case for livepatch is in fact to minimize and control the need to reboot; this is what customers pay us for. For many of them, rebooting every 3 weeks is either impossible or at least very painful. Let's not lose sight of that.

2. The output of canonical-livepatch status clearly says when you are up to date and when a reboot is required. In other words, the client does know when all patches are applied, and alternatively, when it is time to reboot.

3. Every external tool we've had customers ask us (AWS Patch Manager, Azure Patch Management) looks at /var/run/reboot-required and uses that to decide whether to reboot or not.

The net effect of #3 is that canonical-livepatch is rendered useless, basically, because every 3 weeks everything needs to be rebooted. And it's a critical use case, so we do need to address this in order to retain value in the proposition.

I agree with comment #4.

Here are some extra details about the status of livepatch when a kernel upgrade is required.

I am running an 18.04 VM with an old 4.15.0-20-generic kernel from April 2018. Here is status in yaml format:

ubuntu@bioniclivepatcholdkernel:~$ canonical-livepatch status --format yaml
client-version: 9.5.5
architecture: x86_64
cpu-model: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
last-check: 2020-05-15T11:29:29-04:00
boot-time: 2020-05-15T15:28:49Z
uptime: 59s
status:
- kernel: 4.15.0-20.21-generic
  running: true
  livepatch:
    checkState: checked
    patchState: kernel-upgrade-required
    version: "42.1"
    fixes: |-
      * CVE-2018-10323
      * CVE-2018-10840
    [...removing some CVEs to keep this short...]

There are no kernel upgrades pending a reboot on this box, so no /var/run/reboot-required:

ubuntu@bioniclivepatcholdkernel:~$ ls -l /var/run/reboot-required*
ls: cannot access '/var/run/reboot-required*': No such file or directory

If I upgrade to a new kernel, those files are created:

ubuntu@bioniclivepatcholdkernel:~$ ls -l /var/run/reboot-required*
-rw-r--r-- 1 root root 32 May 15 11:37 /var/run/reboot-required
-rw-r--r-- 1 root root 11 May 15 11:37 /var/run/reboot-required.pkgs

ubuntu@bioniclivepatcholdkernel:~$ cat /var/run/reboot-required
*** System restart required ***

ubuntu@bioniclivepatcholdkernel:~$ cat /var/run/reboot-required.pkgs
linux-base

And nothing changes in the output of canonical-livepatch status --format yaml:

ubuntu@bioniclivepatcholdkernel:~$ canonical-livepatch status --format yaml
client-version: 9.5.5
architecture: x86_64
cpu-model: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
last-check: 2020-05-15T11:29:29-04:00
boot-time: 2020-05-15T15:28:49Z
uptime: 9m38s
status:
- kernel: 4.15.0-20.21-generic
  running: true
  livepatch:
    checkState: checked
    patchState: kernel-upgrade-required
    version: "42.1"
    fixes: |-
      * CVE-2018-10323
      * CVE-2018-10840
    [...removing some CVEs to keep this short...]

And if I reboot into a recent kernel, the up to date status is:

ubuntu@bioniclivepatcholdkernel:~$ canonical-livepatch status --format yaml
client-version: 9.5.5
architecture: x86_64
cpu-model: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
last-check: 2020-05-15T11:42:09-04:00
boot-time: 2020-05-15T15:41:28Z
uptime: 50s
status:
- kernel: 4.15.0-99.100-generic
  running: true
  livepatch:
    checkState: checked
    patchState: nothing-to-apply
    version: ""
    fixes: ""

I think the problem is that external tools just simplistically look at /var/run/reboot-required to decide whether to reboot and I don't think we can expect them to be fixed just to handle livepatch.

Chiming in a a potential customer:

I can't figure out how to check when a reboot is required when using livepatch. For example, at the moment a machine I'm testing on has only "linux-base" in /var/run/reboot-required.pkgs.

Livepatch status says: "patch state: ✓ no livepatches needed for this kernel yet".

Do I need to reboot now? I can't figure that out. It would be really helpful if the full decision tree for this was documented.

The attachment "debdiff for unattended-upgrades (focal)" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Thank you for the patches Olivier! LGTM, I've sponsored it for both affected packages.

The only thing I needed to change is the version string for unattended-upgrades 2.3ubuntu1 -> 2.3ubuntu0.2 (as this is an SRU, https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging)

Moving the packages to Fix Released since the patches were already in Jammy:

update-notifier - fixed in 3.192.47: http://changelogs.ubuntu.com/changelogs/pool/main/u/update-notifier/update-notifier_3.192.54/changelog

unattended-upgrades - fixed in 2.8ubuntu1: http://changelogs.ubuntu.com/changelogs/pool/main/u/unattended-upgrades/unattended-upgrades_2.8ubuntu1/changelog

Hello Andreas, or anyone else affected,

Accepted update-notifier into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.30.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Andreas, or anyone else affected,

Accepted unattended-upgrades into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/2.3ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Trying to validate this upload but unattended-upgrades failed to build with:

Applying pinning: PkgFilePin(id=0, priority=-32768)
Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/<<PKGBUILDDIR>>/test/root.unused-deps/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages' a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=106 ID:0>
Using (^linux-image-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^.*-modules-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$|^linux-headers-[1-9][0-9]*\.[0-9]+\.[0-9]+-[0-9]+(-.+)?$) regexp to find kernel packages
Using (^linux-image-5\.4\.0\-109\-generic$|^linux-image-5\.4\.0\-109$|^.*-modules-5\.4\.0\-109\-generic$|^.*-modules-5\.4\.0\-109$|^linux-headers-5\.4\.0\-109\-generic$|^linux-headers-5\.4\.0\-109$) regexp to find running kernel packages
{'old-unused-dependency', 'any-old-unused-modules'}
Keeping auto-removable test-package-dependency package(s) because it would also remove the following packages which should be kept in this step: old-unused-dependency
The following packages are marked for installation or upgrade which is not allowed when performing autoremovals: forbidden-dependency
.
======================================================================
FAIL: test_remove_unused_dependencies_new_unused_only (__main__.TestRemoveUnused)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_remove_unused.py", line 164, in test_remove_unused_dependencies_new_unused_only
    self.assertTrue(needle_kernel_good in haystack,
AssertionError: False is not true : Can not find 'Removing unused kernel packages: linux-image-4.05.0-1021-kvm
' in '2022-05-10 20:56:21,336 INFO Starting unattended upgrades script

The package FTBFS against -proposed but builds successfully against the -release pocket.

Based on the error, it seems related to recent changes in how apt determines kernels that can be autoremoved.

apt (2.0.7) focal; urgency=medium

  * Revert "Protect currently running kernel at run-time"
  * Backport Determine autoremovable kernels at run-time (LP: #1615381) as of
    2.4.5; including the change to only protect two kernels, not last installed
    one (LP: #1968154)

 -- Julian Andres Klode <email address hidden> Mon, 25 Apr 2022 15:58:46 +0200

Hello Andreas, or anyone else affected,

Accepted unattended-upgrades into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/2.3ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Tested successfully on a focal amd64 VM:

Test-case 1 - livepatch enabled
$ canonical-livepatch status
last check: 7 seconds ago
kernel: 5.4.0-121.137-generic
server check-in: succeeded
[...]
$ apt install linux-image-5.13.0-36-generic
[...]
$ ssh olivier@localhost
[...]
0 updates can be applied immediately

Test-case 2 - livepatch disabled

```
$ canonical-livepatch status
Machine is not enabled. Please run 'sudo canonical-livepatch enable' with the token obtained from https://ubuntu.com/livepatch.
$ apt install linux-image-5.13.0-36-generic
[...]
$ ssh olivier@localhost
[...]
*** System restart required ***
```

This bug was fixed in the package unattended-upgrades - 2.3ubuntu0.3

---------------
unattended-upgrades (2.3ubuntu0.3) focal; urgency=medium

  * Fix FTBFS following change in the way apt detects automatically removable
    kernels (LP: #1974223)
  * Work around autopkgtest failure by removing __pycache__ in the test suite
    before the invocation of python-coverage (LP: #1974223)
  * Added new test depends on apt to trigger autopkgtest runs automatically
    when apt is upgraded.

unattended-upgrades (2.3ubuntu0.2) focal; urgency=medium

  [ Brian Murray ]
  * kernel/postinst.d/unattended-upgrades: check to see if canonical-livepatch
    is installed and enabled. In the event that it is and it says a reboot is
    required do not update reboot-required files so that only one reboot
    message appears in the message of the day. (LP: #1747499)

 -- Olivier Gayot <email address hidden> Thu, 19 May 2022 19:38:48 +0200

This bug was fixed in the package update-notifier - 3.192.30.11

---------------
update-notifier (3.192.30.11) focal; urgency=medium

  [ Brian Murray ]
  * data/notify-reboot-required: check to see if canonical-livepatch is
    installed and enabled. In the event that it is and it says a reboot is
    required do not update reboot-required files so that only one reboot
    message appears in the message of the day. (LP: #1747499)

 -- Olivier Gayot <email address hidden> Mon, 25 Apr 2022 16:02:40 +0200

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.