Merge ~mustafakemalgilor/ubuntu/+source/systemd:systemd-fix-lp1978079-efi-pstore-not-cleared-jammy into ~ubuntu-core-dev/ubuntu/+source/systemd:ubuntu-jammy

[Mustafa Kemal Gilor]

Backported a fix from upstream to jammy for the LP issue #1978079:

      * d/p/lp1978079-efi-pstore-not-cleared-on-boot.patch: pstore: Run
        after
        modules are loaded. Thanks to Alexander Graf <email address hidden>.
        LP: #1978079)

Please tag & sponsor.

(See backport merge request for focal: https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/424938)

Steps to reproduce/test:

# In order to be able to reproduce this issue, the system must have EFI-backed pstore.
# To check which kind of backend that pstore, use `cat /sys/module/pstore/parameters/backend`
# If it says `efi`, the steps below are applicable. Otherwise, find an environment that has
# EFI backed pstore.

# Enable the pstore service. This service is supposed to move the data in /sys/fs/pstore
# to the `/var/lib/systemd/pstore` path on boot.
systemctl enable systemd-pstore.service # (or can be vendor enabled)

# Crash the kernel
echo 1 > /proc/sys/kernel/sysrq
echo 1 > /proc/sys/kernel/panic # this is usually set to zero, causing kernel to loop over the panic and freeze
echo "c" > /proc/sysrq-trigger

# The system will reboot itself. Check `/sys/fs/pstore` path first:
ls /sys/fs/pstore # The path should not be empty, which means the systemd-pstore has failed to do its' job
ls /var/lib/systemd/pstore # The path should be empty.

# Apply the fix
sudo add-apt-repository ppa:mustafakemalgilor/lp-1978079-1
sudo apt upgrade

# Crash the kernel
echo 1 > /proc/sys/kernel/sysrq
echo 1 > /proc/sys/kernel/panic # this is usually set to zero, causing kernel to loop over the panic and freeze
echo "c" > /proc/sysrq-trigger

# The system will reboot itself. Check `/sys/fs/pstore` path first:
ls /sys/fs/pstore # The path should be empty
ls /var/lib/systemd/pstore # The path should not be empty

Package Test Results:

autopkgtest [15:49:45]: test systemd-fsckd: [-----------------------
SKIP: root file system is being checked by initramfs already
autopkgtest [15:49:45]: test systemd-fsckd: -----------------------]
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [15:49:46]: test systemd-fsckd: - - - - - - - - - - results - - - - - - - - - -
autopkgtest [15:49:46]: @@@@@@@@@@@@@@@@@@@@ summary
timedated PASS
hostnamed PASS
localed-locale PASS
localed-x11-keymap PASS
logind PASS
unit-config PASS
storage PASS
networkd-test.py PASS
build-login PASS
boot-and-services PASS
udev PASS
unit-tests PASS
tests-in-lxd PASS
upstream-1 PASS
upstream-2 PASS
boot-smoke PASS
systemd-fsckd SKIP exit status 77 and marked as skippable
qemu-system-x86_64: terminating on signal 15 from pid 474221 (/usr/bin/python3)

63e3800... by Mustafa Kemal Gilor on 2022-06-20

* d/p/lp1978079-efi-pstore-not-cleared-on-boot.patch: pstore: Run after
  modules are loaded. Thanks to Alexander Graf <email address hidden>.
  (LP: #1978079)

0569b14... by Mustafa Kemal Gilor on 2022-06-20

updated changelog

[Robie Basak (racb)]

Hi,

Normally the Foundations team reviews systemd changes. Is there a particular reason you requested reviews from the Server Team please?

[Lukas Märdian (slyon)]

Thank you very much! This is quality work and LGTM.

Good work on the detailed reproducer/testcase and thanks for executing autopkgtest ahead of time!

I will probably re-write the d/changelog file a bit, to bring it in line with our "gbp dch" (git-buildpackage) workflow. Other than that, I'll merge it as is.

review: Approve

[Mustafa Kemal Gilor (mustafakemalgilor)]

Thanks for the feedback!

diff --git a/debian/changelog b/debian/changelog
index f9df438..10bce92 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+systemd (249.11-0ubuntu3.4) jammy; urgency=medium
+
+  * d/p/lp1978079-efi-pstore-not-cleared-on-boot.patch: pstore: Run after
+    modules are loaded. Thanks to Alexander Graf <[e-mail]>.
+    (LP: #1978079)
+
+ -- Mustafa Kemal GILOR <[e-mail]>  Mon, 20 Jun 2022 17:14:13 +0300
+
 systemd (249.11-0ubuntu3.3) jammy; urgency=medium
 
   [ Lukas Märdian ]
diff --git a/debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch b/debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
new file mode 100644
index 0000000..2b282d0
--- /dev/null
+++ b/debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
@@ -0,0 +1,41 @@
+From 4dec0b605b378b1a3d54bd738130cc45ff745673 Mon Sep 17 00:00:00 2001
+From: Alexander Graf <[e-mail]>
+Date: Thu, 9 Jun 2022 16:20:43 +0200
+Subject: [PATCH] pstore: Run after modules are loaded
+
+The systemd-pstore service takes pstore files on boot and transfers them
+to disk. It only does it once on boot and only if it finds any. The typical
+location of the pstore on modern systems is the UEFI variable store.
+
+Most distributions ship with CONFIG_EFI_VARS_PSTORE=m. That means, the
+UEFI variable store is only available on boot after the respective module
+is loaded.
+
+In most situations, the pstore service gets loaded before the UEFI pstore,
+so we don't get to transfer logs. Instead, they accumulate, filling up the
+pstore over time, potentially breaking the UEFI variable store.
+
+Let's add a service dependency on any kernel module that can provide a
+pstore to ensure we only scan for pstate after we can actually see pstate.
+
+I have seen live occurences of systems breaking because we did not erase
+the pstates and ran out of UEFI nvram space.
+
+Fixes https://github.com/systemd/systemd/issues/18540
+---
+ units/systemd-pstore.service.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/units/systemd-pstore.service.in b/units/systemd-pstore.service.in
+index 848e311e9642..86de30ad4a72 100644
+--- a/units/systemd-pstore.service.in
++++ b/units/systemd-pstore.service.in
+@@ -15,6 +15,8 @@ ConditionVirtualization=!container
+ DefaultDependencies=no
+ Conflicts=shutdown.target
+ Before=sysinit.target shutdown.target
++After=modprobe@efi_pstore.service [e-mail] modprobe@chromeos_pstore.service [e-mail] modprobe@pstore_zone.service modprobe@pstore_blk.service
++Wants=modprobe@efi_pstore.service [e-mail] modprobe@chromeos_pstore.service [e-mail] modprobe@pstore_zone.service modprobe@pstore_blk.service
+ 
+ [Service]
+ Type=oneshot
diff --git a/debian/patches/series b/debian/patches/series
index 3ba6f06..8dbcd40 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -88,3 +88,4 @@ lp1966381-oomd-calculate-used-memory-with-MemAvailable-instead-of-M.patch
 lp1926860-hwdb-remove-the-tablet-pad-entry-for-the-UC-Logic-1060N.patch
 hwdb-Add-mic-mute-key-mapping-for-HP-Elite-x360.patch
 lp1964494-network-do-not-enable-IPv4-ACD-for-IPv4-link-local-a.patch
+lp1978079-efi-pstore-not-cleared-on-boot.patch