shutdown hangs at "Waiting for process: ..." for 90s, ignoring DefaultTimeoutStopSec
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| systemd (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
| Focal |
Fix Released
|
Medium
|
Unassigned | ||
Bug Description
[Impact]
The systemd shutdown sequence does not honor systemd-system.conf settings when waiting for remaining processes. This means that, for example, if a systemd service specifies KillMode=process and a process remaining from that service does not properly handle SIGTERM, then the remaining process will not be killed until after the compiled-in default value of DefaultTimeoutS
[Test Plan]
* Create a new script, /usr/local/
```
#!/bin/bash
loop_forever() {
while true; do sleep 1; done
}
(
trap 'echo Ignoring SIGTERM...' SIGTERM
loop_forever
)
loop_forever
```
This script will spawn a subshell which will loop forever and ignore
SIGTERM. This will force systemd to wait for the subprocess at
reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec
(DefaultTimeo
* Make the script executable:
$ chmod +x /usr/local/
* Create a systemd service for this script. Add the following to
/etc/
```
[Service]
KillMode=process
ExecStart=
```
* Start the service:
$ systemctl start loop-ignore-
* Edit /etc/systemd/
'DefaultTimeou
e.g. 20s.
* Re-exec the daemon so this new default takes effect:
$ systemctl daemon-reexec
* Reboot, and monitor the logs. Observe that systemd-shutdown will wait
for the loop-ignore-sigterm process for 90s, instead of the 20s
configured earlier.
[Where problems could occur]
The patch moves the reset_arguments() call to the end of main, which means reset_arguments() is no longer called before daemon re-execution (if that branch is taken). If anything in that code path relied on reset_arguments() being called before re-executing, those assumptions could be broken. Any such problems would potentially be seen during daemon re-execution, e.g. when calling systemctl daemon-reexec.
[ Original Description ]
With systemd v245 as shipped with 20.04, the shutdown sequence does not use the value of `DefaultTimeout
This is most visible with services that use `KillMode=process` (docker, k8s, k3s, etc...), especially if the remaining processes do not handle `SIGTERM` or choose to ignore it.
For example:
```
[ OK ] Finished Reboot.
[ OK ] Reached target Reboot.
[ 243.652848 ] systemd-
--- hangs here for 90s even if DefaultTimeoutS
```
The bug has been fixed upstream here: https:/
Marc was kind enough to package the patch for 20.04 so I could test it (https:/
Here's a few github issues I stumbled upon while trying to debug this, along with a short writeup of the workaround I ended up using:
- https:/
- https:/
- https:/
- https:/
Of course, it would be much better if all the processes would properly handle `SIGTERM`, but having a way to enforce a maximum wait time at shutdown is a decent workaround.
Given that the patch is relatively simple, would it be possible to add it the package for 20.04?
Thanks
Related branches
- Lukas Märdian: Approve
-
Diff: 185 lines (+153/-1)4 files modifieddebian/changelog (+12/-1)
debian/patches/lp1958284-core-move-reset_arguments-to-the-end-of-main-s-finish.patch (+48/-0)
debian/patches/pid1-set-SYSTEMD_NSS_DYNAMIC_BYPASS-1-env-var-for-dbus-da.patch (+91/-0)
debian/patches/series (+2/-0)
| description: | updated |
| Launchpad Janitor (janitor) wrote | #1 |
| Changed in systemd (Ubuntu): | |
| status: | New → Confirmed |
| tags: | added: rls-ff-incoming |
| tags: | added: fr-1987 |
| tags: | removed: fr-1987 rls-ff-incoming |
| Changed in systemd (Ubuntu Focal): | |
| status: | New → Confirmed |
| Changed in systemd (Ubuntu Focal): | |
| importance: | Undecided → Medium |
| Marc Deslauriers (mdeslaur) wrote | #2 |
| Lukas Märdian (slyon) wrote | #3 |
| description: | updated |
| Changed in systemd (Ubuntu Focal): | |
| status: | Confirmed → In Progress |
| Changed in systemd (Ubuntu): | |
| status: | Confirmed → Fix Released |
| Brian Murray (brian-murray) wrote Please test proposed package | #4 |
| Changed in systemd (Ubuntu Focal): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed verification-needed-focal |
| Ubuntu SRU Bot (ubuntu-sru-bot) wrote Autopkgtest regression report (systemd/245.4-4ubuntu3.16) | #5 |
| Nick Rosbrook (enr0n) wrote | #6 |
| tags: |
added: verification-done-focal removed: verification-needed-focal |
| Nick Rosbrook (enr0n) wrote | #7 |
| tags: |
added: verification-done removed: verification-needed |
| Launchpad Janitor (janitor) wrote | #8 |
| Changed in systemd (Ubuntu Focal): | |
| status: | Fix Committed → Fix Released |
| Brian Murray (brian-murray) wrote Update Released | #9 |
Status changed to 'Confirmed' because the bug affects multiple users.