Systemd does not send SIGTERM first on shutdown

Bug #1448259 reported by Olli Niemi
44
This bug affects 5 people
Affects Status Importance Assigned to Milestone
systemd
Invalid
Undecided
systemd (Debian)
Fix Released
Unknown
systemd (Fedora)
Invalid
Undecided
systemd (Ubuntu)
Fix Released
High
Martin Pitt
Vivid
Fix Released
High
Martin Pitt

Bug Description

It has been normal that applications first get the SIGTERM signal before SIGKILL on shutdown/reboot in order to successfully finish any pending tasks. Now it seem this logic has been changed to something else, causing problems to mosh and many others:

https://bugs.launchpad.net/ubuntu/+source/mosh/+bug/1446982

SIGTERM suggestion can be seen here:

http://unixhelp.ed.ac.uk/CGI/man-cgi?shutdown+8

I created this error report to find out the correct way for applications to fix this problem or to create one fix to systemd, bringing back the old "BSD shutdown" functionality.

This report is for Ubuntu 15.04.

SRU TEST CASE:
 - Open a terminal, enter some commands, then run "reboot".
 - After a reboot, chances are very high that your bash history does not contain your most recently typed commands
 - With the updated package, the bash history should be intact.

REGRESSION POTENTIAL:
 - The original commit was applied because of an inherent race condition with cgroup's release_agent -- in rare corner cases an nspawn container (probably also LXC) can miss them. In that case it's possible that you instead get a 90s timeout on the unit that is shutting down. But this does not mean data loss, just a rare shutdown hang from containers (for the record, I never actually saw that hanging with LXC), so I think it's a good trade-off.

Revision history for this message
Olli Niemi (olliniem) wrote :
affects: mosh (Ubuntu) → systemd (Ubuntu)
Revision history for this message
Keith Winstein (keithw) wrote :

This does seem to be a systemd bug that is affecting several applications -- basically any application that would like a SIGTERM on reboot/shutdown so that it can save or clean up resources gracefully.

Please see https://bugzilla.redhat.com/show_bug.cgi?id=1141137 (upstream systemd bug)

https://bugzilla.redhat.com/show_bug.cgi?id=1170765 (bug affects bash's attempt to save history)

https://github.com/cockpit-project/cockpit/issues/1155 (bug affects cockpit)

http://lists.freedesktop.org/archives/systemd-devel/2014-October/024452.html (discussion)

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Martin Pitt (pitti)
Changed in systemd (Ubuntu):
importance: Undecided → High
assignee: nobody → Martin Pitt (pitti)
status: Confirmed → In Progress
Martin Pitt (pitti)
tags: added: systemd-bugs
summary: - Systemd has wrong kill mode on shutdown
+ Systemd does not send SIGTERM first on shutdown
Martin Pitt (pitti)
Changed in systemd (Ubuntu Vivid):
milestone: none → vivid-updates
Revision history for this message
Martin Pitt (pitti) wrote :

https://launchpad.net/~pitti/+archive/ubuntu/sru-test now has packages with this fix. I can't upload it yet as there's a previous systemd being tested in -proposed still. Feedback appreciated from all bug reporters, though! (Also for bash history and mosh).

Revision history for this message
Maarten Fonville (maarten-fonville) wrote :

I tested the packages, and bash history is now properly retained during a reboot :-)

Revision history for this message
Olli Niemi (olliniem) wrote :

I also tested tested the packages. The mosh case is now working but the included test.c tester still fails this case.

Revision history for this message
Olli Niemi (olliniem) wrote :

When I compare the documentation I see that with systemd SIGTERM has been removed from it:

Old behavior: http://unixhelp.ed.ac.uk/CGI/man-cgi?shutdown+8
Systemd behavior: http://man7.org/linux/man-pages/man8/shutdown.8.html

Is it now that SIGTERM is sent in case-by-case basis to specific applications (based on error reports)? Should SIGTERM be mentioned in the documentation?

Revision history for this message
Martin Pitt (pitti) wrote :

It's still more or less the same, except that every unit and scope is being shut down individually. "man systemd.kill" has the details.

Revision history for this message
Martin Pitt (pitti) wrote :
Changed in systemd (Ubuntu Vivid):
status: In Progress → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

This hasn't made it to -proposed or -updates yet, setting back to Triaged.

Changed in systemd (Ubuntu Vivid):
status: Fix Committed → Triaged
Martin Pitt (pitti)
description: updated
description: updated
Changed in systemd (Ubuntu):
milestone: vivid-updates → none
Revision history for this message
Chris Halse Rogers (raof) wrote : Please test proposed package

Hello Olli, or anyone else affected,

Accepted systemd into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/219-7ubuntu5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in systemd (Ubuntu Vivid):
status: Triaged → Fix Committed
tags: added: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.9 KiB)

This bug was fixed in the package systemd - 219-8ubuntu1

---------------
systemd (219-8ubuntu1) wily; urgency=medium

  * Merge with Debian experimental branch. Remaining Ubuntu changes:
    - Hack to support system-image read-only /etc, and modify files in
      /etc/writable/ instead.
    - Keep our much simpler udev maintainer scripts (all platforms must
      support udev, no debconf).
    - initramfs init-top: Drop $ROOTDELAY, we do that in a more sensible way
      with wait-for-root. Will get applicable to Debian once Debian gets
      wait-for-root in initramfs-tools.
    - initramfs init-bottom: If LVM is installed, settle udev,
      otherwise we get missing LV symlinks. Workaround for LP #1185394.
    - Add debian/udev.lvm2.init: Dummy SysV init script to satisfy insserv
      dependencies to "lvm2" which is handled with udev rules in Ubuntu.
    - Add debian/udev.lvm2.service to avoid running the dummy lvm2 init
      script.
    - Provide shutdown fallback for upstart. (LP: #1370329)
    - debian/extra/ifup@.service: Additionally run for "auto" class. We don't
      really support "allow-hotplug" in Ubuntu at the moment, so we need to
      deal with "auto" devices appearing after "/etc/init.d/networking start"
      already ran. (LP: #1374521) Also, check if devices are actually defined
      in /etc/network/interfaces as we don't use Debian's net.agent.
      Also run ifup in the background during boot, to avoid blocking
      network.target. (LP: #1425376)
    - ifup@.service: Drop dependency on networking.service (i. e.
      /etc/init.d/networking), and merely ensure that /run/network exists.
      This avoids unnecessary dependencies/waiting during boot and dependency
      cycles if hooks wait for other interfaces to come up (like ifenslave
      with bonding interfaces). (LP: #1414544)
    - Add Get-RTC-is-in-local-time-setting-from-etc-default-rc.patch: In
      Ubuntu we currently keep the setting whether the RTC is in local or UTC
      time in /etc/default/rcS "UTC=yes|no", instead of /etc/adjtime.
      (LP: #1377258)
    - Put session scopes into all cgroup controllers. This makes unprivileged
      user LXC containers work under systemd. (LP: #1346734)
    - systemctl: Don't forward telinit u to upstart. This works around
      upstart's Restart() always reexec'ing /sbin/init on Restart(), even if
      that changes to point to systemd during the upgrade. This avoids running
      systemd during a dist-upgrade. (LP: #1430479)
    - Drop hwdb-update dependency from udev-trigger.service, which got
      introduced in v219-stable. This causes udev and plymouth to start too
      late and isn't really needed in Ubuntu yet as we don't support stateless
      systems yet and handle hwdb.bin updates through dpkg triggers. This can
      be dropped again with initramfs-tools 0.117.
    - Lower Breaks: to plymouth version which has the udev inotify fix in
      Ubuntu.
    - Lower libappamor dep to the Ubuntu version where it moved to /lib.
    - Change systemd-sysv's conflicts to upstart-sysv. (LP: #1422681)
    - Make failure of boot-and-services NSpawn.test_boot non-fatal for now.
      This currently fails when being t...

Read more...

Changed in systemd (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote :

Olli, Maarten, could you please test the package in vivid-proposed instead of my PPA, so that we can officially verify this to get it into vivid-updates? Thank you in advance!

Revision history for this message
Maarten Fonville (maarten-fonville) wrote :

For me the package in vivid-proposed works! :-)

Martin Pitt (pitti)
tags: added: verification-done
removed: verification-needed
tags: added: systemd-boot
removed: systemd-bugs
Changed in systemd (Debian):
status: Unknown → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 219-7ubuntu5

---------------
systemd (219-7ubuntu5) vivid; urgency=medium

  * Revert upstream commit 743970d which immediately SIGKILLs units during
    shutdown. This leads to problems like bash not being able to write its
    history, mosh not saving its state, and similar failed cleanup actions.
    (LP: #1448259)
  * ifup@.service: Set IgnoreOnIsolate, so that "systemctl default" does not
    shut down network interfaces. (LP: #1449380). Add PartOf=network.target,
    so that stopping network.target also stops network interfaces.
  * 75-persistent-net-generator.rules: Fix rules for ibmveth (it's a driver,
    not a subsystem). (LP: #1437375)
  * debian/tests/unit-config: Add tests for systemctl enable/disable on a
    SysV-only unit. Reproduces LP #1447807.
  * Fix systemctl enable for SysV scripts without a native unit. We must not
    try and enable the nonexisting unit then. (LP: #1447807)

 -- Martin Pitt <email address hidden> Thu, 07 May 2015 07:45:34 +0200

Changed in systemd (Ubuntu Vivid):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Olli Niemi (olliniem) wrote :

Tested with Ubuntu Mate 15.04 in Raspberry Pi 2 (test.c, included here). It has systemd 219-7ubuntu5 package. After a reboot one can see that a file "testing.txt" is created but its size is zero, i.e. it was not given enough time to write the file contents. The fix is not working with two laptops and one desktop PC ("testing.txt" file is not created), Ubuntu 15.04. I currently have no access to "development hardware" so I can only test released changes.

Changed in systemd (Debian):
status: Fix Committed → Fix Released
Changed in systemd:
importance: Unknown → Undecided
status: Unknown → Invalid
Changed in systemd (Fedora):
importance: Unknown → Undecided
status: Unknown → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.