output of invoke-rc.d for systemd units un-debuggable on failure

Bug #1596056 reported by Steve Langasek
24
This bug affects 2 people
Affects Status Importance Assigned to Milestone
init-system-helpers (Ubuntu)
Fix Released
Wishlist
Martin Pitt
Xenial
Fix Released
Wishlist
Steve Langasek

Bug Description

[SRU Justification]
Currently if a systemd unit fails to start in a non-interactive upgrade and all you have is a log, it's impossible to debug. Fix this so that logs contain the actual details of the unit failure.

[Test case]
1. Get a root shell.
2. Run "sed -e's,sbin/sshd,sbin/sshd-noexists,; /Alias/d' /lib/systemd/system/ssh.service > /lib/systemd/system/ssh-noexists.service"
3. Run 'systemctl enable ssh-noexists'
4. Run 'ln -s ../init.d/ssh /etc/rc5.d/S02ssh-noexists'
5. Run 'invoke-rc.d ssh-noexists start'
6. Verify that the command directs you to run 'systemctl status' for details, and provides no details.
7. Install init-system-helpers from -proposed.
8. Run 'invoke-rc.d ssh-noexists start'.
9. Verify that the command provides details about the failure to start ssh-noexists service.
10. Run 'systemctl disable ssh-noexists'.
11. Run 'rm -f /lib/systemd/system/ssh-noexists.service'.
12. Run 'rm -f /etc/rc5.d/S02ssh-noexists'

[Regression potential]
Minimal. On the failure case, an additional command is run; the additional command is guarded with || true.

[Original description]

When invoke-rc.d is called on a systemd system, if the unit fails to start, you get output like:

Created symlink /etc/systemd/system/multi-user.target.wants/openafs-fileserver.service → /lib/systemd/system/openafs-fileserver.service.
Job for openafs-fileserver.service failed because the control process exited with error code.
See "systemctl status openafs-fileserver.service" and "journalctl -xe" for details.
invoke-rc.d: initscript openafs-fileserver, action "start" failed.
dpkg: error processing package openafs-fileserver (--configure):
 subprocess installed post-installation script returned error exit status 1

The output shown here comes from systemctl itself, and is usually fine. The admin who ran systemctl can run those other commands to debug.

However, when called by invoke-rc.d, this output is usually seen only in a log file; maybe submitted in a bug report, maybe attached to something like an autopkgtest:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-yakkety/yakkety/armhf/o/openafs/20160624_174535@/log.gz

By the time someone looks at this log output, it is often too late to run those commands in order to debug the failure.

invoke-rc.d should call these commands for us on systemd unit failure, so that the relevant debugging information is included in the log where it can help.

We don't want to call 'journalctl -xe', which might leak information into the log from other jobs, but 'journalctl -x -u <this_unit>' may be appropriate.

Revision history for this message
Martin Pitt (pitti) wrote :

Seems fine to me at first sight. I'd vote for "systemctl status", as that also shows running and exited services with exit codes etc., and the journal tail. We have this shiny new technology now, so let's use it :-)

Just checking with the other init folks.

Changed in init-system-helpers (Ubuntu):
importance: Undecided → Wishlist
status: New → In Progress
Martin Pitt (pitti)
Changed in init-system-helpers (Ubuntu):
assignee: nobody → Martin Pitt (pitti)
Revision history for this message
Martin Pitt (pitti) wrote :
Changed in init-system-helpers (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Robie Basak (racb) wrote :

Would this fix also solve bug 1587791?

Revision history for this message
Martin Pitt (pitti) wrote :

Yes, I believe it would, marking as a duplicate.

Revision history for this message
Martin Pitt (pitti) wrote :

Robie suggested that this would be helpful to backport to xenial, adding task.

Changed in init-system-helpers (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → Wishlist
assignee: nobody → Martin Pitt (pitti)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package init-system-helpers - 1.36

---------------
init-system-helpers (1.36) unstable; urgency=medium

  [ Martin Pitt ]
  * init: Don't depend on sysvinit-core when building on/for Ubuntu and
    derivatives.
  * init: Drop "essential" from the package description as it is not essential
    any more.
  * invoke-rc.d: Quiesce error message if "runlevel" does not exist, which is
    now the case in chroots since init stopped being essential.
    (Closes: #827376)
  * update-rc.d: Testing for /etc/init.d/mountkernfs.sh is not sufficient as
    initscripts might only be unpacked but not configured yet (during
    debootstrap). Thus test if /etc/rcS.d/S??mountkernfs.sh exists. Thanks to
    Felipe Sateler for the idea! (Closes: #824804)
  * init-system-helpers: Ship README.invoke-rc.d and README.policy-rc.d.
  * invoke-rc.d.8: Update the SEE ALSO link to README.policy-rc.d.
    (Closes: #358496)
  * service.8: Document that this starts systemd units as well.
    (Closes: #758139)
  * invoke-rc.d: Show "systemctl status" on a failed (re)start if running
    under systemd. This makes logs with failed package installations much more
    useful, and human users will most likely do exactly that anyway.
    (LP: #1596056)

  [ Luca Boccassi ]
  * dh_systemd_enable: Add support for installing templated service units.
    (Closes: #770344)
  * dh_systemd_enable: Add support for installing path units.
    (Closes: #768609)

 -- Martin Pitt <email address hidden> Tue, 28 Jun 2016 22:10:37 +0200

Changed in init-system-helpers (Ubuntu):
status: Fix Committed → Fix Released
Robie Basak (racb)
tags: added: server-next
Martin Pitt (pitti)
Changed in init-system-helpers (Ubuntu Xenial):
assignee: Martin Pitt (pitti) → nobody
Steve Langasek (vorlon)
description: updated
Steve Langasek (vorlon)
description: updated
Changed in init-system-helpers (Ubuntu Xenial):
assignee: nobody → Steve Langasek (vorlon)
status: Triaged → In Progress
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Steve, or anyone else affected,

Accepted init-system-helpers into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/init-system-helpers/1.29ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in init-system-helpers (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote : [init-system-helpers/xenial] verification still needed

The fix for this bug has been awaiting testing feedback in the -proposed repository for xenial for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
Steve Langasek (vorlon)
description: updated
tags: added: verification-done
removed: removal-candidate verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for init-system-helpers has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package init-system-helpers - 1.29ubuntu4

---------------
init-system-helpers (1.29ubuntu4) xenial; urgency=medium

  * script/invoke-rc.d: backport change from 1.36 so that logs showing a
    failure to start a service with systemd will include the necessary
    information to debug, not just instructions to run an additional
    command. LP: #1596056.

 -- Steve Langasek <email address hidden> Tue, 29 Nov 2016 11:09:41 -0800

Changed in init-system-helpers (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.