Ubuntu
openssh package

openssh-server hangs with AuthorizedKeysCommand

Bug #1877454 reported by Jeremy Norris
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openssh (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Medium
Sergio Durigan Junior

Bug Description

[Impact]

On a default Xenial install, when sshd is configured to obtain the list of allowed keys using AuthorizedKeysCommand (or the list of allowed certificate principals using AuthorizedPrincipalsCommand), and if the command pointed by Authorized{Keys,Principals}Command generates a lot of output, sshd will hang while reading this same output. In a nutshell, the problem happens when the subprocess generates enough data to fill the pipe's buffer; in this scenario, sshd will wait(2) on the subprocess, which will be blocked trying to write the rest of the output.

[Test Case]

In order to reproduce the bug, one can:

$ lxc launch ubuntu-daily:xenial openssh-server-bug1877454
$ lxc shell openssh-server-bug1877454
# ssh-keygen
(no need to choose a passphrase for the key, just hit ENTER on all prompts)
# cat > authkeyscommand.sh << __EOF__
#!/bin/bash

cat /root/.ssh/id_rsa.pub
echo
head -c 1M < /dev/urandom
__EOF__
# chmod +x authkeyscommand.sh
# cat >> /etc/ssh/sshd_config << __EOF__
AuthorizedKeysCommand /root/authkeyscommand.sh
AuthorizedKeysCommandUser root
__EOF__
# systemctl reload sshd.service
# ssh root@127.0.0.1

You will notice that ssh will stay there waiting for sshd's reply, which won't come. The expected result would be for ssh to succeed.

[Regression Potential]

Since the affected code deals with executing a subprocess, reading its output through a pipe, and then relying on wait(2) to determine whether the subprocess exited correctly; and considering that this code is written in C without the help of features like RAII and with the use of goto statements, we are not able to disconsider the chances of making a mistake and forgetting to free a resource or to properly read/write from/to pipes, for example. This is, after all, the reason the bug happened in the first place.

Having said that, openssh contains extensive tests and the code is well organized and relatively easy to follow. Upon close inspection, there doesn't seem to be an evident problem with the backported fixes.

As usual when dealing with a somewhat older distribution, there is always the possibility of encountering problems because we will be recompiling openssh using the most recent versions of its build dependencies.

[Original Description]

Please consider applying this change to openssh-server distributed in Xenial (16.04).
Without it, sshd can sporadically hang when making use of the `AuthorizedKeysCommand` option.

https://github.com/openssh/openssh-portable/commit/ddd3d34e5c7979ca6f4a3a98a7d219a4ed3d98c2

See original description

Related branches

~sergiodj/ubuntu/+source/openssh:bug1877454-hangs-authorizedkeyscommand
Rafael David Tinoco (community): Approve
Canonical Server: Pending requested
Canonical Server Core Reviewers: Pending requested
Diff: 159 lines (+131/-0)
4 files modified
debian/changelog (+14/-0)
debian/patches/authkeyscommand-deadlock-01.patch (+41/-0)
debian/patches/authkeyscommand-deadlock-02.patch (+74/-0)
debian/patches/series (+2/-0)
Revision history for this message
Jeremy Norris (jeremy.norris) wrote :

Additional discussion of this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1496467

https://bugzilla.mindrot.org/show_bug.cgi?id=2496

https://discourse.phabricator-community.org/t/newly-added-ssh-keys-not-working/992

Revision history for this message
Jeremy Norris (jeremy.norris) wrote :

Also see:

https://bugzilla.mindrot.org/show_bug.cgi?id=2655

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks for the report Jeremy,
The fix is in V_7_5_P1 and by that >=Bionic is fixed - I agree to you this only has to target Xenial.

tags: added: bitesize server-next
Changed in openssh (Ubuntu Xenial):
status: New → Confirmed
importance: Undecided → Medium
Changed in openssh (Ubuntu):
status: New → Fix Released
Revision history for this message
Jeremy Norris (jeremy.norris) wrote :

Also see:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=905226

Revision history for this message
Jeremy Norris (jeremy.norris) wrote :

Another link discussing it:
https://secure.phabricator.com/T11827

Sergio Durigan Junior (sergiodj)
Changed in openssh (Ubuntu Xenial):
assignee: nobody → Sergio Durigan Junior (sergiodj)
Sergio Durigan Junior (sergiodj)
description: updated
Sergio Durigan Junior (sergiodj)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Jeremy, or anyone else affected,

Accepted openssh into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:7.2p2-4ubuntu2.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssh (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Just double checked that the fix works and tagged as verification-done-xenial.

tags: added: verification-done-xenial
removed: verification-needed verification-needed-xenial
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Thank you for the verification Sergio! Could you please note down which package version has been used for verification and what kind of verification has been performed?

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote : Re: [Bug 1877454] Re: openssh-server hangs with AuthorizedKeysCommand

On Thursday, June 25 2020, Łukasz Zemczak wrote:

> Thank you for the verification Sergio! Could you please note down which
> package version has been used for verification and what kind of
> verification has been performed?

Hi Łukasz,

The package version I tested is 1:7.2p2-4ubuntu2.10. I performed the
steps listed in the "Test Case" section of the SRU above. Everything
worked as expected:

$ ssh root@127.0.0.1
Warning: Permanently added '127.0.0.1' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 5.4.0-31-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

$

Thanks!

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:7.2p2-4ubuntu2.10

---------------
openssh (1:7.2p2-4ubuntu2.10) xenial; urgency=medium

  * Fix deadlock when AuthorizedKeysCommand produces a large output.
    (LP: #1877454)
    - d/p/authkeyscommand-deadlock-01.patch: Make sure to call fclose(2)
      and assign NULL to the file handler used to read
      "Authorized{Keys,Principal}Command" directives.
    - d/p/authkeyscommand-deadlock-02.patch: Consume entire output
      generated by the command pointed by
      "Authorized{Keys,Principal}Command" in order to avoid sending a
      SIGPIPE to the process.

openssh (1:7.2p2-4ubuntu2.9) xenial; urgency=medium

  * Apply upstream patch to stop using 2020 as a future date in regress
    tests. LP: #1859013

 -- Sergio Durigan Junior <email address hidden> Wed, 13 May 2020 10:12:28 -0400

Changed in openssh (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for openssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.