ssh (client) stucks in an endless loop with 100% CPU

Bug #1646813 reported by George Shuklin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openssh (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

I have automation suite, which uses ssh to connect to servers.

Under high speed conditions (I was never able to reproduce this manually, but many times in the script), ssh can stuck in an endless loop, causing 100% CPU usage.

strace on it looks like this:

ioctl(5, TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(5, SNDCTL_TMR_CONTINUE or TCSETSF, {B38400 opost isig icanon echo ...}) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGTTOU {si_signo=SIGTTOU, si_code=SI_KERNEL} ---
rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call)
ioctl(5, TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(5, SNDCTL_TMR_CONTINUE or TCSETSF, {B38400 opost isig icanon echo ...}) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
(repeated endlessly)

Command line during call:

timeout 120 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o UpdateHostKeys=no -i /tmp/DIBCTL_ssh_keylN8SGlprivate cloud-user@188.42.217.229 uname -a

(IP address and a private key are changed on every run).

This ssh script is called from a shell script, which is in turn called by python supbrocess.check_call() function, with no special settings for stdout/stderr/stdin (all connected to console).

backtrace:

#0 0x00007fe7ad8f4ed8 in tcsetattr () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000055cbfd2544d6 in ?? ()
#2 0x000055cbfd22cf57 in ?? ()
#3 0x000055cbfd2064b0 in ?? ()
#4 0x000055cbfd208bdd in ?? ()
#5 0x000055cbfd208d28 in ?? ()
#6 0x000055cbfd230c59 in ?? ()
#7 0x000055cbfd2088a4 in ?? ()
#8 0x000055cbfd204b44 in ?? ()
#9 0x000055cbfd1f3f26 in ?? ()
#10 0x00007fe7ad8183f1 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x000055cbfd1f5b0a in ?? ()

Package versions:
ii libc6:amd64 2.24-3ubuntu1 amd64 GNU C Library: Shared libraries
ii openssh-client 1:7.3p1-3build1 amd64 secure shell (SSH) client, for secure access to remote machines

uname:
Linux 4.8.0-26-generic #28-Ubuntu SMP Tue Oct 18 14:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: ssh (not installed)
ProcVersionSignature: Ubuntu 4.8.0-26.28-generic 4.8.0
Uname: Linux 4.8.0-26-generic x86_64
ApportVersion: 2.20.3-0ubuntu8
Architecture: amd64
CurrentDesktop: X-Cinnamon
Date: Fri Dec 2 13:29:50 2016
EcryptfsInUse: Yes
InstallationDate: Installed on 2013-07-19 (1231 days ago)
InstallationMedia: Xubuntu 13.04 "Raring Ringtail" - Release i386 (20130423.1)
SourcePackage: openssh
UpgradeStatus: Upgraded to zesty on 2016-11-22 (9 days ago)

Revision history for this message
George Shuklin (george-shuklin) wrote :
description: updated
Revision history for this message
George Shuklin (george-shuklin) wrote :

I was able to debug issue. It happens when 'timeout 120 ssh' is run without --foreground option for timeout and remote server is asking for password. timeout without --foreground, when run from the script will block all input from stdin, which cause ssh to hangs in the loop.

Revision history for this message
Colin Watson (cjwatson) wrote :

SIGTTOU happens when a background process tries to send output to the terminal. If you're getting this then it generally indicates a configuration error. Try using the -n option, and of course making sure that you have suitable public-key authentication arrangements so that ssh doesn't need to prompt for a passphrase?

     -n Redirects stdin from /dev/null (actually, prevents reading from
             stdin). This must be used when ssh is run in the background. A
             common trick is to use this to run X11 programs on a remote
             machine. For example, ssh -n shadows.cs.hut.fi emacs & will
             start an emacs on shadows.cs.hut.fi, and the X11 connection will
             be automatically forwarded over an encrypted channel. The ssh
             program will be put in the background. (This does not work if
             ssh needs to ask for a password or passphrase; see also the -f
             option.)

Revision history for this message
George Shuklin (george-shuklin) wrote :

Yes, exactly. ssh was asked for password auth and wasn't able to ask password from local keyboard. But this shouldn't cause it to hung forever with 100% CPU.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:7.4p1-5

---------------
openssh (1:7.4p1-5) unstable; urgency=medium

  * Create mux socket for regression tests in a temporary directory.
  * Work around clock_gettime kernel bug on Linux x32 (closes: #849923).

 -- Colin Watson <email address hidden> Tue, 03 Jan 2017 14:43:28 +0000

Changed in openssh (Ubuntu):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.