Port parameter sshd_config is 22 AND whatever you specify

Bug #1876320 reported by Adriaan van Nijendaal
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
portable OpenSSH
Unknown
Unknown
openssh (Debian)
Fix Released
Unknown
openssh (Ubuntu)
Fix Released
Low
Unassigned
Focal
Fix Released
Low
Unassigned

Bug Description

[Impact]

 * The "Port" option in sshd_config is accumulative, but due to a bug re-
   adds the default when an include is encountered. Therefore we have these
   cases
   a) Port 722
      Listens on 722 (correct)
   b) Port 722
      Port 2222
      Listens on 722 & 2222 (correct)
   c) Port 722
      include /path/to/otherconfig
      Listens on 722 & 22 (applied defaults as if Port was unset)

 * Of the above (c) is a bug, not documented that way and can lead to open
   ports not expected and not wanted.

[Test Case]

* Test if defaults are applied even if option is specified

Rename sshd_config to something_else and replace sshd_config with two lines to include the original config (now called something_else) and set the Port to 7722:

systemctl stop ssh
mv /etc/ssh/sshd_config /etc/ssh/something_else
cat > /etc/ssh/sshd_config <<EOF
Include /etc/ssh/something_else
Port 7722
EOF
systemctl start ssh
systemctl status ssh
# restore the original config:
mv /etc/ssh/something_else /etc/ssh/sshd_config

Which will show:

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-05-02 15:31:37 UTC; 13s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 45261 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 45271 (sshd)
      Tasks: 1 (limit: 18457)
     Memory: 1.3M
     CGroup: /system.slice/ssh.service
             └─45271 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

May 02 15:31:37 cabernet systemd[1]: Starting OpenBSD Secure Shell server...
May 02 15:31:37 cabernet sshd[45271]: Server listening on 0.0.0.0 port 7722.
May 02 15:31:37 cabernet sshd[45271]: Server listening on :: port 7722.
May 02 15:31:37 cabernet sshd[45271]: Server listening on 0.0.0.0 port 22.
May 02 15:31:37 cabernet sshd[45271]: Server listening on :: port 22.
May 02 15:31:37 cabernet systemd[1]: Started OpenBSD Secure Shell server.

So, NOW it will have ports 22 AND 7722 open!
With the fix this should no more happen.

* Test if multiple options still work

[Regression Potential]

 * The change itself isn't very invasive and I don't expect it to break it
   with crashes or similar.
   But if people didn't realize that this is a bug, they might have a
   config in place and somewhat rely on the broken behavior.
   It is good thou that (a) (b) of above are the common cases and won't
   change.
   Further even if a user used (c) the explicitly configured port will
   still work.
   Fortunately it is early in the Focal lifetime and it was the one
   introducting the 'include' feature - therefore I'd expect not too many
   people using it yet.

[Other Info]

 * n/a

----

On my Ubuntu Server 20.04 LTS with OpenSSH 1:8.2p1-4, I have TWO sshd deamons. One (on port 22) is for internal use, accepts passwords etc. The second (on port 7722) does not allow PAM use and no passwords, allows only one user(name) and uses an alternative autorized_keys file (that only root can edit).

Any parameter FIRST encountered in sshd_config is the one that is accepted; others do not override (like in many other config files). There is one exception: 'Port', which is accumulative. To make life easier, I set the more restrictive parameters for port 7722 first and next include the system-default /etc/ssh/sshd_config.

The /etc/ssh/sshd_config file(s) in Ubuntu Server 20.04 DO NOT specify 'Port' anywhere - the default is 22. But: it is obviously still accumulative: Setting 'Port' to 7722 makes sshd listen on port 7722 AND 22. This is unwanted.

Proposed solution: Remove the accumulative behavior for 'Port' and REQUIRE the 'Port' parameter like before (and maybe have second and later parameters override the earlier ones, like 'everyone else').

Regards,

Adriaan

PS Searching for solutions, I found that specifying 'ListenAddress 0.0.0.0:7722' stops sshd from listening to port 22. This, however, is not documented in 'man 5 sshd_config' and may be an unreliable side-effect.

Related branches

Revision history for this message
Simon Déziel (sdeziel) wrote :

@Adriaan, are there really 2 sshd running? Or is it only one binding to the 2 ports and applying different parameter using Match conditions? Beware what on 20.04, there is support for additional config snippets dropped in /etc/ssh/sshd_config.d/*.conf.

To check for 2 daemons:

sudo ss -nltp | grep sshd

Revision history for this message
Simon Déziel (sdeziel) wrote :

On a stock install, adding "Port 7722" to /etc/ssh/sshd_config and restarting sshd gives me this:

# ss -nltp | grep sshd
LISTEN 0 128 0.0.0.0:7722 0.0.0.0:* users:(("sshd",pid=10651,fd=3))
LISTEN 0 128 [::]:7722 [::]:* users:(("sshd",pid=10651,fd=4))

So 1 daemon, bounding to port 7722 on IPv4 and IPv6 wildcards.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Check also `systemctl cat ssh.service` and `systemctl cat secondssh.service` -- sshd also accepts parameters on the commandline, perhaps the port is being specified outside of the configuration files.

Thanks

Revision history for this message
Adriaan van Nijendaal (adriaan-choam) wrote :
Download full text (6.7 KiB)

Thank you Simon and Seth, for your quick response. I have looked into this further by installing a virgin Ubuntu Server 20.04 with SSH.

Changing the 'Port' in /etc/ssh/sshd_config to 7722 starts sshd on port 7722 and NOT on port 22, just like you said, Simon. BUT:

Rename sshd_config to something_else and replace sshd_config with two lines to include the original config (now called something_else) and set the Port to 7722:

systemctl stop ssh
mv /etc/ssh/sshd_config /etc/ssh/something_else
cat > /etc/ssh/sshd_config <<EOF
Include /etc/ssh/something_else
Port 7722
EOF
systemctl start ssh
systemctl status ssh
# restore the original config:
mv /etc/ssh/something_else /etc/ssh/sshd_config

Which will show:

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-05-02 15:31:37 UTC; 13s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 45261 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 45271 (sshd)
      Tasks: 1 (limit: 18457)
     Memory: 1.3M
     CGroup: /system.slice/ssh.service
             └─45271 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

May 02 15:31:37 cabernet systemd[1]: Starting OpenBSD Secure Shell server...
May 02 15:31:37 cabernet sshd[45271]: Server listening on 0.0.0.0 port 7722.
May 02 15:31:37 cabernet sshd[45271]: Server listening on :: port 7722.
May 02 15:31:37 cabernet sshd[45271]: Server listening on 0.0.0.0 port 22.
May 02 15:31:37 cabernet sshd[45271]: Server listening on :: port 22.
May 02 15:31:37 cabernet systemd[1]: Started OpenBSD Secure Shell server.

So, NOW it will have ports 22 AND 7722 open!

If one sets debug level 3 in /etc/default/ssh (SSHD_OPTS="-d -d -d"), syslog will show that 'something_else' is read from line 1 in sshd_config and that the Port is set afterwards (and not anywhere in /etc/ssh/something_else).

May 2 15:34:01 cabernet systemd[1]: Stopping OpenBSD Secure Shell server...
May 2 15:34:01 cabernet systemd[1]: ssh.service: Succeeded.
May 2 15:34:01 cabernet systemd[1]: Stopped OpenBSD Secure Shell server.
May 2 15:34:01 cabernet systemd[1]: Starting OpenBSD Secure Shell server...
May 2 15:34:01 cabernet sshd[45345]: debug2: load_server_config: filename /etc/ssh/sshd_config
May 2 15:34:01 cabernet sshd[45345]: debug2: load_server_config: done config len = 43
May 2 15:34:01 cabernet sshd[45345]: debug2: parse_server_config_depth: config /etc/ssh/sshd_config len 43
May 2 15:34:01 cabernet sshd[45345]: debug2: /etc/ssh/sshd_config line 1: new include /etc/ssh/something_else
May 2 15:34:01 cabernet sshd[45345]: debug2: /etc/ssh/sshd_config line 1: including /etc/ssh/something_else
May 2 15:34:01 cabernet sshd[45345]: debug2: load_server_config: filename /etc/ssh/something_else
May 2 15:34:01 cabernet sshd[45345]: debug2: load_server_config: done config len = 296
May 2 15:34:01 cabernet sshd[45345]: debug2: parse_server_config_depth: config /etc/ssh/something_else len 296
May 2 15:34:01 cabernet sshd[45345]: debug2: /etc/ssh/something_else line 13: new include /etc/ssh/sshd_config.d/*.conf
May 2 15:34...

Read more...

Revision history for this message
Paride Legovini (paride) wrote :

@Adriaan thanks for providing some minimal steps to reproduce the problem, I indeed can reproduce it. Interestingly reversing the two sshd_config lines, like this:

  Port 7722
  Include /etc/ssh/something_else

causes sshd to listen only on port 7722. I think this is an upstream OpenSSH bug, and should be reported to the upstream portable OpenSSH bug tracker:

  https://bugzilla.mindrot.org/

I had a look at the existing bugs but only found this one related to the Include functionality:

  https://bugzilla.mindrot.org/show_bug.cgi?id=3122

It's a problem specific to Match stanzas, so I don't think it applies here, however it tells us there are probably still some edge cases to iron out. Do you think you can follow up and file a bug upstream? If you do, please link to it here. Thanks!

Changed in openssh (Ubuntu):
status: New → Incomplete
Revision history for this message
Adriaan van Nijendaal (adriaan-choam) wrote :

(Finally) found the time to report it to OpenSSH: https://bugzilla.mindrot.org/show_bug.cgi?id=3169 Thanks, Paride, Seth and Simon, for your help.

Revision history for this message
Paride Legovini (paride) wrote :

Excellent, thanks Adriaan! I linked this bug report to the upstream bug, so its status will be automatically monitored by Launchpad.

Changed in openssh (Ubuntu):
status: Incomplete → Triaged
importance: Undecided → Low
Changed in openssh (Ubuntu Focal):
status: New → Triaged
importance: Undecided → Low
Revision history for this message
Paride Legovini (paride) wrote :

The upstream bug now has a patch attached, so I'm tagging this server-next.

tags: added: server-next
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI: "Include" functionality was added in 8.2 by upstream.
Thereby as already triaged only >=Focal is affected

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I have prepared a PPA with the fix and would appreciate if that could be tried and tested:
@Adriaan - would you be so kind to confirm that this fixes your real case?

PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4080

Furthermore as I expect this will work well I also have merge proposals to fix this in Groovy and later Focal:
https://code.launchpad.net/~paelzer/ubuntu/+source/openssh/+git/openssh/+merge/384813
https://code.launchpad.net/~paelzer/ubuntu/+source/openssh/+git/openssh/+merge/384814

Revision history for this message
Adriaan van Nijendaal (adriaan-choam) wrote :

I have installed (another) fresh Ubuntu Server, got all the updates (apt dist-upgrade), added the PPA (add-apt-repository ppa:ci-train-ppa-service/4080) and updated again: openssh server, client and sftp were replaced. Juggled config files and lines in config files as described above, and:

The problem has gone away.

Question: should I use this PPA for a production system? Or should I (and maybe others) keep using work-arounds and wait for OpenSSH 8.4 to show up?

Revision history for this message
Paride Legovini (paride) wrote :

Thanks for verifying. The patch will be applied to the OpenSSH version already in Focal, so OpenSSH will stay at version 8.2 in Focal.

Christian set up the PPA specifically for testing the patched package, it's not meant for production use. It's unlikely that you'll hit any surprise by using it, but it's entirely up to you. The patched package meant for production will eventually land in focal-updates.

Changed in openssh (Ubuntu):
status: Triaged → In Progress
Changed in openssh (Ubuntu Focal):
status: Triaged → In Progress
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

MPs ready, I asked cjwatson if in Groovy he prefers this as Ubuntu Delta or as a new Debian upload that will sync in. Once Groovy is complete either way we can start the SRU process for focal.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Ok, uploaded to Groovy as Delta for now.

In addition I filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962035 to later on be able to make it a sync again.

Changed in openssh (Debian):
status: Unknown → New
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:8.2p1-4ubuntu1

---------------
openssh (1:8.2p1-4ubuntu1) groovy; urgency=medium

  * d/p/lp-1876320-*: avoid applying defaults for every include statement
    (LP: #1876320)

 -- Christian Ehrhardt <email address hidden> Fri, 29 May 2020 09:37:09 +0200

Changed in openssh (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI
- 1:8.2p1-4ubuntu1 migrated into groovy
- 1:8.3p1-1 got into Debian containing the fix, so this can be a sync again
- added SRU template here
- uploaded to Focal-unapproved

description: updated
Changed in openssh (Debian):
status: New → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Adriaan, or anyone else affected,

Accepted openssh into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.2p1-4ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssh (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (openssh/1:8.2p1-4ubuntu0.1)

All autopkgtests for the newly accepted openssh (1:8.2p1-4ubuntu0.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.44.1-1ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI: The tests were already retried (thanks seb128) and are good now

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Download full text (4.5 KiB)

Retest as-is:

root@f:~# dpkg -l openssh-server
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================================================
ii openssh-server 1:8.2p1-4 amd64 secure shell (SSH) server, for secure access from remote machines
root@f:~# systemctl stop ssh
root@f:~# mv /etc/ssh/sshd_config /etc/ssh/something_else
root@f:~# cat > /etc/ssh/sshd_config <<EOF
> Include /etc/ssh/something_else
> Port 7722
> EOF
root@f:~# systemctl start ssh
root@f:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2020-06-17 11:23:58 UTC; 4s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 60797 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 60798 (sshd)
      Tasks: 1 (limit: 38269)
     Memory: 2.6M
     CGroup: /system.slice/ssh.service
             └─60798 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Jun 17 11:23:58 f systemd[1]: Starting OpenBSD Secure Shell server...
Jun 17 11:23:58 f sshd[60798]: Server listening on 0.0.0.0 port 7722.
Jun 17 11:23:58 f sshd[60798]: Server listening on :: port 7722.
Jun 17 11:23:58 f systemd[1]: Started OpenBSD Secure Shell server.
Jun 17 11:23:58 f sshd[60798]: Server listening on 0.0.0.0 port 22.
Jun 17 11:23:58 f sshd[60798]: Server listening on :: port 22.

Confirmed.

Installing from proposed:
root@f:~# apt install openssh-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  openssh-client openssh-sftp-server
Suggested packages:
  keychain libpam-ssh monkeysphere ssh-askpass molly-guard
The following packages will be upgraded:
  openssh-client openssh-server openssh-sftp-server
3 upgraded, 0 newly installed, 0 to remove and 38 not upgraded.
Need to get 1101 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 openssh-sftp-server amd64 1:8.2p1-4ubuntu0.1 [51.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 openssh-server amd64 1:8.2p1-4ubuntu0.1 [377 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 openssh-client amd64 1:8.2p1-4ubuntu0.1 [672 kB]
Fetched 1101 kB in 0s (2445 kB/s)
Preconfiguring packages ...
(Reading database ... 55167 files and directories currently installed.)
Preparing to unpack .../openssh-sftp-server_1%3a8.2p1-4ubuntu0.1_amd64.deb ...
Unpacking openssh-sftp-server (1:8.2p1-4ubuntu0.1) over (1:8.2p1-4) ...
Preparing to unpack .../openssh-server_1%3a8.2p1-4ubuntu0.1_amd64.deb ...
Unpacking openssh-server (1:8.2p1-4ubuntu0.1) over (1:8.2p1-4) ...
Preparing to unpack .../openssh-client_1%3a8.2p1-4ubuntu0.1_amd64.deb ...
Unpacking openssh-client (1:8.2...

Read more...

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for openssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:8.2p1-4ubuntu0.1

---------------
openssh (1:8.2p1-4ubuntu0.1) focal; urgency=medium

  * d/p/lp-1876320-*: avoid applying defaults for every include statement
    (LP: #1876320)

 -- Christian Ehrhardt <email address hidden> Fri, 29 May 2020 09:37:09 +0200

Changed in openssh (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.