ssh.service and ssh.socket both running.

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Libera.chat.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/2020560/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

This is probably not a bug related to ssh but to the configuration changes introduced by Ubuntu.

It is expected that you have both ssh.socket and ssh.service on your system. The ssh.socket unit is enabled by default and is responsible for listening on the configured port. Once it receives a connection it activates ssh.service (which is what actually spawns sshd).

As the comment in /etc/ssh/sshd_config explains, sshd_config(5) and contain additional documentation about how to configure a non-default port. It may help to read about systemd socket units as well [1].

From what I can tell, you have taken the correct steps to enable port 7022. I do not think there is a bug here, but please let me know if you think I am missing something.

[1] https://www.freedesktop.org/software/systemd/man/systemd.socket.html

Thank you for the prompt reply. If my configuration is correct, why is it that I cannot connect on port 7022? I don't see a listener for it, so I suspect my configuration in /etc/systemd/system/ssh.socket.d/listen.conf is either not being read or is being supplanted by the configuration in /lib/systemd/system/ssh.socket (which only specifies ListenStream=22).

Ok. The solution to this problem was to restart not ssh.service but ssh.socket. Thus:
sudo systemctl daemon-reload
sudo systemctl restart ssh.socket

I think the documentation, such as it is, should make this clear. Googling around gets you several hits that say you should sudo systemctl restart ssh (which is ssh.service) which is incorrect.

According to your output from `systemctl status ssh.socket`, it is listening on both ports. But, to be sure, I tried this in a fresh 23.04 container:

root@lunar:~# systemctl edit ssh.socket # This opens a text editor to create /etc/systemd/system/ssh.socket.d/override.conf
root@lunar:~# cat /etc/systemd/system/ssh.socket.d/override.conf
[Socket]
ListenStream=7022
root@lunar:~# systemctl daemon-reload
root@lunar:~# systemctl restart ssh.socket
root@lunar:~# systemctl status ssh.socket
● ssh.socket - OpenBSD Secure Shell server socket
     Loaded: loaded (/lib/systemd/system/ssh.socket; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/ssh.socket.d
             └─override.conf
     Active: active (listening) since Tue 2023-05-23 18:03:31 UTC; 3s ago
      Until: Tue 2023-05-23 18:03:31 UTC; 3s ago
   Triggers: ● ssh.service
     Listen: [::]:22 (Stream)
             [::]:7022 (Stream)
     CGroup: /system.slice/ssh.socket

May 23 18:03:31 lunar systemd[1]: Stopping ssh.socket - OpenBSD Secure Shell server socket...
May 23 18:03:31 lunar systemd[1]: Listening on ssh.socket - OpenBSD Secure Shell server socket.

Then, after copying my public key to the server, I was able to connect to the instance with:

$ ssh -p 7022 root@10.136.78.53

After that, the server shows:

root@lunar:~# lsof -i :7022
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 28u IPv6 5619190 0t0 TCP *:7022 (LISTEN)
sshd 1054 root 4u IPv6 5619190 0t0 TCP *:7022 (LISTEN)
sshd 1055 root 4u IPv6 5613043 0t0 TCP lunar.lxd:7022->_gateway.lxd:50488 (ESTABLISHED)

Also, configuration files in /etc/systemd/ override corresponding files in /lib/systemd/. See https://www.freedesktop.org/software/systemd/man/systemd.unit.html#System%20Unit%20Search%20Path.

As I noted above, the key was restarting ssh.socket, not ssh. Even though systemctl status ssh.socket reported (claimed) that the service was listening on both 22 and 7022, this didn't come into effect until I restarted ssh.socket. This last step was essential to getting this to work. Restarting ssh.socket did not appear in any of the documentation or Googling hits until I found this reference: https://www.flatcar.org/docs/latest/setup/security/customizing-sshd/

I adjusted our documentation for the next Debian upload (which will get merged into Ubuntu at some point and will close this bug then):

  https://salsa.debian.org/ssh-team/openssh/-/commit/a183f4934873b0dd663177e1772090d1d7a6a631

This bug was fixed in the package openssh - 1:9.3p1-1ubuntu1

---------------
openssh (1:9.3p1-1ubuntu1) mantic; urgency=medium

  * Merge with Debian unstable (LP: #2025664). Remaining changes:
    - debian/rules: modify dh_installsystemd invocations for
      socket-activated sshd
    - debian/openssh-server.postinst: handle migration of sshd_config options
      to systemd socket options on upgrade.
    - debian/README.Debian: document systemd socket activation.
    - debian/patches/socket-activation-documentation.patch: Document in
      sshd_config(5) that ListenAddress and Port no longer work.
    - debian/openssh-server.templates: include debconf prompt explaining
      when migration cannot happen due to multiple ListenAddress values
    - debian/.gitignore: drop file
    - debian/openssh-server.postrm: remove systemd drop-ins for
      socket-activated sshd on purge
    - debian/openssh-server.ucf-md5sum: update for Ubuntu delta
    - debian/openssh-server.tmpfile,debian/systemd/ssh.service: Move
      /run/sshd creation out of the systemd unit to a tmpfile config so
      that sshd can be run manually if necessary without having to create
      this directory by hand.
    - debian/patches/systemd-socket-activation.patch: Fix sshd
      re-execution behavior when socket activation is used
    - debian/tests/systemd-socket-activation: Add autopkgtest for systemd socket
      activation functionality.
    - d/p/test-set-UsePAM-no-on-some-tests.patch: set UsePAM=no for some tests
    - Ensure smooth upgrade path from versions affected by LP: #2020474:
      + debian/openssh-server.postint: do not try to restart systemd units,
        and instead indicate that a reboot is required
      + debian/tests/systemd-socket-activation: Reboot the testbed before starting the test
      + debian/rules: Do not stop ssh.socket on upgrade

openssh (1:9.3p1-1) unstable; urgency=medium

  * Debconf translations:
    - Romanian (thanks, Remus-Gabriel Chelu; closes: #1033178).
  * Properly fix date of 1:3.0.2p1-2 changelog entry (closes: #1034425).
  * New upstream release (https://www.openssh.com/releasenotes.html#9.3p1):
    - [CVE-2023-28531] ssh-add(1): when adding smartcard keys to
      ssh-agent(1) with the per-hop destination constraints (ssh-add -h ...)
      added in OpenSSH 8.9, a logic error prevented the constraints from
      being communicated to the agent. This resulted in the keys being added
      without constraints. The common cases of non-smartcard keys and keys
      without destination constraints are unaffected. This problem was
      reported by Luci Stanescu (closes: #1033166).
    - [SECURITY] ssh(1): Portable OpenSSH provides an implementation of the
      getrrsetbyname(3) function if the standard library does not provide
      it, for use by the VerifyHostKeyDNS feature. A specifically crafted
      DNS response could cause this function to perform an out-of-bounds
      read of adjacent stack data, but this condition does not appear to be
      exploitable beyond denial-of-service to the ssh(1) client.
    - ssh-keygen(1), ssh-keyscan(1): accept -Ohashalg=sha1|sha256 when
      outputting SSHFP fingerprints to allow algorit...