Comment 38 for bug 1591411

Revision history for this message
Pablo Carranza (pcarranza) wrote : Re: [Bug 1591411] Re: systemd-logind must be restarted every ~1000 SSH logins to prevent a ~25 second delay

We can try the proposed patched version in one worker at gitlab. It will
get stressed quite fast

On Nov 14, 2016 13:25, "Łukasz Zemczak" <email address hidden> wrote:

> @Shay: yes, we will prepare an SRU to the currently supported series
> soon - but please note that the current 'fix' is, in fact, just a
> workaround. But it works.
>
> @Simon: I can try finding some people that could reproduce this easily,
> prepare patched-up version of dbus with both the proposed fixes and ask
> them to run tests on them. Would be really cool if a real fix could be
> found this way. I'll take care of this this week and send feedback here
> and the upstream bug.
>
> Thanks again for looking into this further Simon!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1591411
>
> Title:
> systemd-logind must be restarted every ~1000 SSH logins to prevent a
> ~25 second delay
>
> Status in D-Bus:
> Unknown
> Status in systemd:
> Unknown
> Status in dbus package in Ubuntu:
> Fix Released
> Status in systemd package in Ubuntu:
> Fix Released
> Status in dbus source package in Xenial:
> Confirmed
> Status in systemd source package in Xenial:
> Invalid
> Status in dbus source package in Yakkety:
> Confirmed
> Status in systemd source package in Yakkety:
> Invalid
>
> Bug description:
> I noticed on a system that accepts large numbers of SSH connections
> that after awhile, SSH sessions were taking ~25 seconds to complete.
>
> Looking in /var/log/auth.log, systemd-logind starts failing with the
> following:
>
> Jun 10 23:55:28 test sshd[3666]: pam_unix(sshd:session): session opened
> for user ubuntu by (uid=0)
> Jun 10 23:55:28 test systemd-logind[105]: New session c1052 of user
> ubuntu.
> Jun 10 23:55:28 test systemd-logind[105]: Failed to abandon session
> scope: Transport endpoint is not connected
> Jun 10 23:55:28 test sshd[3666]: pam_systemd(sshd:session): Failed to
> create session: Message recipient disconnected from message bus without
> replying
>
> I reproduced this in an LXD container by doing something like:
>
> lxc launch ubuntu:x test
> lxc exec test -- login -f ubuntu
> ssh-import-id <whatever>
>
> Then ran a script as follows (passing in ubuntu@<container-ip>):
>
> while [ 1 ]; do
> (time ssh $1 "echo OK > /dev/null") 2>&1 | grep ^real >> log
> done
>
> In my case, after 1052 logins, the 1053rd and thereafter were taking
> 25+ seconds to complete. Here are some snippets from the log file:
>
> $ cat log | grep 0m0 | wc -l
> 1052
>
> $ cat log | grep 0m25 | wc -l
> 4
>
> $ tail -5 log
> real 0m0.222s
> real 0m25.232s
> real 0m25.235s
> real 0m25.236s
> real 0m25.239s
>
> ProblemType: Bug
> DistroRelease: Ubuntu 16.04
> Package: systemd 229-4ubuntu5
> ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
> Uname: Linux 4.4.0-22-generic x86_64
> ApportVersion: 2.20.1-0ubuntu2
> Architecture: amd64
> Date: Sat Jun 11 00:09:34 2016
> MachineType: Notebook W230SS
> ProcEnviron:
> TERM=xterm-256color
> PATH=(custom, no user)
> ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-22-generic
> root=/dev/mapper/ubuntu--vg-root ro quiet splash
> SourcePackage: systemd
> SystemdDelta:
> [EXTENDED] /lib/systemd/system/rc-local.service →
> /lib/systemd/system/rc-local.service.d/debian.conf
> [EXTENDED] /lib/systemd/system/systemd-timesyncd.service →
> /lib/systemd/system/systemd-timesyncd.service.d/disable-
> with-time-daemon.conf
>
> 2 overridden configuration files found.
> UpgradeStatus: No upgrade log present (probably fresh install)
> dmi.bios.date: 04/15/2014
> dmi.bios.vendor: American Megatrends Inc.
> dmi.bios.version: 4.6.5
> dmi.board.asset.tag: Tag 12345
> dmi.board.name: W230SS
> dmi.board.vendor: Notebook
> dmi.board.version: Not Applicable
> dmi.chassis.asset.tag: No Asset Tag
> dmi.chassis.type: 9
> dmi.chassis.vendor: Notebook
> dmi.chassis.version: N/A
> dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr4.6.5:bd04/15/2014:
> svnNotebook:pnW230SS:pvrNotApplicable:rvnNotebook:
> rnW230SS:rvrNotApplicable:cvnNotebook:ct9:cvrN/A:
> dmi.product.name: W230SS
> dmi.product.version: Not Applicable
> dmi.sys.vendor: Notebook
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/dbus/+bug/1591411/+subscriptions
>