restarting services hangs on systemd-tty-ask-password-agent

Also unable to restart sssd.

Does that happen with any service? Or is that specific to smbd and sssd? I suppose 'service' is not the culprit here, and it also happens if you directly call "sudo systemctl restart smbd.service"?

Can you please do

  sudo systemd-analyze set-log-level debug
  sudo systemctl restart systemd-udevd
  sudo systemd-analyze set-log-level debug
  sudo journalctl -b > /tmp/journal.txt

and attach /tmp/journal.txt here?

Another thing, can you please give me the output of

  systemctl list-jobs
  systemctl status|head

thanks!

I haven't extensively tested all services, but it looks like anything set up similarly to smbd. Some have different behaviour, such as network-manager which immediately returns with:

% sudo restart network-manager
[sudo] password for jgoguen:
restart: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

But in general it looks like these services all fail. If the NM one is actually something else I'll happily submit another bug against the network-manager package.

The commands you asked for:

% systemctl list-jobs
No jobs running.

% systemctl status|head
● jgoguen-test-carbon
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Fri 2015-06-12 13:40:33 PDT; 11min ago
   CGroup: /
           ├─ 1 /sbin/init splash
           ├─914 /sbin/cgmanager -m name=systemd
           ├─system.slice
           │ ├─avahi-daemon.service

> % sudo restart network-manager

That doesn't work. "restart" is an upstart specific command. You need to use the init-neutral "service" or "systemctl".

Will look at the logs soon, thanks!

Ahh, yes, whoops. I tried again with 'systemctl restart network-manager' (and, just in case, I also tried 'systemctl restart network-manager.service') and the behaviour is identical to smbd et. al. - it hangs with subprocesses /bin/systemd-tty-ask-password-agent and /usr/bin/pkttyagent.

Ran into something similar just now. A bit of internet searching suggested<1><2> that it was due to systemd's internal state getting out of sync, and indeed running "systemctl daemon-reload" made it go away.

<1> https://github.com/NixOS/nixpkgs/issues/2584
<2> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774153

Sorry, "systemctl daemon-reexec".

Hi,
I have same problem, when running systemctl stop/start from background bash script...

18581 pts/0 S 0:00 \_ su -
18582 pts/0 S 0:00 \_ -bash
18668 pts/0 S 0:00 \_ /usr/bin/bash /sbin/backupninja -d --run /etc/backup.d/30.redis-restore.sh
18760 pts/0 S 0:00 | \_ /usr/bin/bash /sbin/backupninja -d --run /etc/backup.d/30.redis-restore.sh
18762 pts/0 S 0:00 | | \_ /usr/bin/bash /sbin/backupninja -d --run /etc/backup.d/30.redis-restore.sh
18763 pts/0 S 0:00 | | \_ systemctl stop redis_6379
18764 pts/0 S 0:00 | | \_ /usr/bin/systemd-tty-ask-password-agent --watch
18761 pts/0 S 0:00 | \_ /usr/bin/bash /sbin/backupninja -d --run /etc/backup.d/30.redis-restore.sh

the process just hangs.

I noticed that "systemctl daemon-reexec" just kill somehow last staled process, if I have second staled process.

This is the output after daemon-reexec:

---
Warning! D-Bus connection terminated.
Disconnected from bus.
redis_6379.service - SYSV: Redis is a persistent key-value database
Loaded: loaded (/etc/rc.d/init.d/redis_6379)
Active: deactivating (stop) since Tue 2015-11-24 00:49:05 EET; 8min ago
Control: 17618 (redis_6379)
CGroup: /system.slice/redis_6379.service
├─17263 /usr/local/bin/redis-server *:6379
└─control
├─17618 /bin/sh /etc/rc.d/init.d/redis_6379 stop
└─19422 sleep 1

Nov 24 00:57:53 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:54 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:55 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:56 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:57 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:58 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:57:59 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:58:00 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:58:01 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:58:02 offmedia-bms-stage.localdomain redis_6379[17618]: Waiting for Redis to shutdown ...
Nov 24 00:58:03 offmedia-bms-stage.localdomain systemd[1]: Starting SYSV: Redis is a persistent key-value database...
Info: <<<< finished action /etc/backup.d/30.redis-restore.sh: SUCCESS
Debug: send report to root
Info: FINISHED: 1 actions run. 0 fatal. 0 error. 0 warning.

I didn't see this myself, but this could have been caused by bug 1565617 which is now fixed. Can you please try if you still get this with Ubuntu 16.04?

I was having so many problems with Ubuntu, bugs take so long to get responses, and almost always it's solved by upgrading the package, so I saved myself a bunch of hassle by just switching to Arch.

I'm still seeing this with Ubuntu 16.04.2.

Same here, sample ps auxf output: http://paste.ubuntu.com/26112646/

In the upgrade there was a systemd update at the same time as the other package update, which seems to be the trigger according to the debian bug.

This crap here drives me crazy.

Im having this on an apt-installation of "redis-server".

Ubuntu 18.04

systemctl / networkctl all clean.
kernel is not booted with console parameters (also with console parameters doesnt work)

I tried like everything, even replacing this stupid ..
/bin/systemd-tty-ask-password-agent with /bin/true

Im running everything as root to avoid permission issues and theres no fucking way to bypass this stupid /bin/systemd-tty-ask-password-agent

Im currently having NO WAY to install anything on the system because its stuck on this stupid process...

Fucking get rid of this shit or bring a solution to enter the password.

Already spend 2 hours on it.

These two worked for me...

systemctl start --no-ask-password ...

or...

service ... start

I am running into this problem on Ubuntu 20.04.3 LTS (focal) when running `dpkg --configure -a`, which is hanging when setting up `base-files`. From diagnosis so far, it appears `dpkg --configure -a` runs `/var/lib/dpkg/info/base-files.postinst`, which runs `deb-systemd-invoke start motd-news.timer`, which executes `systemd-tty-ask-password-agent --watch`, which waits forever (without asking for a password on the terminal). Detailed diagnosis:

$ sudo strace -o /tmp/dpkg-strace -tt -ff dpkg --debug=222 --configure -a
Setting up base-files (11ubuntu5.4) ...
D000002: fork/exec /var/lib/dpkg/info/base-files.postinst ( configure 11ubuntu5.3 )

Running this `postinst` script manually with `bash -xv` hangs while executing `deb-systemd-invoke start motd-news.timer`, something also revealed by `pstree` (for a case running without strace):

bash───sudo───dpkg───base-files.post───systemctl───systemd-tty-ask

Strace outputs around 10k lines. One of the strace output files, which appears to correspond to the command `deb-systemd-invoke start motd-news.timer`, ends with a `ppoll`, apparently waiting for a socket. Selection from `deb-systemd-invoke start motd-news.timer` (PID 51072):

10:50:46.673278 close(10) = 0
10:50:46.673319 execve("/usr/bin/deb-systemd-invoke", ["deb-systemd-invoke", "start", "motd-news.timer"], 0x56297d000c88 /* 37 vars */) = 0
10:50:46.673511 brk(NULL) = 0x55fff0539000
10:50:46.673534 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd593b7540) = -1 EINVAL (Invalid argument)
10:50:46.673574 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
10:50:46.673613 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
10:50:46.673641 fstat(3, {st_mode=S_IFREG|0644, st_size=160254, ...}) = 0
10:50:46.673663 mmap(NULL, 160254, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3bdfb5e000
10:50:46.673686 close(3) = 0
10:50:46.673713 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
[...]
10:50:46.708457 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
[...]
10:50:46.710298 sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\4\1\24\0\0\0\2\0\0\0\227\0\0\0\1\1o\0\31\0\0\0/org/fre"..., iov_len=168}, {iov_base="\17\0\0\0motd-news.timer\0", iov_len=20}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 188
10:50:46.710350 recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\2\1\0017\0\0\0\3\0\0\0007\0\0\0\5\1u\0\2\0\0\0", iov_len=24}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 24
10:50:46.710422 recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\7\1s\0\30\0\0\0org.freedesktop.systemd1"..., iov_len=103}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 103
10:50:46.710472 sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\4\19\0\0\0\3\0\0\0\250\0\0\0\1\1o\0002\0\0\0/org/fre"..., iov_len=184}, {iov_base="\35\0\0\0org.freedesktop.systemd1.Uni"..., iov_len=57}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 241
10:50:46.710522 recvmsg(3, {msg_namelen=0}, MSG_DO...

Just faced this bug on 20.04.3 with service apache2 restart.
As noted earlier 'systemctl daemon-reload' did fix it, but this workaround is rather not an obvious solution.