Error message "Operation failed: No such file or directory" on installing/reconfiguring systemd

Can you please do this:

  sudo sed -i '2 s/^/set -x/' /var/lib/dpkg/info/systemd.postinst
  sudo dpkg-reconfigure systemd

and copy&paste the entire output? Thanks!

The error message appears after the line "systemctl enable tmp.mount". Manually calling this command does also show the error:

root@ubuntu:~# systemctl enable tmp.mount
Operation failed: No such file or directory

Ah, so you set RAMTMP=yes in /etc/default/rcS. There's a daemon-reload missing indeed.

Thanks!

An explicit reload should not be necessary for "enable".
I also can't reproduce the problem:

$ cp /usr/share/systemd/tmp.mount /etc/systemd/system/tmp2.mount
$ systemctl enable tmp2.mount
Created symlink from /etc/systemd/system/local-fs.target.wants/tmp2.mount to /etc/systemd/system/tmp2.mount.

> Ah, so you set RAMTMP=yes in /etc/default/rcS.

This file does contain only commented and empty lines on my system.

Ah, I thought this would happen if there already was a tmp.mount in systemd's brain (e. g. from a manual mount or fstab) and then adding the unit to /etc would cause that. But indeed I cannot reproduce this either.

So, Sworddragon, please do the steps in comment #1.

root@ubuntu:~# sed -i '2 s/^/set -x/' /var/lib/dpkg/info/systemd.postinst
root@ubuntu:~# dpkg-reconfigure systemd
+ set -e
+ [ configure = triggered ]
+ dpkg --compare-versions 229-1ubuntu2 lt 214-1
+ dpkg --compare-versions 229-1ubuntu2 lt 218-11~
+ dpkg --compare-versions 229-1ubuntu2 lt 228-5ubuntu3
+ dpkg --compare-versions 229-1ubuntu2 lt 204-9
+ dpkg --compare-versions 229-1ubuntu2 lt 204-8
+ systemd-machine-id-setup
+ addgroup --system systemd-journal
addgroup: The group `systemd-journal' already exists as a system group. Exiting.
+ adduser --quiet --system --group --no-create-home --home /run/systemd --gecos systemd Time Synchronization systemd-timesync
+ adduser --quiet --system --group --no-create-home --home /run/systemd/netif --gecos systemd Network Management systemd-network
+ adduser --quiet --system --group --no-create-home --home /run/systemd/resolve --gecos systemd Resolver systemd-resolve
+ adduser --quiet --system --group --no-create-home --home /run/systemd --gecos systemd Bus Proxy systemd-bus-proxy
+ _update_catalog
+ journalctl --update-catalog
+ setcap cap_dac_override,cap_sys_ptrace=ep /usr/bin/systemd-detect-virt
+ dpkg --compare-versions 229-1ubuntu2 ge 204
+ [ -e /run/systemd/was-enabled ]
+ read UNIT
+ [ <email address hidden> = tmp.mount ]
+ systemctl enable <email address hidden>
+ read UNIT
+ [ remote-fs.target = tmp.mount ]
+ systemctl enable remote-fs.target
+ read UNIT
+ [ tmp.mount = tmp.mount ]
+ dpkg --compare-versions 229-1ubuntu2 lt-nl 220-6~
+ systemctl enable tmp.mount
Operation failed: No such file or directory
+ true
+ read UNIT
+ [ systemd-timesyncd.service = tmp.mount ]
+ systemctl enable systemd-timesyncd.service
+ read UNIT
+ [ -n 229-1ubuntu2 ]
+ _systemctl daemon-reexec
+ [ -d /run/systemd/system ]
+ _systemctl try-restart systemd-networkd.service
+ [ -d /run/systemd/system ]
+ _systemctl try-restart systemd-resolved.service
+ [ -d /run/systemd/system ]
+ _systemctl try-restart systemd-timesyncd.service
+ [ -d /run/systemd/system ]
+ dpkg --compare-versions 229-1ubuntu2 lt-nl 228-5~
+ _systemctl is-active --quiet resolvconf
+ [ -d /run/systemd/system ]
+ _systemctl start systemd-networkd-resolvconf-update.path
+ [ -d /run/systemd/system ]
+ dpkg --compare-versions 229-1ubuntu2 lt 221-1
+ [ configure = configure ]
+ [ -d /run/systemd/system ]
+ dpkg-maintscript-helper rm_conffile /etc/bash_completion.d/systemctl-bash-completion.sh 204-1~ -- configure 229-1ubuntu2
+ dpkg-maintscript-helper rm_conffile /etc/bash_completion.d/systemd-bash-completion.sh 204-1~ -- configure 229-1ubuntu2
+ dpkg-maintscript-helper mv_conffile /etc/systemd/systemd-logind.conf /etc/systemd/logind.conf 204-1~ -- configure 229-1ubuntu2
+ dpkg-maintscript-helper mv_conffile /etc/systemd/systemd-journald.conf /etc/systemd/journald.conf 204-1~ -- configure 229-1ubuntu2
+ dpkg-maintscript-helper rm_conffile /etc/dbus-1/system.d/org.freedesktop.machine1.conf 228-5~ -- configure 229-1ubuntu2
+ dpkg-maintscript-helper rm_conffile /etc/X11/xinit/xinitrc.d/50-systemd-user.sh 228-3~ -- configure 229-1ubuntu2

Thanks! Much clearer now, and trivially reproducible: Mount /tmp manually or from /etc/fstab), and try to run "systemctl enable tmp.mount".

Fixed by http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=c59ff5c9ec6

@pitti: will /etc/systemd/system/tmp.mount override a /tmp mount point from /etc/fstab (/run vs /etc)?

@Michael: Indeed /etc/ will override /run. My gut feeling is that /etc/fstab should have precedence over /etc/default/rcS, so I guess we should refine the check to not copy the unit if there already is a tmp.mount? This can only be done when already running systemd, though.

This bug was fixed in the package systemd - 229-2ubuntu1

---------------
systemd (229-2ubuntu1) xenial; urgency=medium

  * Merge with Debian unstable. Remaining Ubuntu changes:
    - Hack to support system-image read-only /etc, and modify files in
      /etc/writable/ instead.

    Upgrade fixes, keep until 16.04 LTS release:
    - systemd Conflicts/Replaces/Provides systemd-services.
    - Remove obsolete systemd-logind upstart job.
    - Clean up obsolete /etc/udev/rules.d/README.
    - systemd.postinst: Migrate mountall specific fstab options to standard
      util-linux "nofail" option.
    - systemctl: Don't forward telinit u to upstart. This works around
      upstart's Restart() always reexec'ing /sbin/init on Restart(), even if
      that changes to point to systemd during the upgrade. This avoids running
      systemd during a dist-upgrade. (LP: #1430479)
    - Provide shutdown fallback for upstart. (LP: #1370329)
    - Break lvm (<< 2.02.133-1ubuntu1) and remove our dummy /etc/init.d/lvm2
      on upgrades, as it's shipped by lvm2 now.
    - Make udev break on mdadm << 3.3-2ubuntu3, as udev's init script dropped
      the "Provides: raid-mdadm".
    - Clean up /var/log/udev on upgrade (which is written under upstart, but
      not under systemd). (LP: #1537211)
    - Migrate existing s390x network configuration to new names. (LP: #1526808)
    - systemd.postinst: Bump Version comparison for migrating the UTC setting
      from /etc/default/rcS to /etc/adjtime, to run it for upgrades to 16.04.

systemd (229-2) unstable; urgency=medium

  * time-util: map ALARM clockids to non-ALARM clockids in now(), to work on
    architectures which don't support CLOCK_BOOTTIME_ALARM. Fixes FTBFS on
    many architectures.
  * debian/systemd.postinst: Add missing newline to /etc/adjtime migration.
    (See #699554)
  * debian/systemd.postinst: Only try to enable tmp.mount if we actually
    copied it to /etc. Don't try to enable a generated unit. (LP: #1545707)
  * debian/tests/boot-and-services: Increase timeouts of test_bash_crash from
    5 to 10 seconds, and sync the journal after every iteration.
  * debian/extra/checkout-upstream: Try again after one minute if git checkout
    fails, to avoid failures from transient network errors.
  * debian/tests/systemd-fsckd: Use grub.d/50-cloudimg-settings.cfg as a
    template for generating our custom one instead of 90-autopkgtest.cfg. The
    latter does not exist on non-x86 architectures and is not relevant for
    this test.
  * debian/tests/boot-and-services: Skip journal test for test_bash_crash when
    running against upstream, as this currently fails most of the time. To be
    investigated.
  * debian/tests/networkd: Skip test_coldplug_dhcp_ip6 when running against
    upstream, as this is brittle there. To be investigated.
  * debian/tests/bootchart: Skip test if bootchart is not available or
    testing in upstream mode. bootchart got removed from master and will be
    moved to a separate repository.
  * debian/tests/boot-and-services: Show verbose journal output on failure in
    nspawn test, and sync journal before.
  * Move systemd-coredump socket and service into systemd-coredump bina...

I'm still seeing the issue in this version of systemd.

Still get it with 229-3ubuntu1 on xenial 64 bits

Status changed to 'Confirmed' because the bug affects multiple users.

So this isn't the code path of "Do a one-time migration of the RAMTMP setting", but the code path of "Re-run systemctl enable for any service that was enabled when preinst was run", as per the trace in comment 7.

+ read UNIT
+ [ tmp.mount = tmp.mount ]
+ dpkg --compare-versions 229-1ubuntu2 lt-nl 220-6~
+ systemctl enable tmp.mount
Operation failed: No such file or directory
+ true

Obviously $2 is greater than 220-6~, it's "229-1ubuntu2" (see beginning of the trace), so the if failed on that condition. But as it's in /run/systemd/was-enabled, it must have been added by "save_is_enabled tmp.mount" in systemd.preinst.

Can you please give me the output of

  systemctl is-enabled tmp.mount ; echo $?
  sudo systemctl status tmp.mount
  ls -l /etc/systemd/system/tmp.mount

? I think what happens is that tmp.mount is enabled in /run only, not in /etc/, so this cannot be enabled like that.

FTR, it's not just about having a /run/systemd/generator/tmp.mount if you have an fstab entry like "none /tmp/ tmpfs defaults 0 0", because with that "systemctl --quiet is-enabled tmp.mount" fails and thus shouldn't be added to was-enabled. So I still cannot reproduce this nor have an idea what's going wrong here.

oem@u64:~$ systemctl is-enabled tmp.mount ; echo $?
Failed to get unit file state for tmp.mount: No such file or directory 1

oem@u64:~$ sudo systemctl status tmp.mount
[sudo] password for oem:
● tmp.mount
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

oem@u64:~$ ls -l /etc/systemd/system/tmp.mount
ls: cannot access '/etc/systemd/system/tmp.mount': No such file or directory

 cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda4 during installation
UUID=7c755ed6-51cc-4b75-88ac-9c75acf82749 / ext4 errors=remount-ro 0 1
# swap was on /dev/sdb2 during installation
UUID=0a9ca7f0-6eeb-4b21-b70f-670fa600de16 none swap sw 0 0

ls -la /etc/systemd/system/

drwxr-xr-x 18 root root 4096 Apr 6 09:01 .
drwxr-xr-x 5 root root 4096 Apr 12 13:08 ..
drwxr-xr-x 2 root root 4096 Aug 29 2014 bluetooth.target.wants
lrwxrwxrwx 1 root root 37 Aug 29 2014 dbus-org.bluez.service -> /lib/systemd/system/bluetooth.service
lrwxrwxrwx 1 root root 40 Aug 29 2014 dbus-org.freedesktop.Avahi.service -> /lib/systemd/system/avahi-daemon.service
lrwxrwxrwx 1 root root 40 Aug 29 2014 dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service
lrwxrwxrwx 1 root root 53 Aug 29 2014 dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service
lrwxrwxrwx 1 root root 36 Feb 4 16:34 dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service
drwxr-xr-x 2 root root 4096 Mar 14 12:23 default.target.wants
lrwxrwxrwx 1 root root 35 Apr 5 12:14 display-manager.service -> /lib/systemd/system/lightdm.service
drwxr-xr-x 2 root root 4096 Apr 18 2015 display-manager.service.wants
drwxr-xr-x 2 root root 4096 Oct 26 2014 getty.target.wants
drwxr-xr-x 2 root root 4096 Aug 29 2014 graphical.target.wants
drwxr-xr-x 2 root root 4096 Jan 4 2015 hibernate.target.wants
drwxr-xr-x 2 root root 4096 Jan 4 2015 hybrid-sleep.target.wants
drwxr-xr-x 2 root root 4096 Feb 17 23:15 multi-user.target.wants
drwxr-xr-x 2 root root 4096 Jan 28 11:58 network-online.target.wants
drwxr-xr-x 2 root root 4096 Feb 18 21:33 paths.target.wants
drwxr-xr-x 2 root root 4096 Aug 29 2014 printer.target.wants
drwxr-xr-x 2 root root 4096 Aug 29 2014 shutdown.target.wants
drwxr-xr-x 2 root root 4096 Feb 14 15:13 sockets.target.wants
drwxr-xr-x 2 root root 4096 Jan 4 2015 suspend.target.wants
drwxr-xr-x 2 root root 4096 Mar 31 15:58 sysinit.target.wants
lrwxrwxrwx 1 root root 35 Aug 29 2014 syslog.service -> /lib/systemd/system/rsyslog.service
drwxr-xr-x 2 root root 4096 Apr 6 09:01 timers.target.wants

Scanning the system, tmp.mount is only found into /ust/share/systemd/

oem@u64:~$ ls -la /usr/share/systemd/
total 28
drwxr-xr-x 2 root root 4096 Apr 12 13:08 .
drwxr-xr-x 342 root root 12288 Apr 13 09:22 ..
-rw-r--r-- 1 root root 3564 Apr 12 12:34 kbd-model-map
-rw-r--r-- 1 root root 146 Apr 12 12:34 language-fallback-map
-rw-r--r-- 1 root root 661 Apr 12 12:34 tmp.mount

oem@u64:~$ sudo dpkg-reconfigure systemd
[sudo] password for oem:
addgroup: The group `systemd-journal' already exists as a system group. Exiting.

Here is the requested output:

root@ubuntu:~# systemctl is-enabled tmp.mount ; echo $?
Failed to get unit file state for tmp.mount: No such file or directory
0
root@ubuntu:~# systemctl status tmp.mount
Failed to connect to bus: No such file or directory
root@ubuntu:~# ls -l /etc/systemd/system/tmp.mount
ls: cannot access '/etc/systemd/system/tmp.mount': No such file or directory

Executing "dpkg-reconfigure systemd" does now also not show the error anymore. But if I'm not wrong I remember to had this for a short time in the past too and shortly after this the error appeared again. I think I should wait at least until I upgrade systemd to a new version again and see if this error doesn't appear there anymore too.

# systemctl is-enabled tmp.mount ; echo $?
Failed to get unit file state for tmp.mount: No such file or directory
0

Ouch, so systemctl exiting successfully (with 0) for a nonexisting tmp.mount is indeed bad, and would cause this bug. dino99 correctly gets an "1" there, so dino99 and Sworddragon do *not* experience the same bug here. As Sworddragon reported this, let's concentrate on this issue now.

Sworddragon, can you please look for any tmp.mount units you may have around, with

  find /{etc,lib,run}/systemd -name tmp.mount | xargs ls -l

?

Tested on 2 xenial install: one fresh, and a wily to xenial dist-upgrade: get the same result

oem@u64:~$ sudo find /{etc,lib,run}/systemd -name tmp.mount | xargs ls -l
[sudo] password for oem:
total 261428
-rw------- 1 oem oem 267624448 Feb 12 07:42 core
drwxr-xr-x 2 oem oem 4096 Sep 1 2014 Desktop
drwxr-xr-x 2 oem oem 4096 Apr 16 08:39 Documents
drwxr-xr-x 2 oem oem 4096 Apr 4 13:06 Downloads
-rw-r--r-- 1 oem oem 8980 Aug 29 2014 examples.desktop
drwxrwxr-x 3 oem oem 4096 Nov 1 06:33 git
drwxrwxr-x 3 oem oem 4096 Aug 30 2014 git_projects
-rw-rw-r-- 1 oem oem 493 Apr 10 17:36 hdparm.txt
drwxr-xr-x 2 oem oem 4096 Mar 14 17:05 kern
drwxr-xr-x 2 oem oem 4096 Aug 29 2014 Music
-rw------- 1 oem oem 9924 Apr 3 13:58 nohup.out
drwxr-xr-x 2 oem oem 4096 Mar 24 2015 Pictures
drwxr-xr-x 2 oem oem 4096 Aug 29 2014 Public
drwxr-xr-x 2 oem oem 4096 Aug 29 2014 Templates
drwxr-xr-x 2 oem oem 4096 Aug 29 2014 Videos

On both install, tmp.mount is only found into /usr/share/systemd/, like #20 above let suppose too.

@dino99: I know, you already said that in comment #18, but for you "systemctl is-enabled tmp.mount" correctly exits with 1.

And I thought the return code of 0 is another bug and have created another report for it (which got already marked as a duplicate of this bug). Anyways, here is the requested output:

root@ubuntu:~/tmp# find /{etc,lib,run}/systemd -name tmp.mount | xargs ls -l
find: ‘/run/systemd’: No such file or directory
total 0

Thanks sworddragon. Let's try that then:

  sudo SYSTEMD_LOG_LEVEL=debug strace -fvvs1024 -o /tmp/t systemctl enable tmp.mount

please copy&paste the entire output and attach /tmp/t. I doubt that this already tells me what's going on, but at least this is easy to do.

Please leave the status at incomplete until I actually understand what's going on here.

Has using -vv any additional effect (because only -v is documented in the strace manpage)? Anyways, in the attachments is the requested output.

In this strace systemctl now exited with 1 as expected, so we have a classic heisenbug :-/ Can you confirm that if you run it without strace it still exits with 0?

No, it does now return with 1 too.

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]