Installer crash when trying to unmount target

Bug #1934775 reported by Alfonso Sanchez-Beato
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Fix Released
Undecided
Unassigned
subiquity
New
Undecided
Unassigned

Bug Description

Installing has crashed in the latest stages of the installation, apparently due to not being able to unmount /target/sys. The error I see in the log is:

 Running command ['umount', '/target/sys/firmware/efi/efivars'] with allowed return codes [0] (capture=False)
 Running command ['umount', '/target/sys'] with allowed return codes [0] (capture=False)
 umount: /target/sys: target is busy.

Mount shows that there is something else still mounted in target/sys:

sysfs on /target/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /target/sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)

This fails:
# umount /target/sys
But this is successful:
# umount /target/sys/fs/fuse/connections
# umount /target/sys

See attached file with the crash. This happens with the revision in edge for arm64:

edge arm64 21.06.01+git14.aa0dfb01 2543 2021-07-02T04:58:24.297995+00:00

Tags: fr-1497

Related branches

Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :
description: updated
tags: added: fr-1497
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I guess this can be fixed by manually bind mounting /target/sys/fs/fuse/connections as we currently do for /target/sys/firmware/efi/efivars. But I don't understand why either of these is needed, unless /sys/fs/fuse/connections is not mounted before the install starts, because this sequence works for me:

root@venerated-bobcat:~/t# mkdir a a/b c
root@venerated-bobcat:~/t# mount -t fusectl fusectl a/b
root@venerated-bobcat:~/t# mount --bind a c
root@venerated-bobcat:~/t# umount c

This sequence does not:

root@venerated-bobcat:~/t# mkdir a a/b c
root@venerated-bobcat:~/t# mount --bind a c
root@venerated-bobcat:~/t# mount -t fusectl fusectl a/b
root@venerated-bobcat:~/t# umount c
umount: /root/t/c: target is busy.

But afaics /sys/fs/fuse/connections is mounted at startup (it's "wanted" by sysinit.target). So I don't know what's going on basically. But I can probably fix it anyway if it's urgent.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Is there any way I can poke around on a system where this happens?

Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

I have checked and for this system /sys/fs/fuse/connections is not mounted at startup:

root@ubuntu-server:/# mount | grep fuse
root@ubuntu-server:/# ls /sys/fs/fuse/connections
ls: cannot access '/sys/fs/fuse/connections': No such file or directory

Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

Some more observations. The fs-fuse mount unit fails:

root@ubuntu-server:/# systemctl status sys-fs-fuse-connections.mount
* sys-fs-fuse-connections.mount - FUSE Control File System
     Loaded: loaded (/lib/systemd/system/sys-fs-fuse-connections.mount; static; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Thu 2021-05-27 15:16:21 UTC; 1 months 16 days ago
      Where: /sys/fs/fuse/connections
       What: fusectl
       Docs: https://www.kernel.org/doc/Documentation/filesystems/fuse.txt
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems

May 27 15:16:19 ubuntu-server systemd[1]: Condition check resulted in FUSE Control File System being skipped.
May 27 15:16:21 ubuntu-server systemd[1]: Condition check resulted in FUSE Control File System being skipped.
May 27 15:16:21 ubuntu-server systemd[1]: Condition check resulted in FUSE Control File System being skipped.
May 27 15:16:21 ubuntu-server systemd[1]: Condition check resulted in FUSE Control File System being skipped.

The condition that fails is apparently

ConditionPathExists=/sys/fs/fuse/connections

Which indeed does not exist. It seems to be created when a FUSE mount happens:

root@ubuntu-server:/# ls /sys/fs/fuse
ls: cannot access '/sys/fs/fuse': No such file or directory
root@ubuntu-server:/# mkdir xx
root@ubuntu-server:/# mount -t fusectl fusectl xx
root@ubuntu-server:/# ls xx
root@ubuntu-server:/# mount | grep fuse
fusectl on /xx type fusectl (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
root@ubuntu-server:/# umount xx
root@ubuntu-server:/# ls -d /sys/fs/fuse/connections
/sys/fs/fuse/connections

At that point I can restart successfully the mount unit with "systemctl restart sys-fs-fuse-connections.mount". And with that I am also able to finish successfully the installation.

To me, the ConditionPathExists=/sys/fs/fuse/connections is quite strange, and it seems like it is a dependency that is fulfilled when the mount unit is run. So the mount unit expects something has already mounted fusectl but its mission is to mount it too??

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

huh. does modprobe fuse make the mount happen?

Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

Ok, the problem seems to be that the fuse module needs to be manually loaded. Running

# modprobe fuse

makes sys-fs-fuse-connections.mount re-start and it finishes successfully. The difference here is that in the Ubuntu generic kernel we have CONFIG_FUSE_FS=y, while in bluefield kernel it is a module.

We can force the loading of this kernel module either by configuration or from subiquity. Or maybe change order of things as in comment #2 so pre-loading the module is not needed.

Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote : Re: [Bug 1934775] Re: Installer crash when trying to unmount target

On Tue, Jul 13, 2021 at 11:01 AM Michael Hudson-Doyle <
<email address hidden>> wrote:

> huh. does modprobe fuse make the mount happen?
>

Just answered to that before seeing this comment :D

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1934775
>
> Title:
> Installer crash when trying to unmount target
>
> Status in subiquity:
> New
>
> Bug description:
> Installing has crashed in the latest stages of the installation,
> apparently due to not being able to unmount /target/sys. The error I
> see in the log is:
>
> Running command ['umount', '/target/sys/firmware/efi/efivars'] with
> allowed return codes [0] (capture=False)
> Running command ['umount', '/target/sys'] with allowed return codes [0]
> (capture=False)
> umount: /target/sys: target is busy.
>
> Mount shows that there is something else still mounted in target/sys:
>
> sysfs on /target/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> fusectl on /target/sys/fs/fuse/connections type fusectl
> (rw,nosuid,nodev,noexec,relatime)
>
> This fails:
> # umount /target/sys
> But this is successful:
> # umount /target/sys/fs/fuse/connections
> # umount /target/sys
>
> See attached file with the crash. This happens with the revision in
> edge for arm64:
>
> edge arm64 21.06.01+git14.aa0dfb01 2543
> 2021-07-02T04:58:24.297995+00:00
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/subiquity/+bug/1934775/+subscriptions
>

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I'm not sure the suggestion in #2 will work in fact, as I don't think /sys/fs/fuse/connections will even exist when setting up /target as a chroot.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

This bug is fixed with commit 1f663e51 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=1f663e51

Changed in curtin:
status: New → Fix Committed
Revision history for this message
Dan Bungert (dbungert) wrote : Fixed in curtin version 21.3.

This bug is believed to be fixed in curtin in version 21.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.