fstrim: cannot open /dev/.lxd-mounts: Permission denied

Status changed to 'Confirmed' because the bug affects multiple users.

On my containers using the "dir" storage backend, /etc/cron.weekly/fstrim outputs:

fstrim: cannot open /dev/.lxd-mounts: Permission denied
fstrim: /dev/lxd: FITRIM ioctl failed: Operation not permitted
fstrim: /: FITRIM ioctl failed: Operation not permitted

From one of those containers:

# mount | grep sda
/dev/sda on / type ext4 (rw,noatime,errors=remount-ro,data=ordered)
/dev/sda on /dev/lxd type ext4 (rw,noatime,errors=remount-ro,data=ordered)
/dev/sda on /dev/.lxd-mounts type ext4 (rw,noatime,errors=remount-ro,data=ordered)

Fails also on Proxmox LXC https://forum.proxmox.com/threads/deleting-data-in-lxc-lvm-thin-doesnt-release-space.32179/#post-173302

I started to read that bug a couple of days ago

Maybe using 'ConditionVirtualization=!container' should be enought to prevent fstrim from running inside container, I'll give it a try later this week. If its work as expected, I'll submit it upstream.

'container' type would cover the following:

openvz OpenVZ/Virtuozzo
lxc Linux container implementation by LXC
lxc-libvirt Linux container implementation by libvirt
systemd-nspawn systemd's minimal container implementation, see systemd-nspawn(1)
docker Docker container manager
podman Podman container manager
rkt rkt app container runtime
wsl Windows Subsystem for Linux

Reference:
https://www.freedesktop.org/software/systemd/man/systemd-detect-virt.html#

hmmm it is installed under examples/ so not used at all by default.

# dpkg -l util-linux
/usr/share/doc/util-linux/examples/fstrim.service

We could also implement a check mechanism inside "/etc/cron.weekly/fstrim" using'systemd-detect-virt', if it's a container don't run it, otherwise let's do it.

[/etc/cron.weekly/fstrim] # script as-is
#!/bin/sh
# trim all mounted file systems which support it
/sbin/fstrim --all || true

from:
#!/bin/sh
# trim all mounted file systems which support it
/sbin/fstrim --all || true

to:

#!/bin/sh
# trim all mounted file systems which support it
if ! /usr/bin/systemd-detect-virt -q -c; then
/sbin/fstrim --all || true
fi

or even better I think:

#!/bin/sh
# trim all mounted file systems which support it
+if /usr/bin/systemd-detect-virt -q -c; then
+exit 0
+fi
/sbin/fstrim --all || true

Ok more investigation revealed the following

X is using the cron weekly for fstrim, while late version uses fstrim.timer.

So I proposed we fix it with the "ConditionVirtualization=!container" in Bionic and late.

and for X by fixing the cron.weekly/fstrim script that way:

#!/bin/sh
# trim all mounted file systems which support it
+if /usr/bin/systemd-detect-virt -q -c; then
+exit 0
+fi
/sbin/fstrim --all || true

I'll submit the fstrim.timer change upstream.

Upstream BUG:
https://github.com/karelzak/util-linux/issues/840

Upstream PR:
https://github.com/karelzak/util-linux/pull/841

Status changed to 'Confirmed' because the bug affects multiple users.

Upstream PR has been approved/merged.

Uploaded in active devel release (Eoan).

This bug was fixed in the package util-linux - 2.34-0.1ubuntu2

---------------
util-linux (2.34-0.1ubuntu2) eoan; urgency=medium

  * d/p/prevent-fstrim-inside-container.patch:
    - Prevent fstrim to run inside a container environment.
    (LP: #1589289)

 -- Eric Desrochers <email address hidden> Wed, 21 Aug 2019 13:19:03 +0000

Hello Tamas, or anyone else affected,

Accepted util-linux into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/util-linux/2.33.1-0.1ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Tamas, or anyone else affected,

Accepted util-linux into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/util-linux/2.31.1-0.4ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Tamas, or anyone else affected,

Accepted util-linux into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/util-linux/2.27.1-6ubuntu3.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Bionic verification]

Confirming that I'm using a lxd container running Bionic:

# systemd-detect-virt
lxc

# lsb_release -cs
bionic

* rmadison:
 util-linux | 2.31.1-0.4ubuntu3.3 | bionic-updates
 util-linux | 2.31.1-0.4ubuntu3.4 | bionic-proposed

* With current bionic-updates package:

# dpkg -l | grep -i util-linux
ii util-linux 2.31.1-0.4ubuntu3.3 amd64 miscellaneous system utilities

# systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
   Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Tue 2019-08-27 13:59:23 UTC; 2min 29s ago
  Trigger: Mon 2019-09-02 00:00:00 UTC; 5 days left
     Docs: man:fstrim

Aug 27 13:59:23 lxcfstrim systemd[1]: Started Discard unused blocks once a week.

* With the bionic-proposed package:

# dpkg -l | grep -i util-linux
ii util-linux 2.31.1-0.4ubuntu3.4 amd64 miscellaneous system utilities

# systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
   Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2019-08-27 14:02:50 UTC; 23s ago
  Trigger: n/a
Condition: start condition failed at Tue 2019-08-27 14:03:13 UTC; 1s ago
           └─ ConditionVirtualization=!container was not met
     Docs: man:fstrim

Aug 27 13:59:23 lxcfstrim systemd[1]: Started Discard unused blocks once a week.
Aug 27 14:02:50 lxcfstrim systemd[1]: Stopped Discard unused blocks once a week.
Aug 27 14:02:50 lxcfstrim systemd[1]: Stopping Discard unused blocks once a week.

[Xenial verification]

Confirming that I'm using a lxd container running Xenial:

# systemd-detect-virt
lxc

# lsb_release -cs
xenial

* rmadison:
 util-linux | 2.27.1-6ubuntu3.7 | xenial-updates
 util-linux | 2.27.1-6ubuntu3.8 | xenial-proposed

* With current xenial-updates package:

$ dpkg -l | grep -i util-linux
ii util-linux 2.27.1-6ubuntu3.7 amd64 miscellaneous system utilities

$ sh -xv /etc/cron.weekly/fstrim
#!/bin/sh
# trim all mounted file systems which support it
/sbin/fstrim --all || true
+ /sbin/fstrim --all
fstrim: /: FITRIM ioctl failed: Operation not permitted
+ true

* With current xenial-proposed package:

$ dpkg -l | grep -i util-linux
ii util-linux 2.27.1-6ubuntu3.8 amd64 miscellaneous system utilities

$ sh -xv /etc/cron.weekly/fstrim
#!/bin/sh
# Prevent fstrim to run inside a container environment (LP: #1589289)
if /usr/bin/systemd-detect-virt -q -c; then
exit 0
fi
+ /usr/bin/systemd-detect-virt -q -c
+ exit 0

[Disco verification]

Confirming that I'm using a lxd container running Disco:

# systemd-detect-virt
lxc

# lsb_release -cs
disco

* rmadison:
 util-linux | 2.33.1-0.1ubuntu2 | disco | source, amd64, arm64, armhf, i386, ppc64el, s390x
 util-linux | 2.33.1-0.1ubuntu3 | disco-proposed | source, amd64, arm64, armhf, i386, ppc64el, s390x

* With current disco-updates package:

$ dpkg -l | grep -i util-linux
ii util-linux 2.33.1-0.1ubuntu2 amd64 miscellaneous system utilities

$ systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
   Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Tue 2019-08-27 14:18:00 UTC; 40s ago
  Trigger: Mon 2019-09-02 00:00:00 UTC; 5 days left
     Docs: man:fstrim

Aug 27 14:18:00 lxcdfstrim systemd[1]: Started Discard unused blocks once a week.

* With current disco-proposed package:

$ dpkg -l | grep -i util-linux
ii util-linux 2.33.1-0.1ubuntu3 amd64 miscellaneous system utilities

$ systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
   Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2019-08-27 14:19:25 UTC; 43s ago
  Trigger: n/a
Condition: start condition failed at Tue 2019-08-27 14:20:06 UTC; 2s ago
           └─ ConditionVirtualization=!container was not met
     Docs: man:fstrim

Aug 27 14:18:00 lxcdfstrim systemd[1]: Started Discard unused blocks once a week.
Aug 27 14:19:25 lxcdfstrim systemd[1]: fstrim.timer: Succeeded.
Aug 27 14:19:25 lxcdfstrim systemd[1]: Stopped Discard unused blocks once a week.
Aug 27 14:19:25 lxcdfstrim systemd[1]: Stopping Discard unused blocks once a week.
Aug 27 14:19:25 lxcdfstrim systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped.
Aug 27 14:20:06 lxcdfstrim systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped.

All regressions for B/D are recurrent patterns that were already failing before that SRU.

For x, there is a chromium-browser failures. It's definitely a chromium-browser thing, not related to the current SRU.

# logs:
Traceback (most recent call last):
  File "/tmp/autopkgtest.Gv56G4/build.V4H/src/debian/tests/chromium-version", line 13, in <module>
    version = driver.capabilities['browserVersion']
KeyError: 'browserVersion'

A new chromium-browser upstream version has been released on "Aug 10 2019":

chromium-browser (76.0.3809.100-0ubuntu0.16.04.1) xenial; urgency=medium

  * Upstream release: 76.0.3809.100
    - CVE-2019-5868: Use-after-free in PDFium ExecuteFieldAction.
    - CVE-2019-5867: Out-of-bounds read in V8.

 -- Olivier Tilloy <email address hidden> Sat, 10 Aug 2019 15:49:36 +0200

Autopkgtest script for that package probably need some rework to adapt to new code reality.

- Eric

All autopkgtests for the newly accepted util-linux (2.31.1-0.4ubuntu3.4) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

network-manager/1.10.6-2ubuntu1.1 (arm64)
openjdk-8/8u222-b10-1ubuntu1~18.04.1 (arm64, ppc64el, armhf, i386, amd64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#util-linux

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted util-linux (2.33.1-0.1ubuntu3) for disco have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/240-6ubuntu5.3 (amd64, ppc64el)
openjdk-8/8u222-b10-1ubuntu1~19.04.1 (armhf, arm64, amd64, s390x, i386, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/disco/update_excuses.html#util-linux

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted util-linux (2.27.1-6ubuntu3.8) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

chromium-browser/76.0.3809.100-0ubuntu0.16.04.1 (i386, amd64, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#util-linux

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

The verification of the Stable Release Update for util-linux has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package util-linux - 2.33.1-0.1ubuntu3

---------------
util-linux (2.33.1-0.1ubuntu3) disco; urgency=medium

  * d/p/prevent-fstrim-inside-container.patch:
    - Prevent fstrim to run inside a container environment.
    (LP: #1589289)

 -- Eric Desrochers <email address hidden> Thu, 22 Aug 2019 23:40:49 +0000

This bug was fixed in the package util-linux - 2.31.1-0.4ubuntu3.4

---------------
util-linux (2.31.1-0.4ubuntu3.4) bionic; urgency=medium

  * d/p/prevent-fstrim-inside-container.patch:
    - Prevent fstrim to run inside a container environment.
    (LP: #1589289)

 -- Eric Desrochers <email address hidden> Thu, 22 Aug 2019 23:47:46 +0000

As for xenial's chromium-browser test failures - could you please file a bug about the failing architectures? I would then include the bug number in the hint. This way we won't forget about the test issues.

@sil2100,
https://launchpad.net/bugs/1842426

Excellent! Let me proceed in that case.

This bug was fixed in the package util-linux - 2.27.1-6ubuntu3.8

---------------
util-linux (2.27.1-6ubuntu3.8) xenial; urgency=medium

  * d/p/prevent-fstrim-inside-container.patch:
    - Prevent fstrim to run inside a container environment.
    (LP: #1589289)

  * d/fstrim-all.cron: Prevent cron.weekly to fstrim if inside a
    container.

 -- Eric Desrochers <email address hidden> Thu, 22 Aug 2019 23:56:21 +0000