Ubuntu
systemd package

systemd: core: Fix edge case when processing /proc/self/mountinfo

Bug #1795764 reported by Joshua R. Poulson on 2018-10-02

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	systemd (Ubuntu)	Fix Released	Medium	Unassigned
	Xenial	Fix Released	Medium	Eric Desrochers

Bug Description

[Impact]

kubernetes loaded inactive dead transient mount points grows
https://github.com/kubernetes/kubernetes/issues/57345

[Test Case]

# cd /tmp
# mkdir -p bind-test/abc
# mount --bind bind-test bind-test
# mount -t tmpfs tmpfs bind-test/abc
# umount bind-test/abc
# systemctl list-units --all | grep bind-test
tmp-bind\x2dtest-abc.mount loaded inactive dead /tmp/bind-test/abc
tmp-bind\x2dtest.mount loaded active mounted /tmp/bind-test

Expected outcome (w/ the fix) :

[Regression Potential]

This is a adapted version of 2 upstream fixes as the original upstream commit has been made on top on 2 functions mount_setup_new_unit() & mount_setup_existing_unit() that not yet exist systemd 229. It is easily adaptable because the current function mount_setup_unit() is dealing with both of at the moment instead of being individually separate in two distinct function.

It is an adaptation of commits :
[65d36b495] core: Fix edge case when processing /proc/self/mountinfo
[03b8cfede] core: make sure to init mount params before calling mount_is_extrinsic()

This patch changes mount_setup_unit() to prevent the just_mounted mount setup flag from being overwritten if it is set to true. This will allow all mount units created from /proc/self/mountinfo entries to be initialised properly.

Additionally, the patch got the blessing of 'xnox' who looked at it and mention it looks fine to him.

[Pending SRU]

Note: No autopkgtests has been reported since systemd (21.5) ... between 21.5 and now (21.11) everything released has been about security fixes :

systemd (229-4ubuntu21.11) xenial; urgency=medium ==> Current SRU
systemd (229-4ubuntu21.10) xenial-security; urgency=medium
systemd (229-4ubuntu21.9) xenial-security; urgency=medium
systemd (229-4ubuntu21.8) xenial-security; urgency=medium
systemd (229-4ubuntu21.6) xenial-security; urgency=medium
systemd (229-4ubuntu21.5) xenial; urgency=medium ==> Previous SRU

Note: I don't know the level of adoption of Netplan in Xenial, but I suspect it is low as Netplan replaced ifupdown as the default configuration utility starting with Ubuntu 17.10 Artful only AFAIK.

* Regression in autopkgtest for nplan (s390x): test log
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/s390x/n/nplan/20181023_132448_031b9@/log.gz

Error:
modprobe: FATAL: Module cfg80211 not found in directory /lib/modules/4.4.0-138-generic

Justification:
This above seems to be a recurrent failure since a couple of release already. This wasn't introduce by this particular SRU.

I don't think having wifi module is relevant in s390x anyway, so most likely the module is not there on purpose for kernel w/ s390x architecture.

* Regression in autopkgtest for nplan (amd64): test log
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/amd64/n/nplan/20181217_010129_e07e2@/log.gz

Error: (Ran on autopkgtest Ubuntu infra)
test_bond_mode_balance_rr_pps (__main__.TestNetworkManager) ... Error: Could not create NMClient object: Cannot invoke method; proxy is for a well-known name without an owner and proxy was constructed with the G_DBUS_PROXY_FLAGS_DO_NOT_AUTO_START flag.FAIL

test_bridge_priority (__main__.TestNetworkManager) ... Error: Could not create NMClient object: Cannot invoke method; proxy is for a well-known name without an owner and proxy was constructed with the G_DBUS_PROXY_FLAGS_DO_NOT_AUTO_START flag.FAIL

test_dhcp6 (__main__.TestNetworkManager) ... Error: Could not create NMClient object: Cannot invoke method; proxy is for a well-known name without an owner and proxy was constructed with the G_DBUS_PROXY_FLAGS_DO_NOT_AUTO_START flag.FAIL

Justification:

The test are "passing" if ran manually in my own HW :

autopkgtest [15:29:15]: host <MY_HOSTNAME>; command line: /usr/bin/autopkgtest nplan -U --apt-pocket=proposed --log-file /tmp/adt-proposed.out --- qemu /var/lib/libvirt/images/autopkgtest-xenial-amd64.img
.....
Setting up systemd (229-4ubuntu21.11) ...
Setting up nplan (0.32~16.04.6) ...
Setting up network-manager (1.2.6-0ubuntu0.16.04.3) ...
....
test_dhcp6 (__main__.TestNetworkManager) ... ok
test_bridge_priority (__main__.TestNetworkManager) ... ok
test_bond_mode_balance_rr_pps (__main__.TestNetworkManager) ... ok

I *think* that these test may eventually works, but is it really worth it to retry them until success if it works fine outside the Ubuntu infra w/ the same proposed packages ? (See attachment: autopkgtest_result.txt for the whole ADT log)

* Regression in autopkgtest for nplan (armhf): test log
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/armhf/n/nplan/20181217_132248_e07e2@/log.gz

Error:
Traceback (most recent call last):
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 717, in mainloop
    command()
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 646, in command
    r = f(c, ce)
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 342, in cmd_reboot
    caller.hook_wait_reboot()
  File "/home/ubuntu/autopkgtest/virt/autopkgtest-virt-lxd", line 230, in hook_wait_reboot
    wait_booted()
  File "/home/ubuntu/autopkgtest/virt/autopkgtest-virt-lxd", line 104, in wait_booted
    VirtSubproc.check_exec(['lxc', 'exec', container_name, '--', 'sh', '-ec', '[ ! -d /run/systemd/system ] || systemctl start network-online.target'], timeout=60)
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 183, in check_exec
    stdout=stdout, stderr=subprocess.PIPE)
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 144, in execute_timeout
    (out, err) = sp.communicate(instr)
  File "/usr/lib/python3.5/subprocess.py", line 1062, in communicate
    stderr = self.stderr.read()
  File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 64, in alarm_handler
    raise Timeout()
VirtSubproc.Timeout

Justification:
It is a recurrent failure since "systemd/229-4ubuntu21.3".
Nothing to do with the current SRU "systemd/229-4ubuntu21.11".
Since then quite a few SRU has been approved so I'm not too worry here.

* Regression in autopkgtest for systemd (s390x): test log
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/s390x/s/systemd/20181213_162040_4f06f@/log.gz

Error:
FileNotFoundError: [Errno 2] No such file or directory: '/boot/grub/grub.cfg'

Justification:
This above seems to be a recurrent failure since a couple of release already. This wasn't introduce by this particular SRU.

* Regression in autopkgtest for snapd (i386): test log
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/i386/s/snapd/20181213_212830_ea0ac@/log.gz

Error:
2018-12-13 21:28:16 Failed tasks: 1
- autopkgtest:ubuntu-16.04-i386:tests/main/dirs-not-shared-with-host:alternatives
error: unsuccessful run

Justification:
This above seems to be a recurrent failure since a couple of release already. This wasn't introduce by this particular SRU.

*
*

[Other Info]

One line fix in https://github.com/systemd/systemd/pull/7811/files

Referenced issue: https://github.com/systemd/systemd/issues/7798

Related kubernetes issue: https://github.com/kubernetes/kubernetes/issues/57345

systemd v237 has this fix, but we'd like to have it fixed in 16.04.

It only affect systemd for Xenial, later release already has the fix:

$ git describe --contains 65d36b495
v237~140

[Original Description]

From the PR:
Currently, if there are two /proc/self/mountinfo entries with the same
mount point path, the mount setup flags computed for the second of
these two entries will overwrite the mount setup flags computed for
the first of these two entries. This is the root cause of issue #7798.
This patch changes mount_setup_existing_unit to prevent the
just_mounted mount setup flag from being overwritten if it is set to
true. This will allow all mount units created from /proc/self/mountinfo
entries to be initialized properly.

One line fix in https://github.com/systemd/systemd/pull/7811/files

Referenced issue: https://github.com/systemd/systemd/issues/7798

Related kubernetes issue: https://github.com/kubernetes/kubernetes/issues/57345

See original description

Tags:

CVE References

Joshua R. Poulson (jrp) on 2018-10-02

Changed in systemd (Ubuntu):
status:	New → Confirmed

Eric Desrochers (slashd) on 2018-10-04

tags:	added: sts
Changed in systemd (Ubuntu Xenial):
status:	New → Confirmed
description:	updated
Changed in systemd (Ubuntu):
status:	Confirmed → Fix Released

Eric Desrochers (slashd) on 2018-10-04

Changed in systemd (Ubuntu Xenial):
importance:	Undecided → Medium

Eric Desrochers (slashd) on 2018-11-30

description:

updated

Revision history for this message

Eric Desrochers (slashd) wrote on 2018-12-11:

systemd-lp1795764-xenial.debdiff Edit (3.7 KiB, text/plain)

debdiff for xenial [systemd-lp1795764-xenial.debdiff]

Eric Desrochers (slashd) on 2018-12-11

description:	updated
description:	updated
Changed in systemd (Ubuntu Xenial):
status:	Confirmed → In Progress
assignee:	nobody → Eric Desrochers (slashd)
description:	updated
description:	updated

Eric Desrochers (slashd) on 2018-12-11

description:

updated

Eric Desrochers (slashd) on 2018-12-13

description:

updated

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2018-12-13: Please test proposed package

Hello Joshua, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Xenial):
status:	In Progress → Fix Committed
tags:	added: verification-needed verification-needed-xenial

Eric Desrochers (slashd) on 2018-12-13

description:	updated
description:	updated
description:	updated
description:	updated

Eric Desrochers (slashd) on 2018-12-13

description:

updated

Eric Desrochers (slashd) on 2018-12-13

description:	updated
description:	updated
description:	updated

Eric Desrochers (slashd) on 2018-12-17

description:	updated
description:	updated

Eric Desrochers (slashd) on 2018-12-17

description:

updated

Eric Desrochers (slashd) on 2018-12-17

description:

updated

Eric Desrochers (slashd) on 2018-12-17

description:	updated
description:	updated

Eric Desrochers (slashd) on 2018-12-17

description:	updated
description:	updated

Eric Desrochers (slashd) on 2018-12-17

description:	updated
description:	updated

Revision history for this message

Eric Desrochers (slashd) wrote on 2018-12-17:

autopkgtest_result.txt Edit (54.8 KiB, text/plain)

Regression in autopkgtest for nplan (amd64): test log

It's failling in the Ubuntu infra but passing when ran manually in my own HW, here's the full output in attachment: autopkgtest_result.txt

- Eric

description:

updated

Eric Desrochers (slashd) on 2018-12-17

description:

updated

Revision history for this message

Eric Desrochers (slashd) wrote on 2018-12-17:

[VERIFICATION XENIAL]

I confirm that this specific bug is now fix with that proposed package.

# Confirmation of the use of proposed systemd packages:

# Confirmation that I ran the test on xenial

root@ubuntu:/tmp# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"

# Reproducer as describe in upstream bug and above in the SRU tmpl:

root@ubuntu:/tmp# mkdir -p bind-test/abc
root@ubuntu:/tmp# mount --bind bind-test bind-test
root@ubuntu:/tmp# mount -t tmpfs tmpfs bind-test/abc
root@ubuntu:/tmp# umount bind-test/abc
root@ubuntu:/tmp# systemctl list-units --all | grep bind-test
tmp-bind\x2dtest.mount loaded active mounted /tmp/bind-test

I also did some dogfooding here and there, BUT, since systemd is a pretty critical piece, I would appreciate more feedback from other impacted users before I can proceed with the release in xenial-updates.

Meaning I won't change this bug to "verification-done-xenial" until then. I'll return from vacation on the week of January 7th (mid-week) which will give us a reasonable amount of time for impacted users and other volunteers to test this new proposed package.

- Eric

[VERIFICATION XENIAL]

I confirm that this specific bug is now fix with that proposed package.

# Confirmation of the use of proposed systemd packages:

ii  libpam-systemd:amd64                229-4ubuntu21.11                           amd64        system and service manager - PAM module
ii  libsystemd0:amd64                   229-4ubuntu21.11                           amd64        systemd utility library
ii  libudev1:amd64                      229-4ubuntu21.11                           amd64        libudev shared library
ii  systemd                             229-4ubuntu21.11                           amd64        system and service manager
ii  systemd-sysv                        229-4ubuntu21.11                           amd64        system and service manager - SysV links
ii  udev                                229-4ubuntu21.11                           amd64        /dev/ and hotplug management daemon

# Confirmation that I ran the test on xenial

root@ubuntu:/tmp# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"

# Reproducer as describe in upstream bug and above in the SRU tmpl:

root@ubuntu:/tmp# mkdir -p bind-test/abc
root@ubuntu:/tmp# mount --bind bind-test bind-test
root@ubuntu:/tmp# mount -t tmpfs tmpfs bind-test/abc
root@ubuntu:/tmp# umount bind-test/abc
root@ubuntu:/tmp# systemctl list-units --all | grep bind-test
  tmp-bind\x2dtest.mount loaded    active   mounted   /tmp/bind-test

- Eric

Revision history for this message

David Coronel (davecore) wrote on 2018-12-17:

I tested the systemd 229-4ubuntu21.11 package from xenial-proposed in Ubuntu 16.04 in KVM and also confirm I cannot reproduce the issue with this updated package:

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial

# dpkg -l | grep 229-4ubuntu21.11
ii libpam-systemd:amd64 229-4ubuntu21.11 amd64 system and service manager - PAM module
ii libsystemd0:amd64 229-4ubuntu21.11 amd64 systemd utility library
ii systemd 229-4ubuntu21.11 amd64 system and service manager

# mkdir -p bind-test/abc
# mount --bind bind-test bind-test
# mount -t tmpfs tmpfs bind-test/abc
# umount bind-test/abc
# systemctl list-units --all | grep bind-test
root-bind\x2dtest.mount loaded active mounted /root/bind-test

+1, LGTM

Eric Desrochers (slashd) on 2018-12-19

description:

updated

Revision history for this message

Eric Desrochers (slashd) wrote on 2019-01-06:

Hi Joshua,

Have you been able to test the systemd proposed package using your k8s setup reproducer ?

I'll be curious to hear about the outcome before I request the release of this package to its final destination: xenial-updates and mark the end of this particular SRU.

Regards,
Eric

Revision history for this message

Eric Desrochers (slashd) wrote on 2019-01-14:

#10

A security fix has superseded the current SRU. The SRU will need to be restarted from scratch.

debian/changelog
--
systemd (229-4ubuntu21.15) xenial-security; urgency=medium

* SECURITY UPDATE: memory corruption in journald via attacker controlled alloca
- debian/patches/CVE-2018-16864.patch: journald: do not store the iovec
entry for process commandline on the stack
- CVE-2018-16864
* SECURITY UPDATE: memory corruption in journald via attacker controlled alloca
- debian/patches/CVE-2018-16865_1.patch: journald: set a limit on the
number of fields (1k)
- debian/patches/CVE-2018-16865_2.patch: journal-remote: set a limit on the
number of fields in a message
- CVE-2018-16865
* SECURITY UPDATE: out-of-bounds read in journald
- debian/patches/CVE-2018-16866.patch: journal: fix syslog_parse_identifier()
- CVE-2018-16866
* SECURITY UPDATE: symlink mishandling in systemd-tmpfiles
- debian/patches/CVE-2018-6954.patch: don't resolve pathnames when traversing
recursively through directory trees
- debian/patches/CVE-2018-6954_2.patch: backport the remaining patches to
resolve this completely
- CVE-2018-6954

* Fix LP: #1804603 - btrfs-util: unbreak tmpfiles' subvol creation
- add debian/patches/btrfs-util-unbreak-tmpfiles-subvol-creation.patch
- update debian/patches/series
* Fix LP: #1804864 - test: Set executable bits on TEST-22-TMPFILES shell scripts
- add debian/patches/test-Set-executable-bits-on-TEST-22-TMPFILES-shell-script.patch
- update debian/patches/series

-- Chris Coulson <email address hidden> Thu, 10 Jan 2019 00:15:47 +0000
--

Changed in systemd (Ubuntu Xenial):
status:	Fix Committed → In Progress

Eric Desrochers (slashd) on 2019-02-19

Changed in systemd (Ubuntu Xenial):
assignee:	Eric Desrochers (slashd) → Dimitri John Ledkov (xnox)

Eric Desrochers (slashd) on 2019-02-28

Changed in systemd (Ubuntu Xenial):
assignee:	Dimitri John Ledkov (xnox) → Eric Desrochers (slashd)

Revision history for this message

Steve Langasek (vorlon) wrote on 2019-03-02:

#11

Hello Joshua, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.17 in a few hours, and then in the -proposed repository.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Xenial):
status:	In Progress → Fix Committed

Revision history for this message

Eric Desrochers (slashd) wrote on 2019-03-04:

#12

[VERIFICATION XENIAL]

Using the protocol mentioned in the [Test Case] with systemd from xenial-proposed, version : 229-4ubuntu21.17. I can no longer reproduce the situation.

# systemctl list-units --all | grep bind-test
mnt-bind\x2dtest.mount loaded active mounted /mnt/bind-test

- Eric

tags:	added: verification-done-xenial removed: verification-needed-xenial
tags:	added: verification-done removed: verification-needed

Revision history for this message

Dan Streetman (ddstreet) wrote on 2019-03-04:

#13

root@systemd-x:/tmp# dpkg -l | grep libsystemd0
ii libsystemd0:amd64 229-4ubuntu21.16 amd64 systemd utility library
root@systemd-x:/tmp# mkdir -p bind-test/abc
root@systemd-x:/tmp# mount --bind bind-test bind-test
root@systemd-x:/tmp# mount -t tmpfs tmpfs bind-test/abc
root@systemd-x:/tmp# umount bind-test/abc
root@systemd-x:/tmp# systemctl list-units --all | grep bind-test
tmp-bind\x2dtest-abc.mount loaded inactive dead /tmp/bind-test/abc
tmp-bind\x2dtest.mount loaded active mounted /tmp/bind-test

root@systemd-x:/tmp# dpkg -l |grep libsystemd0
ii libsystemd0:amd64 229-4ubuntu21.17 amd64 systemd utility library
root@systemd-x:/tmp# mkdir -p bind-test/abc
root@systemd-x:/tmp# mount --bind bind-test bind-test
root@systemd-x:/tmp# mount -t tmpfs tmpfs bind-test/abc
root@systemd-x:/tmp# umount bind-test/abc
root@systemd-x:/tmp# systemctl list-units --all | grep bind-test
tmp-bind\x2dtest.mount loaded active mounted /tmp/bind-test

Revision history for this message

Dan Streetman (ddstreet) wrote on 2019-03-04:

#14

oops, @slashd already verified :)

Revision history for this message

Dan Streetman (ddstreet) wrote on 2019-03-06:

#15

for autopkgtest regression failure justifications, see bug 1755863

Mathew Hodson (mhodson) on 2019-03-10

Changed in systemd (Ubuntu):
importance:	Undecided → Medium

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2019-03-11: Update Released

#16

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2019-03-11:

#17

This bug was fixed in the package systemd - 229-4ubuntu21.17

---------------
systemd (229-4ubuntu21.17) xenial; urgency=medium

  [ Victor Tapia ]
  * d/p/stop-mount-error-propagation.patch:
    keep mount errors local to the failing mount point instead of blocking
    the processing of all mounts (LP: #1755863)

  [ Eric Desrochers ]
  * d/p/fix-egde-case-when-processing-proc-self-mountinfo.patch:
    Mounting any file system to a mount point in a directory
    that is bind mounted to itself will create an inactive
    mount unit. (LP: #1795764)

-- Dan Streetman <email address hidden> Thu, 28 Feb 2019 17:50:50 -0500