Ubuntu
systemd package

Docker containers lose their cgroup after systemd reload

Bug #1546214 reported by PierreF
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
docker.io (Ubuntu)
Fix Released
Undecided
Martin Pitt
systemd (Ubuntu)
Won't Fix
Undecided
Unassigned

Bug Description

After a Systemd reload & any service restart, docker top no longer show process of containers:

To reproduce this issue, do the following step:

# docker run -d --name test busybox sleep 1d
# docker top test
UID PID PPID C STIME TTY TIME CMD
root 26416 1072 1 18:05 ? 00:00:00 sleep 1d
# systemctl --system daemon-reload && systemctl restart atd.service
# docker top test
UID PID PPID C STIME TTY TIME CMD
[ no process listed... but sleep is still running]

Note: this idea of restarting any service restart come from patch https://lists.freedesktop.org/archives/systemd-devel/2014-September/023276.html (which is applied to Systemd package in Ubuntu)

After few searching, this seems to be due to process from the container being moved in other cgroup by Systemd. More details on https://github.com/docker/docker/issues/20152

Depending on version of Systemd (Wily or Xenial), this issue:

* Wily: Happend with Docker 1.10 (with default option)
* Wily: Does NOT happend with Docker 1.10 and --exec-opt native.cgroupdriver=systemd
* Wily: Does NOT happend with Docker 1.9
* Xenial: Does always happend (Docker 1.9, 1.10 with or without native.cgroupdriver=systemd)

I don't know if this issue is a Systemd issue, a Docker issue... or in middle.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in docker.io (Ubuntu):
status: New → Confirmed
Changed in systemd (Ubuntu):
status: New → Confirmed
Revision history for this message
Michael Crosby (crosbymichael) wrote :

This is not a docker bug it affects any type of cgroup made anywhere in the cgroup hierarchy.

Ex:

 I have service A that forks off child B. I place B in a cgroup that I made at /sys/fs/cgroup/cpu/mycgroup. Reload and restart a service and boom, systemd deletes /sys/fs/cgroup/cpu/mycgroup. /sys/fs/cgroup is not even in any of the systemd controlled cgroup paths, it's just nuking things in the cgroup root that it did not create.

Also this is a security issue.

Revision history for this message
Martin Pitt (pitti) wrote :

I was about to point out the Delegate= option if docker.service wants to maintain its sub-cgroups by itself -- but it seems that already landed upstream two days ago: https://github.com/jheiss/docker/commit/64b87f0

There have been plenty of discussions about that "who owns the cgroup hiearchy"/"single writer" topic, and I don't think this will change anytime soon (nor will we change the behaviour downstream), so I'll close the systemd task.

I'll see to pulling this patch into Xenial's docker.

Changed in docker.io (Ubuntu):
status: Confirmed → In Progress
Changed in systemd (Ubuntu):
status: Confirmed → Won't Fix
Revision history for this message
Michael Crosby (crosbymichael) wrote :

Sounds good Martin. Thanks, the Delegate=yes fixes the issue and was merged into Docker's systemd service file.

Martin Pitt (pitti)
Changed in docker.io (Ubuntu):
status: In Progress → Fix Committed
assignee: nobody → Martin Pitt (pitti)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package docker.io - 1.10.2-0ubuntu4

---------------
docker.io (1.10.2-0ubuntu4) xenial; urgency=medium

  * Add debian/patches/upstream-delegate.patch: Add "Delegate=yes" to docker's
    service file, so that it can manage cgroups by itself. Patch cherry-picked
    from upstream master. (LP: #1546214)

 -- Martin Pitt <email address hidden> Sun, 13 Mar 2016 22:50:51 +0100

Changed in docker.io (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.