[SRU] unattended-upgrade still restarts blacklisted daemons
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
docker.io (Ubuntu) |
Fix Released
|
Critical
|
Bryce Harrington | ||
Xenial |
Fix Released
|
Critical
|
Bryce Harrington | ||
Bionic |
Fix Released
|
Critical
|
Unassigned | ||
Focal |
Fix Released
|
Critical
|
Unassigned | ||
Groovy |
Fix Released
|
Critical
|
Unassigned | ||
Hirsute |
Fix Released
|
Critical
|
Bryce Harrington |
Bug Description
[Impact]
Docker uses containerd under the hood. When containerd is upgraded it stops and restarts its service; docker stops when containerd stops but doesn’t restart. Particularly when doing unattended upgrades, an SRU fix rolled out for containerd can result in unexpected and widespread service outages for docker.
[Test Case]
$ sudo apt install docker.io
$ sudo systemctl start docker
$ systemctl status docker | grep Active
Active: active (running) since[...]
$ systemctl status containerd | grep Active
Active: active (running) since[...]
$ docker pull ubuntu/redis:latest
$ docker run -e REDIS_PASSWORD=1234 --network host \
--name test-redis -d ubuntu/redis:latest
$ telnet localhost 6379
$ docker container logs test-redis
$ sudo apt install --reinstall containerd
$ systemctl status containerd | grep Active
Active: active (running) since
$ systemctl status docker | grep Active
Active: inactive (dead) since [...]; 8s ago
$ docker container logs test-redis
[Where Problems Could Occur]
The challenge with this issue is addressing all important corner cases, and as such the biggest risk is that we miss a corner case and fail to keep the two services running when they should. Areas to watch will be failures during start/stop/
[Original Report]
Hello,
Today plenty of our systems running ubuntu 20.04 were restarting the docker daemon, even if i blacklisted the docker package. Since docker has an dependency on containerd thats the reason why it was restarted. IMO the blacklist should also check the full tree of dependencies... This should NOT happen!
From the log you find:
2020-12-01 06:40:13,881 INFO Starting unattended upgrades script
2020-12-01 06:40:13,882 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,
2020-12-01 06:40:13,882 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:13,882 INFO Initial whitelist (not strict):
2020-12-01 06:40:19,139 INFO Packages that will be upgraded: containerd qemu-block-extra qemu-kvm qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils
2020-12-01 06:40:19,140 INFO Writing dpkg log to /var/log/
2020-12-01 06:40:46,996 INFO All upgrades installed
2020-12-01 06:40:50,732 INFO Starting unattended upgrades script
2020-12-01 06:40:50,732 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,
2020-12-01 06:40:50,733 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:50,733 INFO Initial whitelist (not strict):
Also this happened for us on plenty of our servers almost at the same (why the unattended updates are not spread over time?), which destroyed the second time an production environment.
This is not how unattended-upgraded should be, sadly this package lost our trust and we disable it and schedule the 'unattended updates' now on our own.
PS: Not to say that on some servers the docker daemon did not even restart..
Related branches
- Sergio Durigan Junior (community): Approve
- Canonical Server: Pending requested
-
Diff: 117 lines (+85/-0)4 files modifieddebian/changelog (+12/-0)
debian/patches/do-not-bind-docker-to-containerd.patch (+64/-0)
debian/patches/series (+1/-0)
debian/rules (+8/-0)
- Sergio Durigan Junior (community): Approve
- Canonical Server: Pending requested
-
Diff: 132 lines (+90/-1)5 files modifieddebian/changelog (+16/-0)
debian/control (+1/-1)
debian/patches/do-not-bind-docker-to-containerd.patch (+64/-0)
debian/patches/series (+1/-0)
debian/rules (+8/-0)
- Sergio Durigan Junior (community): Approve
- Canonical Server: Pending requested
-
Diff: 115 lines (+85/-0)4 files modifieddebian/changelog (+12/-0)
debian/patches/do-not-bind-docker-to-containerd.patch (+64/-0)
debian/patches/series (+1/-0)
debian/rules (+8/-0)
- Sergio Durigan Junior (community): Approve
- Canonical Server: Pending requested
-
Diff: 116 lines (+85/-0)4 files modifieddebian/changelog (+12/-0)
debian/patches/do-not-bind-docker-to-containerd.patch (+64/-0)
debian/patches/series (+1/-0)
debian/rules (+8/-0)
- Steve Langasek (community): Approve
- Michael Hudson-Doyle: Pending requested
- Paulo Flabiano Smorigo: Pending requested
- git-ubuntu bot: Pending requested
- Dimitri John Ledkov: Pending requested
- Canonical Server: Pending requested
- Canonical Server packageset reviewers: Pending requested
-
Diff: 111 lines (+80/-0)4 files modifieddebian/changelog (+12/-0)
debian/patches/do-not-bind-docker-to-containerd.patch (+64/-0)
debian/patches/series (+1/-0)
debian/rules (+3/-0)
description: | updated |
description: | updated |
summary: |
- unattended-upgrade still restarts blacklisted daemons + [SRU] unattended-upgrade still restarts blacklisted daemons |
no longer affects: | unattended-upgrades (Ubuntu) |
no longer affects: | unattended-upgrades (Ubuntu Xenial) |
no longer affects: | unattended-upgrades (Ubuntu Bionic) |
no longer affects: | unattended-upgrades (Ubuntu Focal) |
no longer affects: | unattended-upgrades (Ubuntu Groovy) |
no longer affects: | unattended-upgrades (Ubuntu Hirsute) |
We have at least 4 servers on which docker daemon was shutdown due to this unattended upgrade, and was not restarted.