Ubuntu
docker.io package

After restarting docker daemon, unable to restart existing containers

Bug #1404300 reported by James Page
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
docker.io (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Unassigned
Utopic
Won't Fix
High
Unassigned
Vivid
Fix Released
High
Unassigned

Bug Description

Impact:
Docker users are unable to upgrade packaging or restart the docker service without cleaning up stale mounts first; impacted containers will not start post restart of the docker daemon.

Test case:
NOTE: this must be done on a ext4 filesystem - btrfs users are not impacted.
<create a docker image>
docker run -d <uuid of image>
sudo service docker stop
sudo service docker start
docker ps -a
docker start <uuid of container>
container will fails to start as in original bug report.

Regression potential:
Minimal - the init script and upstart configuration will attempt cleanup of stale mounts in pre-start.

Original bug report:
Whilst testing security updates for bug 1396572, I noticed that after restarting docker, existing containers where no longer running and I was unable to restart them:

ubuntu@juju-jp-machine-24:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dd470c1d3430 38e88eeb106a memcached /bin/bash About an hour ago Exited (0) About a minute ago stupefied_newton
ubuntu@juju-jp-machine-24:~$ docker start stupefied_newton
Error response from daemon: Cannot start container stupefied_newton: Error getting container dd470c1d3430b651bca3ca3b7f35235529f5eaf7db03a460157d646655fbe020 from driver devicemapper: Error mounting '/dev/mapper/docker-253:1-131799-dd470c1d3430b651bca3ca3b7f35235529f5eaf7db03a460157d646655fbe020' on '/var/lib/docker/devicemapper/mnt/dd470c1d3430b651bca3ca3b7f35235529f5eaf7db03a460157d646655fbe020': device or resource busy
2014/12/19 16:13:18 Error: failed to start one or more containers

This appears to be related to https://github.com/docker/docker/issues/5684

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: docker.io 1.0.1~dfsg1-0ubuntu1~ubuntu0.14.04.1
ProcVersionSignature: User Name 3.13.0-43.72-generic 3.13.11.11
Uname: Linux 3.13.0-43-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
Date: Fri Dec 19 16:13:00 2014
Ec2AMI: ami-0000002c
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
SourcePackage: docker.io
UpgradeStatus: No upgrade log present (probably fresh install)

See original description
Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

Bug raised against trusty version; however I see this issue in 14.10 and vivid development as well.

Revision history for this message
James Page (james-page) wrote :

Best advice from upstream issue:

cat /proc/mounts | grep "mapper/docker" | awk '{print $2}' | sudo xargs -r umount

James Page (james-page)
Changed in docker.io (Ubuntu Vivid):
importance: Undecided → High
Changed in docker.io (Ubuntu Trusty):
importance: Undecided → High
Changed in docker.io (Ubuntu Utopic):
importance: Undecided → High
Changed in docker.io (Ubuntu Vivid):
status: New → In Progress
Changed in docker.io (Ubuntu Utopic):
status: New → In Progress
Changed in docker.io (Ubuntu Trusty):
status: New → In Progress
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package docker.io - 1.3.3~dfsg1-2ubuntu2

---------------
docker.io (1.3.3~dfsg1-2ubuntu2) vivid; urgency=medium

  * d/p/device-mapper-cleanup.patch: Cleanup any stale docker mounts
    from previous shutdown (LP: #1404300).
 -- James Page <email address hidden> Thu, 22 Jan 2015 08:50:14 +0000

Changed in docker.io (Ubuntu Vivid):
status: In Progress → Fix Released
Revision history for this message
Robie Basak (racb) wrote :

I can reproduce this on Trusty (1.0.1~dfsg1-0ubuntu1~ubuntu0.14.04.1) but not on my test 1.6.0 package that didn't include the workaround in Wily. So I presume this is somehow fixed upstream and a patch is no longer needed. When we push 1.6 back into stable releases that should resolve the outstanding bug tasks.

Exact steps I used:

sudo -i
docker pull ubuntu:14.04
docker run -d ubuntu:14.04 /bin/sleep 3600
service docker stop
service docker start
docker ps -a
docker start <uuid of container>

In addition, I used a proxy setting in /etc/default/docker.io (Trusty) and /etc/default/docker (Wily). On Trusty, export= is needed as it's sourced from an upstart job. On Wily, export= must be absent as it uses systemd EnvironmentFile syntax.

Also the service name on Trusty is docker.io, not docker.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package docker.io - 1.6.2~dfsg1-1ubuntu4~14.04.1

---------------
docker.io (1.6.2~dfsg1-1ubuntu4~14.04.1) trusty; urgency=medium

  * Backport to Ubuntu 14.04 (LP: #1454719).
  * Disabled
    - d/p/lxc.autodev-support.patch to minimise regression risk as
      it is not relevant for the version of LXC on Trusty (1.0.3-0ubuntu3).
    - d/p/update-go.net-golang.org.patch: there has been a url
      canonical name change upstream, but keeping this patch on involves
      backporting golang to 1.4 which is undesirable for this backport
      (golang-go.net-dev needs golang-x-text, which does not build
      successfully without a 1.4 backport).
    - Wily related fixes:
      + d/p/golang-1.5-wily.patch to fix FTBFS with golang-1.5 build on wily
      + d/p/ppc64el-wily.patch to fix ppc64le FTBFS on wily (LP: #1488668)
      + d/p/libcontainer_arm64_syscall_dup2_to_dup3-c_changes.patch (LP: #1488669)
      + d/p/libcontainer_arm64_syscall_dup2_to_dup3-golang_changes.patch (LP: #1488669)
      + d/rules to build with golang-go on arm64 (LP: #1488669)
      + d/control to build with golang-go on arm64 (LP: #1488669)
  * Reverted:
    d/rules: http://anonscm.debian.org/cgit/docker/docker.io.git/diff/?id=b1458f5
    commit to preserve docker.io symlink.

 -- Pierre-André MOREY <email address hidden> Tue, 22 Sep 2015 13:47:53 +0200

Changed in docker.io (Ubuntu Trusty):
status: In Progress → Fix Released
Revision history for this message
Rolf Leggewie (r0lf) wrote :

utopic has seen the end of its life and is no longer receiving any updates. Marking the utopic task for this ticket as "Won't Fix".

Changed in docker.io (Ubuntu Utopic):
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.