docker: Error response from daemon: error creating aufs mount

Bug #1560685 reported by Matt Bruzek
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
docker.io (Ubuntu)
Fix Released
High
Serge Hallyn

Bug Description

I am running 16.04 Ubuntu and trying to run docker within lxd (I know crazy right?!). While inside the LXC container we are unable to run docker because there is an error trying to mount aufs.

We found a workaround to this problem was to add the overlay driver to /etc/default/docker:
DOCKER_OPTS=--storage-driver=overlay

Distribution: Ubuntu
Distribution version: 16.04

# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.0-13-generic
Operating System: Ubuntu 16.04
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.04 GiB
Name: xenial
ID: CHCK:H2MG:HNB5:2XDN:KDOP:3U2Q:XLDG:VWF3:UXNR:XIIW:EE3P:YXIF
WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
03f4658f8b78: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:8be990ef2aeb16dbcb9271ddfe2610fa6658d13f6dfb8bc72074cc1ca36966a7
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: error creating aufs mount to /var/lib/docker/aufs/mnt/0df532e33493853f62ea882f5e
5ce585d7fc4b02753b2252ffd607f3800fb95b-init: operation not permitted.
See 'docker run --help'.

Can we either fix the aufs bug or enable the overlay driver by default?

Revision history for this message
Matt Bruzek (mbruzek) wrote :

The official Docker docs refer to aufs in the Ubuntu extras: https://docs.docker.com/engine/installation/linux/ubuntulinux/

Revision history for this message
Matt Bruzek (mbruzek) wrote :

The lxc/lxd team is wondering why the overlay driver is not included in docker by default: https://github.com/lxc/lxd/issues/1791

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Correction - it is included by default; we are wondering why it was not *used* by default in your case.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Could you please show what:

ps -ef | grep docker
dpkg -l | grep docker
cat /proc/filesystems

show?

Revision history for this message
Matt Bruzek (mbruzek) wrote :

Serge,

$ ps -ef | grep docker
root 1387 1 0 07:44 ? 00:00:10 /usr/bin/docker daemon -H fd://
mbruzek 6392 18048 0 21:27 pts/19 00:00:00 grep --color=auto docker
100000 31515 28211 0 21:21 ? 00:00:00 /usr/bin/docker daemon -H fd://

$ dpkg -l | grep docker
ii docker.io 1.10.2-0ubuntu4 amd64 Linux container runtime
rc lxc-docker-1.10.0-dev 1.10.0~dev~git20151104.194544.0.0cdc96c amd64 Linux container runtime
rc lxc-docker-1.8.0-dev 1.8.0~dev~git20150811.005309.0.ca8f246 amd64 Linux container runtime
rc lxc-docker-1.9.0-dev 1.9.0~dev~git20151029.220813.0.2eaa25d amd64 Linux container runtime

$ cat /proc/filesystems
nodev sysfs
nodev rootfs
nodev ramfs
nodev bdev
nodev proc
nodev cpuset
nodev cgroup
nodev tmpfs
nodev devtmpfs
nodev debugfs
nodev tracefs
nodev securityfs
nodev sockfs
nodev bpf
nodev pipefs
nodev devpts
        ext3
        ext2
        ext4
nodev hugetlbfs
        vfat
nodev ecryptfs
        fuseblk
nodev fuse
nodev fusectl
nodev pstore
nodev mqueue
nodev autofs
nodev zfs
nodev binfmt_misc
nodev aufs
nodev overlayfs
nodev overlay

$ uname -a
Linux warhorse 4.4.0-13-generic #29-Ubuntu SMP Fri Mar 11 19:31:18 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

In an email you asked further questions:

> Can you tell us the background of the host? Is it a clean newly installed machine or vm?

My own development system upgraded to 16.04 from 15.04.

> How was lxd installed?

lxd was installed from the archive.
dpkg -l | grep lxd
ii lxd 2.0.0~rc4-0ubuntu1 amd64

> How was docker.io installed in the container?

First I got into the container `lxc exec xenial /bin/bash`
I installed docker.io with `apt-get update && apt-get install docker.io`.
Running `docker info` shows the aufs driver selected
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs

> Can you easily reproduce this with the same iso (or whatever you used) for the host?

Yes this is still reproducible on my system. The steps are:
$ lxc launch images:ubuntu/xenial/amd64 xenial -p default -p docker
$ lxc exec xenial /bin/bash
# apt-get update && apt-get install -y docker.io
# docker info
# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
03f4658f8b78: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:8be990ef2aeb16dbcb9271ddfe2610fa6658d13f6dfb8bc72074cc1ca36966a7
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: error creating aufs mount to /var/lib/docker/aufs/mnt/e174f7d65f4b75fd5f1ea763a3f74cd13bf43ed68cb22b594d5b142b7201ca8c-init: operation not permitted.
See 'docker run --help'.

Revision history for this message
Matt Bruzek (mbruzek) wrote :

It should be noted that I can run the hello-world docker container on my host system, outside of lxd.

Changed in docker.io (Ubuntu):
importance: Undecided → High
status: New → Triaged
Changed in docker.io (Ubuntu):
assignee: nobody → Serge Hallyn (serge-hallyn)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package docker.io - 1.10.3-0ubuntu4

---------------
docker.io (1.10.3-0ubuntu4) xenial; urgency=medium

  * Drop d/patches/libnetwork-s390x.patch, no longer needed after golang
    updates.

 -- Michael Hudson-Doyle <email address hidden> Mon, 04 Apr 2016 16:27:28 +1200

Changed in docker.io (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.