This is a regression in Xenial's kernel 4.4.0-9 or 4.4.0-10. See comment #3 for simple reproducer.
ORIGINAL BUG REPORT
===================
I'm investigating some failures in autopkgtest's testsuite, and stumbled over something really weird: In an ephemeral container it is apparently not possible any more to chmod files that started out being root owned and got chowned later:
$ sudo lxc-start-ephemeral -o adt-wily
(log in as ubuntu/ubuntu)
ubuntu@adt-wily-hvzj1eoa:~$ echo hello | sudo tee /tmp/testfile
[sudo] password for ubuntu:
hello
ubuntu@adt-wily-hvzj1eoa:~$ sudo chown ubuntu:ubuntu /tmp/testfile
ubuntu@adt-wily-hvzj1eoa:~$ chmod +x /tmp/testfile
chmod: changing permissions of ‘/tmp/testfile’: Operation not permitted
However, if the file was *not* previously chowned, it works as expected:
ubuntu@adt-wily-hvzj1eoa:~$ echo hello > /tmp/testfile2
ubuntu@adt-wily-hvzj1eoa:~$ chmod +x /tmp/testfile2
ubuntu@adt-wily-hvzj1eoa:~$ chmod -x /tmp/testfile2
(no errors and testfile2 becomes executable)
There is no visible permission difference in the files at all, other than being group-writable (but changing the group w bit in either direction does not change the error at all):
-rw-r--r-- 1 ubuntu ubuntu 6 Mar 11 10:26 /tmp/testfile
-rw-rw-r-- 1 ubuntu ubuntu 6 Mar 11 10:26 /tmp/testfile2
ubuntu@adt-wily-hvzj1eoa:~$ stat /tmp/testfile*
File: ‘/tmp/testfile’
Size: 6 Blocks: 8 IO Block: 4096 regular file
Device: 15h/21d Inode: 28 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ ubuntu) Gid: ( 1000/ ubuntu)
Access: 2016-03-11 10:26:19.574364117 +0100
Modify: 2016-03-11 10:26:19.574364117 +0100
Change: 2016-03-11 10:26:21.930343210 +0100
Birth: -
File: ‘/tmp/testfile2’
Size: 6 Blocks: 8 IO Block: 4096 regular file
Device: 15h/21d Inode: 29 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 1000/ ubuntu) Gid: ( 1000/ ubuntu)
Access: 2016-03-11 10:26:58.730145919 +0100
Modify: 2016-03-11 10:26:58.730145919 +0100
Change: 2016-03-11 10:27:44.530203985 +0100
Birth: -
There are also no ACLs involved (I checked with getfacl).
This does not happen with a normal lxc-start, so this might very well be a bug in Linux' overlayfs. However, it also does not happen with the more modern "sudo lxc-copy -n adt-wily --ephemeral --foreground" -- bug perhaps this isn't using overlayfs?
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: lxc 2.0.0~rc9-0ubuntu1
ProcVersionSignature: Ubuntu 4.4.0-11.26-generic 4.4.4
Uname: Linux 4.4.0-11-generic x86_64
ApportVersion: 2.20-0ubuntu3
Architecture: amd64
CurrentDesktop: i3
Date: Fri Mar 11 10:21:20 2016
EcryptfsInUse: Yes
PackageArchitecture: all
SourcePackage: lxc
UpgradeStatus: No upgrade log present (probably fresh install)
defaults.conf:
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx
dnsmasq.conf:
enable-tftp
tftp-root=/tmp/tftp
dhcp-boot=pxelinux.0
lxc.conf: lxc.lxcpath = /srv/lxc
I tried to purge all lxc/lxd packages and install LXC 2.0.0~rc1-0ubuntu1 again, but this now not working at all any more on xenial's kernel -- the container never starts and I get a lot of apparmor violations.