Semaphores cannot be created in lxc container

This is easily and repeatably duplicatable on in my lxc lucid container. The error for the last test (with the multiprocessing SemLock OSError seems the most likely to be a shared root cause to me. Interestingly, the OSError was slightly different for me, with a newer lxc: "OSError: [Errno 38] Function not implemented". I checked the syslog and some random other log files, without really knowing where to look, and saw nothing of interest.

As you'd expect from the traceback, you can repeat this error with a single Python call to the standard library.

>>> from multiprocessing import synchronize
>>> synchronize.Lock()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 117, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 49, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented

38 is ENOSYS

I talked with hallyn and this is an lxc issue. Adding this line to the container's fstab will fix the problem:
none dev/shm tmpfs defaults 0 0

He is updating the ubuntu template with this change.

There are a few parts to this. At core, we need /dev/shm to be a symbolic link to /dev/shm, which *is* mounted.

/etc/init/mounted-dev.conf is supposed to create that symbolic link. However, it does not run because we never mount /dev.

If it did run, it would do the wrong thing. That is because the initscripts postinst leaves a /dev/shm directory. mounted-dev.conf simply does

[ -e /dev/shm ] || ln -s /run/shm /dev/shm

That would cause /dev/shm/shm to be the symbolic link to /run/shm.

The simplest solution is probably to rmdir $rootfs/dev/shm at the end of the container creation templates.

Note, the fix will only affect new containers.

The WORKAROUND for existing containers is to manually rmdir /var/lib/lxc/{container}/rootfs/dev/shm

For P we are working around this in lxc. No change is needed in P in initscripts.

For Q and beyond, initscripts should be fixed. If running from debootstrap, /dev/shm should be created as a symlink. That is not happening. Note that a system which mounts /dev as devtmpfs (or tmpfs) will do the right thing on reboot. But a system which has /dev on rootfs is stuck with /dev/shm as whatever initscripts' postinst left it as.

This bug was fixed in the package lxc - 0.7.5-3ubuntu51

---------------
lxc (0.7.5-3ubuntu51) precise; urgency=low

  * 0070-templates-rmdir-dev-shm: in precise containers, rmdir $rootfs/dev/shm
    and and create it as a symbolic link to /run/shm. (LP: #974584)
 -- Serge Hallyn <email address hidden> Thu, 12 Apr 2012 09:54:22 -0500

Sorry, the workaround I listed above was for precise, and actually was incomplete (you also need to ln -s /run/shm $rootfs/dev/shm).

For older releases, the /run transition was not done, so /dev/shm is supposed to be mounted by mountall through /lib/init/fstab. So the workaround is to add a dev/shm entry to /var/lib/lxc/<container>/fstab, or to add it to /var/lib/lxc/<container>/rootfs/lib/init/fstab.lxc.

It would appear part of the problem is simply that ischroot, provided by debianutils, does not work. initscripts' postinst counts on ischroot working.

This has taken me longer than it should have because I made some bad initial assumptions.

The appearance of a /dev/shm directory was a bit of a false flag. That was happening because the chroot case higher in initscripts.postinst was failing to create the /dev/shm symbolic link in the first place. It's source directory (/run/shm) needed to first be created, and then the arguments to compat_link() were reversed. I assumed they were correct because they were the same for the /var/lock symlink which is correctly created - but in fact the /var/lock symlink exists before that block of code.

Here is a debdiff which appears to fix the /dev/shm symlink creation in a container for me.

Per irc discussion with slangasek, further patching is needed to make sure that the /run/lock and /run->/var/run update works right in an upgrade with /run transition.

I can confirm Serge's analysis here. I'm surprised that this hasn't bitten us more severely before now, given that I see all my compat symlinks are wrong in my (iteratively upgraded) precise chroots here; apparently things really don't care so much about the compat these days.

We should definitely fix this for .1.

(thinking out loud)

my patch works for a fresh debootstrap. It should have zero effect for a machine install, since it is only in chroots. Similarly, containers are treated as hosts so will not be affected when upgraded under lxc.

Cases to still test, then, include an update from oneiric or lucid to precise inside a chroot or schroot, as well as an oneiric container image upgraded under chroot (not inside lxc). Not sure what else.

My patch is however not right - it somehow leaves /run/shm mounted.

I wonder if it would be overkill to detect a /dev/shm mount and non-existent /run/shm and, if both are the case, move the mount. Otherwise, throw our hands up and assume user did something weird by hand.

Ah, a simple 'chroot dpkg -i *.deb' of course fails because ischroot fails with /proc not mounted.

Do we care about that? The fix is simple, but if we don't care about it then I won't add it.

New debdiff which also makes sure proc is mounted before ischroot is called, so that 'chroot precise-rootfs dpkg -i *.deb' succeeds and does not leave dangling /run/shm mounts.

The attachment in #15 (sysvinit-shm2.debdiff) is the best fix in my opinion. I don't have upload rights, and this ought to be reviewed anyway. If noone speaks up here reviewing the patch in a few days, I'll ping on irc.

Doing
   debootstrap sid sid
   chroot sid dpkg -i /var/cache/apt/archives/initscr*.deb

(on a sid host) also leaves a mount on sid/run/shm.

From the debian bug, here is my new suggestion for a replacement for the ischroot case in initscripts.postinst:

=============================
if ischroot; then
        # Symlink /var/run from /run
        # Note var/run is relative
        if compat_link /var/run /run; then
                # Symlink /var/lock from /run/lock
                # Note that it's really /var/run/lock
                compat_link /var/lock /run/lock

                # Symlink /dev/shm from /run/shm
                # Note that it's really /var/run/shm
  [ ! -d /run/shm ] && mkdir /run/shm
  if [ ! mountpoint -q /dev ]; then
   [ ! -d /dev/shm ] && compat_link /run/shm /dev/shm
  else
   compat_link /dev/shm /run/shm
  fi
        fi
# Host system, not a chroot.
else
=============================

So if the chroot has /dev/ bind-mounted from the host, assume they
always will, and will want /run/shm as a symlink to /dev/shm.

Otherwise if /dev/shm exists, leave it alone, because we're definately
not in debootstrap.

Otherwise, assume /dev and /run won't get cleaned up at 'shutdown'
(chroot exit) so set up *exactly* what we want to see as the end
result: /dev/shm as a symbolic link to /run/shm.

Here is version tested and working in debootstrap.

diff -u sysvinit-2.88dsf/debian/changelog sysvinit-2.88dsf/debian/changelog
--- sysvinit-2.88dsf/debian/changelog
+++ sysvinit-2.88dsf/debian/changelog
@@ -1,3 +1,12 @@
+sysvinit (2.88dsf-13.10ubuntu12) quantal; urgency=low
+
+ * initscripts.postinst: if /dev is not a separate partition and we're in a
+ chroot, then create /run/shm and make /dev/shm a symbolic link to it, as
+ we would expect to find in a upgraded and rebooted running system.
+ (LP: #974584) (Closes: #674178)
+
+ -- Serge Hallyn <email address hidden> Wed, 30 May 2012 12:17:37 -0500
+
 sysvinit (2.88dsf-13.10ubuntu11) precise; urgency=low

   * Only try to move links in /etc/rc{0,6}.d that match "S0*". LP: #941867.
diff -u sysvinit-2.88dsf/debian/initscripts.postinst sysvinit-2.88dsf/debian/initscripts.postinst
--- sysvinit-2.88dsf/debian/initscripts.postinst
+++ sysvinit-2.88dsf/debian/initscripts.postinst
@@ -255,7 +255,12 @@

   # Symlink /dev/shm from /run/shm
   # Note that it's really /var/run/shm
- compat_link /dev/shm /run/shm
+ if ! mountpoint -q /dev ; then
+ [ ! -d /run/shm ] && mkdir -p /run/shm
+ [ ! -d /dev/shm ] && compat_link /run/shm /dev/shm
+ else
+ compat_link /dev/shm /run/shm
+ fi
  fi
 # Host system, not a chroot.
 else

I think there are the following possible valid end states that we want to get to at the end of the postinst:

 - we're not in a chroot. /dev/shm is bind mounted to /run/shm, with the reboot script handling the fix-up to make /dev/shm a symlink to /run/shm before next boot.
 - we're in a chroot and /dev is not bind mounted. sysvshm should be mounted *somewhere* within the chroot, but we don't know where this will be done and we're not going to do it ourselves. So dictate that this should be /run/shm, and make /dev/shm a symlink there.
 - we're in a chroot and /dev is bind mounted, and /dev/shm is a symlink to /run/shm (because the parent environment is also a recent Debian/Ubuntu release). Make the /run/shm directory.
 - we're in a chroot, /dev is bind mounted, and /dev/shm is also bind mounted. This is not a good situation to be in, because it means that the future correctness of the chroot is dependent on the setup remaining the same on the host system - which it won't if the host system is an older Debian/Ubuntu release that is later upgraded. But we do the best we can for now, and make /run/shm a symlink to /dev/shm.
 - we're in a chroot, /dev/ is bind mounted, and /dev/shm is a non-bind-mounted directory. We can't fix this, so we should just create the /run/shm directory and leave /dev/shm alone.

Does that make sense?

If it does make sense, the implementation that follows is:

if mountpoint -q /dev && mountpoint -q /dev/shm; then
  compat_link /dev/shm /run/shm
else
  [ -d /run/shm ] || mkdir -p /run/shm
  mountpoint -q /dev/ || compat_link /run/shm /dev/shm
fi

BTW, I realize there's another possible scenario, which is that /dev/shm is a bind mount when /dev itself is not. We should probably handle that the same as if they both are (make /run/shm a symlink to /dev/shm), in which case the first check just needs to be "if mountpoint -q /dev/shm".

Comment #20 looks very much correct.

The shm_overview(7) man page does say that it always uses a tmpfs filesystem, so does that mean that there is no way for /dev/shm to be a unmounted-over directory holding valuable shm state?

In any case, both your code in comment #21 and the update you propose there would suffice to fix the bug in debootstrap creating images not usable in a chroot/container.

As this bug is fixed in debian, should the patch be cherrypicked into ubuntu, or should i open a bug requesting al merge of 2.88dsf-28?

This bug was fixed in the package sysvinit - 2.88dsf-13.10ubuntu12

---------------
sysvinit (2.88dsf-13.10ubuntu12) quantal; urgency=low

  * Cherry-pick fix from git to better handle chroot in postinst and change
    the check to allow for the case where /dev isn't a mountpoint but /dev/shm
    is one. (LP: #974584)
  * Add logic to detect broken environments (mostly LXC) where both /dev/shm
    and /run/shm exist as directories (and aren't bind-mounts).
    In such case, move the content of /dev/shm to /run/shm and make /dev/shm
    a symlink to /run/shm.
 -- Stephane Graber <email address hidden> Fri, 20 Jul 2012 16:55:04 -0400

SRU team: Please wait for matching lxc upload before accepting into -proposed or moving to -updates or creating container will fail because of the broken check in lxc-ubuntu and lxc-ubuntu-cloud.

Hello Gary, or anyone else affected,

Accepted sysvinit into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/sysvinit/2.88dsf-13.10ubuntu11.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Gary, or anyone else affected,

Accepted lxc into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/lxc/0.7.5-3ubuntu61 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package lxc - 0.8.0~rc1-4ubuntu21

---------------
lxc (0.8.0~rc1-4ubuntu21) quantal; urgency=low

  [ Stéphane Graber ]
  * Fix lxc-ubuntu and lxc-ubuntu-cloud to fix the /dev/shm workaround to only
    trigger when /dev/shm is not a symlink. (LP: #974584)

  [ Serge Hallyn ]
  * lxc.lxc-net.upstart: replace the check for USE_LXC_BRIDGE (which could be
    changed from true to false after starting lxc-net) with one for the
    existence /var/run/lxc. (LP: #1019290)
  * 0095-lxc-clone-change-uuid-on-xfs.patch: give each cloned xfs-backed
    lvm partition a unique uuid so they can be mounted simultaneously.
    (LP: #1013549)
  * 0096-lxc-wait-add-timeout.patch: patch submitted upstream to add a timeout
    option to lxc-wait. (LP: #1020179)
 -- Serge Hallyn <email address hidden> Thu, 26 Jul 2012 17:40:36 +0000

Tested before/after for both lxc and sysvinit on precise with precise and quantal containers, everything seems to work as expected.

As discussed in #ubuntu-release, a new lxc package was accepted resetting the testing period to 0 today.

As both sysvinit and lxc absolutely NEED to land at the same time for this bug to get fixed without causing any regression, I'd appreciate if whoever copies sysvinit copies lxc as well, regardless of how many days are left for lxc.

lxc_0.7.5-3ubuntu61 was properly tested days ago and I have confirmed that lxc_0.7.5-3ubuntu62 is good too.

So when sysvinit reaches 7 days in -proposed, the matching change in lxc will too and so both should be considered good to go.

Here's the 61 to 62 delta: http://paste.ubuntu.com/1123344/

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package lxc - 0.7.5-3ubuntu62

---------------
lxc (0.7.5-3ubuntu62) precise-proposed; urgency=low

  * lxc.postrm: support "purge" command (LP: #1029716)
 -- Serge Hallyn <email address hidden> Fri, 27 Jul 2012 03:31:39 +0000

This bug was fixed in the package sysvinit - 2.88dsf-13.10ubuntu11.1

---------------
sysvinit (2.88dsf-13.10ubuntu11.1) precise-proposed; urgency=low

  * Cherry-pick fix from git to better handle chroot in postinst and change
    the check to allow for the case where /dev isn't a mountpoint but /dev/shm
    is one. (LP: #974584)
  * Add logic to detect broken environments (mostly LXC) where both /dev/shm
    and /run/shm exist as directories (and aren't bind-mounts).
    In such case, move the content of /dev/shm to /run/shm and make /dev/shm
    a symlink to /run/shm.
 -- Stephane Graber <email address hidden> Fri, 20 Jul 2012 16:55:04 -0400