when booting oneiric cloud-images under libvirt, no pty is available

In the container's rootfs, can you edit /etc/init/lxcguest.conf to have its pre-start script also do

   ls -l /dev/ptmx >> /debugout
   ls -l /dev/pts >> /debugout
   cat /proc/self/mounts >> /debugout
   cat /proc/self/mountinfo >> /debugout

and post /debugout here?

Shouldn't this be filed against libvirt rather than lxc?

I dont know if it should be against libvirt rather than lxc. lxcguest does things to make ubuntu work as an lxc guest. I figured it was at least relevant.

Here is how you can reproduce, given oneiric:

$ sudo apt-get install cgroup-bin
$ sudo apt-get install libvirt-bin python-libvirt libvirt0
$ sudo adduser $USER libvirtd
$ exit # come back in to get into libvirt group

$ wget https://cloud-images.ubuntu.com/server/releases/natty/release-20110426/ubuntu-11.04-server-uec-amd64.tar.gz
$ tar -Sxvzf ubuntu-11.04-server-uec-amd64.tar.gz
$ mkdir mp
$ sudo mount -o loop natty-server-uec-amd64.img mp
$ sudo rsync -axHAS mp/ root
$ wget "http://smoser.brickies.net/git/?p=tildabin.git;a=blob_plain;f=lxc-libvirt-root;hb=HEAD" -O lxc-libvirt-root
$ chmod 755 lxc-libvirt-root
$ ./lxc-libvirt-root root/ --init /bin/bash

At this point, you can see a strange mount table at:

root@ubuntu:/# cat /proc/1/mountinfo
74 81 0:25 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
78 54 253:16 /root / rw,relatime - ext3 /dev/vdb rw,errors=continue,barrier=0,data=ordered
79 78 0:25 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
81 78 0:29 / /dev rw,relatime - tmpfs /dev rw
82 78 0:27 / /proc rw,relatime - proc /proc rw
83 78 0:30 / /sys rw,relatime - sysfs /sys rw
root@ubuntu:/# cat /proc/mounts
rootfs / rootfs rw 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0
/dev/vdb / ext3 rw,relatime,errors=continue,barrier=0,data=ordered 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0
/dev /dev tmpfs rw,relatime 0 0
/proc /proc proc rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

As a point of reference, I did the same as above, but on a natty host:

root@ubuntu:/# cat /proc/1/mountinfo
68 75 0:24 / /dev/pts rw,relatime - devpts devpts rw,mode=600,ptmxmode=666
72 50 252:0 /home/ubuntu/root / rw,relatime - ext4 /dev/disk/by-label/uec-rootfs rw,barrier=1,data=ordered
73 72 0:24 / /dev/pts rw,relatime - devpts devpts rw,mode=600,ptmxmode=666
75 72 0:28 / /dev rw,relatime - tmpfs /dev rw
76 72 0:26 / /proc rw,relatime - proc /proc rw
77 72 0:29 / /sys rw,relatime - sysfs /sys rw

root@ubuntu:/# cat /proc/mounts
rootfs / rootfs rw 0 0
devpts /dev/pts devpts rw,relatime,mode=600,ptmxmode=666 0 0
/dev/disk/by-label/uec-rootfs / ext4 rw,relatime,barrier=1,data=ordered 0 0
devpts /dev/pts devpts rw,relatime,mode=600,ptmxmode=666 0 0
/dev /dev tmpfs rw,relatime 0 0
/proc /proc proc rw,relatime 0 0
/sys /sys sysfs rw,relatime 0 0

So, although it does seem strange to me that /dev is mounted *over* /dev/pts, it was that way in natty also.

Diagnosis: udev looks through /dev, making sure all devices have the expected device type and permissions. If not, it replaces them.

lxc creates /dev/ptmx and bind mounts /dev/pts/ptmx on top of it, so it has the expected device type.

libvirt creates /dev/ptmx as a symlink to /dev/pts/ptmx. This is not the expected device type, so udev replaces it. A bare ptmx device is not usable in the container as it is tied to the initial devpts namespace.

There are three ways to solve this. One is a small patch to libvirt to do the same thing lxc does. Another is to patch udev to allow ptmx to be a symlink. The third is to have another upstart job shipped with the lxcguest package, which starts on starting udev and replaces ptmx with a bind mount.

Since a bind mount from /dev/pts/ptmx is always correct (with any modern kernel), I suppose a fourth solution would be to have /etc/init/udev.conf create the expected ptmx as a bind mount from /dev/pts/ptmx unconditionally, even if it is not a container.

Note I've pinged the libvirt community on irc about the viability of the libvirt patch upstream.

Status changed to 'Confirmed' because the bug affects multiple users.

oneiric has seen the end of its life and is no longer receiving any updates. Marking the oneiric task for this ticket as "Won't Fix".