lxc on precise is not working with lucid containers (container does not reach runlevel 2)

Hi Gary,

when we've seen this before it has been because the container was waiting for a dhcp response. It has generally been on virbr0, and when virbr0 had stp on.

Can you show what 'brctl show' on the host gives? Do your host logs show any problems with dnsmasq on lxcbr0?

I reported on IRC to Serge that virbr0 did indeed have STP enabled, but that I did not see any obvious dnsmasq errors. He already had diagnosed the problem & said as a workaround I could just change the 'start on' in /etc/init/console.conf in my container to 'start on mounted MOUNTPOINT=/run'. stgraber's new upstart will fix it properly when it hits the archive.

Thanks, Gary.

Note that the upstart 'fix' will basically be the same as the suggested workaround.

I think the stp on for virbr0 is actually a red herring here, for two reasons. First, you show in the Description that you are using lxcbr0, which should have stp off. Second, in my own experiments, when the console has been slow to come up, the network was actually up and i could ssh in before the console came up. So i think the current 'start on', which is 'stopped rc runlevel=2345]' is jsut bad - that is, there is something running long (maybe udev?) which keeps us getting to *stopped* rc runlevel 2.

I tried the fix and I got the same symptom.

For clarity:

- I wiped out /var/lxc/cache before I started because lxc-create wasn't working. That got lxc-create working. I used the same commands I listed in the bug description.

- I changed the container's console.conf as described:

$ cat /var/lib/lxc/lp/rootfs/etc/init/console.conf
# /dev/console - getty
#
# This service maintains a getty on /dev/console from the point the
# system is started until it is shut down again.
# It only runs in lxc containers.
# Libvirt links /dev/tty1 to /dev/pts/0.

## start on stopped rc RUNLEVEL=[2345]

start on mounted MOUNTPOINT=/run
stop on runlevel [!2345]

env container
env LIBVIRT_LXC_UUID
pre-start script
 [ "x$container" = "xlxc" ] && exit 0;
 stop;
 exit 0;
end script

respawn

exec /sbin/getty -8 38400 /dev/console

- sudo lxc-start -n lp then hangs with the exact same symptoms described initially.

Thanks

Reproduced. Note that during this time I can ssh in just fine. The container does have an ip address! (you can find it with tail /var/log/daemon.log on the host)

@Gary

terribly sorry, 'mounted=/run' won't work in your containers as you are using lxcguest.

You should be safe to just make it 'start on started lxcguest' (really even 'start on startup', which I just tried and worked fine)

It seems to be a bug in lxcguest. runlevel shows 'unknown', whereas on a real host or precise container it shows 'N 2'

Serge, cool. The workaround is fine. Thanks.

rc-sysinit.conf never starts because it doesn't see net-device-up for lo. When I do:

sudo initctl emit --no-wait net-device-up IFACE=lo

then the container transitions to runlevel 'N 2'.

When I first created lxcguest, one of the main things it did was emit that event. Then we stopped having to, but the reasons were always a bit too mysterious for me.

Assigning to stgraber as I *believe* he will know offhand what's going on. If not, please feel free to reassign to me.

It turns out the original problem never went away. The container never gets a net-device-up event for lo, even in precise. You can verify this trivially with two upstart jobs, /etc/init/lxc{lo,eth}.conf:

==== lxclo.conf ====
description "detect net-device-up IFACE=lo"
start on net-device-up IFACE=lo
==== lxceth.conf ===
description "detect net-device-up IFACE=eth0"
start on net-device-up IFACE=eth0
====================

Start the container and check 'status lxclo' and 'status lxceth'.

The reason this happens is clear in /etc/network/if-up.d/upstart. The reason
this stopped mattering is that rc-sysinit.conf switched from being

start on filesystem and net-device-up IFACE=lo

to being

start on (filesystem and static-network-up) or failsafe-boot

Where static-network-up does not require lo to come up.

Right, so we need for anything before Oneiric to have that emit in lxcguest, since Oneiric it should be properly handled by networking.conf bringing up lo and by the static-network-up event.

Starting with Precise where we won't have lxcguest, the same trick as Oneiric will work so we won't need to emit net-device-up either.

This bug was fixed in the package lxc - 0.7.5-3ubuntu18

---------------
lxc (0.7.5-3ubuntu18) precise; urgency=low

  * lxcguest.lxcguest.upstart: emit the net-device-up IFACE=lo event, so
    that any upstart jobs waiting on it (esp rc-sysinit before oneiric) will
    proceed. (LP: #924337)
  * 0034-fix-lxc-execute-reboot.patch: fix bad handling of 'exit 0' for
    lxc-execute introduced with the container reboot handling. (LP: #927863)
  * debian/lxcguest.lxcmount.upstart: add '--no-wait' to emit to make sure we
    don't wait for the event to be processed.
  * 0035-lxc-init-ignore-shm.patch: if lxc-init can't mount /dev/shm, don't
    fail on account of that. (LP: #927883)
  * debian/lxc.init: if the network is already up, exit before setting the
    trap EXIT.
 -- Serge Hallyn <email address hidden> Mon, 06 Feb 2012 17:37:37 -0600

Hello Gary, or anyone else affected,

Accepted lxc into oneiric-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Confirmed, lxcguest from oneiric-proposed fixes this in an oneiric guest.

This bug was fixed in the package lxc - 0.7.5-0ubuntu8.5

---------------
lxc (0.7.5-0ubuntu8.5) oneiric-proposed; urgency=low

  * lxcguest.lxcguest.upstart: emit the net-device-up IFACE=lo event, so
    that any upstart jobs waiting on it (esp rc-sysinit before oneiric) will
    proceed. (LP: #924337)
  * 0035-lxc-init-ignore-shm.patch: if lxc-init can't mount /dev/shm, don't
    fail on account of that. (LP: #927883)
 -- Serge Hallyn <email address hidden> Mon, 06 Feb 2012 19:06:46 -0600

This happens to me on a Natty container:

salgado@delgadito:~$ cat /var/lib/lxc/patchmetrics/rootfs/etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.04
DISTRIB_CODENAME=natty
DISTRIB_DESCRIPTION="Ubuntu 11.04"
salgado@delgadito:~$ sudo lxc-start -n patchmetrics
init: procps main process (26) terminated with status 255
init: udev-fallback-graphics main process (46) terminated with status 1
init: plymouth main process (6) killed by ABRT signal
init: plymouth-splash main process (52) terminated with status 2

Confirmed - in fact /etc/init/lxcguest.conf is missing altogether, causing other problems as well.

I'll push a fix to natty-proposed. Thanks for pointing this out.

Hello Gary, or anyone else affected,

Accepted lxc into natty-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Salgado, would you be able to do this verification? If not, let me know, and I'll do it.

Just confirmed; this did fix a newly created Natty container for me. Thanks!

Based on comment above, marking verification-done

This bug was fixed in the package lxc - 0.7.4-0ubuntu7.3

---------------
lxc (0.7.4-0ubuntu7.3) natty-proposed; urgency=low

  * lxcguest.lxcguest.upstart: emit the net-device-up IFACE=lo event, so
    that any upstart jobs waiting on it (esp rc-sysinit before oneiric) will
    proceed. (LP: #924337)
  * debian/rules: install lxcguest.lxcguest.upstart (as it was not in the
    natty package before)
 -- Serge Hallyn <email address hidden> Wed, 28 Mar 2012 13:58:23 -0500

Is it possible that this bug returned?

I updated the host to ubuntu saucy and lxc was working fine with the virtual machine "jenkins". Once I updated the guest system to saucy as well, it seemed not to boot, but I was wrong. It never reached the runlevel (5), which it did before.

Here is the output of lxc-ls

# lxc-ls --fancy
NAME STATE IPV4 IPV6 AUTOSTART
-------------------------------------------------
jenkins RUNNING 192.168.1.30 - NO
php-54-32 STOPPED - - NO
test STOPPED - - NO

as you can see it is running, but not reachable by ssh. so I entered:

# lxc-attach -n jenkins -- runlevel
unknown

it doesn't know in which runlevel it is.

even though the network is up within the guest system:

# lxc-attach -n jenkins -- ifconfig
eth0 Link encap:Ethernet HWaddr 10:16:3e:53:5b:7a
          inet addr:192.168.1.30 Bcast:192.168.1.255 Mask:255.255.255.0
          inet6 addr: fe80::1216:3eff:fe53:5b7a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:277 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:44733 (44.7 KB) TX bytes:648 (648.0 B)

as soon as I force the runlevel with:

lxc-attach -n jenkins -- init 5

everything is working, until I reboot the guest again.

Hi,

I suspect you're suffering from the veth offloading feature in saucy kernel. If possible please flush the lxc cache (sudo rm -rf /var/cache/lxc/*) and re-create the containers, or else chroot into the containers to update them. Newer userspace will work around the "feature".

If that doesn't help, please open a new bug.