[Impact]
This affects anyone using lxc-start-ephemeral as part of an automated process for which intermittent failures are a problem. This includes the people who developed the initial version of the script, the Launchpad developers. Our automated test suite will fail, stopping our landing tools, whenever this failure is triggered.
[Development Fix]
1. no longer look in the container's file system for a dhcp table to get the ip of the container; instead, look in the host's network information. This is more reliable and ready sooner. r101 of quantal lxc package has this change.
2. increase the timeout waiting for the containers' network and sshd to be ready.
Note that the current increase, from 30 retries @ 1/sec to 60 retries @ 1/sec, is insufficient for the people who filed the bug, unfortunately. Making the retry count configurable would be ideal. Increasing it to 300 would be sufficient, based on our experience so far. stgraber suggested using the `parallel -l maxload` construct to keep the starts from being overwhelmed by load. Unfortunately, we believe that this is insufficient for at least two reasons. First, the point of our effort is to do a lot of work in parallel, with an lxc per core. The work we have to do takes more than half an hour. Waiting for load to decrease would miss the point of the effort. Second, it doesn't seem that cpu contention is always the problem, from watching top.
[Stable Fix]
[stgraber will need to specify]
[Text Case]
1. Create an lxc container (which has sshd running and your home directory mounter, as is the default). For the sake of these instructions, we will call it "lxctest".
2. Run something like this. Replace "username" with your user name. You might need to do this more or fewer times; we've seen it most easily on a 32 core (16 core hyperthreaded) machine trying to run 32 concurrent callsTo make this less annoying, you could create a temporary passphraseless ssh key.
parallel -j 16 bash -c "lxc-start-ephemeral -u gary -o lpdev -- 'cat /etc/hostname'" -- 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Do this a few times.
Broken Behavior: At least one of the times, at least one of these fail (emitting an error message rather than the hostname) either because the code could not get the ip address in time, or because the container's sshd wasn't ready in time.
Fixed Behavior: You get all 16 hostnames.
[Regression Potential]
The increased timeout might cause some code to wait longer than before to discover that something is wrong. The improved ip code should have no negative effect.
[Original Report]When lxc-start-ephemeral is given a command to run (-- do_something) it wants to use lxc-attach to run the command, but lxc-attach is not ready yet. Instead, it parses the dhcp leases to figure out the IP for the container, and then tries to use ssh to run the command.
Twice today in tests involving lxc-start-ephemeral, the dhcp leases were unavailable and lxc-start-ephemeral failed. The machine was under fairly heavy load and was virtualized (EC2).
I'd like to try and make this less fragile. As discussed on IRC, using lxcip (http://bazaar.launchpad.net/~launchpad/lpsetup/trunk/files/head:/lplxcip/) should make this more reliable. Perhaps increasing the timeout in that code might be useful as well.
Thanks, Gary. Do you have a debdiff for fixing this using lxcip in either precise or q?